SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Further bounding the sum over  $t$  above by its positive part, we can expand the sum over Base-Alg(s,m)  $\in \mathrm{PROPER}(t_{\ell},t_{\ell +1})\cup \mathrm{SUBPROPER}(t_{\ell},t_{\ell +1})$  to be over all Base-Alg(s,m), or obtain:
$$
\sum_ {a = 1} ^ {K} \sum_ {\text {B a s e - A l g} (s, m)} \mathbf {1} \left\{B _ {s, m} = 1 \right\} \left(\sum_ {t = s \vee t _ {\ell} ^ {a}} ^ {M (s, m)} \frac {\delta_ {t} \left(\hat {a} _ {t} , a\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \mathbf {1} \left\{a \in \mathcal {A} _ {t} \right\}\right) _ {+}, \tag {11}
$$
where the sum is over all replays  $\operatorname{Base - Alg}(s, m)$ , i.e.  $s \in \{t_{\ell} + 1, \dots, t_{\ell + 1} - 1\}$  and  $m \in \{2, 4, \dots, 2^{\lceil \log(T) \rceil}\}$ . It then remains to bound the contributed relative regret of each  $\operatorname{Base - Alg}(s, m)$  in the interval  $[s \vee t_{\ell}^{a}, M(s, m)]$ , which will follow similarly to the previous steps. Fix  $s, m$  and suppose  $t_{\ell}^{a} + 1 \leq M(s, m)$  since otherwise  $\operatorname{Base - Alg}(s, m)$  contributes no regret in (11).
Then, following similar reasoning as before, i.e. combining our concentration bound (9) with the eviction criterion (3), we have for a fixed arm  $a$ :
$$
\sum_ {t = s \vee t _ {\ell} ^ {a}} ^ {M (s, m)} \frac {\delta_ {t} (\hat {a} _ {t} , a)}{| \mathcal {A} _ {t} |} \leq \frac {c _ {2} \log (T) \sqrt {K m \vee K ^ {2}}}{\min _ {t \in [ s , M (s , m) ]} | \mathcal {A} _ {t} |},
$$
Plugging this into (11) and switching the ordering of the outer double sum, we obtain (now for clarity overloading the notation  $M(s,m,a)$  to also depend on the reference arm  $a$ ):
$$
\sum_ {\operatorname {B a s e - A l g} (s, m)} \mathbf {1} \left\{B _ {s, m} = 1 \right\} \cdot c _ {2} \log (T) \sqrt {K m \vee K ^ {2}} \sum_ {a = 1} ^ {K} \frac {1}{\min  _ {t \in [ s , M (s , m . a) ]} | \mathcal {A} _ {t} |}.
$$
We claim the above innermost sum over  $a$  is at most  $\log(K)$ . For a fixed Base-Alg  $(s, m)$ , if  $a_k$  is the  $k$ -th arm in  $[K]$  to be evicted by Base-Alg  $(s, m)$  or any of its children, then  $\min_{t \in [s, M(s, m, a_k)]} |\mathcal{A}_t| \geq K + 1 - k$ . Thus, our claim follows from  $\sum_{k=1}^{K} \frac{1}{K+1-k} \leq \log(K)$ .
Let  $R(m) \coloneqq c_2 \log(K) \log(T) \sqrt{Km \vee K^2}$  which is the bound we've obtained so far on the relative regret for a single Base-Alg(s,m). Then, plugging  $R(m)$  into (11) gives:
$$
\begin{array}{l} \mathbb {E} \left[ \mathbf {1} \{\mathcal {E} _ {1} \} \sum_ {a = 1} ^ {K} \sum_ {t = t _ {\ell} ^ {a}} ^ {t _ {\ell + 1} - 1} \frac {\delta_ {t} (\hat {a} _ {t} , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{a \in \mathcal {A} _ {t} \} \right] \leq \mathbb {E} _ {t _ {\ell}} \left[ \mathbb {E} \left[ \sum_ {\text {B a s e - A l g} (s, m)} \mathbf {1} \{B _ {s, m} = 1 \} \cdot R (m) \mid t _ {\ell} \right] \right] \\ = \mathbb {E} _ {t _ {\ell}} \left[ \sum_ {s = t _ {\ell}} ^ {T} \sum_ {m} \mathbb {E} [ \mathbf {1} \{B _ {s, m} = 1 \} \cdot \mathbf {1} \{s <   t _ {\ell + 1} \} \mid t _ {\ell} ] \cdot R (m) \right]. \\ \end{array}
$$
Next, we observe that  $B_{s,m}$  and  $\mathbf{1}\{s < t_{\ell + 1}\}$  are independent conditional on  $t_\ell$  since  $\mathbf{1}\{s < t_{\ell + 1}\}$  only depends on the scheduling and observations of base algorithms scheduled before round  $s$ . Thus,
recalling that  $\mathbb{P}(B_{s,m} = 1) = 1 / \sqrt{m\cdot(s - t_{\ell})}$
$$
\begin{array}{l} \mathbb {E} [ \mathbf {1} \{B _ {s, m} = 1 \} \cdot \mathbf {1} \{s <   t _ {\ell + 1} \} \mid t _ {\ell} ] = \mathbb {E} [ \mathbf {1} \{B _ {s, m} = 1 \} \mid t _ {\ell} ] \cdot \mathbb {E} [ \mathbf {1} \{s <   t _ {\ell + 1} \} \mid t _ {\ell} ] \\ = \frac {1}{\sqrt {m \cdot (s - t _ {\ell})}} \cdot \mathbb {E} [ \mathbf {1} \{s <   t _ {\ell + 1} \} \mid t _ {\ell} ]. \\ \end{array}
$$
Plugging this into our expectation from before and unconditioning, we obtain:
$$
\mathbb {E} \left[ \sum_ {s = t _ {\ell} + 1} ^ {t _ {\ell + 1} - 1} \sum_ {n = 1} ^ {\lceil \log (T) \rceil} \frac {1}{\sqrt {2 ^ {n} \cdot (s - t _ {\ell})}} \cdot R \left(2 ^ {n}\right) \right] \leq c _ {3} \log^ {3} (T) \mathbb {E} _ {t _ {\ell}, t _ {\ell + 1}} \left[ \sqrt {K \left(t _ {\ell + 1} - t _ {\ell}\right) \vee K ^ {2}} \right]. \tag {12}
$$
Then, it suffices to bound  $\sqrt{K(t_{\ell + 1} - t_{\ell})\lor K^2}$ . First, we claim that every phase  $[\tau_i,\tau_{i + 1})$  is length at least  $K / 4$ . Observe by our notion of significant regret, that an arm  $a$  incurring significant regret on the interval  $[s_1,s_2]$  means
$$
\sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(a _ {t} ^ {*}, a\right) \geq \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)} \Rightarrow 2 \cdot \left(s _ {2} - s _ {1}\right) \geq \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)} \Rightarrow s _ {2} - s _ {1} \geq K / 4.
$$
Thus, each significant phase (Definition 3) must be at least  $K / 4$  rounds long meaning  $\tau_{i + 1} - \tau_i = (\tau_{i + 1} - \tau_i)\lor K / 4$ . This will allow us to remove the "  $\lor K^2$ " in (12). In particular, since the episode length  $t_{\ell +1} - t_\ell$  in (12) can be upper bounded by the combined length of all significant phases  $[\tau_i,\tau_{i + 1})$  intersecting episode  $[t_{\ell},t_{\ell +1})$ , (12) gives us the desired bound.
C.4 Bounding  $\mathbb{E}[\sum_{t = t_\ell}^{t_{\ell +1} - 1}\delta_t(a_t^\sharp ,\hat{a}_t)]$  : Regret of Candidate Arm to Safe Arm
We first invoke an elementary lemma based on SST and STI to further help us decompose the regret.
Lemma 13. For any three arms  $a, b, c$ , under SST  $\cap$  STI:
$$
\delta_ {t} (a, c) \leq 2 \cdot \delta_ {t} (a, b) + \delta_ {t} (b, c) + 3 \cdot \delta_ {t} \left(a _ {t} ^ {*}, a\right),
$$
where  $a_{t}^{*}$  is the winner arm.
Proof. We handle all the different orderings:
(a)  $a\succ_{t}b,c$  : this already follows from Lemma 8 since then  $\delta_t(a,c)\leq 2\cdot \delta_t(a,b) + \delta_t(b,c)$
(b)  $c\succ_{t}a\succ_{t}b\colon \delta_{t}(a,c)\leq 0\leq \delta_{t}(a,b)$  and  $\delta_t(a^*,b)\geq \delta_t(c,b)$  by SST. Summing these together gives the result.
(c)  $b\succ_{t}a\succ_{t}c\colon \delta_{t}(a,c)\leq \delta_{t}(b,c)$  and  $\delta_t(a_t^*,a)\geq \delta_t(b,a)$  by SST. Summing these together gives the result.
(d)  $b, c \succ_{t} a$ :  $\delta_t(a^*, a)$  dominates the first two terms on the desired inequality's RHS.
Then, using Lemma 13, we further decompose the regret about the last master arm  $a_{\ell}$  defined in Section 4, which is the last arm to be evicted from  $\mathcal{A}_{\mathrm{master}}$  in episode  $[t_{\ell}, t_{\ell + 1})$ . We have
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, \hat {a} _ {t}\right) \leq 2 \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {\ell}\right) + \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {\ell}, \hat {a} _ {t}\right) + 3 \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {*}, a _ {t} ^ {\sharp}\right). \tag {13}
$$
As said earlier, the sum  $\sum_{t = t_{\ell}}^{t_{\ell +1} - 1}\delta_t(a_t^*,a_t^\sharp)$  is of the right order. Meanwhile, the sum  $\sum_{t = t_{\ell}}^{t_{\ell +1} - 1}\delta_t(a_\ell ,\hat{a}_t)$  is bounded using our candidate arm switching criterion (4). If  $\hat{a}_t = a_\ell$  for every round  $t\in [t_{\ell},t_{\ell +1})$  we are already done. Otherwise, let  $m_{\ell}$  be the last round that  $a_{\ell}$  is not the candidate arm  $\hat{a}_t$ . Then, we must have that since arm  $a_{\ell}$  is not evicted until round  $t_{\ell +1} - 1$ :
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \hat {\delta} _ {t} (a _ {\ell}, \hat {a} _ {t}) = \sum_ {t = t _ {\ell}} ^ {m _ {\ell} - 1} \hat {\delta} _ {t} (a _ {\ell}, \hat {a} _ {t}) \leq C \log (T) \sqrt {K \cdot (m _ {\ell} - t _ {\ell}) \vee K ^ {2}}
$$
Then, by concentration (Proposition 10) and the fact from earlier that each phase  $\left[\tau_{i},\tau_{i + 1}\right)$  is at least  $K / 4$  rounds (so that "  $\vee K^2$  can be removed in the above), we have that  $\sum_{t = t_{\ell}}^{t_{\ell +1} - 1}\delta_{t}(a_{\ell},\hat{a}_{t})$  is of the right order.
Then, turning back to (13), it remains to bound the regret of  $a_{\ell}$  to  $a_{t}^{\sharp}$  over the episode  $[t_{\ell}, t_{\ell + 1})$ .
C.5 Bounding  $\mathbb{E}[\sum_{t = t_\ell}^{t_{\ell +1} - 1}\delta_t(a_t^\sharp ,a_\ell)]$  : Regret of Last Master Arm to Safe Arm
First, following the outline of Section 4, we recall the definition of the last safe arm  $\alpha_{t}^{\sharp}$  at round  $t$  which is the last arm to incur significant regret in the unique phase  $[\tau_{i},\tau_{i + 1})$  containing round  $t$ .
We next formally define a bad segment, alluded to in Section 4. In what follows, bad segments will be defined with respect to a fixed arm  $a$  and conditional on the episode start time  $t_\ell$ . We will then show that, with respect to any arm  $a$ , not too many bad segments will elapse before  $a$  is evicted from  $\mathcal{A}_{\mathrm{master}}$ . In particular, this will hold for  $a = a_{\ell}$  which will ultimately be used to bound  $\delta_t(a_t^\sharp, a_\ell)$  across the episode  $[t_\ell, t_{\ell+1})$ .
Definition 5. Fix the episode start time  $t_\ell$ , and let  $[\tau_i, \tau_{i+1})$  be any phase intersecting  $[t_\ell, T)$ . For any arm  $a$ , define rounds  $s_{i,0}(a), s_{i,1}(a), s_{i,2}(a) \ldots \in [t_\ell \vee \tau_i, \tau_{i+1})$  recursively as follows: let  $s_{i,0}(a) := t_\ell \vee \tau_i$  and define  $s_{i,j}(a)$  as the smallest round in  $(s_{i,j-1}(a), \tau_{i+1})$  such that arm  $a$  satisfies for some fixed  $c_4 > 0$ :
$$
\sum_ {t = s _ {i, j - 1} (a)} ^ {s _ {i, j} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \geq c _ {4} \log (T) \sqrt {K \cdot \left(s _ {i , j} (a) - s _ {i , j - 1} (a)\right)}, \tag {14}
$$
if such a round  $s_{i,j}(a)$  exists. Otherwise, we let the  $s_{i,j}(a) \coloneqq \tau_{i+1} - 1$ . We refer to any interval  $[s_{i,j-1}(a), s_{i,j}(a))$  as a critical segment, and as a bad segment (w.r.t. arm  $a$ ) if (14) above holds.
Note that the above definition only depends on the arm  $a$  and the episode start time  $t_\ell$  and that, conditional on these variables, they are fixed in the environment. Observe also that the arm  $a_t^\sharp$  is fixed within any critical segment  $[s_{i,j-1}(a), s_{i,j}(a)) \subseteq [\tau_i, \tau_{i+1})$  since a significant shift does not occur inside  $[\tau_i, \tau_{i+1})$ .
Now relating this notion of a bad segment to our goal of bounding regret, a given bad segment  $[s_{i,j}(a), s_{i,j}(a))$  only contributes order  $\sqrt{K \cdot (s_{i,j}(a) - s_{i,j-1}(a))}$  to the regret of  $a$  to  $a_t^\sharp$ . At the same time, we claim that a well-timed replay (see Definition 6 below) running from  $s_{i,j-1}(a)$  to  $s_{i,j}(a)$  is capable of evicting arm  $a$ . This in turn allows us to reduce the regret bounding to studying the number and lengths of bad segments which elapse before one is detected by such a replay.
We first define such a well-timed and perfect replay.
Definition 6. Let  $\tilde{s}_{i,j}(a) := \lceil \frac{s_{i,j}(a) + s_{i,j+1}(a)}{2} \rceil$  denote the approximate midpoint of  $[s_{i,j}(a), s_{i,j+1}(a)]$ . Given a bad segment  $[s_{i,j}(a), s_{i,j+1}(a)]$ , define a perfect replay w.r.t.  $[s_{i,j}(a), s_{i,j+1}(a)]$  as a call of Base-Alg(tstart, m) where  $t_{\text{start}} \in [s_{i,j}(a), \tilde{s}_{i,j}(a)]$  and  $m \geq s_{i,j+1}(a) - s_{i,j}(a)$
Next, we analyze the behavior of a perfect replay on the bad segment  $[s_{i,j}(a), s_{i,j+1}(a)]$ . Going forward, we will use the simpler notation  $a_i^\sharp$  to denote the last safe arm of a phase  $[\tau_i, \tau_{i+1})$ , known in context.
Proposition 14. Suppose the good event  $\mathcal{E}_1$  holds (cf. Proposition 10). Let  $[s_{i,j}(a), s_{i,j+1}(a))$  be a bad segment with respect to arm  $a$ . Fix an integer  $m \geq s_{i,j+1}(a) - s_{i,j}(a)$ . Then, if a perfect replay with respect to  $[s_{i,j}(a), s_{i,j+1}(a))$  is scheduled, arm  $a$  will be evicted from  $\mathcal{A}_{master}$  by round  $s_{i,j+1}(a)$ .
Proof. Suppose event  $\mathcal{E}_1$  (i.e., our concentration bound (9)) holds. We first observe that by elementary calculations and the definition of the rounds  $s_{i,j}(a)$ , we have (in an identical fashion to Lemma 4 of Suk and Kpotufe [2022]):
$$
\sum_ {t = \tilde {s} _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {i} ^ {\sharp}, a\right) \geq \frac {c _ {4}}{4} \log (T) \sqrt {K \left(s _ {i , j + 1} (a) - \tilde {s} _ {i , j} (a)\right)}, \tag {15}
$$
where  $\tilde{s}_{i,j}(a)$  is the midpoint of  $[s_{i,j}(a), s_{i,j+1}(a))$  as defined in Definition 6. The above will come in handy in showing arm  $a$  is evicted over the second half of the bad segment  $[\tilde{s}_{i,j}(a), s_{i,j+1}(a)]$ .
Next, following the intuition given in Section 4, in order to relate  $\delta_t(a_i^\sharp, a)$  to  $\delta_t(\hat{a}_t, a)$ , we again use SST and STI via Lemma 13 on inequality (15):
$$
\sum_ {t = \tilde {s} _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} 2 \cdot \delta_ {t} \left(a _ {i} ^ {\sharp}, \hat {a} _ {t}\right) + \delta_ {t} \left(\hat {a} _ {t}, a\right) + 3 \cdot \delta_ {t} \left(a _ {t} ^ {*}, a _ {i} ^ {\sharp}\right) \geq \frac {c _ {4}}{4} \log (T) \sqrt {K \left(s _ {i , j + 1} (a) - \tilde {s} _ {i , j} (a)\right)}. \tag {16}
$$
We next show that  $\sum_{t = \tilde{s}_{i,j}(a)}^{s_{i,j + 1}(a)}\delta_t(a_i^\sharp ,\hat{a}_t)$  and  $\sum_{t = \tilde{s}_{i,j}(a)}^{s_{i,j + 1}(a)}\delta_t(a_t^*,a_i^\sharp)$  on the above LHS are small.
First, it is clear that any perfect replay  $\mathrm{Base - Alg}(t_{\mathrm{start}},m)$  will not evict  $a_i^\sharp$  since otherwise it incurs significant regret within phase  $[\tau_i,\tau_{i + 1})$  (see the earlier Lemma 11). At the same time, by the candidate arm switching criterion (4) and concentration:
$$
\sum_ {t = \tilde {s} _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {i} ^ {\sharp}, \hat {a} _ {t}\right) \leq c _ {5} \log (T) \sqrt {K \left(s _ {i , j + 1} (a) - \tilde {s} _ {i , j} (a)\right)}.
$$
Meanwhile, by the definition of significant regret (Definition 3),
$$
\sum_ {t = \tilde {s} _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {*}, a _ {i} ^ {\sharp}\right) \geq \sqrt {K \left(s _ {i , j + 1} (a) - \tilde {s} _ {i , j} (a)\right)}.
$$
Thus, for sufficiently large  $c_4 > 0$  in the definition of bad segments (Definition 5), we have that the above two inequalities can be combined with (16) to yield:
$$
\sum_ {t = \tilde {s} _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} \delta_ {t} (\hat {a} _ {t}, a) \geq \sqrt {K \left(s _ {i , j + 1} (a) - \tilde {s} _ {i , j} (a)\right)}.
$$
If arm  $a$  is evicted from  $\mathcal{A}_{\mathrm{master}}$  before round  $s_{i,j + 1}(a)$ , then we are already done. Otherwise, using the fact that  $\mathbb{E}[\hat{\delta}_t(\hat{a}_t,a)|\mathcal{F}_{t - 1}] = \delta_t(\hat{a}_t,a)$  for any round  $t\in [\tilde{s}_{i,j}(a),s_{i,j + 1}(a)]$  with  $a\in \mathcal{A}_t$ , we have that arm  $a$  will be evicted at round  $s_{i,j + 1}(a)$  using the above inequality and concentration.
It remains to show that, for any arm  $a$ , a perfect replay is scheduled w.h.p. before too much regret is incurred on the elapsed bad segments w.r.t.  $a$ . In particular, this will hold for the last master arm  $a_{\ell}$ , allowing us to bound the remaining term  $\mathbb{E}[\sum_{t = t_{\ell}}^{t_{\ell + 1} - 1} \delta_t(a_t^\sharp, a_{\ell})]$ . The argument will be identical to that of Appendix B.2 of Suk and Kpotufe [2022].
First, fix an arm  $a$  and an episode start time  $t_\ell$ . Then, define the bad round  $s(a) > t_\ell$  as follows:
Definition 7. (bad round) For a fixed round  $t_\ell$  and arm  $a$ , the bad round  $s(a) > t_\ell$  is defined as the smallest round which satisfies, for some fixed  $c_6 > 0$ :
$$
\sum_ {(i, j)} \sqrt {s _ {i , j + 1} (a) - s _ {i , j} (a)} > c _ {6} \log (T) \sqrt {s (a) - t _ {\ell}}, \tag {17}
$$
where the above sum is over all pairs of indices  $(i,j) \in \mathbb{N} \times \mathbb{N}$  such that  $[s_{i,j}(a), s_{i,j+1}(a))$  is a bad segment with  $s_{i,j+1}(a) < s(a)$ .
Our goal is then to then to show that arm  $a$  is evicted by some perfect replay scheduled within episode  $[t_{\ell}, t_{\ell + 1})$  with high probability before the bad round  $s(a)$  occurs. Going forward, to simplify notation we will drop the dependence on the fixed arm  $a$  in some variables.
For each bad segment  $[s_{i,j}(a), s_{i,j+1}(a))$ , recall that  $\tilde{s}_{i,j}(a)$  is the approximate midpoint between  $s_{i,j}(a)$  and  $s_{i,j+1}(a)$  (see Definition 6). Next, let  $m_{i,j} := 2^n$  where  $n \in \mathbb{N}$  satisfies:
Plainly,  $m_{i,j}$  is a dyadic approximation of the bad segment length. Next, recall that the Bernoulli  $B_{t,m}$  decides whether  $\operatorname{BaseAlg}(t,m)$  is scheduled at round  $t$  (see Line 6 of Algorithm 1). If for
some  $t \in [s_{i,j}(a), \tilde{s}_{i,j}(a)]$ ,  $B_{t,m_{i,j}} = 1$ , i.e. a perfect replay is scheduled, then  $a$  will be evicted from  $\mathcal{A}_{\mathrm{master}}$  by round  $s_{i,j+1}(a)$  (Proposition 14). We will show this happens with high probability via concentration on the sum
$$
S (a, t _ {\ell}) := \sum_ {(i, j): s _ {i, j + 1} (a) <   s (a)} \sum_ {t = s _ {i, j} (a)} ^ {\tilde {s} _ {i, j} (a)} B _ {t, m _ {i, j}},
$$
Note that the random variable  $S(a, t_{\ell})$  only depends on the replay scheduling probabilities  $\{B_{s,m}\}_{s,m}$  given a fixed arm  $a$  and episode start time  $t_{\ell}$ , since the bad round  $s(a)$  is also fixed given these quantities. This means that  $S(a, t_{\ell})$  is an independent sum of Bernoulli random variables  $B_{t,m_{i,j}}$ , conditional on  $t_{\ell}$ . Then, a Chernoff bound over the randomness of  $S(a, t_{\ell})$ , conditional on  $t_{\ell}$  yields
$$
\mathbb {P} \left(S (a, t _ {\ell}) \leq \frac {\mathbb {E} [ S (a , t _ {\ell}) \mid t _ {\ell} ]}{2} \mid t _ {\ell}\right) \leq \exp \left(- \frac {\mathbb {E} [ S (a , t _ {\ell}) \mid t _ {\ell} ]}{8}\right).
$$
The above RHS error probability is bounded above above by  $1 / T^3$  by observing:
$$
\mathbb {E} \left[ S (a, t _ {\ell}) \mid t _ {\ell} \right] \geq \sum_ {(i, j)} \sum_ {t = s _ {i, j} (a)} ^ {\tilde {s} _ {i, j} (a)} \frac {1}{\sqrt {m _ {i , j} \cdot (t - t _ {\ell})}} \geq \frac {1}{4} \sum_ {(i, j)} \sqrt {\frac {s _ {i , j + 1} (a) - s _ {i , j} (a)}{s (a) - t _ {\ell}}} \geq \frac {c _ {6}}{4} \log (T),
$$
for  $c_6 > 0$  large enough, where the last inequality follows from (17) in the definition of the bad round  $s(a)$  (Definition 7). Taking a further union bound over the choice of arm  $a \in [K]$  gives us that  $S(a, t_{\ell}) > 1$  for all choices of arm  $a$  (define this as the good event  $\mathcal{E}_2(t_{\ell})$ ) with probability at least  $1 - K / T^3$ . This means arm  $a$  will be evicted before round  $s(a)$  with high probability.
Recall on the event  $\mathcal{E}_1$  the concentration bounds of Proposition 10 hold. Then, on  $\mathcal{E}_1 \cap \mathcal{E}_2(t_\ell)$ , letting  $a = a_\ell$  in the preceding arguments we must have  $t_{\ell + 1} - 1 \leq s(a_\ell)$ . Thus, by the definition of the bad round  $s(a_\ell)$  (Definition 7), we must have:
$$
\sum_ {\left[ s _ {i, j} \left(a _ {\ell}\right), s _ {i, j + 1} \left(a _ {\ell}\right)\right): s _ {i, j + 1} \left(a _ {\ell}\right) <   t _ {\ell + 1} - 1} \sqrt {s _ {i , j + 1} \left(a _ {\ell}\right) - s _ {i , j} \left(a _ {\ell}\right)} \leq c _ {6} \log (T) \sqrt {t _ {\ell + 1} - t _ {\ell}}. \tag {18}
$$
Thus, by (14) in the definition of bad segments (Definition 5), over the bad segments  $[s_{i,j}(a_\ell), s_{i,j+1}(a_\ell))$  which elapse before the end of the episode  $t_{\ell+1} - 1$ , the regret of  $a_\ell$  to  $a_t^\sharp$  is at most order  $\log^2(T) \sqrt{K \cdot (t_{\ell+1} - t_\ell)}$ .
Over each non-bad critical segment  $[s_{i,j}(a_\ell), s_{i,j+1}(a_\ell))$ , the regret of playing arm  $a_\ell$  to  $a_i^\sharp$  is at most  $\log(T) \sqrt{\tau_{i+1} - \tau_i}$  since there is at most one non-bad critical segment per phase  $[\tau_i, \tau_{i+1})$  (follows from Definition 5).
So, we conclude that on event  $\mathcal{E}_1\cap \mathcal{E}_2(t_\ell)$
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {\ell}\right) \leq c _ {7} \log^ {2} (T) \sum_ {i \in \text {P H A S E S} \left(t _ {\ell}, t _ {\ell + 1}\right)} \sqrt {K \left(\tau_ {i + 1} - \tau_ {i}\right)}.
$$
Taking expectation, we have by conditioning first on  $t_\ell$  and then on event  $\mathcal{E}_1\cap \mathcal{E}_2(t_\ell)$ :
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {\ell}\right) \right] \leq \mathbb {E} _ {t _ {\ell}} \left[ \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} (t _ {\ell}) \right\} \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {\ell}\right) \mid t _ {\ell} \right] \right] \\ + T \cdot \mathbb {E} _ {t _ {\ell}} \left[ \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} ^ {c} \cup \mathcal {E} _ {2} ^ {c} (t _ {\ell}) \right\} \mid t _ {\ell} \right] \right] \\ \leq c _ {7} \log^ {2} (T) \mathbb {E} _ {t _ {\ell}} \left[ \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \left(t _ {\ell}\right) \right\} \sum_ {i \in \text {P H A S E S} \left(t _ {\ell}, t _ {\ell + 1}\right)} \sqrt {K \left(\tau_ {i + 1} - \tau_ {i}\right)} \mid t _ {\ell} \right] \right] \\ + \frac {2 K}{T ^ {2}} \\ \leq c _ {7} \log^ {2} (T) \mathbb {E} \left[ \mathbf {1} \{\mathcal {E} _ {1} \} \sum_ {i \in \text {P H A S E S} \left(t _ {\ell}, t _ {\ell + 1}\right)} \sqrt {\tau_ {i + 1} - \tau_ {i}} \right] + \frac {2}{T}, \\ \end{array}
$$
where in the last step we bound  $\mathbf{1}\{\mathcal{E}_1\cap \mathcal{E}_2(t_\ell)\} \leq \mathbf{1}\{\mathcal{E}_1\}$  and apply tower law again. This concludes the proof.