SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
c _ {2 7} \left(\log (T) \sqrt {K \cdot \left(n _ {B ^ {\prime}} \left([ \tilde {s} , s _ {i , j + 1} (a) ]\right) \vee K\right)} + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} \left([ \tilde {s}, s _ {i, j + 1} (a) ]\right)\right). \tag {32}
$$
We first upper bound the "variance" term, or the first term on the R.H.S. above. Let  $W \doteq s_{i,j+1}(a) - \tilde{s}$ . By Lemma 10, we have
$$
\begin{array}{l} \log (T) \sqrt {K \cdot \left(n _ {B ^ {\prime}} \left([ \tilde {s} , s _ {i , j + 1} (a) ]\right) \vee K\right)} \leq c _ {8} \left(\log (T) \cdot W ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}} + \log^ {3 / 2} (T) + K \log (T) \right. \\ \left. + \log^ {5 / 4} (T) \cdot K ^ {\frac {1 + 3 d / 4}{2 + d}} \cdot W ^ {\frac {1 / 2}{2 + d}}\right). \\ \end{array}
$$
Now, we also have
$$
s _ {i, j + 1} (a) - s _ {i, j} (a) \geq W \Rightarrow \log (T) \cdot \left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}} \geq \log (T) \cdot W ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}}
$$
Next, we note that by the definition of a proper bad segment ((29) in Definition 12) that
$$
2 \cdot (s _ {i, j + 1} (a) - s _ {i, j} (a)) \geq \sum_ {t = s _ {i, j} (a)} ^ {s _ {i, j} (a)} \delta_ {t} (a _ {t} ^ {\sharp}, a) \geq c _ {2 4} \log (T) \cdot (s _ {i, j + 1} (a) - s _ {i, j} (a)) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
This implies  $(s_{i,j+1}(a) - s_{i,j}(a))^{\frac{1}{2+d}} \geq \frac{c_{24}}{2} \log(T)$ . By similar reasoning, we have  $s_{i,j+1}(a) - s_{i,j}(a) \geq c_{28}K$ . From this, we conclude for  $c_{24} > 0$  large enough:
$$
\log (T) \cdot (s _ {i, j + 1} (a) - s _ {i, j} (a)) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}} \geq \log^ {3 / 2} (T) + K \log (T) + \log^ {5 / 4} (T) \cdot K ^ {\frac {1 + 3 d / 4}{2 + d}} \cdot W ^ {\frac {1 / 2}{2 + d}}.
$$
Thus, (32) is shown.
Now, we define a well-timed or perfect replay which, if scheduled, will detect the badness of arm  $a$  (in the sense of (5)) in bin  $B$  over a proper bad segment  $[s_{i,j}(a), s_{i,j+1}(a)]$ . The simplest such perfect replay is one which is scheduled directly from rounds  $s_{i,j}(a)$  to  $s_{i,j+1}(a)$ . We in fact show there is a spectrum of replays (of size the length of the segment  $\Omega(s_{i,j+1}(a) - s_{i,j}(a))$ ) each of which can detect arm  $a$  is bad in bin  $B$ , possibly by using a larger ancestor bin  $B' \supseteq B$ .
Definition 13 (Perfect Replay). For a fixed proper bad segment  $[s_{i,j}(a), s_{i,j+1}(a))$ , define a perfect replay as a Base-Alg  $(t_{\mathrm{start}}, M)$  with  $t_{\mathrm{start}} \in [s_{i,j+1}(a) - K2^{(m-2)(2+d)} + 1, s_{i,j+1}(a) - K2^{(m-2)(2+d)-1}]$  (where  $m \in \mathbb{N} \cup \{0\}$  is as in Lemma 13) and  $t_{\mathrm{start}} + M \geq s_{i,j+1}(a)$ .
The following proposition analyzes the behavior of a perfect replay and shows, if scheduled, it will in fact evict arm  $a$  from  $\mathcal{A}(B)$  within a proper bad segment  $[s_{i,j}(a), s_{i,j+1}(a)]$ .
Proposition 15 (Perfect Replay Evicts Bad Arm in Proper Bad Segment). Suppose event  $\mathcal{E}_1 \cap \mathcal{E}_2$  holds (see Notation 2). Fix a bin  $B$  at level  $r$ . Let  $[s_{i,j}(a), s_{i,j+1}(a))$  be a proper bad segment defined with respect to arm  $a$ . Let Base-Alg  $(t_{\text{start}}, M)$  be a perfect replay as defined above which becomes active at  $t_{\text{start}}$  (i.e.,  $Z_{t_{\text{start}}, M} = 1$ ) for a fixed integer  $M \geq s_{i,j+1}(a) - s_{i,j}(a)$ . Then:
(i) Let  $B'$  be the bin at level  $r_{\tilde{s} - s_{i,j}(a)}$  where  $\tilde{s} \doteq s_{i,j+1}(a) - K2^{(m-2)(2+d)-1} (m \in \mathbb{N} \cup \{0\})$  is as in Lemma 13, as in Lemma 14. Then, there is a "safe arm"  $a^\sharp(B')$  which will not be evicted from  $\mathcal{A}(B')$  by Base-Alg  $(t_{\mathrm{start}}, M)$  (or any of its children) before round  $s_{i,j+1}(a) + 1$ .
(ii) If  $a \in \mathcal{A}_t$  for all rounds  $t \in [\tilde{s}, s_{i,j+1}(a))$  where  $X_t \in B$ ,  $w$  then arm  $a$  will be excluded from  $\mathcal{A}(B)$  by round  $s_{i,j+1}(a)$ .
Proof. For (i), we can define the "safe arm"  $a^\sharp(B')$  in a similar fashion to how  $a_t^\sharp$  was defined. Let  $t_i(B')$  be the last round in  $[\tilde{s}, s_{i,j+1}(a)]$  such that  $X_{t_i(B')} \in B'$ . Then, since  $s_{i,j+1}(a) < \tau_{i+1}$ , we have that at round  $t_i(B')$ , there is a safe arm  $a^\sharp(B')$  which does not satisfy  $(\star)$  for any bin  $B''$  intersecting  $B'$  and interval of rounds  $I \subseteq [\tilde{s}, s_{i,j+1}(a)]$ . Once Base-Alg  $(t_{\text{start}}, M)$  is scheduled, it (or any of its children) cannot evict arm  $a^\sharp(B')$  from  $B'$  as doing so would imply it has significant regret in some bin intersecting  $B'$  (following the same calculations as in Lemma 11).
We next turn to (ii). We first suppose that arms  $a$  is active in bin  $B'$  from rounds  $\tilde{s}$  to  $s_{i,j+1}(a)$  (we'll carefully argue later this is indeed the case). We first observe  $\mathbb{E}[\hat{\delta}_t^B(a^\sharp(B), a) \mid \mathcal{F}_{t-1}] = \delta_t(a_i^\sharp(B), a)$  for any round  $t \in [\tilde{s}, s_{i,j+1}(a)]$  such that  $X_t \in B'$ . We next observe that:
$$
\sum_ {t = \bar {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a ^ {\sharp} \left(B ^ {\prime}\right), a\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\} \geq \sum_ {t = \bar {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\} - \sum_ {t = \bar {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a ^ {\sharp} \left(B ^ {\prime}\right)\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\}. \tag {33}
$$
By Lemma 14, the first term on the R.H.S. is at least
$$
c _ {2 6} \left(\log (T) \sqrt {K \cdot \left(n _ {B ^ {\prime}} \left([ \tilde {s} , s _ {i , j + 1} (a) ]\right) \vee K\right)} + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} \left([ \tilde {s}, s _ {i, j + 1} (a) ]\right)\right).
$$
Meanwhile, the second term on the R.H.S. of (33) is at most the same order by the definition of  $a^{\sharp}(B)$ . Thus, choosing  $c_{26}$  large enough gives us that
$$
\begin{array}{l} \sum_ {t = \tilde {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a ^ {\sharp} \left(B ^ {\prime}\right), a\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\} \geq c _ {2 9} \left(\log (T) \sqrt {K \cdot \left(n _ {B ^ {\prime}} \left([ \tilde {s} , s _ {i , j + 1} (a) ]\right) \vee K\right)} \right. \\ + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} \left(\left[ \tilde {s}, s _ {i, j + 1} (a) \right]\right)). \\ \end{array}
$$
Then, combining the above with our eviction criterion (5) and concentration (10), we have that arm  $a$  will be evicted in the bin  $B' \supseteq B$  at level  $r_{s_{i,j+1}(a) - \tilde{s}}$  by round  $s_{i,j+1}(a)$ .
Finally, it remains to show that, within Base-Alg  $(t_{\mathrm{start}},M)$  's play, arm  $a$  will not be evicted in any child of  $B^{\prime}$  before round  $s_{i,j + 1}(a)$ . This will follow from the fact that any perfect replay must use a level in  $\mathcal{R}$  of size at least  $r_W$ . In particular, by Definition 13, the starting round  $t_\mathrm{start}$  is "close enough" to the critical round  $s_{i,j + 1}(a) - K2^{(m - 2)(2 + d) - 1}$  so that it will not use a different level than the perfect replay which starts exactly at this critical round.
Formally, we have that the smallest level a perfect replay can use is  $r_{\tilde{W}}$  where  $\tilde{W} \doteq K \cdot 2^{(m-2)(2+d)} - 1$ .
Next, note that  $s_{i,j + 1}(a) - t_{\mathrm{start}} \leq K \cdot 2^{(m - 2)(2 + d)} - 1$  and so
$$
\left(\frac {K}{s _ {i , j + 1} (a) - t _ {\mathrm {s t a r t}}}\right) ^ {\frac {1}{2 + d}} \geq \left(\frac {K}{K \cdot 2 ^ {(m - 2) (2 + d)} - 1}\right) ^ {\frac {1}{2 + d}} \geq 2 ^ {- (m - 2)}.
$$
Thus,  $r_{\tilde{W}} \geq 2^{-(m-2)}$ . On the other hand,
$$
\left(\frac {K}{s _ {i , j + 1} (a) - \tilde {s}}\right) ^ {\frac {1}{2 + d}} = \frac {1}{2 ^ {m - 2 - \frac {1}{2 + d}}} \in [ 2 ^ {- (m - 2)}, 2 ^ {- (m - 3)}).
$$
Thus,  $2^{-(m-2)} = r_{s_{i,j+1}(a) - \tilde{s}}$  is also the level used to detect that arm  $a$  is bad in bin  $B'$ . Thus, we conclude that  $r_{\tilde{W}}$  is no smaller than the level  $r_{s_{i,j+1}(a) - \tilde{s}}$  used to evict arm  $a$  in bin  $B'$ . This means arm  $a$  cannot be evicted in a child of  $B'$  before round  $s_{i,j+1}(a)$ , if  $a$  is not already evicted in  $B$ .
Next, we show for any arm  $a$  (in particular,  $a = a_{r}(B)$ ), a perfect replay characterized by Definition 13 is scheduled with high probability if too many bad segments w.r.t.  $a$  elapse, thus bounding the regret of  $a$  to  $a_{i}^{\sharp}(B)$  over the phases  $[\tau_{i},\tau_{i + 1})$  intersecting block  $[s_{\ell}(r),e_{\ell}(r)]$ .
D.7 Bounding the Regret of the Last Master Arm  $a_{r}(B)$  to the Last Safe Arm  $a_{t}^{\sharp}$
Next, we bound the regret of a fixed arm  $a$  to  $a_{t}^{\sharp}$  over the bad segments w.r.t.  $a$  in  $B$ . Recall from earlier (28) that our remaining goal is to establish the following bound for every bin  $B \in \mathcal{T}_r$  at level  $r$ :
$$
\mathbb {E} \left[ \max  _ {a \in [ K ]} ^ {e _ {\ell} (r) \wedge E (B, a)} \sum_ {t = s _ {\ell} (r)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \right\} \right] \lesssim \sum_ {i \in \text {P H A S E S} (\ell , r)} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}},
$$
Note that the bad segments (Definition 12) are defined for a level  $r$ , block start time  $s_{\ell}(r)$ , and phase  $[\tau_i, \tau_{i+1})$ . In particular, they are defined independent of any choice of bin  $B$  at level  $r$ . We'll similarly define a bad round  $s(a)$  which will indicate when "too many" bad segments w.r.t.  $a$  have elapsed, irrespective of a choice of bin  $B$ . Then, we'll argue that for any bin  $B$  at level  $r$ ,  $E(B, a) \leq s(a)$ .
It should be understood that in what follows, we condition on the block start time  $s_{\ell}(r)$ . First, fix an arm  $a$  and define the bad round  $s(a) > s_{\ell}(r)$  as the smallest round which satisfies, for some fixed  $c_{30} > 0$ :
$$
\sum_ {(i, j)} \left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1 + d}{2 + d}} > c _ {3 0} \log (T) \left(s (a) - t _ {\ell}\right) ^ {\frac {1 + d}{2 + d}} \tag {34}
$$
where the above sum is over all pairs of indices  $(i,j) \in \mathbb{N} \times \mathbb{N}$  such that  $[s_{i,j}(a), s_{i,j+1}(a)]$  is a proper bad segment with  $s_{i,j+1}(a) < s(a)$ . We will show that, for any bin  $B$  at level  $r$ , arm  $a$  is evicted from  $\mathcal{A}(B)$  episode  $\ell$  with high probability by the time the bad round  $s(a)$  occurs.
For each proper bad segment  $[s_{i,j}(a), s_{i,j+1}(a))$ , let  $\tilde{s}_{i,j}(a) \doteq s_{i,j+1}(a) - K2^{(m-2)(2+d)-1}$  denote the "critical point" of the bad segment as in Lemma 13 and also let  $m_{i,j} \doteq 2^n$  where  $n \in \mathbb{N}$  satisfies:
$$
2 ^ {n} \geq s _ {i, j + 1} (a) - s _ {i, j} (a) > 2 ^ {n - 1}.
$$
Next, recall that the Bernoulli  $Z_{M,t}$  decides whether Base-Alg  $(t,M)$  activates at round  $t$  (see Line 6 of Algorithm 1). If for some  $t \in [\hat{s}_{i,j}(a),\tilde{s}_{i,j}(a)]$  where  $\hat{s}_{i,j}(a) \doteq s_{i,j + 1}(a) - K2^{(m - 2)(2 + d)} + 1$ ,  $Z_{m_{i,j},t} = 1$ , i.e. a perfect replay is scheduled, then  $a$  will be evicted from  $\mathcal{A}(B)$  by round  $s_{i,j + 1}(a)$  (Proposition 15).
We will show this happens with high probability via concentration on the sum  $\sum_{(i,j)}\sum_{t}Z_{m_{i,j},t}$  where  $j,i,t$  run through all  $t\in [\hat{s}_{i,j}(a),\tilde{s}_{i,j}(a))$  and all proper bad segments  $[s_{i,j}(a),s_{i,j + 1}(a))$  with  $s_{i,j + 1}(a) < s(a)$ . Note that these random variables, conditional on  $\mathbf{X}_T$ , depend only on the fixed arm  $a$ , the block start time  $s_\ell (r)$ , and the randomness of scheduling replays on Line 6. In particular, the  $Z_{m_{i,j},t}$  are independent conditional on  $t_\ell$ .
Then, a Chernoff bound over the randomization of CMETA on Line 6 of Algorithm 1 conditional on  $t_{\ell}$  yields
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {(i, j)} \sum_ {t} Z _ {m _ {i, j}, t} \leq \frac {\mathbb {E} \left[ \sum_ {(i , j)} \sum_ {t} Z _ {m _ {i , j} , t} \mid s _ {\ell} (r) , \mathbf {X} _ {T} \right]}{2} \Bigg | s _ {\ell} (r), \mathbf {X} _ {T}\right) \\ \leq \exp \left(- \frac {\mathbb {E} [ \sum_ {(i , j)} \sum_ {t} Z _ {m _ {i , j} , t} \mid s _ {\ell} (r) , \mathbf {X} _ {T} ]}{8}\right). \\ \end{array}
$$
We claim the error probability on the R.H.S. above is at most  $1 / T^3$ . To this end, we compute:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {(i, j)} \sum_ {t} Z _ {m _ {i, j}, t} \Bigg | s _ {\ell} (r), \mathbf {X} _ {T} \right] \geq \sum_ {(i, j)} \sum_ {t = \hat {s} _ {i, j} (a)} ^ {\tilde {s} _ {i, j} (a)} \left(\frac {1}{m _ {i , j}}\right) ^ {\frac {1}{2 + d}} \left(\frac {1}{t - t _ {\ell}}\right) ^ {\frac {1 + d}{2 + d}} \\ \geq \frac {1}{4} \sum_ {(i, j)} m _ {i, j} ^ {\frac {1 + d}{2 + d}} \left(\frac {1}{s (a) - t _ {\ell}}\right) ^ {\frac {1 + d}{2 + d}} \\ \geq \frac {c _ {3 0}}{4} \log (T), \\ \end{array}
$$
where the last inequality follows from (34). The R.H.S. above is larger than  $24\log (T)$  for  $c_{30}$  large enough, showing that the error probability is small. Taking a further union bound over the choice of arm  $a\in [K]$  gives us that  $\sum_{(i,j)}\sum_tZ_{m_{i,j},t} > 1$  for all choices of arm  $a$  (define this as the good event  $\mathcal{E}_3(s_\ell (r)))$  with probability at least  $1 - K / T^3$ .
Recall on the event  $\mathcal{E}_1 \cap \mathcal{E}_2$  the concentration bounds of Proposition 7 and Lemma 8 hold. Then, on  $\mathcal{E}_1 \cap \mathcal{E}_2 \cap \mathcal{E}_3(s_\ell(r))$ , we must have for each bin  $B \in \mathcal{T}_r$  at level  $r$ ,  $E(B, a) \leq s(a)$  since otherwise  $a$  would have been evicted in  $\mathcal{A}(B)$  by some perfect replay before the end of the block  $e_\ell(r)$  by virtue of  $\sum_{(i,j)} \sum_t Z_{m_{i,j},t} > 1$  for arm  $a$ . Thus, by the definition of the bad round  $s(a)$  (34), we must have:
$$
\sum_ {\left[ s _ {i, j} (a), s _ {i, j + 1} (a)\right): s _ {i, j + 1} (a) <   e _ {\ell} (r)} \left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1 + d}{2 + d}} \leq c _ {3 0} \log (T) \left(e _ {\ell} (r) - t _ {\ell}\right) ^ {\frac {1 + d}{2 + d}} \tag {35}
$$
Thus, by (29) in Definition 12, over the proper bad segments  $[s_{i,j}(a), s_{i,j+1}(a))$  which elapse before round  $e_{\ell}(r) \wedge E(B, a)$  in phase  $[\tau_i, \tau_{i+1})$ : the regret is at most
$$
\sum_ {(i, j)} \log (T) \cdot K ^ {\frac {1}{2 + d}} m _ {i, j} ^ {\frac {1 + d}{2 + d}} \leq \log^ {2} (T) \cdot K ^ {\frac {1}{2 + d}} \cdot \left(e _ {\ell} (r) - t _ {\ell}\right) ^ {\frac {1 + d}{2 + d}}
$$
Over each non-proper bad segment  $[s_{i,j}(a), s_{i,j-1}(a))$  and the last segment  $[s_{i,j}(a), e_\ell(r) \wedge E(B, a)]$ , the regret of playing arm  $a$  to  $a_t^\sharp$  is at most  $\log(T) \cdot K^{\frac{1}{2+d}} m_{i,j}^{\frac{1+d}{2+d}}$  since there is at most one non-proper bad segment per phase  $[\tau_i, \tau_{i+1})$  (see (29) in Definition 12).
So, we conclude that on event  $\mathcal{E}_1\cap \mathcal{E}_2\cap \mathcal{E}_3(s_\ell (r))$  : for any bin  $B\in \mathcal{T}_r$
$$
\max _ {a \in [ K ]} \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r) \wedge E (B, a)} \delta_ {t} (a _ {t} ^ {\sharp}, a) \leq 2 c _ {3 0} \log^ {2} (T) \sum_ {i \in \mathrm {P H A S E S} (\ell , r)} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
Let event  $\mathcal{G} \doteq \mathcal{E}_1 \cap \mathcal{E}_2$ . Then, taking expectation, we have by conditioning first on  $s_\ell(r)$  and then on event  $\mathcal{G} \cap \mathcal{E}_3(s_\ell(r))$ :
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {a \in [ K ]} \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r) \wedge E (B, a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mathbf {1} \{\mathcal {G} \} \middle | \mathbf {X} _ {T} \right] \\ \leq \mathbb {E} _ {s _ {\ell} (r)} \left[ \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {G} \cap \mathcal {E} _ {3} \left(s _ {\ell} (r)\right) \right\} \max  _ {a \in [ K ]} \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r) \wedge E (B, a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {r} (B)\right) \middle | s _ {\ell} (r) \right] \middle | \mathbf {X} _ {T} \right] \\ + T \cdot \mathbb {E} _ {s _ {\ell} (r)} \left[ \mathbb {E} \left[ \mathbf {1} \{\mathcal {G} \cap \mathcal {E} _ {3} ^ {c} (s _ {\ell} (r)) \} \Bigg | s _ {\ell} (r) \right] \Bigg | \mathbf {X} _ {T} \right] \\ \end{array}
$$
We first handle the first double expectation on the R.H.S. above. We have this is at most
$$
\begin{array}{l} \leq 2 c _ {3 0} \log^ {2} (T) \mathbb {E} _ {s _ {\ell} (r)} \left[ \mathbb {E} \left[ \mathbf {1} \{\mathcal {G} \cap \mathcal {E} _ {3} (t _ {\ell}) \} \sum_ {i \in \mathrm {P H A S E S} (\ell , r)} K ^ {\frac {1}{2 + d}} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} \Bigg | s _ {\ell} (r) \right] \Bigg | \mathbf {X} _ {T} \right] \\ \leq 2 c _ {3 0} \log^ {2} (T) \mathbb {E} \left[ \mathbf {1} \{\mathcal {G} \} \sum_ {i \in \mathrm {P H A S E S} (\ell , r)} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} \middle | \mathbf {X} _ {T} \right], \\ \end{array}
$$
where in the last step we bound  $\mathbf{1}\{\mathcal{G} \cap \mathcal{E}_3(s_\ell(r))\} \leq \mathbf{1}\{\mathcal{G}\}$  and apply tower law again. The above R.H.S. is of the right order w.r.t. (19). So, it remains to bound
$$
T \cdot \mathbb {E} _ {s _ {\ell} (r)} \left[ \mathbb {E} \left[ \mathbf {1} \{\mathcal {G} \cap \mathcal {E} _ {3} ^ {c} (s _ {\ell} (r)) \} \Bigg | s _ {\ell} (r) \right] \Bigg | \mathbf {X} _ {T} \right].
$$
We first observe that events  $\mathcal{G}$  and  $\mathcal{E}_3^c (s_\ell (r))$  are independent conditional on  $\mathbf{X}_T$  and  $s_{\ell}(r)$  since the former event only depends on the distribution of  $Y_{s_{\ell}(r)},\ldots ,Y_{T}$  while the latter event depends on the distribution of the Bernoulli's  $\{Z_{M,t}\}_{t > s_{\ell}(r),M}$  (which are independent). Then, writing  $\mathbf{1}\{\mathcal{G}\cap \mathcal{E}_3^c (s_\ell (r))\} = \mathbf{1}\{\mathcal{G}\} \cdot \mathbf{1}\{\mathcal{E}_3^c (s_\ell (r))\}$ , the above becomes at most  $\mathbb{E}[\mathbf{1}\{\mathcal{G}\} \cdot (K / T^2)\mid \mathbf{X}_T]$  which is also of the right order w.r.t. (19).
E Proof of Corollary 5
The proof of Corollary 5 will follow in a similar fashion to the proof of Corollary 2 in Suk and Kpotufe [2022], which relates the total-variation rates to significant shifts in the non-stationary MAB setting. A novel difficulty here is that our notion of significant shift  $\tau_{i}(\mathbf{X}_{T}),\tilde{L} (\mathbf{X}_{T})$  (Definition 6) depends on the full context sequence  $\mathbf{X}_T$ , and so it is not clear how the (random) significant phases  $[\tau_i(\mathbf{X}_T),\tau_{i + 1}(\mathbf{X}_T))$  relate to the total-variation  $V_{T}$ , which is a deterministic quantity.
Our strategy will be to first convert the regret rate of Theorem 3 into one which depends on a weaker worst-case notion of significant shift which does not depend on the observed  $\mathbf{X}_T$ . Although this notion of shift is weaker, it will be easier to relate to the total-variation quantity  $V_{T}$ .
Recall that  $\delta_t^a (x)\doteq \max_{a'\in [K]}\delta_t^{a',a}(x)$  and  $\delta_t^{a',a}(x)\doteq f_t^{a'}(x) - f_t^a (x)$  are the gap functions in mean rewards.
Definition 14 (worst-case sig shift). Let  $\tau_0 = 1$ . Then, recursively for  $i \geq 0$ , the  $(i + 1)$ -th worst-case significant shift is recorded at time  $\tilde{\tau}_{i+1}$ , which denotes the earliest time  $\tilde{\tau} \in (\tilde{\tau}_i, T]$  such that there exists  $x \in \mathcal{X}$  such that for every arm  $a \in [K]$ , there exists round  $s \in [\tilde{\tau}_i, \tilde{\tau}]$ , such that  $\delta_s^a(x) \geq \left(\frac{K}{t - \tilde{\tau}_i}\right)^{\frac{1}{2 + d}}$ .
We will refer to intervals  $[\tilde{\tau}_i, \tilde{\tau}_{i+1}), i \geq 0$ , as worst-case (significant) phases. The unknown number of such phases (by time  $T$ ) is denoted  $\tilde{L}_{\mathrm{pop}} + 1$ , whereby  $[\tilde{\tau}_{\tilde{L}_{\mathrm{pop}}}, \tilde{\tau}_{\tilde{L}_{\mathrm{pop}} + 1})$ , for  $\tau_{\tilde{L}_{\mathrm{pop}} + 1} \doteq T + 1$ , denotes the last phase.
We next claim that
$$
\mathbb {E} _ {\mathbf {X} _ {T}} \left[ \sum_ {i = 0} ^ {\tilde {L} (\mathbf {X} _ {T})} \big (\tau_ {i + 1} (\mathbf {X} _ {T}) - \tau_ {i} (\mathbf {X} _ {T}) \big) ^ {\frac {1 + d}{2 + d}} \right] \leq c _ {2 4} \sum_ {i = 0} ^ {\tilde {L} _ {\mathrm {p o p}}} (\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}) ^ {\frac {1 + d}{2 + d}}.
$$
This follows since the experienced significant phases  $\left[\tau_{i}(\mathbf{X}_{T}),\tau_{i + 1}(\mathbf{X}_{T})\right)$  interleave the population analogues  $[\tilde{\tau}_i,\tilde{\tau}_{i + 1})$  in the following sense: at each significant shift  $\tau_{i + 1}(\mathbf{X}_T)$ , for each arm  $a\in [K]$ , there is a round  $s\in [\tau_i(\mathbf{X}_T),\tau_{i + 1}(\mathbf{X}_T)]$  such that for  $\delta_s(X_{\tau_{i + 1}}) > \left(\frac{K}{\tau_{i + 1} - \tau_i}\right)^{\frac{1}{2 + d}}$ . This means there must be a worst-case significant shift  $\tilde{\tau}_j$  in the interval  $[\tau_i(\mathbf{X}_T),\tau_{i + 1}(\mathbf{X}_T)]$  since the criterion of Definition 14 is triggered at  $x = X_{\tau_{i + 1}}$ . This in turn allows us to conclude that each worst-case significant phase  $[\tilde{\tau}_i,\tilde{\tau}_{i + 1})$  can intersect at most two significant phases  $[\tau_i(\mathbf{X}_T),\tau_{i + 1}(\mathbf{X}_T))$ .
Thus, by the sub-additivity of the function  $x \mapsto x^{\frac{1 + d}{2 + d}}$  (dropping dependence on  $\mathbf{X}_T$  in  $\tilde{L}$ ,  $\tau_i$  to ease notation):
$$
\begin{array}{l} \sum_ {i = 0} ^ {\tilde {L}} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} \leq \sum_ {i = 0} ^ {\tilde {L}} \sum_ {j: [ \tilde {\tau} _ {j}, \tilde {\tau} _ {j + 1}) \cap [ \tau_ {i}, \tau_ {i + 1}) \neq \emptyset} | [ \tilde {\tau} _ {j}, \tilde {\tau} _ {j + 1}) \cap [ \tau_ {i}, \tau_ {i + 1}) | ^ {\frac {1 + d}{2 + d}} \\ \leq c _ {2 4} \sum_ {j = 0} ^ {\tilde {L} _ {\mathrm {p o p}}} \left(\tilde {\tau} _ {j + 1} - \tilde {\tau} _ {j}\right) ^ {\frac {1 + d}{2 + d}}, \\ \end{array}
$$
where we use Jensen's inequality for  $a^p + b^p \leq 2^{1 - p}(a + b)^p$  for  $p \in (0,1)$  and  $a, b \geq 0$  in the last step to re-combine the subintervals of each worst-case significant phase  $[\tilde{\tau}_j, \tilde{\tau}_{j+1})$ .
Then, it suffices to show
$$
\sum_ {j = 0} ^ {\tilde {L} _ {\mathrm {p o p}}} \left(\tilde {\tau} _ {j + 1} - \tilde {\tau} _ {j}\right) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} \lesssim T ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} + \left(V _ {T} \cdot K\right) ^ {\frac {1}{3 + d}} \cdot T ^ {\frac {2 + d}{3 + d}}. \tag {36}
$$
To start, fix a worst-case significant phase  $\left[\tilde{\tau}_i,\tilde{\tau}_{i + 1}\right)$  such that  $\tau_{i + 1} < T + 1$ . By Definition 14, there exists a context  $x_{i}\in \mathcal{X}$  such that for arm  $a_i\in \mathrm{argmax}_{a\in [K]}f_{\tilde{\tau}_{i + 1}}^a (x_i)$  we have there exists a round  $t_i\in [\tau_i,\tau_{i + 1}]$  such that:
$$
\delta_ {t _ {i}} ^ {a _ {i}} (x _ {i}) > \left(\frac {K}{\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}}\right) ^ {\frac {1}{2 + d}}.
$$
On the other hand,  $\delta_{\tilde{\tau}_{i + 1}}^{a_i}(x_i) = 0$  by the definition of arm  $a_i$  being the best at  $x_{i}$  at round  $\tilde{\tau}_{i + 1}$ . Thus, letting  $\tilde{a}_i\in \mathrm{argmax}_{a\in [K]}f_{t_i}^a (x_i)$  be an optimal arm at  $x_{i}$  at round  $t_i$ , we have: