SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our strategy, similar to that of Section B.1 in Suk and Kpotufe [2022], is to partition the rounds when  $a$  is played by a replay after round  $t_r^a$  according to which replay is active and not accounted for by another replay. This involves carefully identifying a subclass of replays whose durations while playing  $a$  in  $B$  span all the rounds where  $a$  is played in  $B$  after  $t_r^a$ . Then, we cover the times when  $a$  is played by a collection of intervals corresponding to the schedules of this subclass of replays, on each of which we can employ the eviction criterion (5) and concentration bound as done earlier.
For this purpose, we first define the following terminology (which is all w.r.t. a fixed arm  $a$ , along with fixed block  $[e_{\ell}(r), s_{\ell}(r)]$  and bin  $B \in \mathcal{T}_r$ ):
Definition 11.
(i) For each scheduled and activated Base-Alg  $(s, m)$ , let the round  $M(s, m)$  be the minimum of two quantities: (a) the last round in  $[s, s + m]$  when arm  $a$  is retained in  $\mathcal{A}(B)$  by Base-Alg  $(s, m)$  and all of its children, and (b) the last round that Base-Alg  $(s, m)$  is active and not permanently interrupted by another replay. Call the interval  $[s, M(s, m)]$  the active interval of Base-Alg  $(s, m)$ .
(ii) Call a replay Base-Alg  $(s, m)$  proper if there is no other scheduled replay Base-Alg  $(s', m')$  such that  $[s, s + m] \subset (s', s' + m')$  where Base-Alg  $(s', m')$  will become active again after round  $s + m$ . In other words, a proper replay is not scheduled inside the scheduled range of rounds of another replay. Let  $\mathrm{PROPER}(s_{\ell}(r), e_{\ell}(r))$  be the set of proper replays scheduled to start in the block  $[s_{\ell}(r), e_{\ell}(r)]$ .
(iii) Call a scheduled replay Base-Alg  $(s, m)$  a sub-replay if it is non-proper and if each of its ancestor replays (i.e., previously scheduled replays whose durations have not concluded) Base-Alg  $(s', m')$  satisfies  $M(s', m') < s$ . In other words, a sub-replay either permanently interrupts its parent or does not, but is scheduled after its parent (and all its ancestors) stops playing arm  $a$  in  $B$ . Let SUBPROPER  $(s_{\ell}(r), s_{\ell}(r))$  be the set of all sub-replays scheduled before round  $t_{\ell+1}$ .
Equipped with this language, we now show some basic claims which essentially reduce analyzing the complicated hierarchy of replays to analyzing the active intervals of replays in  $\mathrm{PROPER}(s_{\ell}(r), e_{\ell}(r)) \cup \mathrm{SUBPROPER}(s_{\ell}(r), s_{\ell}(r))$ .
Proposition 12. The active intervals
$$
\{[ s, M (s, m) ]: \text {B a s e - A l g} (s, m) \in \text {P R O P E R} (s _ {\ell} (r), e _ {\ell} (r)) \cup \text {S u b P R O P E R} (s _ {\ell} (r), s _ {\ell} (r)) \},
$$
are mutually disjoint.
Proof. Clearly, the classes of replays  $\mathrm{PROPER}(t_{\ell}, t_{\ell+1})$  and  $\mathrm{SUBPROPER}(s_{\ell}(r), s_{\ell}(r))$  are disjoint. Next, we show the respective active intervals  $[s, M(s, m)]$  and  $[s', M(s', m')]$  of any two Base-Alg  $(s, m)$  and Base-Alg  $(s', m') \in \mathrm{PROPER}(s_{\ell}(r), e_{\ell}(r)) \cup \mathrm{SUBPROPER}(s_{\ell}(r), s_{\ell}(r))$  are disjoint.
Figure 2: Shown are replay scheduled durations (in gray) with dots marking when arm  $a$  is reintroduced to  $\mathcal{A}_t$ . Black segments indicate the active intervals  $[s, M(s, m)]$  for proper replays and sub-replays. Note that the rounds where  $a \in \mathcal{A}_t$  in the left unlabeled replay's duration are accounted for by the larger proper replay.
1. Proper replay vs. sub-replay: a sub-replay can only be scheduled after the round  $M(s, m)$  of the most recent proper replay Base-Alg  $(s, m)$  (which is necessarily an ancestor). Thus, the active intervals of proper replays and sub-replays are disjoint.
2. Two distinct proper replays: two such replays can only intersect by one permanently interrupting the other, and since  $M(s,m)$  always occurs before the permanent interruption of Base-Alg  $(s,m)$ , we have the active intervals of two such replays are disjoint.
3. Two distinct sub-replays: consider two non-proper replays Base-Alg  $(s,m)$  ,Base-Alg  $(s^{\prime},m^{\prime})\in \mathrm{SUBPROPER}(s_{\ell}(r),s_{\ell}(r))$  with  $s^\prime >s$  . Suppose their active intervals intersect and that Base-Alg  $(s,m)$  is an ancestor of Base-Alg  $(s^{\prime},m^{\prime})$  Then, if Base-Alg  $(s^{\prime},m^{\prime})$  is a sub-replay, we must have  $s^\prime >M(s,m)$  , which means that  $[s^{\prime},M(s^{\prime},m^{\prime})]$  and  $[s,M(s,m)]$  are disjoint.
Next, we claim that the active intervals  $[s,M(s,m)]$  for Base-Alg  $(s,m)\in \mathrm{PROPER}(t_{\ell},t_{\ell +1})\cup$  SUBPROPER  $(s_{\ell}(r),s_{\ell}(r))$  contain all the rounds where  $a$  is played in  $B$  after being evicted from  $\mathcal{A}_{\mathrm{master}}(B)$ . To show this, we first observe that for each round  $t$  when a replay is active, there is a unique proper replay associated to  $t$ , namely the proper replay scheduled most recently. Next, note that any round  $t > t_r^a$  where  $X_{t}\in B$  and where arm  $a\in \mathcal{A}_t$  must belong to the active interval  $[s,M(s,m)]$  of this unique proper replay Base-Alg  $(s,m)$  associated to round  $t$ , or else satisfies  $t > M(s,m)$  in which case a unique sub-replay Base-Alg  $(s',m')\in \mathrm{SUBPROPER}(s_{\ell}(r),s_{\ell}(r))$  is active at round  $t$  and not yet permanently interrupted by round  $t$ . Thus, it must be the case that  $t\in [s',M(s',m')]$ .
Overloading notation, we'll let  $\mathcal{A}_t(B)$  be the value of  $\mathcal{A}(B)$  for the Base-Alg active at round  $t$ . Next, note that every round  $t \in [s, M(s, m)]$  for a proper or subproper Base-Alg  $(s, m)$  is clearly a round where  $a \in \mathcal{A}_t(B)$  and no such round is accounted for twice by Proposition 12. Thus,
$$
\{t \in (t _ {r} ^ {a}, e _ {\ell} (r) ]: a \in \mathcal {A} _ {t} (B) \} = \quad \bigsqcup_ {[ s, M (s, m) ].}
$$
Base-Alg  $(s,m)\in \mathrm{PROPER}(s_{\ell}(r),e_{\ell}(r))\cup \mathrm{SUBPROPER}(s_{\ell}(r),s_{\ell}(r))$
Then, we can rewrite the second double sum in (21) as:
$$
\sum_ {a = 1} ^ {K} \sum_ {\text {B a s e - A l g} (s, m) \in \text {P R O P E R} (s _ {\ell} (r), e _ {\ell} (r)) \cup \text {S U B P R O P E R} (s _ {\ell} (r), s _ {\ell} (r))} Z _ {m, s} \cdot \sum_ {t = s \vee t _ {r} ^ {a}} ^ {M (s, m)} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \}.
$$
Recall in the above that the Bernoulli R.V.  $Z_{m,s}$  (see Line 6 of Algorithm 1) decides whether  $\operatorname{Base - Alg}(s,m)$  is scheduled.
Further bounding the sum over  $t$  above by its positive part, we can expand the middle sum above over Base-Alg  $(s,m) \in \mathrm{PROPER}(t_{\ell},t_{\ell +1}) \cup \mathrm{SUBPROPER}(s_{\ell}(r),s_{\ell}(r))$  to instead be over all Base-Alg  $(s,m)$ , or obtain:
$$
\sum_ {a = 1} ^ {K} \sum_ {\text {B a s e - A l g} (s, m)} Z _ {m, s} \cdot \left(\sum_ {t = s \vee t _ {r} ^ {a}} ^ {M (s, m)} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \}\right) _ {+},
$$
where the sum is over all replays Base-Alg  $(s,m)$ , i.e.  $s\in \{t_{\ell} + 1,\dots ,t_{\ell +1} - 1\}$  and  $m\in$ $\{2,4,\ldots ,2^{\lceil \log (T)\rceil}\}$ . It then remains to bound the contributed relative regret of each Base-Alg  $(s,m)$  in the interval  $[s\vee t_r^a,M(s,m)]$ , which will follow similarly to the previous steps in bounding the first double sum of (21).
We first have (now overloading the notation  $M(s, m)$  as  $M(s, m, a)$  for clarity), i.e. combining our concentration bound (10) with the eviction criterion (5) and applying Lemma 9:
$$
\begin{array}{l} \sum_ {t = s \lor t _ {r} ^ {a}} ^ {M (s, m, a)} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \} \\ \leq \frac {c _ {2 1} \left(\log^ {1 / 2} (T) r ^ {d} \cdot K ^ {\frac {1}{2 + d}} \cdot M (s , m , a) ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) \cdot M (s , m , a) \cdot \mu (B)}\right)}{\min _ {t \in [ s , M (s , m , a) ]} | \mathcal {A} _ {t} |} \\ \end{array}
$$
Thus, it remains to bound
$$
\sum_ {a, s, m} Z _ {m, s} \left(\frac {\log^ {1 / 2} (T) r ^ {d} K ^ {\frac {1}{2 + d}} M (s , m , a) ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) M (s , m , a) \mu (B)}}{\min _ {t \in [ s , M (s , m , a) ]} | \mathcal {A} _ {t} |}\right).
$$
Swapping the outer two sums and, similar to before, recognizing that  $\sum_{a=1}^{K} \frac{1}{\min_{t \in [s, M(s, m, a)]} |\mathcal{A}_t|} \leq \log(K)$  by summing over arms in the order they are evicted by Base-Alg  $(s, m)$ , we have that it remains to bound:
$$
\log (K) \sum_ {\text {B a s e - A l g} (s, m)} Z _ {m, s} \cdot c _ {2 1} \left(\log^ {1 / 2} (T) \cdot r ^ {d} \cdot K ^ {\frac {1}{2 + d}} \cdot \tilde {m} ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) \cdot \tilde {m} \cdot \mu (B)}\right), \tag {24}
$$
where  $\tilde{m} \doteq m \wedge (e_{\ell}(r) - s_{\ell}(r))$  (note we may freely restrict all active intervals to the current block  $[s_{\ell}(r), e_{\ell}(r)]$ ). Let
$$
\begin{array}{l} R (m, B) \doteq \left(c _ {2 1} \left(\log^ {1 / 2} (T) \cdot r ^ {d} \cdot K ^ {\frac {1}{2 + d}} \cdot \tilde {m} ^ {\frac {1 + d}{2 + d}} + (K \wedge \tilde {m}) \log (T) \right. \right. \\ \left. \left. + \sqrt {\log (T) \cdot \tilde {m} \cdot \mu (B)}\right)\right) \wedge n _ {B} ([ s, s + m ]). \\ \end{array}
$$
Then, in light of the previous calculations,  $R(m,B)$  is an upper bound on the within-bin  $B$  regret contributed by a replay of total duration  $m$  (note we can always coarsely upper bound this regret by  $n_B([s,s + m])$ .
Then, plugging  $R(m,B)$  into (24) gives via tower law:
$$
\mathbb {E} \left[ \mathbb {E} \left[ \sum_ {s, m} Z _ {m, s} \cdot R (m, B) \mid s _ {\ell} (r) \right] \mid \mathbf {X} _ {T} \right] = \mathbb {E} \left[ \sum_ {s = s _ {\ell} (r)} ^ {T} \sum_ {m} \mathbb {E} [ Z _ {m, s} \mathbf {1} \{s \leq e _ {\ell} (r) \} \mid s _ {\ell} (r) ] R (m, B) \mid \mathbf {X} _ {T} \right] \tag {25}
$$
Next, we observe that  $Z_{m,s}$  and  $\mathbf{1}\{s \leq e_{\ell}(r)\}$  are independent conditional on  $s_{\ell}(r)$  since  $\mathbf{1}\{s \leq e_{\ell}(r)\}$  only depends on the scheduling and observations of base algorithms scheduled before round  $s$ . Additionally, conditional on  $s_{\ell}(r)$ , the episode start time  $t_{\ell}$  is also fixed since the two are deterministically related (see Fact 3). Then, we have that:
$$
\mathbb {P} (Z _ {m, s} = 1) = \mathbb {E} [ Z _ {m, s} \mid s _ {\ell} (r) ] = \mathbb {E} [ Z _ {m, s} \mid t _ {\ell}, s _ {\ell} (r) ] = \left(\frac {1}{m}\right) ^ {\frac {1}{2 + d}} \cdot \left(\frac {1}{s - t _ {\ell}}\right) ^ {\frac {1 + d}{2 + d}}.
$$
$$
\begin{array}{l} \mathbb {E} \left[ Z _ {m, s} \cdot \mathbf {1} \left\{s \leq e _ {\ell} (r) \right\} \mid s _ {\ell} (r) \right] = \mathbb {E} \left[ Z _ {m, s} \mid s _ {\ell} (r), t _ {\ell} \right] \cdot \mathbb {E} \left[ \mathbf {1} \left\{s \leq e _ {\ell} (r) \right\} \mid s _ {\ell} (r), t _ {\ell} \right] \\ = \left(\frac {1}{m}\right) ^ {\frac {1}{2 + d}} \cdot \left(\frac {1}{s - t _ {\ell}}\right) ^ {\frac {1 + d}{2 + d}} \cdot \mathbb {E} [ \mathbf {1} \{s \leq e _ {\ell} (r) \} \mid s _ {\ell} (r) ]. \\ \end{array}
$$
Plugging this into (25) and unconditioning, we obtain:
$$
\mathbb {E} \left[ \sum_ {s = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \sum_ {n = 1} ^ {\lceil \log (T) \rceil} \left(\frac {1}{2 ^ {n}}\right) ^ {\frac {1}{2 + d}} \left(\frac {1}{s - t _ {\ell}}\right) ^ {\frac {1 + d}{2 + d}} \cdot R \left(2 ^ {n}, B\right) \Bigg | \mathbf {X} _ {T} \right] \tag {26}
$$
We first evaluate the inner sum over  $n$ . Note that
$$
\begin{array}{l} \sum_ {n = 1} ^ {\lceil \log (T) \rceil} \left(\frac {1}{2 ^ {n}}\right) ^ {\frac {1}{2 + d}} \cdot \left(2 ^ {n} \wedge \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} \leq \log (T) \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {d}{2 + d}} \right. \\ \sum_ {n = 1} ^ {\lceil \log (T) \rceil} \left(\frac {1}{2 ^ {n}}\right) ^ {\frac {1}{2 + d}} \sqrt {2 ^ {n} \wedge \left(e _ {\ell} (r) - s _ {\ell} (r)\right)} \leq \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {d / 2}{2 + d}} \\ \sum_ {n = 1} ^ {\lceil \log (T) \rceil} \left(\frac {1}{2 ^ {n}}\right) ^ {\frac {1}{2 + d}} (K \wedge 2 ^ {n}) \leq \log (T) \cdot K ^ {\frac {1 + d}{2 + d}}. \\ \end{array}
$$
Next, we plug in the above displays into (26). In particular, multiplying the above displays by  $(s - t_{\ell})^{-\frac{1 + d}{2 + d}}$  and taking a further sum over  $s \in [s_{\ell}(r), e_{\ell}(r)]$  gives an upper bound of:
$$
\begin{array}{l} \left(e _ {\ell} (r) - t _ {\ell}\right) ^ {\frac {1}{2 + d}} \left(\left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {d}{2 + d}} K ^ {\frac {1}{2 + d}} \cdot r ^ {d} \cdot \log^ {3 / 2} (T) \right. \\ \left. + \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {d / 2}{2 + d}} \sqrt {\log (T) \cdot r ^ {d}} + K ^ {\frac {1 + d}{2 + d}} \log (T)\right). \\ \end{array}
$$
First, we note the first term inside the parentheses above inside dominates the second term for all values of  $K$ ,  $e_{\ell}(r)$ ,  $s_{\ell}(r)$ ,  $T$ .
Next, note from Fact 4 that  $e_{\ell}(r) - t_{\ell} \leq c_{10}(e_{\ell}(r) - s_{\ell}(r))$  and so the above is at most:
$$
r ^ {d} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} \log^ {3 / 2} (T) + \log (T) K ^ {\frac {1 + d}{2 + d}} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1}{2 + d}}. \tag {27}
$$
We next recall from Fact 4 that each block  $[s_{\ell}(r), e_{\ell}(r)]$  is at least  $K$  rounds long. Thus,
$$
r ^ {d} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} \geq c _ {2 2} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}}.
$$
Thus, the second term of (27) is at most the order of the first term.
Showing (a) is order (19) then follows from writing  $e_{\ell}(r) - s_{\ell}(r)$  as the sum of effective phase lengths  $T(i, r, \ell)$  (see Definition 10) of the phases  $[\tau_i, \tau_{i+1})$  intersecting block  $[s_{\ell}(r), e_{\ell}(r)]$ , and using the sub-additivity of  $x \mapsto x^{\frac{1 + d}{2 + d}}$ .
- Bounding the Regret of the Last Master Arm  $a_{r}(B)$  to the Last Safe Arm  $a_{t}^{\sharp}$ . Before we proceed, we first convert  $\sum_{t=s_{\ell}(r)}^{e_{\ell}(r)} \delta_{t}(a_{t}^{\sharp}, a_{r}(B)) \cdot 1\{X_{t} \in B\}$  into a more convenient form in terms of the masses  $\mu(B)$ . By concentration (12) of Proposition 7, we have
$$
\begin{array}{l} \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {r} (B)\right) \cdot \mathbf {1} \{X _ {t} \in B \} \leq \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a _ {r} (B)\right) \cdot \mu (B) \\ + c _ {2} \left(\log (T) + \sqrt {\log (T) \left(e _ {\ell} (r) - s _ {\ell} (r)\right) \cdot \mu (B)}\right). \\ \end{array}
$$
We first show the two concentration error terms on the R.H.S. above are negligible with respect to the desired bound (19). The  $\log (T)$  term is clearly of the right order, whereas the other term is handled by Lemma 10, by which
$$
\sqrt {(e _ {\ell} (r) - s _ {\ell} (r)) \cdot \mu (B)} \leq c _ {7} (e _ {\ell} (r) - s _ {\ell} (r)) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {d / 2}{2 + d}} \leq c _ {2 3} \cdot r ^ {d} \cdot (e _ {\ell} (r) - s _ {\ell} (r)) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
By similar arguments to before, where we write  $e_{\ell}(r) - s_{\ell}(r) = \sum_{i \in \mathrm{PHASES}(\ell, r)} T(i, r, \ell)$  and use the sub-additivity of the function  $x \mapsto x^{\frac{1 + d}{2 + d}}$ , the above is of the right order w.r.t. (19).
Thus, going forward, by the strong density assumption (Assumption 2) and in light of (19), it suffices to show for any fixed arm  $a \in [K]$  (which we will take to be  $a_r(B)$  in the end):
$$
\sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r) \wedge E (B, a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \lesssim \sum_ {i \in \text {P H A S E S} (\ell , r)} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}}, \tag {28}
$$
where  $E(B, a)$  is the last round in block  $[s_{\ell}(r), e_{\ell}(r)]$  for which  $a \in \mathcal{A}_{\mathrm{master}}(B)$ .
This aggregate gap is the most difficult quantity to bound since arm  $a_{t}^{\sharp}$  may have been evicted from  $\mathcal{A}_{\mathrm{master}}(B)$  before round  $t$  and, thus, we rely on our replay scheduling (Line 6 of Algorithm 2) to bound the regret incurred while waiting to detect a large aggregate value of  $\delta_t(a_t^\sharp, a)$ .
In an abuse of notation, we'll conflate  $e_{\ell}(r)$  with the anticipated block end time based on  $s_{\ell}(r)$ ; that is, the end block time if no episode restart occurs within the block. Now, for each phase  $[\tau_i, \tau_{i+1})$  which intersects the block  $[s_{\ell}(r), e_{\ell}(r)]$ , our strategy will be to map out in time the local bad segments or subintervals of  $[\tau_i, \tau_{i+1})$  where a fixed arm  $a$  incurs significant regret to arm  $a_t^\sharp$  in bin  $B$ , roughly in the sense of  $(\star)$ . The argument will conclude by arguing that a well-timed replay is scheduled w.h.p. to detect some local bad segment in  $B$ , before too many elapse.
In particular, conditional on just the block start time  $s_{\ell}(r)$ , we define the bad segments for a fixed arm  $a$  and then argue that if too many bad segments w.r.t.  $a$  elapse in the block's anticipated set of rounds  $[s_{\ell}(r), e_{\ell}(r)]$ , then arm  $a$  will be evicted in bin  $B$ . Crucially, this will hold uniformly over all arms  $a$  and, in particular, for arm  $a \doteq a_r(B)$ . This will then bound the regret of  $a_r(B)$  in block  $[s_{\ell}(r), e_{\ell}(r)]$  in the sense of (28).
Notation. Going forward, we will drop the dependence on the level  $r$ , block  $[s_{\ell}(r), e_{\ell}(r)]$ , and episode  $[t_{\ell}, t_{\ell + 1})$  in certain definitions as they are fixed momentarily. Recall from Appendix D.2 that  $a_{t}^{\sharp}$  is the local last safe arm of the last round  $t_{i}(B) \in [\tau_{i}, \tau_{i + 1})$  such that  $X_{t_{i}(B)} \in B$  where  $B$  is the bin at level  $r_{\tau_{i + 1} - \tau_{i}}$  containing  $X_{t}$  (see Definition 9).
We first introduce the notion of a bad segment of rounds which is a minimal period where large regret in the sense of  $(\star)$  within bin  $B$  is detectable by a well-timed replay.
Definition 12. Fix an arm  $a$  and  $s_\ell(r)$ , and let  $[\tau_i, \tau_{i+1})$  be any phase intersecting  $[s_\ell(r), e_\ell(r)]$ . Define rounds  $s_{i,0}(a), s_{i,1}(a), s_{i,2}(a) \ldots \in [t_\ell \vee \tau_i, \tau_{i+1})$  recursively as follows: let  $s_{i,0}(a) \doteq t_\ell \vee \tau_i$  and define  $s_{i,j}(a)$  as the smallest round in  $(s_{i,j-1}(a), \tau_{i+1} \wedge e_\ell(r))$  such that arm  $a$  satisfies for some fixed  $c_{21} > 0$ :
$$
\sum_ {t = s _ {i, j - 1} (a)} ^ {s _ {i, j} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \geq c _ {2 4} \log (T) \cdot \left(s _ {i, j} (a) - s _ {i, j - 1} (a)\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}. \tag {29}
$$
Otherwise, we let the  $s_{i,j}(a) \doteq \tau_{i+1} - 1$ . We refer to the interval  $[s_{i,j-1}(a), s_{i,j}(a))$  as a bad segment. We call  $[s_{i,j-1}(a), s_{i,j}(a))$  a proper bad segment if (29) above holds.
It will in fact suffice to constrain our attention to proper bad segments, since non-proper bad segments  $[s_{i,j-1}(a), s_{i,j}(a))$  (where  $s_{i,j}(a) = \tau_{i+1} - 1$  and (29) is reversed) will be negligible in the regret analysis since there is at most one non-proper bad segment per phase  $[\tau_i, \tau_{i+1})$  (i.e., the regret of each non-proper bad segment is at most the R.H.S. of (28)).
We first establish some elementary facts about proper bad segments which will later serve useful in analyzing the detectability of  $(\star)$  along such segments of time.
Lemma 13. Let  $[s_{i,j}(a), s_{i,j+1}(a))$  be a proper bad segment defined w.r.t. arm  $a$ . Let  $m \in \mathbb{N} \cup \{0\}$  be such that  $r_{s_{i,j+1}(a) - s_{i,j}(a)} = 2^{-m}$ . Then, for some  $c_{25} = c_{25}(d) > 0$  depending on the dimension  $d$ :
$$
\sum_ {t = s _ {i, j + 1} (a) - K 2 ^ {(m - 2) (2 + d)} - 1} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \geq c _ {2 5} \log (T) \cdot K ^ {\frac {1}{2 + d}} \left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1 + d}{2 + d}}. \tag {30}
$$
Proof. First, we may assume  $s_{i,j+1}(a) - s_{i,j}(a) \geq 4 \cdot K$  by choosing  $c_{24}$  in (29) large enough (this will make  $m - 1 \geq 0$ ).
First, observe by the definition of  $r_{s_{i,j+1}(a) - s_{i,j}(a)}$  (Notation 1) that
$$
K 2 ^ {(m - 1) (2 + d)} \leq s _ {i, j + 1} (a) - s _ {i, j} (a) <   K 2 ^ {m (2 + d)}. \tag {31}
$$
Now, let  $\tilde{s} \doteq s_{i,j+1}(a) - K2^{(m-2)(2+d)-1}$ . Then, we have by (29) in the construction of the  $s_{i,j}(a)$ 's (Definition 12) that:
$$
\begin{array}{l} \sum_ {t = \tilde {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) = \sum_ {t = s _ {i, j} (a)} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) - \sum_ {t = s _ {i, j} (a)} ^ {\tilde {s}} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \\ \geq c _ {2 4} \log (T) K ^ {\frac {1}{2 + d}} \left(\left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1 + d}{2 + d}} - \left(\tilde {s} - s _ {i, j} (a)\right) ^ {\frac {1 + d}{2 + d}}\right) \\ \end{array}
$$
Let  $m_{i,j}(a)\doteq s_{i,j + 1}(a) - s_{i,j}(a)$ . Then, we have by (31) that:
$$
m _ {i, j} (a) \leq K 2 ^ {m (2 + d)} \Rightarrow \tilde {s} - s _ {i, j} (a) = m _ {i, j} (a) - K 2 ^ {(m - 2) (2 + d) - 1} \leq m _ {i, j} (a) \cdot (1 - 2 ^ {- 2 (2 + d) - 1}).
$$
Plugging this into our earlier bound the constants in our updated lower bound scale like:
$$
1 - \left(1 - \frac {1}{2 ^ {2 (2 + d) + 1}}\right) ^ {\frac {1 + d}{2 + d}} > 0.
$$
But, this last term is positive for all  $d \in \mathbb{N} \cup \{0\}$  and only depends on  $d$ .
Lemma 14 (Aggregate Gap Dominates Concentration Error). Fix a bin  $B$  at level  $r$ . Let  $[s_{i,j}(a), s_{i,j+1}(a))$  be a proper bad segment and let  $B' \supseteq B$  be the bin at level  $r_{s_{i,j+1}(a)-\tilde{s}}$  where  $\tilde{s} \doteq s_{i,j+1}(a) - K2^{(m-2)(2+1)-1}$  is as in Lemma 13. Then, for some  $c_{26} > 0$ :
$$
\begin{array}{l} \sum_ {t = \tilde {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\} \geq c _ {2 6} \left(\log (T) \sqrt {K \cdot \left(n _ {B ^ {\prime}} ([ \tilde {s} , s _ {i , j + 1} (a) ])\right) \vee K)} \right. \\ + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} ([ \tilde {s}, s _ {i, j + 1} (a) ]). \\ \end{array}
$$
Proof. We have via concentration ((12) of Lemma 8), Lemma 13, and the strong density assumption (Assumption 2):
$$
\begin{array}{l} \sum_ {t = \tilde {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mathbf {1} \left\{X _ {t} \in B ^ {\prime} \right\} \geq \sum_ {t = \tilde {s}} ^ {s _ {i, j + 1} (a)} \delta_ {t} \left(a _ {t} ^ {\sharp}, a\right) \cdot \mu \left(B ^ {\prime}\right) \\ - c _ {2} \left(\log (T) + \sqrt {\log (T) \left(s _ {i , j + 1} (a) - \tilde {s}\right) \cdot \mu \left(B ^ {\prime}\right)}\right) \\ \geq c _ {2 5} \log (T) \left(s _ {i, j + 1} (a) - s _ {i, j} (a)\right) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}} \\ - c _ {2} \left(\log (T) + \sqrt {\log (T) \left(s _ {i , j + 1} (a) - \tilde {s}\right) \cdot \mu \left(B ^ {\prime}\right)}\right). \\ \end{array}
$$
Now, the first term on the final R.H.S. above dominates the other two terms for large enough  $c_{25}$  and via strong density assumption (Assumption 2).
Thus, it suffices to show
$$
c _ {2 5} \log (T) (s _ {i, j + 1} (a) - s _ {i, j} (a)) ^ {\frac {1}{2 + d}} \cdot K ^ {\frac {1 + d}{2 + d}} \geq
$$