SlowGuess/NeurIPS_2023_OCR huggingface dataset:
over context  $\mathcal{X}_l$ , where  $C = 2\sqrt{2}(10\sqrt{\log(rSK / \delta)} + 1)$ . This means that,
$$
\sum_ {i \in \mathcal {X} _ {l}} \frac {\nu (i)}{\nu (\mathcal {X} _ {l})} \left(\max  _ {j} A (i, j) - A (i, \pi_ {l} (i))\right) = \frac {C \varepsilon}{\sqrt {\nu (\mathcal {X} _ {l})}}.
$$
As a result, the total suboptimality of  $\pi^{\mathrm{out}}$  can be upper bounded as following
$$
\begin{array}{l} \sum_ {i \in [ S ]} \nu (i) \left(\max  _ {j \in [ K ]} A (i, j) - A (i, \pi^ {\mathrm {o u t}} (i))\right) \\ \leq \sum_ {l \in [ L - 1 ]} \sum_ {i \in \mathcal {X} _ {l}} \nu (i) \left(\max  _ {j} A (i, j) - A (i, \pi_ {l} (i))\right) + \sum_ {i \in \mathcal {X} _ {L}} \nu (i) \\ \leq C \sum_ {l \in [ L - 1 ]} \nu (\mathcal {X} _ {l}) \times \frac {\varepsilon}{\sqrt {\nu (\mathcal {X} _ {l})}} + \varepsilon = C \sqrt {L} \varepsilon , \\ \end{array}
$$
where the final equality uses Cauchy-Schwartz inequality and  $L \leq \log(S / \varepsilon) + 1$ .
C.7 Proof of Theorem 3
Proof. Since  $r \leq S$ , we have
$$
\Omega (\min \{r (S + K), S K \}) = \left\{ \begin{array}{l l} \Omega (S K), & r \geq K, \\ \Omega (r S), & r <   K \& S \geq K, \\ \Omega (r K), & r <   K \& S <   K. \end{array} \right.
$$
Case (1) We construct the following hard instance:
- The reward after pulling action  $j \in [K]$  in block  $b \in [r]$  is sampled from distribution Bernoulli  $(1/2 + \varepsilon \mathbb{1}(b = j))$ .
- The context distribution is uniform over  $[S]$ . For each  $i \in [S]$ , we sample  $g(i)$  uniformly at random from  $[K]$ .
The above instance is the standard one commonly used in proving lower bounds for contextual bandit with  $S$  contexts and  $K$  arms [e.g., Lattimore and Szepesvári, 2020]. And learning an  $\varepsilon$ -optimal policy for the above hard instance requires at least  $\Omega (SK / \varepsilon^2)$  samples.
Case (2) We only need to slightly modify the above hard instance:
- The reward after pulling action  $j \in [K]$  in block  $b \in [r]$  is sampled from distribution Bernoulli  $(1/2 + \varepsilon \mathbb{1}(b = j))$ .
- The context distribution is uniform over  $[S]$ . For each  $i \in [S]$ , we sample  $g(i)$  uniformly at random from  $[r]$ .
The above modified problem is equivalent to the original one with  $S$  contexts but  $r$  arms. As a result, a lower bound of form  $\Omega (Sr / \varepsilon^2)$  holds.
Case (3) Similarly, we slightly modify the first hard instance:
- For each block  $b \in [r]$ , sample  $j_b^\star$  uniformly at random from  $[K]$ . The reward after pulling action  $j \in [K]$  in block  $b \in [r]$  is sampled from distribution Bernoulli  $(1/2 + \varepsilon \mathbb{1}(j = j_b^\star))$ .
- The context distribution is uniform over  $[r]$  and  $g(i) = \min \{i, r\}$ .
The above modified problem is equivalent to the original one with  $K$  arms but  $r$  contexts. As a result, a lower bound of form  $\Omega(Kr / \varepsilon^2)$  holds.
In this section, we first show key lemmas to prove Theorem 4. The proofs of the lemmas are deferred to Appendix E.
In the following discussion, we consider a single phase (i.e. fix an error  $\varepsilon_h$ ). Similar to the analysis of other phased elimination algorithms, we have to show that in a phase specified by error level  $\varepsilon_h$  with high probability, (i) the optimal arm is not eliminated and (ii) all  $\omega(\widetilde{\varepsilon}_h)$ -suboptimal arms are eliminated, that is, all arms in  $\mathrm{GOOD}_h(\mathcal{C})$  for all  $\mathcal{C}$  are  $O(\widetilde{\varepsilon}_h)$ -optimal.
The following lemma is the counterpart of Lemma 11, which ensures that every arm is played in every block.
Lemma 14. With probability at least  $1 - 2\delta_h$ , for any cluster  $\mathcal{C} \in \mathcal{P}_h$  and any block  $b$  so that  $\mathcal{B}(b) \subseteq \mathcal{C}$ , we have that: for any  $j \in \mathrm{GOOD}_h(\mathcal{C})$  there exists  $(i,j) \in \mathcal{D}_h$  satisfying  $g(i) = b$ .
The above lemma ensures that every block has a least one context  $i$  assigned to include  $(i,j)$  in  $\mathcal{D}_h$ . Thus, every arm is explored for every block.
Next, we define an event  $\mathcal{E}_h$  under which the estimates  $\widehat{A}_h$  are good:
$$
\mathcal {E} _ {h} = \left\{\left| \widehat {A} _ {h} (i, j) - A (i, j) \right| \leq \frac {\widetilde {\varepsilon} _ {h}}{4}, \forall (i, j) \in \mathcal {D} _ {h} \right\}.
$$
The level of precision  $\frac{\overline{\varepsilon}_h}{4}$  is more accurate than the elimination step and will be helpful in the analysis. The following lemma states that  $\mathcal{E}_h$  is a high probability event.
Lemma 15. Event  $\mathcal{E}_h$  holds with probability  $1 - \delta_h$ .
Next, let  $j_{b} = \arg \max_{j\in [K]} \mu (b,j)$  be the optimal arm in block  $b$ . The next lemma says that the optimal arm  $j_{b}$  is not eliminated during the execution of the algorithm.
Lemma 16. Assume action  $j_b \in \mathsf{GOOD}_h(\mathcal{C})$  for every block  $b \in [r]$ , and its corresponding partition  $\mathcal{P}_h \ni \mathcal{C} \supseteq \mathcal{B}(b)$ . Then for every block  $b \in [r]$ ,  $j_b$  is not eliminated from  $\mathsf{GOOD}_{h+1}(\mathcal{C})$  with probability at least  $1 - 3\delta_h$ .
The high-level idea of the proof is that the error of the estimated mean is smaller than  $\widetilde{\varepsilon}_h$ , so  $\widehat{A}_t(i,j_b)$  will not be much worse than other arms, given that its true mean is largest. The next lemma shows that arms in  $\mathrm{GOOD}_{h + 1}(\mathcal{C})$  are all  $O(\varepsilon_h)$ -optimal. Formally, we say an arm  $j$  in a block  $b$  is  $\varepsilon$ -optimal if  $\max_{j'} \mu(b,j') - \mu(b,j) \leq \varepsilon$ . Similarly, we say an arm  $j$  in a block  $b$  is  $\varepsilon$ -suboptimal if  $\max_{j'} \mu(b,j') - \mu(b,j) \geq \varepsilon$ .
Lemma 17. For any block  $b \in [r]$  and its corresponding cluster  $\mathcal{P}_h \ni \mathcal{C} \supseteq \mathcal{B}(b)$ , all  $3\widetilde{\varepsilon}_h$ -suboptimal arms in block  $b$  are eliminated in  $\mathrm{GOOD}_{h+1}(\mathcal{C})$  with probability at least  $1 - 3\delta_h$ . Consequently, only  $6\widetilde{\varepsilon}_h$ -optimal arms in block  $b$  are played in phase  $h + 1$  for every context in block  $b$ .
Finally, we need to argue that Algorithm 5 is not called too many times. To show this, we first provide a general guarantee of Algorithm 5.
Lemma 18. Suppose we have context  $i_u \in \mathcal{B}(b_u) \subseteq \mathcal{C}$  in block  $b_u$  and context  $i_l \in \mathcal{B}(b_l) \subseteq \mathcal{C}$  in block  $b_l$  so that
$$
A (i _ {u}, j) - A (i _ {l}, j) = \mu (b _ {u}, j) - \mu (b _ {l}, j) > \frac {3 r}{2} \sqrt {\widetilde {\lg}} \cdot \varepsilon^ {\prime}.
$$
Then Algorithm 5 separates  $\mathcal{B}(b_u)$  and  $\mathcal{B}(b_l)$  perfectly with probability at least  $1 - 2\delta'$ . In other words, there exist indices  $c_u$  and  $c_l$ ,  $c_u \neq c_l$ , such that  $\mathcal{B}(b_u) \subseteq \mathcal{P}_{c_u}$  and  $\mathcal{B}(b_l) \subseteq \mathcal{P}_{c_l}$ .
Consequently, with a high probability, Algorithm 4 makes progress in terms of clustering contexts every time calling Algorithm 5 and the number of calls is bounded by  $r$ , as shown in the following lemma.
Lemma 19. When Algorithm 5 is called by Algorithm 4, it separates at least two blocks and never separates contexts in the same block with probability at least  $1 - 5\delta_h$ . Consequently, Algorithm 5 is called at most  $r$  times.
We are now ready to bound regret.
Proof of Theorem 4. Since there are at most  $O(\log_T)$  phases, it suffices to bound regret at a single phase  $h$ . Conditioned on all the high probability good events in the previous lemmas, we first bound the total number of timesteps spent in phase  $h$ . In the data collection stage, we use at most  $L_{h} = \frac{r(S + K)\widetilde{\lg}_{h}}{\varepsilon_{h}^{2}}$  samples; as for the clustering stage, by Lemma 19, we know the total length of executing Algorithm 5 is at most
$$
r L ^ {\prime} = r S \widetilde {\mathrm {l g}} / \varepsilon^ {\prime 2} = 1 6 S r ^ {3} \widetilde {\mathrm {l g}} / \varepsilon_ {h} ^ {2} = \widetilde {O} \left(\frac {r ^ {3} S}{\varepsilon_ {h} ^ {2}}\right).
$$
Thus, the length of phase  $h$  is the minimum of  $T$  and  $\widetilde{O}\left(\frac{r^3(S + K)}{\varepsilon_h^2}\right)$ . Therefore, by Lemma 17, the regret is at most (recall that  $\widetilde{\varepsilon}_h = \sqrt{\log_h} \cdot \varepsilon_h = \widetilde{O}(\varepsilon_h)$ )
$$
\begin{array}{l} 6 \widetilde {\varepsilon} _ {h} \cdot \min  \left\{T, \widetilde {O} \left(\frac {r ^ {3} (S + K)}{\varepsilon_ {h} ^ {2}}\right) \right\} = \min  \left\{6 T \widetilde {\varepsilon} _ {h}, \widetilde {O} \left(\frac {r ^ {3} (S + K)}{\varepsilon_ {h}}\right) \right\} \\ = \min  \left\{\widetilde {O} \left(T \varepsilon_ {h}\right), \widetilde {O} \left(\frac {r ^ {3} (S + K)}{\varepsilon_ {h}}\right) \right\} \\ = \widetilde {O} \left(\sqrt {r ^ {3} (S + K) T}\right). \\ \end{array}
$$
On the other hand, the bad events happen with probability  $O(\delta_h) = O(\varepsilon_h^2 / (r^3 SK))$ . In this case the regret contributes at most  $\widetilde{O}(1)$ .
E Missing Proofs in Appendix D
E.1 Proof of Lemma 14
Proof. Note that under the uniform block assumption, we have  $\omega(b) = \frac{1}{r} \geq \frac{\varepsilon_h}{r}$ . Thus, the proof follows directly from the proof of Lemma 11 when  $\varepsilon_b = \varepsilon_h$ ,  $\delta = \varepsilon_h^2$ , and  $n = n_h$ . The only difference is when applying Lemma 6, Lemma 11 uses  $\mathcal{K} = [K]$  but we need  $\mathcal{K} = \mathrm{GOOD}_h(\mathcal{C})$  here.
E.2 Proof of Lemma 15
Proof. Fix an  $(i,j)$  pair. Applying Lemma 7, we have with probability  $1 - \frac{\delta_h}{SK}$ ,
$$
\left| \widehat {A} _ {h} (i, j) - A (i, j) \right| \leq 2 \sqrt {\frac {\log (S K / \delta_ {h})}{2 ^ {(n _ {h})}}} \leq \frac {\widetilde {\varepsilon} _ {h}}{4}.
$$
We complete the proof by applying a union bound on all  $(i,j)$  pairs in  $\mathcal{D}_h$ .
E.3 Proof of Lemma 16
Proof. We prove by contradiction. Assume that  $j_b$  is removed from  $\mathrm{GOOD}_h(\mathcal{C})$ . Let
$$
j ^ {\prime} = \underset {j \in \operatorname {G O O D} _ {h} (\mathcal {C})} {\arg \max } \overline {{\mu}} _ {h} (\mathcal {C}, j)
$$
be the action achieving the highest empirical mean. By Lemma 14 there exists a context  $i' \in \mathcal{B}(b)$  so that  $(i', j') \in \mathcal{D}_h$ . Moreover, the assumption that  $j_b$  is eliminated implies that the condition at Line 7 of Algorithm 4 does not hold for the current partition  $\mathcal{P}_h$ , which further implies that there exists context  $\bar{i} \in \mathcal{C}$  so that
$$
\widehat {A} _ {h} (\bar {i}, j ^ {\prime}) - \widehat {A} _ {h} (i ^ {\prime}, j ^ {\prime}) = \overline {{\mu}} _ {h} (\mathcal {C}, j ^ {\prime}) - \widehat {A} _ {h} (i ^ {\prime}, j ^ {\prime}) <   \widetilde {\varepsilon} _ {h}.
$$
On the other hand, by the assumption that  $j_b$  is eliminated, again by Lemma 14 we have that there exists a context  $i'' \in \mathcal{B}(b), (i'', j_b) \in \mathcal{D}_h$  such that
$$
\overline {{\mu}} _ {h} (\mathcal {C}, j ^ {\prime}) - \widehat {A} _ {h} (i ^ {\prime \prime}, j _ {b}) \geq \overline {{\mu}} _ {h} (\mathcal {C}, j ^ {\prime}) - \overline {{\mu}} _ {h} (\mathcal {C}, j _ {b}) > 2 \widetilde {\varepsilon} _ {h}.
$$
Combining two inequalities, we have
$$
\widehat {A} _ {h} \left(i ^ {\prime \prime}, j _ {b}\right) + \widetilde {\varepsilon} _ {h} <   \overline {{\mu}} _ {h} (\mathcal {C}, j ^ {\prime}) - \widetilde {\varepsilon} _ {h} <   \widehat {A} _ {h} \left(i ^ {\prime}, j ^ {\prime}\right).
$$
This means that  $\mu (b,j_b) + \frac{\widetilde{\varepsilon}_h}{2} <  \mu (b,j')$  under  $\mathcal{E}_h$ , which contradicts the optimality of  $j_{b}$ . Therefore, we conclude that  $j_{b}$  is not eliminated.
E.4 Proof of Lemma 17
Proof. By Lemma 16,  $j_b$  is not eliminated for any block  $b \in [r]$  with probability at least  $1 - 3\delta_h$ . Thus, for any block  $b \in [r]$  and arm  $j \in \mathrm{GOOD}_h(\mathcal{C})$  so that  $\mu(b, j_b) - \mu(b, j) > 3\widetilde{\varepsilon}_h$ , we have
$$
\max  _ {j ^ {\prime}} \bar {\mu} _ {h} (\mathcal {C}, j ^ {\prime}) - \bar {\mu} _ {h} (\mathcal {C}, j) \geq \bar {\mu} _ {h} (\mathcal {C}, j _ {b}) - \bar {\mu} _ {h} (\mathcal {C}, j) \geq \mu (b, j _ {b}) - \mu (b, j) - 2 \cdot \frac {\widetilde {\varepsilon} _ {h}}{4} > 2 \widetilde {\varepsilon} _ {h}.
$$
Therefore,  $j$  is eliminated at Line 14.
E.5 Proof of Lemma 18
Proof. With probability  $1 - \frac{\delta_h}{S}$ , context  $i$  receives at least  $2 / \varepsilon'^2 \geq 2^{n'}$  samples by Lemma 9. Thus,  $\widehat{A}(i,j)$  is well defined for every  $i \in \mathcal{C}$  with probability  $1 - \delta_h$ . Moreover, by Lemma 7 and a union bound, we have for every context  $i$ , with probability at least  $1 - \delta_h$ ,
$$
\left| A (i, j) - \widehat {A} (i, j) \right| \leq \frac {\sqrt {\widetilde {\lg}} \cdot \varepsilon^ {\prime}}{4} \tag {8}
$$
Clearly, we have  $\widehat{A}(i_u) \geq \widehat{A}(i_l)$  under Eq. (8). Consequently, to simplify the notation, we do the following modification on labels of contexts and blocks. First, we restrict the game to  $\mathcal{C}$ , where there are  $S'$  contexts and  $r'$  blocks; also, we relabel contexts so that  $i_u = 1$ ,  $i_l = S'$ , and
$$
\widehat {A} \left(i _ {u}\right) = \widehat {A} (1) \geq \widehat {A} (2) \geq \dots \geq \widehat {A} \left(S ^ {\prime} - 1\right) \geq \widehat {A} \left(S ^ {\prime}\right) = \widehat {A} \left(i _ {l}\right).
$$
Finally, given a context  $i \in [S']$ , we define
$$
\overline {{\mu}} (b) = \max  _ {i ^ {\prime} \in [ S ^ {\prime} ], g (i ^ {\prime}) = b} \widehat {A} (i ^ {\prime}, j) \quad \text {a n d} \quad \underline {{\mu}} (b) = \min  _ {i ^ {\prime} \in [ S ^ {\prime} ], g (i ^ {\prime}) = b} \widehat {A} (i ^ {\prime}, j)
$$
for its block  $b = g(i)$  and we relabel blocks so that  $b_{u} = 1 \leq g(i) \leq r' = b_{l}$  and
$$
\bar {\mu} \left(b _ {u}\right) = \bar {\mu} (1) \geq \bar {\mu} \left(b _ {u} - 1\right) \geq \dots \geq \bar {\mu} \left(b _ {l} + 1\right) \geq \bar {\mu} \left(r ^ {\prime}\right) = \bar {\mu} \left(b _ {l}\right).
$$
It is not hard to see that this modification is without loss of generality. We show next that there exists a context  $i$ ,  $i_u < i \leq i_l$ , such that Line 7 of Algorithm 5 holds, that is,
$$
\widehat {A} (i - 1, j) - \widehat {A} (i, j) \geq \sqrt {\widetilde {\lg}} \cdot \varepsilon^ {\prime}. \tag {9}
$$
We prove this by contradiction. Assume Eq. (9) doesn't hold for any  $k$ .
$$
\begin{array}{l} A \left(i _ {u}, j\right) - A \left(i _ {l}, j\right) \leq \left| A \left(i _ {u}, j\right) - \bar {\mu} \left(b _ {u}\right) \right| + \left| A \left(i _ {l}, j\right) - \bar {\mu} \left(b _ {l}\right) \right| + \bar {\mu} \left(b _ {u}\right) - \bar {\mu} \left(b _ {l}\right) \\ = \frac {\sqrt {\widetilde {\lg}} \cdot \varepsilon}{4} + \frac {\sqrt {\widetilde {\lg}} \cdot \varepsilon}{4} + \sum_ {b = b _ {u} + 1} ^ {b _ {l}} \bar {\mu} (i _ {b - 1}) - \bar {\mu} (i _ {b}) \\ <   \frac {\sqrt {\widetilde {\lg}} \cdot \varepsilon}{2} + \sum_ {b = b _ {u} + 1} ^ {b _ {l}} \bar {\mu} \left(i _ {b - 1}\right) - \underline {{\mu}} \left(i _ {b}\right) + \sqrt {\widetilde {\lg}} \cdot \varepsilon^ {\prime} \\ \leq \frac {\sqrt {\tilde {\lg}} \cdot \varepsilon}{2} + \frac {3}{2} \cdot (r - 1) \sqrt {\tilde {\lg}} \cdot \varepsilon^ {\prime} \\ <   \frac {3}{2} \cdot r \sqrt {\tilde {\mathrm {l g}}} \cdot \varepsilon^ {\prime}, \\ \end{array}
$$
which contradicts the condition that  $A(i_u, j) - A(i_l, j) \geq \frac{3r}{2} \sqrt{\widetilde{\lg}} \cdot \varepsilon$ . Therefore, we conclude that Eq. (9) holds for some  $k$ .
E.6 Proof of Lemma 19
Proof. The proof follows directly from Lemma 18 and the fact that Algorithm 4 use  $\varepsilon' = \frac{\varepsilon_h}{4r}$  and thus
$$
A (\overline {{i}}, j) - A (\underline {{\dot {i}}}, j) \geq \frac {\widetilde {\varepsilon} _ {h}}{2} \geq \frac {\sqrt {\widetilde {\lg} _ {h}} \cdot \varepsilon_ {h}}{2} > \frac {3}{2} \cdot r \sqrt {\widetilde {\lg}} \cdot \varepsilon^ {\prime},
$$
which satisfies the condition of Lemma 18.
F Non-uniform Context Distribution
We show a reduction to problems with approximately uniform context distributions. The cost of this reduction is an extra  $\widetilde{O}(\sqrt{ST})$  additive regret and an extra  $O(\log(ST))$  multiplicative factor in regret. The idea is to learn the context distribution in the first  $\widetilde{O}(\sqrt{ST})$  timesteps. With high probability, we can estimate  $\nu(i)$  for any context  $i$  with a constant multiplicative error as long as  $\nu(i) = \widetilde{\Omega}(1/\sqrt{ST})$ . Then we split the contexts into several buckets so that contexts within the same bucket have the same probability up to a constant factor. For contexts  $i$  with  $\nu(i) = o(1/\sqrt{ST})$ , we can not estimate the probability properly but we can simply ignore such contexts and suffer regret at most  $O(T \cdot S \cdot 1/\sqrt{ST}) = O(\sqrt{ST})$ . We then run the algorithm that handles uniform context distribution for each bucket separately. Since there are  $O(\log(ST))$  buckets, the overall regret is  $O(\log(ST))$  times maximum regret over all subsets (or  $\sqrt{\log(ST)}$  with refined analysis using a Cauchy-Schwarz inequality).
G Proof of Theorem 5
$$
\varepsilon_ {h, b} = \max  \left\{1, \frac {1}{r \omega (b)} \right\} \varepsilon_ {h}, \quad \text {f o r} b \in [ r ].
$$
Also, for every phase  $h$  and every level  $n$ , we define
$$
\widetilde {\varepsilon} _ {h, n} = \sqrt {\frac {\widetilde {\mathrm {l g}} _ {h}}{2 ^ {n}}}.
$$
Next, we show the counterpart of Lemma 14 in the non-uniform case.
Lemma 20. With probability at least  $1 - 2\delta_h$ , for any cluster  $\mathcal{C} \in \mathcal{P}_h$ , any block  $b$  with  $\mathcal{B}(b) \subseteq \mathcal{C}$ , we have that: for any level  $n \leq \lceil \log(1 / \varepsilon_{h,b}^2) \rceil$ , action  $j \in \mathrm{GOOD}_{h,n}(\mathcal{C})$ , there exists  $(i,j) \in \mathcal{D}_h$  satisfying  $g(i) = b$ .
Proof. Fix a block  $b \in [r]$ . For each accuracy level, recall in the step of data collection, we sample  $L = r(S + K) \widetilde{\lg}_h 2^n$  contexts i.i.d. from  $\nu$ . By Lemma 9, with probability  $1 - \frac{\delta_h}{r}$ , at least  $L\omega(b)/2$  of them are from block  $b$  as
$$
L \omega (b) \geq \frac {8 L}{S} \geq \widetilde {\operatorname {I g}} \geq 1 6 \log (r / \delta_ {h}),
$$
where the first inequality comes from the near-uniform context distribution assumption. Since we add a context-action pair into  $\mathcal{D}_n$  once we have collected
$$
2 ^ {n} \leq 2 ^ {\lceil \log (1 / \varepsilon_ {h, b} ^ {2}) \rceil} <   2 ^ {\log (1 / \varepsilon_ {h, b} ^ {2}) + 1} = 2 / \varepsilon_ {h, b} ^ {2}
$$
samples for estimating its reward. Note that in the end, there are at most  $|\mathcal{B}(b)|(2^n - 1)$  samples from block  $b$  unused. Thus, with probability  $1 - \frac{\delta_h}{S}$ , the number of the context-action pairs, where the contexts are from block  $b$ , that are added into  $\hat{\mathcal{D}}_n$  is at least
$$
\begin{array}{l} \frac {L \omega (b) / 2 - | \mathcal {B} (b) | (2 ^ {n} - 1)}{2 ^ {n}} \geq \frac {L \omega (b) / 2}{2 / \varepsilon_ {h , b} ^ {2}} - S \quad \left(2 ^ {n} \leq 2 / \varepsilon_ {h, b} ^ {2} \text {a n d} | \mathcal {B} (b) | \leq S\right) \\ \geq \frac {L \varepsilon_ {h} / (2 r \varepsilon_ {h , b})}{2 / \varepsilon_ {h , b} ^ {2}} - S \quad (\text {b y}) \\ = 1 6 (S + K) \log (r S K / \delta_ {h}) - S \quad (\text {t h e v a l u e o f} L) \\ \geq K \log (r S K / \delta_ {h}). \\ \end{array}
$$
Conditioned on this event, with probability  $1 - \frac{\delta_h}{r}$ , for any  $j \in \mathrm{GOOD}_{h,n}(\mathcal{C})$ , there exists  $(i,j) \in \mathcal{D}_n$  satisfying  $g(i) = b$  by Lemma 6. Therefore, the lemma holds for block  $b$  with probability at least  $1 - \frac{2\delta_h}{r}$ . We complete the proof by a union bound on all blocks.
Now define a good event  $\mathcal{E}_h$  as
$$
\mathcal {E} _ {h} = \left\{\left| \widehat {A} _ {h, n} (i, j) - A (i, j) \right| \leq \frac {1}{4} \widetilde {\varepsilon} _ {h, n}, \forall (i, j) \in \mathcal {D} _ {h}, n \in [ N _ {h} ] \right\}.
$$
Lemma 21. Event  $\mathcal{E}_h$  holds with probability  $1 - \delta_h$ .
Proof. Fix an  $(i,j)$  pair and level  $n$ . Applying Lemma 7, we have with probability  $1 - \frac{\delta_h}{SKN_h}$ ,
$$
\left| \widehat {A} _ {h, n} (i, j) - A (i, j) \right| \leq 2 \sqrt {\frac {\log \left(S K N _ {h} / \delta_ {h}\right)}{2 ^ {n}}} \leq \frac {1}{4} \widetilde {\varepsilon} _ {h, n}.
$$
We complete the proof by applying a union bound on all  $(i,j)$  pairs in  $\mathcal{D}_h$  and all levels  $n\in [N_h]$ .
In the following, we present and then prove the counterpart of Lemma 16 for Algorithm 6.
Lemma 22. Assume action  $j_b \in \mathsf{GOOD}_{h,n}(\mathcal{C})$  for  $b \in [r]$  with  $\omega(b) \geq \frac{2\varepsilon_h}{r}$ , and its corresponding cluster  $\mathcal{P}_h \ni \mathcal{C} \supseteq \mathcal{B}(b)$ . Then  $j_b$  is not eliminated from  $\mathsf{GOOD}_{h+1,n}(\mathcal{C})$  with probability at least  $1 - 3\delta_h$  for any level  $n \leq \lceil \log(1 / \varepsilon_{h,b}^2) \rceil$ .
Proof. We prove by induction on  $n$ . The base case is  $n = 1$ , which satisfies the condition of Lemma 20 as
$$
\log \left(\frac {1}{\varepsilon_ {h , b} ^ {2}}\right) \geq \log \left(\frac {\omega (b) ^ {2} r ^ {2}}{\varepsilon_ {h} ^ {2}}\right) \geq \log (4) \geq 1 = n.
$$
For the inductive step we prove by contradiction. Assume that  $j_{b}$  is eliminated in  $\mathrm{GOOD}_{h,n + 1}(\mathcal{C})$  but not eliminated in  $\mathrm{GOOD}_{h,n}(\mathcal{C})$  for  $n\geq 2$ . This means that the following inequality hold:
$$
\max _ {j ^ {\prime} \in \mathrm {G O O D} _ {h, n} (\mathcal {C})} \overline {{\mu}} _ {h, n} (\mathcal {C}, j ^ {\prime}) - \overline {{\mu}} _ {h, n} (\mathcal {C}, j _ {b}) > 2 \widetilde {\varepsilon} _ {h, n}
$$
Let  $j' = \arg \max_{j \in \mathrm{GOOD}(n, \mathcal{C})} \overline{\mu}_{h,n}(\mathcal{C}, j)$ . By Lemma 20 there exists a context  $i' \in \mathcal{B}(b)$  so that  $\psi(n, i') \ni j'$  as  $n \leq \log(1 / \varepsilon_{h,b}^2)$ . Moreover, the assumption that  $j_b$  is eliminated implies that the condition at Line 8 does not hold for the current partition  $\mathcal{P}_h$ , which further implies that
$$
\overline {{\mu}} _ {h, n} (\mathcal {C}, j ^ {\prime}) - \widehat {A} _ {h, n} (i ^ {\prime}, j ^ {\prime}) <   \widetilde {\varepsilon} _ {h, n}
$$
On the other hand, by the assumption that  $j_{b}$  is eliminated, again by Lemma 20 we have that there exists a context  $i^{\prime \prime}\in \mathcal{B}(b),\psi (n,i^{\prime \prime})\ni j_{b}$  such that
$$
\overline {{\mu}} _ {h, n} (\mathcal {C}, j ^ {\prime}) - \widehat {A} _ {h, n} (i ^ {\prime \prime}, j _ {b}) > 2 \widetilde {\varepsilon} _ {h, n}
$$
Combining two inequalities, we have  $\widehat{A}_{h,n}(i'', j_b) + \widetilde{\varepsilon}_{h,n} < \widehat{A}_{h,n}(i', j')$ . This means that  $\mu(b, j_b) + \frac{\widetilde{\varepsilon}_{h,n}}{2} < \mu(b, j')$  by Lemma 21, which contradicts the optimality of  $j_b$ . Therefore, we conclude that  $j_b$  is not eliminated.
Now we present a key lemma similar to Lemma 17.
Lemma 23. For any block  $b \in [r]$  with  $\omega(b) \geq \frac{2\varepsilon_h}{r}$  and its corresponding cluster  $\mathcal{P}_h \ni \mathcal{C} \supseteq \mathcal{B}(b)$ , all  $3\widetilde{\varepsilon}_{h,n}$ -suboptimal arms in block  $b$  are eliminated in  $\mathrm{GOOD}_{h+1,n}(\mathcal{C})$  for any level  $n \leq \lceil \log(1 / \varepsilon_{h,b}^2) \rceil$  with probability at least  $1 - 3\delta_h$ . Consequently, only  $6\widetilde{\varepsilon}_{h,n}$ -optimal arms in block  $b$  are played in phase  $h+1$  for every context in block  $b$ .
Proof. By Lemma 22,  $j_b$  is not eliminated with high probability, so for every pair of block  $b \in [r]$  with  $\omega(b) \geq \frac{2\varepsilon_h}{r}$  and arm  $j \in \mathrm{GOOD}_{h,n}(\mathcal{C})$  so that  $\mu(b,j_b) - \mu(b,j) > 3\widetilde{\varepsilon}_{h,n}$ , we have