SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} \leq 6 8 \sqrt {A T \beta} \cdot \log (4 \delta^ {- 1}) \\ = 6 8 \beta \cdot \log (4 \delta^ {- 1}) \cdot \sqrt {A T / \beta} \\ \leq 6 8 \beta \cdot \log (4 \delta^ {- 1}) \cdot \left(2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon))\right) \\ \leq 1 3 6 \beta \cdot \log (4 \delta^ {- 1}) \cdot T _ {\epsilon} + 3 8 0 8 A ^ {2} \beta^ {2} \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log^ {2} (4 / (\delta \epsilon)) \\ \end{array}
$$
where we apply the condition (15) in the second inequality.
Proof of the number of queries. To show an upper bound for the number of queries, we also consider two cases. First, when
$$
2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) <   \sqrt {A T / \beta}, \tag {16}
$$
we can invoke Lemma 21 and get that  $\lambda_t = 0$  for all  $t \in [T]$ . Hence, similar to the proof of Lemma 17, we have
$$
\begin{array}{l} \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} Z _ {t} \\ = \sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) <   \epsilon \right\} + \sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) \geq \epsilon \right\} \\ = T _ {\epsilon} + \sum_ {t = 1} ^ {T} Z _ {t} \sup  _ {a, b \in \mathcal {A} _ {t}} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \epsilon \right\} \\ \leq T _ {\epsilon} + \sum_ {t = 1} ^ {T} Z _ {t} \sum_ {a, b} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \epsilon \right\} \\ \leq T _ {\epsilon} + A ^ {2} \underbrace {\sum_ {t = 1} ^ {T} Z _ {t} \underset {a , b \sim p _ {t}} {\mathbb {E}} \mathbb {1} \left\{\sup  _ {f , f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t} , a , b) - f ^ {\prime} (x _ {t} , a , b) \geq \epsilon \right\}} _ {(*)} \\ \end{array}
$$
where the second inequality holds as  $p_t(a)$  is uniform for any  $a, b$  when  $\lambda_t = 0$ . We apply Lemma 4 and Lemma 10 to (*) and obtain
$$
\begin{array}{l} (*) \leq 2 \sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a _ {t}, b _ {t}) - f ^ {\prime} (x _ {t}, a _ {t}, b _ {t}) \geq \epsilon \right\} + 8 \log (\delta^ {- 1}) \\ \leq 2 \left(\frac {4 \beta}{\epsilon^ {2}} + 1\right) \dim_ {E} (\mathcal {F}; \epsilon) + 8 \log (\delta^ {- 1}) \\ \leq \frac {1 0 \beta}{\epsilon^ {2}} \cdot \dim_ {E} (\mathcal {F}; \epsilon) + 8 \log (\delta^ {- 1}). \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} \leq T _ {\epsilon} + \frac {1 0 A ^ {2} \beta}{\epsilon^ {2}} \cdot \dim_ {E} (\mathcal {F}; \epsilon) + 8 A ^ {2} \log (\delta^ {- 1}) \\ \leq 8 T _ {\epsilon} ^ {2} \beta / A + 6 2 7 2 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \epsilon)}{\epsilon^ {2}} \cdot \log^ {2} (2 / (\delta \epsilon)) \\ \end{array}
$$
where the second line corresponds to the upper bound derived from the alternative case, which is shown below.
When the contrary of (16) holds, i.e.,
$$
2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) \geq \sqrt {A T / \beta}.
$$
Squaring both sides and leveraging the inequality  $(a + b)^2 \leq 2a^2 + 2b^2$ , we obtain
$$
8 T _ {\epsilon} ^ {2} + 6 2 7 2 A ^ {4} \beta^ {2} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \epsilon)}{\epsilon^ {2}} \cdot \log^ {2} (2 / (\delta \epsilon)) \geq A T / \beta
$$
$$
T \leq 8 T _ {\epsilon} ^ {2} \beta / A + 6 2 7 2 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \epsilon)}{\epsilon^ {2}} \cdot \log^ {2} (2 / (\delta \epsilon)).
$$
We note that we always have  $\mathrm{Queries}_T^{\mathrm{CB}} \leq T$  and thus
$$
\operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} \leq T \leq 8 T _ {\epsilon} ^ {2} \beta / A + 6 2 7 2 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \epsilon)}{\epsilon^ {2}} \cdot \log^ {2} (2 / (\delta \epsilon)).
$$
Minimizing on  $\epsilon$ . Given that the aforementioned proofs hold for any threshold  $\epsilon$ , we can select the specific value of  $\epsilon$  that minimizes the upper bounds. Hence, we deduce the desired result.
C.6 Proof of Theorem 4
Proof of Theorem 4. The upper bound of the number of queries is straightforward: Algorithm 2 is simply running  $H$  instances of Algorithm 1, so the total number of queries is simply the sum of these  $H$  instances. For bounding the regret, we have
$$
\begin{array}{l} \operatorname {R e g r e t} _ {T} ^ {\mathrm {I L}} = \sum_ {t = 1} ^ {T} V _ {0} ^ {\pi_ {e}} \left(x _ {t, 0}\right) - V _ {0} ^ {\pi_ {t}} \left(x _ {t, 0}\right) \\ \leq \sum_ {h = 0} ^ {H - 1} \sum_ {t = 1} ^ {T} \mathbb {E} _ {x _ {t, h}, a _ {t, h} \sim d _ {x _ {t, 0}, h} ^ {\pi_ {t}}} \left[ Q _ {h} ^ {\pi_ {e}} \left(x _ {t, h}, \pi_ {h} ^ {\pi_ {e}} \left(x _ {t, h}\right)\right) - Q _ {h} ^ {\pi_ {e}} \left(x _ {t, h}, a _ {t, h}\right) \right] \\ \leq \sum_ {h = 0} ^ {H - 1} \sum_ {t = 1} ^ {T} \mathbb {E} _ {x _ {t, h}, a _ {t, h} \sim d _ {x _ {t, 0}, h} ^ {\pi_ {t}}} \left[ Q _ {h} ^ {\pi_ {e}} \left(x _ {t, h}, \pi_ {h} ^ {+} \left(x _ {t, h}\right)\right) - Q _ {h} ^ {\pi_ {e}} \left(x _ {t, h}, a _ {t, h}\right) \right] \\ - \sum_ {h = 0} ^ {H - 1} \sum_ {t = 1} ^ {T} \underset {x _ {t, h} \sim d _ {x _ {t, 0}, h} ^ {\pi_ {t}}} {\mathbb {E}} \left[ A _ {h} ^ {\pi_ {e}} \left(x _ {t, h}, \pi_ {h} ^ {+} \left(x _ {t, h}\right)\right) \right] \\ \leq H \cdot \mathbb {E} \left[ \operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} \right] - \operatorname {A d v} _ {T}. \\ \end{array}
$$
where the first inequality holds by Lemma 5, and we denote  $\pi_h^+(x_{t,h}) = \arg \max_a Q_h^{\pi_e}(x_{t,h}, a)$  in the second inequality. Then, we can plug the upper bound of  $\mathrm{Regret}_T^{\mathrm{CB}}$  (Theorem 1). Moreover, we need to take a union bound over all  $h \in [H]$ .