SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem 3.4. Note that  $R_{1}(\gamma) = R_{2}(\gamma + \delta_{\gamma, \epsilon})$  implies  $R_{2}^{-1}(R_{1}(\gamma)) = \gamma + \delta_{\gamma, \epsilon}$  and  $R_{1}^{-1}(R_{2}(\gamma + \delta_{\gamma, \epsilon})) = \gamma$ . Obviously  $R_{2}^{-1}(R_{1}(0)) = 0$  and  $R_{2}^{-1}(R_{1}(1)) = 1$ . On a high level, our proof technique is to split the  $D_{\alpha}(P_{1}, P_{2})$  into two parts using the quantile-based representation of  $\alpha$ -divergence in Lemma 3.5, and then use Jensen's inequality (since  $\frac{1}{\alpha(\alpha - 1)} x^{1 - \alpha}$  is a convex function of  $x \geq 0$  for any  $\alpha \neq 0, 1$ ) to derive a bound for  $\delta_{\gamma, \epsilon}$ .
By Lemma 3.5 and Jensen's inequality,
$$
\begin{array}{l} D _ {\alpha} (P _ {1}, P _ {2}) \\ = \frac {1}{\alpha (\alpha - 1)} \int_ {0} ^ {1} \left(\frac {d}{d u} R _ {2} ^ {- 1} \left(R _ {1} (u)\right)\right) ^ {1 - \alpha} d u - \frac {1}{\alpha (\alpha - 1)} \\ = \frac {1}{\alpha (\alpha - 1)} \int_ {0} ^ {\gamma} \left(\frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u))\right) ^ {1 - \alpha} d u + \frac {1}{\alpha (\alpha - 1)} \int_ {\gamma} ^ {1} \log \left(\frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u))\right) ^ {1 - \alpha} d u - \frac {1}{\alpha (\alpha - 1)} \\ \geq \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {1}{\gamma} \int_ {0} ^ {\gamma} \frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u)) d u\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1}{1 - \gamma} \int_ {\gamma} ^ {1} \frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u)) d u\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)} \\ = \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {R _ {2} ^ {- 1} (R _ {1} (\gamma)) - R _ {2} ^ {- 1} (R _ {1} (0))}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {R _ {2} ^ {- 1} (R _ {1} (1)) - R _ {2} ^ {- 1} (R _ {1} (\gamma))}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)} \\ = \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)} \tag {5} \\ \end{array}
$$
First, we note that for any  $\gamma \in (0,1)$  and any  $\delta_{\gamma,\epsilon} \in [-\gamma, 1 - \gamma]$ , the above inequality is indeed achievable. To see this, consider
$$
p _ {2} (x) = \left\{ \begin{array}{l l} \frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma} p _ {1} (x) & \text {i f} x _ {1} <   R _ {1} (\gamma) \\ \frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma} p _ {1} \left(x _ {1}\right) & \text {i f} x _ {1} > R _ {1} (\gamma) \end{array} \right. \tag {6}
$$
Simple calculations give
$$
\int p _ {2} (x) = \frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma} F _ {1} (R _ {1} (\gamma)) + \frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma} (1 - F _ {1} (R _ {1} (\gamma))) = \frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma} \gamma + \frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma} (1 - \gamma) = 1
$$
where  $F_{1}$  is the cdf of  $P_{1}$ , and
$$
\begin{array}{l} D _ {\alpha} (P _ {1}, P _ {2}) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int \left(\frac {p _ {2} (x)}{p _ {1} (x)}\right) ^ {1 - \alpha} p _ {1} (x) d x - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} F _ {1} (R _ {1} (\gamma)) + \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} (1 - F _ {1} (R _ {1} (\gamma))) - 1\right) \\ = \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)}. \\ \end{array}
$$
Therefore, the  $P_{1}$  and  $P_{2}$  constructed in Equation (6) show that the inequality (5) is indeed achievable.
Next, we consider the function
$$
g (\delta) = \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)}.
$$
Taking the derivative of  $g(\delta)$  with respect to  $\delta$ , we obtain
$$
\begin{array}{l} g ^ {\prime} (\delta) = - \frac {1}{\alpha} \left(\frac {\gamma + \delta}{\gamma}\right) ^ {- \alpha} + \frac {1}{\alpha} \left(\frac {1 - \gamma - \delta}{1 - \gamma}\right) ^ {- \alpha}. \\ g ^ {\prime \prime} (\delta) = \frac {1}{\gamma} \left(\frac {\gamma + \delta}{\gamma}\right) ^ {- \alpha - 1} + \frac {1}{1 - \gamma} \left(\frac {1 - \gamma - \delta}{1 - \gamma}\right) ^ {- \alpha - 1} > 0. \\ \end{array}
$$
It is easy to see that  $g(\delta)$  is strictly decreasing when  $\delta \in (-\gamma, 0)$  and strictly increasing when  $\delta \in (0, 1 - \gamma)$  with minimum  $g(0) = 0$ . This fact shows that  $\epsilon \geq g(\delta)$  is equivalent to  $\delta$  in an interval around 0, i.e.,  $\delta \in [\underline{\delta}, \overline{\delta}] \subset [-\gamma, 1 - \gamma]$ . More explicitly, we can obtain an explicit bound for  $\underline{\delta}, \overline{\delta}$ . In the following, we will show that in some cases, the bound for  $\delta$  becomes too loose in the sense that  $\underline{\delta}$  can be close to  $-\gamma$  or  $\overline{\delta}$  can be close to  $1 - \gamma$ , in which cases we cannot obtain any useful information.
1) Suppose that  $\alpha > 1$ . When  $\delta_{\gamma, \epsilon} > 0$ , we have
$$
\frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} \geq \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + 1 - \gamma}{\gamma}\right) ^ {1 - \alpha} = \frac {\gamma^ {\alpha}}{\alpha (\alpha - 1)} \geq 0
$$
as  $0 < \delta_{\gamma, \epsilon} \leq 1 - \gamma$ . Therefore  $\epsilon \geq D_{\alpha}(P_1, P_2)$  implies that
$$
\frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} \leq \epsilon + \frac {1}{\alpha (\alpha - 1)}
$$
$$
\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma} \geq \left(\frac {\epsilon \alpha (\alpha - 1) + 1}{1 - \gamma}\right) ^ {\frac {1}{1 - \alpha}}.
$$
Remark. Note that when  $\alpha > 1$ , the lower bound of  $\delta_{\gamma, \epsilon}$  can be similarly derived but it becomes vicious. To see this, we notice that whenever
$$
\left(\epsilon \alpha (\alpha - 1) + \gamma\right) ^ {\frac {1}{1 - \alpha}} \gamma^ {\frac {\alpha}{\alpha - 1}} - \gamma \leq \delta_ {\gamma , \epsilon} <   0, \tag {7}
$$
$$
\frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)} \leq \frac {1 - \gamma}{\alpha (\alpha - 1)} - \frac {1}{\alpha (\alpha - 1)} = \frac {- \gamma}{\alpha (\alpha - 1)}
$$
as  $-\gamma \leq \delta_{\gamma ,\epsilon} < 0$  and
$$
\frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} \leq \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + (\epsilon \alpha (\alpha - 1) + \gamma) ^ {\frac {1}{1 - \alpha}} \gamma^ {\frac {\alpha}{\alpha - 1}} - \gamma}{\gamma}\right) ^ {1 - \alpha} = \epsilon + \frac {\gamma}{\alpha (\alpha - 1)}
$$
which ensures  $\epsilon \geq D_{\alpha}(P_1, P_2)$ . However, the lower bound in Equation (7) is too loose:
$$
\lim _ {\epsilon \rightarrow + \infty} (\epsilon \alpha (\alpha - 1) + \gamma) ^ {\frac {1}{1 - \alpha}} \gamma^ {\frac {\alpha}{\alpha - 1}} - \gamma = - \gamma
$$
which can be sufficiently close to  $-\gamma$  if  $\epsilon$  is sufficiently large.
b) Suppose that  $\alpha < 0$ . When  $\delta_{\gamma, \epsilon} < 0$ , we have
$$
\frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} \geq \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma - \gamma}{\gamma}\right) ^ {1 - \alpha} = 0
$$
as  $-\gamma \leq \delta_{\gamma, \epsilon} < 0$ . Therefore  $\epsilon \geq D_{\alpha}(P_1, P_2)$  implies that
$$
\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma} \leq \left(\frac {\epsilon \alpha (\alpha - 1) + 1}{1 - \gamma}\right) ^ {\frac {1}{1 - \alpha}}.
$$
$$
\delta_ {\gamma , \epsilon} \geq 1 - \gamma - (\epsilon \alpha (\alpha - 1) + 1) ^ {\frac {1}{1 - \alpha}} (1 - \gamma) ^ {\frac {\alpha}{\alpha - 1}} \tag {8}
$$
Remark. Note that when  $\alpha < 0$ , the upper bound of  $\delta_{\gamma, \epsilon}$  can be similarly derived but it becomes vicious. To see this, we notice that whenever
$$
0 \leq \delta_ {\gamma , \epsilon} <   \min  \left\{\left(\epsilon \alpha (\alpha - 1) + \gamma\right) ^ {\frac {1}{1 - \alpha}} \gamma^ {\frac {\alpha}{\alpha - 1}} - \gamma , 1 - \gamma \right\} \tag {9}
$$
which ensures  $\epsilon \geq D_{\alpha}(P_1, P_2)$ . However, the lower bound in Equation (9) is too loose:
$$
\lim _ {\epsilon \rightarrow + \infty} (\epsilon \alpha (\alpha - 1) + \gamma) ^ {\frac {1}{1 - \alpha}} \gamma^ {\frac {\alpha}{\alpha - 1}} - \gamma = + \infty
$$
so the lower bound in Equation (9) can be  $1 - \gamma$  if  $\epsilon$  is large.
c) Note that when  $\alpha \in (0,1)$ ,  $\alpha (\alpha -1) < 0$  and thus we have
$$
\frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} \leq \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma - \gamma}{\gamma}\right) ^ {1 - \alpha} = 0
$$
$$
\frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} \leq \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - (1 - \gamma)}{1 - \gamma}\right) ^ {1 - \alpha} = 0
$$
as  $-\gamma \leq \delta_{\gamma, \epsilon} \leq 1 - \gamma$ . Hence, as long as  $\epsilon \geq \frac{-1}{\alpha(\alpha - 1)}$ , we always have
$$
\frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)} \leq \epsilon
$$
for any  $\delta_{\gamma,\epsilon} \in [-\gamma, 1 - \gamma]$ . Moreover, as we discussed below Equation (5), for any  $\delta_{\gamma,\epsilon} \in [-\gamma, 1 - \gamma]$ , there exist two distributions  $P_1$  and  $P_2$  such that
$$
D _ {\alpha} (P _ {1}, P _ {2}) = \frac {\gamma}{\alpha (\alpha - 1)} \left(\frac {\gamma + \delta_ {\gamma , \epsilon}}{\gamma}\right) ^ {1 - \alpha} + \frac {1 - \gamma}{\alpha (\alpha - 1)} \left(\frac {1 - \gamma - \delta_ {\gamma , \epsilon}}{1 - \gamma}\right) ^ {1 - \alpha} - \frac {1}{\alpha (\alpha - 1)},
$$
which shows that  $D_{\alpha}(P_1, P_2) \leq \epsilon$  holds for such  $P_1$  and  $P_2$ . This implies that the condition  $D_{\alpha}(P_1, P_2) \leq \epsilon$  cannot control the quantile shift between  $P_1$  and  $P_2$  in general when  $\alpha \in (0, 1)$ .
Proof of Lemma 3.5. Let  $F_{1}$  and  $F_{2}$  be the cdfs of  $P_{1}$  and  $P_{2}$  respectively. Since  $F_{1}$  and  $F_{2}$  are absolute continuous and strictly increasing (as they have positive densities), we have that  $F_{i}(R_{i}(u)) = u$  for  $0 \leq u \leq 1$ . Taking the derivative with respect to both sides of  $F_{i}(R_{i}(u)) = u$ , we obtain
$$
R _ {i} ^ {\prime} (u) p _ {i} \left(R _ {i} (u)\right) = 1 \tag {10}
$$
where  $R_i'(u) \coloneqq \frac{dR_i(u)}{du}$ . Note that we have the following equality:
$$
\begin{array}{l} \frac {d}{d u} R _ {2} ^ {- 1} (R _ {1} (u)) = R _ {1} ^ {\prime} (u) \frac {d R _ {2} ^ {- 1} (v)}{d v} | _ {v = R _ {1} (u)} \\ = R _ {1} ^ {\prime} (u) \frac {1}{R _ {2} ^ {\prime} \left(R _ {2} ^ {- 1} (v)\right)} | _ {v = R _ {1} (u)} \\ = R _ {1} ^ {\prime} (u) p _ {2} \left(R _ {2} \left(R _ {2} ^ {- 1} (v)\right)\right) | _ {v = R _ {1} (u)} \quad \text {b y} (10) \\ = R _ {1} ^ {\prime} (u) p _ {2} \left(R _ {1} (u)\right). (11) \\ \end{array}
$$
Using integration by substitution, for  $\alpha$ -divergence, we obtain that
$$
\begin{array}{l} D _ {\alpha} \left(P _ {1}, P _ {2}\right) = \frac {1}{\alpha (\alpha - 1)} \left(\int p _ {1} (x) ^ {\alpha} p _ {2} (x) ^ {1 - \alpha} d x - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int \left(\frac {p _ {1} (x)}{p _ {2} (x)}\right) ^ {\alpha - 1} p _ {1} (x) d x - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {1} \left(\frac {p _ {1} \left(R _ {1} (u)\right)}{p _ {2} \left(R _ {1} (u)\right)}\right) ^ {\alpha - 1} p _ {1} \left(R _ {1} (u)\right) d R _ {1} (u) - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {1} \left(\frac {1}{R _ {1} ^ {\prime} (u) p _ {2} \left(R _ {1} (u)\right)}\right) ^ {\alpha - 1} p _ {1} \left(R _ {1} (u)\right) R _ {1} ^ {\prime} (u) d u - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {1} \left(R _ {1} ^ {\prime} (u) p _ {2} \left(R _ {1} (u)\right)\right) ^ {1 - \alpha} d u - 1\right) \quad \text {b y} \tag {10} \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {1} \left(\frac {d}{d u} R _ {2} ^ {- 1} \left(R _ {1} (u)\right)\right) ^ {1 - \alpha} d u - 1\right) \quad \text {b y} \\ \end{array}
$$
Similarly, for KL divergence, we obtain that
$$
\begin{array}{l} K L (P _ {1}, P _ {2}) = \int_ {0} ^ {1} \log \left(\frac {p _ {1} (R _ {1} (u))}{p _ {2} (R _ {1} (u))}\right) p _ {1} (R _ {1} (u)) d R _ {1} (u) \\ = \int_ {0} ^ {1} \log \left(\frac {1}{R _ {1} ^ {\prime} (u) p _ {2} (R _ {1} (u))}\right) p _ {1} (R _ {1} (u)) R _ {1} ^ {\prime} (u) d u \\ = - \int_ {0} ^ {1} \log \left(R _ {1} ^ {\prime} (u) p _ {2} \left(R _ {1} (u)\right)\right) d u \quad \text {b y} \tag {10} \\ = - \int_ {0} ^ {1} \log \left(\frac {d}{d u} R _ {2} ^ {- 1} \left(R _ {1} (u)\right)\right) d u \quad \text {b y} \\ \end{array}
$$
Appendix C Proofs of Results in Section 3.3
We first prove two useful lemmas.
Lemma C.1. Under the same condition in Theorem 3.7, the quantiles of the approximate distributions  $qu_{j}(t)$  chosen by the EBUCB algorithm satisfies the following bound:
$$
\underline {{u}} _ {j} (t) \leq q u _ {j} (t) \leq \overline {{u}} _ {j} (t)
$$
$$
\overline {{u}} _ {j} (t) = \underset {x > \frac {S _ {j} (t)}{N _ {j} (t)}} {\arg \max} \left\{d \left(\frac {S _ {j} (t)}{N _ {j} (t)}, x\right) \leq \frac {\zeta \tilde {\alpha} _ {1} \log (t) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1})}{N _ {j} (t)} \right\}.
$$
$$
\underline {{u}} _ {j} (t) = \underset {x > \frac {S _ {j} (t)}{N _ {j} (t) + 1}} {\arg \max} \left\{d \left(\frac {S _ {j} (t)}{N _ {j} (t) + 1}, x\right) \leq \frac {\zeta \tilde {\alpha} _ {2} \log (t) + c \tilde {\alpha} _ {2} \log (\log T) - \log (M _ {\epsilon , 2}) - \log (N _ {j} (t) + 2)}{N _ {j} (t) + 1} \right\},
$$
Proof of Lemma C.1. Recall that  $M_{\epsilon,1} = (\epsilon \alpha_1(\alpha_1 - 1) + 1)^{\frac{1}{1 - \alpha_1}} < 1$ ,  $\tilde{\alpha}_1 = \frac{\alpha_1}{\alpha_1 - 1} > 0$ ,  $M_{\epsilon,2} = (\epsilon \alpha_2(\alpha_2 - 1) + 1)^{\frac{1}{1 - \alpha_2}} > 1$ ,  $\tilde{\alpha}_2 = \frac{\alpha_2}{\alpha_2 - 1} > 0$ . First we notice that by Theorem 3.4 part a) (where  $P_1$
corresponds to  $Q_{t,j - 1}$ ), we have
$$
\begin{array}{l} q u _ {j} (t) = Q u \left(1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}}, Q _ {t - 1, j}\right) \\ = Q u (1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}} + \delta_ {1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}}, \epsilon}, \Pi_ {t - 1, j}) \\ \leq Q u (1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}} + \frac {1}{t ^ {\zeta} (\log T) ^ {c}} - M _ {\epsilon , 1} \frac {1}{t ^ {\zeta \tilde {\alpha} _ {1}} (\log T) ^ {c \tilde {\alpha} _ {1}}}, \Pi_ {t - 1, j}) \\ = Q u (1 - \frac {M _ {\epsilon , 1}}{t ^ {\zeta \tilde {\alpha} _ {1}} (\log T) ^ {c \tilde {\alpha} _ {1}}}, \Pi_ {t - 1, j}) \\ \end{array}
$$
since  $D_{\alpha_1}(Q_{t,j - 1},\Pi_{t,j - 1})\leq \epsilon$  and we have use the fact that  $Qu$  is non-decreasing. Now we apply the proof of Lemma 1 in [21], the tight bounds of the quantiles of the Beta distributions, to obtain
$$
\begin{array}{l} Q u (1 - \frac {M _ {\epsilon , 1}}{t ^ {\zeta \hat {\alpha} _ {1}} (\log T) ^ {c \hat {\alpha} _ {1}}}, \Pi_ {t - 1, j}) \\ \leq \underset {x > \frac {S _ {j} (t)}{N _ {j} (t)}} {\arg \max } \left\{d \left(\frac {S _ {j} (t)}{N _ {j} (t)}, x\right) \leq \frac {\log \left(\frac {1}{\frac {M _ {\epsilon , 1}}{t ^ {\zeta \bar {\alpha} _ {1}} (\log T) ^ {c \bar {\alpha} _ {1}}}}\right)}{N _ {j} (t)} \right\} \\ \underset {x > \frac {S _ {j} (t)}{N _ {j} (t)}} {\leq \arg \max } \left\{d \left(\frac {S _ {j} (t)}{N _ {j} (t)}, x\right) \leq \frac {\zeta \tilde {\alpha} _ {1} \log (t) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1})}{N _ {j} (t)} \right\}. \\ = \bar {u} _ {j} (t) \\ \end{array}
$$
Similarly, by Theorem 3.4 part b) (where  $P_{1}$  corresponds to  $Q_{t,j - 1}$ ), we have that
$$
\begin{array}{l} q u _ {j} (t) = Q u \left(1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}}, Q _ {t - 1, j}\right) \\ = Q u (1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}} + \delta_ {1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}}, \epsilon}, \Pi_ {t - 1, j}) \\ \geq Q u \left(1 - \frac {1}{t ^ {\zeta} (\log T) ^ {c}} + \frac {1}{t ^ {\zeta} (\log T) ^ {c}} - M _ {\epsilon , 2} \frac {1}{t ^ {\zeta \tilde {\alpha} _ {2}} (\log T) ^ {c \tilde {\alpha} _ {2}}}, \Pi_ {t - 1, j}\right) \\ = Q u \left(1 - \frac {M _ {\epsilon , 2}}{t ^ {\zeta \tilde {\alpha} _ {2}} (\log T) ^ {c \tilde {\alpha} _ {2}}}, \Pi_ {t - 1, j}\right) \\ \end{array}
$$
since  $D_{\alpha_2}(\Pi_{t,j - 1},Q_{t,j - 1})\leq \epsilon$  and we have use the fact that  $Qu$  is non-decreasing. Now we apply the proof of Lemma 1 in [21], the tight bounds of the quantiles of the Beta distributions, to obtain
$$
\begin{array}{l} Q u (1 - \frac {M _ {\epsilon , 2}}{t ^ {\zeta \bar {\alpha} _ {2}} (\log T) ^ {c \bar {\alpha} _ {2}}}, \Pi_ {t - 1, j}) \\ \geq \operatorname *{arg  max}_{x > \frac{S_{j}(t)}{N_{j}(t) + 1}}\left\{d\left(\frac{S_{j}(t)}{N_{j}(t) + 1},x\right)\leq \frac{\log\left(\frac{1}{\frac{M_{\epsilon,2}}{t^{\zeta\bar{\alpha}2}}(\log T)^{c\bar{\alpha}2}}(N_{j}(t) + 2)\right)}{N_{j}(t) + 1}\right\} \\ \geq \underset {x > \frac {S _ {j} (t)}{N _ {j} (t) + 1}} {\arg \max } \left\{d \left(\frac {S _ {j} (t)}{N _ {j} (t) + 1}, x\right) \leq \frac {\zeta \tilde {\alpha} _ {2} \log (t) + c \tilde {\alpha} _ {2} \log (\log T) - \log (M _ {\epsilon , 2}) - \log (N _ {j} (t) + 2)}{N _ {j} (t) + 1} \right\} \\ = \underline {{u}} _ {j} (t) \\ \end{array}
$$
$$
\underline {{u}} _ {j} (t) \leq q u _ {j} (t) \leq \overline {{u}} _ {j} (t).
$$
Based on Lemma C.1, we can obtain a UCB-type decomposition of the number of draws of any sub-optimal action  $j \geq 2$  as follows.
Lemma C.2. Under the same condition in Theorem 3.7, we have that for any constant  $\beta_T$ ,
$$
\begin{array}{l} N _ {2} (T) \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \underline {{u}} _ {1} (t) \right\} \tag {12} \\ + \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} \leq \bar {u} _ {2} (t)\right) \cap \left(A _ {t} = 2\right) \right\}. \\ \end{array}
$$
Proof of Lemma C.2. We have that, by definition,
$$
\begin{array}{l} N _ {2} (T) = \sum_ {t = 1} ^ {T} \mathbb {1} \left\{A _ {t} = 2 \right\} \\ = \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} > q _ {1} (t)\right) \cap \left(A _ {t} = 2\right) \right\} + \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} \leq q _ {1} (t)\right) \cap \left(A _ {t} = 2\right) \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > q _ {1} (t) \right\} + \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} \leq q _ {1} (t)\right) \cap \left(A _ {t} = 2\right) \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \underline {{u}} _ {1} (t) \right\} + \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} \leq \bar {u} _ {2} (t)\right) \cap (A _ {t} = 2) \right\} \\ \end{array}
$$
where the last inequality follows from the fact that  $q_{1}(t) \geq \underline{u}_{1}(t)$  and when  $A_{t} = 2$ ,  $q_{1}(t) \leq q_{2}(t) \leq \overline{u}_{2}(t)$ .
Therefore, to obtain Theorem 3.7, it is sufficient to analyze the two terms in Lemma C.2.
Proof of Theorem 3.7. Without loss of generality, we let  $j = 2$ . (Note that we have assumed the action 1 is optimal.) By Lemma C.2, we only need to bound the two following two terms:
$$
D _ {1} := \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \underline {{u}} _ {1} (t) \right\}, \quad D _ {2} := \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\left(\mu_ {1} - \beta_ {T} \leq \bar {u} _ {2} (t)\right) \cap \left(A _ {t} = 2\right) \right\}
$$
Let  $\beta_{T} = \sqrt{\frac{1}{\log T}}$ . We further split  $D_{1}$  into two parts:
$$
D _ {1, 1} := \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \underline {{u}} _ {1} (t), N _ {1} (t) + 2 \leq (\log T) ^ {2} \right\},
$$
$$
D _ {1, 2} := \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \underline {{u}} _ {1} (t), N _ {1} (t) + 2 \geq (\log T) ^ {2} \right\}.
$$
Step 1: Consider  $D_{1,1}$
$$
\frac {\zeta \tilde {\alpha} _ {2} \log (t) + c \tilde {\alpha} _ {2} \log (\log T) - \log (M _ {\epsilon , 2}) - \log (N _ {1} (t) + 2)}{N _ {1} (t) + 1} \geq \frac {\zeta \tilde {\alpha} _ {2} \log (t) + (c \tilde {\alpha} _ {2} - 2) \log (\log T) - \log (M _ {\epsilon , 2})}{N _ {1} (t) + 1}
$$
in  $\underline{u}_1(t)$  when  $N_{1}(t) + 2\leq (\log T)^{2}$ . Hence we have that
$$
\underline{u}_{1}(t)\geq \operatorname *{arg  max}_{x > \frac{S_{1}(t)}{N_{1}(t) + 1}}\left\{d\left(\frac{S_{1}(t)}{N_{1}(t) + 1},x\right)\leq \frac{\zeta\tilde{\alpha}_{2}\log(t) + (c\tilde{\alpha}_{2} - 2)\log(\log T) - \log(M_{\epsilon ,2})}{N_{1}(t) + 1}\right\} := \tilde{u}_{1}(t)
$$
when  $N_{1}(t) + 2\leq (\log T)^{2}$ . This shows that
$$
D _ {1, 1} \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} > \tilde {u} _ {1} (t) \right\}
$$
Similarly to the proof in [21], with a straightforward adaptation of the proof of theorem 10 in [15], we obtain the following self-normalized inequality
Lemma C.3.
$$
\mathbb {P} \left(\mu_ {1} > \underline {{\tilde {u}}} _ {1} (t)\right) \leq (\bar {\delta} \log (t) + 1) \exp (- \bar {\delta} + 1)
$$
$$
\bar {\delta} = \zeta \tilde {\alpha} _ {2} \log (t) + (c \tilde {\alpha} _ {2} - 2) \log (\log T) - \log (M _ {\epsilon , 2}).
$$
Lemma C.3 leads to the following upper bound of  $D_{1,1}$ :