SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} [ D _ {1, 1} ] \leq \sum_ {t = 1} ^ {T} \mathbb {P} (\mu_ {1} > \tilde {u} _ {1} (t)) \\ \leq 1 + \sum_ {t = 2} ^ {T} \left(\left(\zeta \tilde {\alpha} _ {2} \log (t) + \left(c \tilde {\alpha} _ {2} - 2\right) \log (\log T) - \log \left(M _ {\epsilon , 2}\right)\right) \log (t) + 1\right) \frac {M _ {\epsilon , 2} e}{t ^ {\zeta \tilde {\alpha} _ {2}} (\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \\ \leq 1 + \sum_ {t = 2} ^ {T} \left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2} - 2 + 1\right) \frac {M _ {\epsilon , 2} e}{t ^ {\zeta \tilde {\alpha} _ {2}} (\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \\ \leq 1 + \left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) \frac {M _ {\epsilon , 2} e}{(\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \sum_ {t = 2} ^ {T} \int_ {t - 1} ^ {t} \frac {1}{x ^ {\zeta \tilde {\alpha} _ {2}}} d x \\ \leq 1 + \left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) \frac {M _ {\epsilon , 2} e}{(\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \int_ {T} ^ {1} \frac {1}{x ^ {\zeta \tilde {\alpha} _ {2}}} d x \\ \leq \left\{ \begin{array}{l l} 1 + \Big (\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2} \Big) \frac {M _ {\epsilon , 2} e}{(\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \frac {T ^ {1 - \zeta \tilde {\alpha} _ {2}} - 1}{1 - \zeta \tilde {\alpha} _ {2}} \leq 1 + \frac {\Big (\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2} \Big) M _ {\epsilon , 2} e T ^ {1 - \zeta \tilde {\alpha} _ {2}}}{1 - \zeta \tilde {\alpha} _ {2}} & \text {i f} 0 <   \zeta \tilde {\alpha} _ {2} <   1 \\ 1 + \Big (\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2} \Big) \frac {M _ {\epsilon , 2} e}{(\log T) ^ {c \tilde {\alpha} _ {2} - 4}} \log (T) \leq 1 + \Big (\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2} \Big) M _ {\epsilon , 2} e & \text {i f} \zeta \tilde {\alpha} _ {2} \geq 1 \end{array} \right. \\ \end{array}
$$
for  $c\tilde{\alpha}_2 \geq 5$ .
Step 2: Consider  $D_{1,2}$ . Note that in this term, the optimal action 1 has been sufficiently drawn to be well estimated, so we can use a loose bound
$$
\mathbb {E} [ D _ {1, 1} ] \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mu_ {1} - \beta_ {T} > \frac {S _ {1} (t)}{N _ {1} (t) + 1}, N _ {1} (t) + 2 \geq (\log T) ^ {2} \right\}
$$
This right-hand side only depends on the draws from action 1 and has been studied in Theorem 1 in [21], so we apply their results:
$$
\mathbb {E} [ D _ {1, 2} ] \leq \frac {1}{T - 1}.
$$
Step 3: Consider  $D_{2}$ . Using the same technique as in lemma 7 in [15],  $D_{2}$  is bounded by
$$
D _ {2} \leq \sum_ {s = 1} ^ {T} \mathbb {1} \left\{s d ^ {+} \left(\hat {\mu} _ {2} (t), \mu_ {1} - \beta_ {T}\right) \leq \zeta \tilde {\alpha} _ {1} \log (T) + c \tilde {\alpha} _ {1} \log (\log T) - \log \left(M _ {\epsilon , 1}\right) \right\}
$$
For  $\xi >0$  , we let
$$
K _ {T, \epsilon} = \frac {(1 + \xi) (\zeta \tilde {\alpha} _ {1} \log (T) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1})}{d (\mu_ {2} , \mu_ {1})}.
$$
Then  $D_{2}$  can be rewritten as
$$
\begin{array}{l} D _ {2} \leq \sum_ {s = 1} ^ {\lfloor K _ {T, \epsilon} \rfloor} 1 + \sum_ {s = \lfloor K _ {T, \epsilon} \rfloor + 1} ^ {T} \mathbb {1} \left\{K _ {T, \epsilon} d ^ {+} (\hat {\mu} _ {2} (t), \mu_ {1} - \beta_ {T}) \leq \zeta \tilde {\alpha} _ {1} \log (T) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1}) \right\} \\ = \left\lfloor K _ {T, \epsilon} \right\rfloor + \sum_ {s = \left\lfloor K _ {T, \epsilon} \right\rfloor + 1} ^ {T} \mathbb {1} \left\{d ^ {+} \left(\hat {\mu} _ {2} (t), \mu_ {1} - \beta_ {T}\right) \leq \frac {d \left(\mu_ {2} , \mu_ {1}\right)}{1 + \xi} \right\} \\ \leq K _ {T, \epsilon} + \sum_ {s = \left\lfloor K _ {T, \epsilon} \right\rfloor + 1} ^ {T} \mathbb {1} \left\{d ^ {+} \left(\hat {\mu} _ {2} (t), \mu_ {1}\right) \leq \frac {d \left(\mu_ {2} , \mu_ {1}\right)}{1 + \xi} + \beta_ {T} \frac {2}{\mu_ {1} \left(2 - \mu_ {1}\right)} \right\} \\ \end{array}
$$
where the last inequality follows from the same technique in the proof of Theorem 1 in [21], by noting that the function  $g(q) = d^{+}(\hat{\mu}_{2}(s),q)$  is convex and differentiable and  $g^{\prime}(q) = \frac{q - \hat{\mu}_{2}(s)}{q(1 - q)}\mathbb{1}\{(q > \hat{\mu}_{2}(s)\}$ . Hence, for  $T\geq \exp \left(\left(\frac{2(1 + \xi)(1 + \xi / 2)}{\xi\mu_{1}(1 - \mu_{1})d(\mu_{2},\mu_{1})}\right)^{2}\right)$ , we obtain  $\frac{d(\mu_2,\mu_1)}{1 + \xi} +\beta_T\frac{2}{\mu_1(1 - \mu)}\leq \frac{d(\mu_2,\mu_1)}{1 + \xi / 2}$ . Following the proof of Theorem 1 in [21] (as well as [15]), we obtain
$$
\begin{array}{l} \mathbb {E} \left[ D _ {2} \right] \leq K _ {T, \epsilon} + \sum_ {s = \lfloor K _ {T, \epsilon} \rfloor + 1} ^ {T} \mathbb {P} \left(d ^ {+} (\hat {\mu} _ {2} (t), \mu_ {1}) \leq \frac {d \left(\mu_ {2} , \mu_ {1}\right)}{1 + \xi / 2}\right) \\ \leq K _ {T, \epsilon} + \frac {(1 + \xi / 2) ^ {2}}{\xi^ {2} (\min  \left\{\mu_ {2} (1 - \mu_ {2}) , \mu_ {1} (1 - \mu_ {1}) \right\}) ^ {2}} \\ \end{array}
$$
Step 4: Combing the above results, we obtain that if  $0 < \zeta \tilde{\alpha}_{2} < 1$
$$
\begin{array}{l} \mathbb {E} \left[ N _ {2} (T) \right] \leq \mathbb {E} \left[ D _ {1, 1} \right] + \mathbb {E} \left[ D _ {1, 2} \right] + \mathbb {E} \left[ D _ {2} \right] \\ \leq 1 + \frac {\left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) M _ {\epsilon , 2} e T ^ {1 - \zeta \tilde {\alpha} _ {2}}}{1 - \zeta \tilde {\alpha} _ {2}} + \frac {(1 + \xi) (\zeta \tilde {\alpha} _ {1} \log (T) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1})}{d (\mu_ {2} , \mu_ {1})} \\ + \frac {(1 + \xi / 2) ^ {2}}{\xi^ {2} (\min  \left\{\mu_ {2} (1 - \mu_ {2}) , \mu_ {1} (1 - \mu_ {1}) \right\}) ^ {2}} \\ \leq \frac {\left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) M _ {\epsilon , 2} e T ^ {1 - \zeta \tilde {\alpha} _ {2}}}{1 - \zeta \tilde {\alpha} _ {2}} + o \left(T ^ {1 - \zeta \tilde {\alpha} _ {2}}\right). \\ \end{array}
$$
and if  $\zeta \tilde{\alpha}_{2} \geq 1$ ,
$$
\begin{array}{l} \mathbb {E} \left[ N _ {2} (T) \right] \leq \mathbb {E} \left[ D _ {1, 1} \right] + \mathbb {E} \left[ D _ {1, 2} \right] + \mathbb {E} \left[ D _ {2} \right] \\ \leq 1 + \left(\zeta \tilde {\alpha} _ {2} + c \tilde {\alpha} _ {2}\right) M _ {\epsilon , 2} e + \frac {(1 + \xi) (\zeta \tilde {\alpha} _ {1} \log (T) + c \tilde {\alpha} _ {1} \log (\log T) - \log (M _ {\epsilon , 1})}{d (\mu_ {2} , \mu_ {1})} \\ + \frac {(1 + \xi / 2) ^ {2}}{\xi^ {2} (\min  \left\{\mu_ {2} (1 - \mu_ {2}) , \mu_ {1} (1 - \mu_ {1}) \right\}) ^ {2}} \\ \leq \frac {(1 + \xi) \zeta \tilde {\alpha} _ {1}}{d (\mu_ {2} , \mu_ {1})} \log (T) + o (\log T). \\ \end{array}
$$
Note that the final step (Step 4) in the above proof provides an exact finite-time regret bound that holds for any time horizon  $T$ . It explicitly expresses the  $o(\cdot)$  term in Theorem 3.7. The error terms  $M_{\epsilon,1}^{-1}$  and  $M_{\epsilon,2}$  depending on  $\epsilon$  appear in this regret bound explicitly. Obviously,  $M_{\epsilon,1}^{-1}$  and  $M_{\epsilon,2}$  in the bound increase as  $\epsilon$  increases.
In general, our above derivations depend on bounds for specific distributions (Beta posterior distributions with inference errors in our setting). It is a direction to generalize these results to more general bandit problems or more general families of distributions. For instance, [20] extends the setting of Beta posterior distribution to the exponential family that includes Gaussian (without approximate inference). Combining [20] with our techniques in Sections 3.2 and 3.3 may lead to analyzing the exponential family with approximate inference. This, however, requires some additional careful technical derivation beyond our current bounds in Bernoulli with approximate inference, which is a future research direction.
Appendix D Proofs of Results in Section 3.4
Proof of Theorem 3.12. We can explicitly construct such a distribution  $Q_{t}$  as follows:
$$
q _ {t, 2} (x _ {2}) = \pi_ {t, 2} (x _ {2})
$$
$$
q _ {t, 1} \left(x _ {1}\right) = \left\{ \begin{array}{l l} \frac {1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)}{F _ {t , 1} \left(b _ {t}\right)} \pi_ {t, 1} \left(x _ {1}\right) & \text {i f} 0 <   x _ {1} <   b _ {t} \\ \frac {1}{r} \pi_ {t, 1} \left(x _ {1}\right) & \text {i f} b _ {t} <   x _ {1} <   1 \end{array} \right. \tag {13}
$$
where  $F_{t,1}$  is the cumulative distribution function (cdf) of  $\Pi_{t,1}$  and  $r > 1$ ,  $b_t \in (0,1)$  will be specified later.
First, note that by setting
$$
q _ {t, 2} = \pi_ {t, 2},
$$
we have  $D_{\alpha}(\Pi_{t,2},Q_{t,2}) = 0$  and  $q_{t,2} = \pi_{t,2}$  with the same support [0,1], satisfying Assumptions 3.6 and 3.11 on action  $j = 2$ .
We set  $b_{t} = Qu\left(\frac{1}{2}, \Pi_{t,2}\right) \in (0,1)$ , the  $\frac{1}{2}$ -quantile of the distribution  $\Pi_{t,2}$  (or equivalently,  $Q_{t,2}$ ). Let  $F_{t,1}$  be the cdf of  $\Pi_{t,1}$ . We have  $F_{t,1}(b_{t}) \in (0,1)$ . For  $r > 1$ , we set
$$
q _ {t, 1} (x _ {1}) = \left\{ \begin{array}{l l} \frac {1 - \frac {1}{r} (1 - F _ {t , 1} (b _ {t}))}{F _ {t , 1} (b _ {t})} \pi_ {t, 1} (x _ {1}) & \text {i f} 0 <   x _ {1} <   b _ {t} \\ \frac {1}{r} \pi_ {t, 1} (x _ {1}) & \text {i f} b _ {t} <   x _ {1} <   1 \end{array} \right.
$$
Step 1: We show that  $q_{t,1}(x_1)$  is indeed a density satisfying Assumption 3.6 on action  $j = 1$ . First of all, it is obvious that  $q_{t,1} > 0$  on  $(0,1)$  as  $\pi_{t,1} > 0$  on  $(0,1)$ . Moreover
$$
\begin{array}{l} \int_ {0} ^ {1} q _ {t, 1} \left(x _ {1}\right) d x _ {1} = \int_ {0} ^ {b _ {t}} q _ {t, 1} \left(x _ {1}\right) d x _ {1} + \int_ {b _ {t}} ^ {1} q _ {t, 1} \left(x _ {1}\right) d x _ {1} \\ = \int_ {0} ^ {b _ {t}} \frac {1 - \frac {1}{r} (1 - F _ {t , 1} (b _ {t}))}{F _ {t , 1} (b _ {t})} \pi_ {t, 1} (x _ {1}) d x _ {1} + \int_ {b _ {t}} ^ {1} \frac {1}{r} \pi_ {t, 1} (x _ {1}) d x _ {1} \\ = \frac {1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)}{F _ {t , 1} \left(b _ {t}\right)} F _ {t, 1} \left(b _ {t}\right) + \frac {1}{r} \left(1 - F _ {t, 1} \left(b _ {t}\right)\right) \\ = 1. \\ \end{array}
$$
Step 2: We show that there exists an  $r > 1$  (independent of  $t$ ) such that  $q_{t,1}$  satisfies Assumption 3.11 on action  $j = 1$ .
We have that when  $\alpha < 0$  or  $0 < \alpha < 1$ :
$$
\begin{array}{l} D _ {\alpha} \left(Q _ {t, 1}, \Pi_ {t, 1}\right) = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {b _ {t}} \pi_ {t, 1} \left(x _ {1}\right) \left(\frac {\pi_ {t , 1} \left(x _ {1}\right)}{q _ {t , 1} \left(x _ {1}\right)}\right) ^ {- \alpha} d x _ {1} + \int_ {b _ {t}} ^ {1} \pi_ {t, 1} \left(x _ {1}\right) \left(\frac {\pi_ {t , 1} \left(x _ {1}\right)}{q _ {t , 1} \left(x _ {1}\right)}\right) ^ {- \alpha} d x _ {1} - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {b _ {t}} \pi_ {t, 1} \left(x _ {1}\right) \left(\frac {F _ {t , 1} \left(b _ {t}\right)}{1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)}\right) ^ {- \alpha} d x _ {1} + \int_ {b _ {t}} ^ {1} \pi_ {t, 1} \left(x _ {1}\right) r ^ {- \alpha} d x _ {1} - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\left(\frac {F _ {t , 1} (b _ {t})}{1 - \frac {1}{r} (1 - F _ {t , 1} (b _ {t}))}\right) ^ {- \alpha} F _ {t, 1} (b _ {t}) + r ^ {- \alpha} (1 - F _ {t, 1} (b _ {t})) - 1\right) \\ \end{array}
$$
$$
\frac {F _ {t , 1} \left(b _ {t}\right)}{1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)} \leq \frac {r - 1 + F _ {t , 1} \left(b _ {t}\right)}{1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)} = r
$$
as  $r > 1$ . Hence we have
$$
D _ {\alpha} \left(Q _ {t, 1}, \Pi_ {t, 1}\right) \leq \frac {1}{\alpha (\alpha - 1)} \left(r ^ {- \alpha} F _ {t, 1} \left(b _ {t}\right) + r ^ {- \alpha} \left(1 - F _ {t, 1} \left(b _ {t}\right)\right) - 1\right) = \frac {1}{\alpha (\alpha - 1)} \left(r ^ {- \alpha} - 1\right).
$$
Then for  $1 < r < (\epsilon \alpha(\alpha - 1) + 1)^{-\frac{1}{\alpha}}$  (only if  $\epsilon \alpha(\alpha - 1) + 1 > 0$ , otherwise we put  $+\infty$  as the upper bound on  $r$ ), we have that
$$
D _ {\alpha} (Q _ {t, 1}, \Pi_ {t, 1}) \leq \epsilon .
$$
When  $\alpha = 0$ :
$$
\begin{array}{l} D _ {0} \left(Q _ {t, 1}, \Pi_ {t, 1}\right) = K L \left(\Pi_ {t, 1}, Q _ {t, 1}\right) \\ = \int_ {0} ^ {b _ {t}} \pi_ {t, 1} \left(x _ {1}\right) \log \left(\frac {\pi_ {t , 1} \left(x _ {1}\right)}{q _ {t , 1} \left(x _ {1}\right)}\right) d x _ {1} + \int_ {b _ {t}} ^ {1} \pi_ {t, 1} \left(x _ {1}\right) \log \left(\frac {\pi_ {t , 1} \left(x _ {1}\right)}{q _ {t , 1} \left(x _ {1}\right)}\right) d x _ {1} \\ = \int_ {0} ^ {b _ {t}} \pi_ {t, 1} \left(x _ {1}\right) \log \left(\frac {F _ {t , 1} \left(b _ {t}\right)}{1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)}\right) d x _ {1} + \int_ {b _ {t}} ^ {1} \pi_ {t, 1} \left(x _ {1}\right) \log (r) d x _ {1} \\ = \log \left(\frac {F _ {t , 1} \left(b _ {t}\right)}{1 - \frac {1}{r} \left(1 - F _ {t , 1} \left(b _ {t}\right)\right)}\right) F _ {t, 1} \left(b _ {t}\right) + \log (r) \left(1 - F _ {t, 1} \left(b _ {t}\right)\right) \\ \end{array}
$$
$$
\frac {F _ {t , 1} (b _ {t})}{1 - \frac {1}{r} (1 - F _ {t , 1} (b _ {t}))} \leq \frac {r - 1 + F _ {t , 1} (b _ {t})}{1 - \frac {1}{r} (1 - F _ {t , 1} (b _ {t}))} = r
$$
$$
K L \left(\Pi_ {t, 1}, Q _ {t, 1}\right) \leq \log (r) F _ {t, 1} \left(b _ {t}\right) + \log (r) \left(1 - F _ {t, 1} \left(b _ {t}\right)\right) = \log (r).
$$
Then for  $1 < r < e^{\epsilon}$ , we have that
$$
D _ {0} \left(Q _ {t, 1}, \Pi_ {t, 1}\right) = K L \left(\Pi_ {t, 1}, Q _ {t, 1}\right) \leq \log (r) \leq \log \left(e ^ {\epsilon}\right) = \epsilon .
$$
Step 3: We show that the probability of sampling from  $Q_{t - 1}$  choosing action 2 is greater than a positive constant  $\frac{1}{2}\left(1 - \frac{1}{r}\right)$ , which thus leads to a linear regret.
In fact, the probability of sampling from  $Q_{t-1}$  choosing action 2 is given by  $\mathbb{P}_{Q_{t-1}}(x_2 \geq x_1)$ . Therefore we have that
$$
\mathbb {P} _ {Q _ {t - 1}} \left(x _ {2} \geq x _ {1}\right) \geq \mathbb {P} _ {Q _ {t - 1}} \left(x _ {2} \geq b _ {t - 1} \geq x _ {1}\right) = \mathbb {P} _ {Q _ {t - 1, 2}} \left(x _ {2} \geq b _ {t - 1}\right) \mathbb {P} _ {Q _ {t - 1, 1}} \left(x _ {1} \leq b _ {t - 1}\right)
$$
since  $Q_{t - 1,1}$  and  $Q_{t - 1,2}$  are independent.
$$
\mathbb {P} _ {Q _ {t - 1, 2}} (x _ {2} \geq b _ {t - 1}) = \frac {1}{2}
$$
since  $b_{t-1}$  is the  $\frac{1}{2}$ -quantile of the distribution  $\Pi_{t-1,2}$  and  $p_{t-1,2} > 0$  on  $(0,1)$ .
$$
\mathbb {P} _ {Q _ {t - 1, 1}} (x _ {1} \leq b _ {t - 1}) = 1 - \mathbb {P} _ {Q _ {t - 1, 1}} (x _ {1} \geq b _ {t - 1}) = 1 - \frac {1}{r} (1 - F _ {t - 1, 1} (b _ {t - 1})) \geq 1 - \frac {1}{r}
$$
by our construction of  $Q_{t - 1,1}$ . Therefore we have that
$$
\mathbb {P} _ {Q _ {t - 1}} \left(x _ {2} \geq x _ {1}\right) \geq \frac {1}{2} \left(1 - \frac {1}{r}\right) > 0.
$$
We conclude that the lower bound of the average expected regret is given by
$$
R (T, \mathcal {A}) = \sum_ {j = 1} ^ {2} (\mu_ {1} - \mu_ {j}) \mathbb {E} [ N _ {j} (t) ] \geq (\mu_ {1} - \mu_ {2}) \frac {T}{2} (1 - \frac {1}{r}) = \Omega (T)
$$
leading to a linear regret.
Proof of Theorem 3.13. We can explicitly construct such a distribution  $Q_{t}$  as follows:
$$
q _ {t, 1} (x _ {1}) = \pi_ {t, 1} (x _ {1})
$$
$$
q _ {t, 2} \left(x _ {2}\right) = \left\{ \begin{array}{l l} \frac {1}{r} \pi_ {t, 2} \left(x _ {2}\right) & \text {i f} 0 <   x _ {1} <   b _ {t} \\ \frac {1 - \frac {1}{r} F _ {t , 2} \left(b _ {t}\right)}{1 - F _ {t , 2} \left(b _ {t}\right)} \pi_ {t, 2} \left(x _ {2}\right) & \text {i f} b _ {t} <   x _ {1} <   1 \end{array} \right. \tag {14}
$$
where  $F_{t,2}$  is the cdf of  $\Pi_{t,2}$  and  $r > 1$ ,  $b_{t} \in (0,1)$  will be specified later.
$$
q _ {t, 1} = \pi_ {t, 1},
$$
we have  $D_{\alpha}(\Pi_{t,1},Q_{t,1}) = 0$  and  $q_{t,1} = \pi_{t,1}$  with the same support [0, 1], satisfying Assumptions 3.6 and 3.11 on action  $j = 1$ .
We set  $b_{t} = Qu(\gamma_{t + 1},\Pi_{t,1}) \in (0,1)$ , the  $\gamma_{t + 1}$ -quantile of the distribution  $\Pi_{t,1}$  (or equivalently,  $Q_{t,1}$ ). Let  $F_{t,2}$  be the cdf of  $\Pi_{t,2}$ . We have  $F_{t,2}(b_t) \in (0,1)$ . For  $r > 1$ , we set
$$
q _ {t, 2} (x _ {2}) = \left\{ \begin{array}{l l} \frac {1}{r} \pi_ {t, 2} (x _ {2}) & \text {i f} 0 <   x _ {2} <   b _ {t} \\ \frac {1 - \frac {1}{r} F _ {t , 2} (b _ {t})}{1 - F _ {t , 2} (b _ {t})} \pi_ {t, 2} (x _ {2}) & \text {i f} b _ {t} <   x _ {2} <   1 \end{array} \right.
$$
Step 1: We show that  $q_{t,2}(x_2)$  is indeed a density satisfying Assumption 3.6 on action  $j = 2$ . First of all, it is obvious that  $q_{t,2} > 0$  on  $(0,1)$  as  $\pi_{t,2} > 0$  on  $(0,1)$ . Moreover
$$
\begin{array}{l} \int_ {0} ^ {1} q _ {t, 2} (x _ {2}) d x _ {2} = \int_ {0} ^ {b _ {t}} q _ {t, 2} (x _ {2}) d x _ {2} + \int_ {b _ {t}} ^ {1} q _ {t, 2} (x _ {2}) d x _ {2} \\ = \int_ {0} ^ {b _ {t}} \frac {1}{r} \pi_ {t, 2} (x _ {2}) d x _ {2} + \int_ {b _ {t}} ^ {1} \frac {1 - \frac {1}{r} F _ {t , 2} (b _ {t})}{1 - F _ {t , 2} (b _ {t})} \pi_ {t, 2} (x _ {2}) d x _ {2} \\ = \frac {1}{r} F _ {t, 2} \left(b _ {t}\right) + \frac {1 - \frac {1}{r} F _ {t , 2} \left(b _ {t}\right)}{1 - F _ {t , 2} \left(b _ {t}\right)} \left(1 - F _ {t, 2} \left(b _ {t}\right)\right) \\ = 1. \\ \end{array}
$$
Step 2: We show that there exists an  $r > 1$  (independent of  $t$ ) such that  $q_{t,2}$  satisfies Assumption 3.11 on action  $j = 2$ .
$$
\begin{array}{l} D _ {\alpha} (Q _ {t, 2}, \Pi_ {t, 2}) = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {b _ {t}} \pi_ {t, 2} (x _ {2}) \left(\frac {\pi_ {t , 2} (x _ {2})}{q _ {t , 2} (x _ {2})}\right) ^ {- \alpha} d x _ {2} + \int_ {b _ {t}} ^ {1} \pi_ {t, 2} (x _ {2}) \left(\frac {\pi_ {t , 2} (x _ {2})}{q _ {t , 2} (x _ {2})}\right) ^ {- \alpha} d x _ {2} - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(\int_ {0} ^ {b _ {t}} \pi_ {t, 2} (x _ {2}) r ^ {- \alpha} d x _ {2} + \int_ {b _ {t}} ^ {1} \pi_ {t, 2} (x _ {2}) \left(\frac {1 - F _ {t , 2} (b _ {t})}{1 - \frac {1}{r} F _ {t , 2} (b _ {t})}\right) ^ {- \alpha} d x _ {2} - 1\right) \\ = \frac {1}{\alpha (\alpha - 1)} \left(r ^ {- \alpha} F _ {t, 2} (b _ {t}) + \left(\frac {1 - F _ {t , 2} (b _ {t})}{1 - \frac {1}{r} F _ {t , 2} (b _ {t})}\right) ^ {- \alpha} (1 - F _ {t, 2} (b _ {t})) - 1\right) \\ \end{array}
$$
$$
\frac {1 - F _ {t , 2} \left(b _ {t}\right)}{1 - \frac {1}{r} F _ {t , 2} \left(b _ {t}\right)} \leq \frac {r - F _ {t , 2} \left(b _ {t}\right)}{1 - \frac {1}{r} F _ {t , 2} \left(b _ {t}\right)} = r
$$
$$
D _ {\alpha} \left(\Pi_ {t, 2}, Q _ {t, 2}\right) \leq \frac {1}{\alpha (\alpha - 1)} \left(r ^ {- \alpha} F _ {t, 2} (b _ {t}) + r ^ {- \alpha} \left(1 - F _ {t, 2} (b _ {t})\right) - 1\right) = \frac {1}{\alpha (\alpha - 1)} \left(r ^ {- \alpha} - 1\right).
$$
$$
D _ {\alpha} (Q _ {t, 2}, \Pi_ {t, 2}) \leq \epsilon .
$$
When  $\alpha = 0$
$$
\begin{array}{l} D _ {0} (Q _ {t, 2}, \Pi_ {t, 2}) = K L (\Pi_ {t, 2}, Q _ {t, 2}) = \int_ {0} ^ {b _ {t}} \pi_ {t, 2} (x _ {2}) \log \left(\frac {\pi_ {t , 2} (x _ {2})}{q _ {t , 2} (x _ {2})}\right) d x _ {2} + \int_ {b _ {t}} ^ {1} \pi_ {t, 2} (x _ {2}) \log \left(\frac {\pi_ {t , 2} (x _ {2})}{q _ {t , 2} (x _ {2})}\right) d x _ {2} \\ = \int_ {0} ^ {b _ {t}} \pi_ {t, 2} (x _ {2}) \log (r) d x _ {2} + \int_ {b _ {t}} ^ {1} \pi_ {t, 2} (x _ {2}) \log \left(\frac {1 - F _ {t , 2} (b _ {t})}{1 - \frac {1}{r} F _ {t , 2} (b _ {t})}\right) d x _ {2} \\ = \log (r) F _ {t, 2} (b _ {t}) + \log \left(\frac {1 - F _ {t , 2} (b _ {t})}{1 - \frac {1}{r} F _ {t , 2} (b _ {t})}\right) (1 - F _ {t, 2} (b _ {t})) \\ \end{array}
$$
$$
K L \left(\Pi_ {t, 2}, Q _ {t, 2}\right) \leq \log (r) F _ {t, 2} \left(b _ {t}\right) + \log (r) \left(1 - F _ {t, 2} \left(b _ {t}\right)\right) = \log (r).
$$
$$
D _ {0} \left(Q _ {t, 2}, \Pi_ {t, 2}\right) = K L \left(\Pi_ {t, 2}, Q _ {t, 2}\right) \leq \log (r) = \log \left(e ^ {\epsilon}\right) = \epsilon .
$$
Therefore, we conclude that there exists an  $r > 1$  (independent of  $t$ ) such that  $q_{t,2}$  satisfies Assumption 3.11 on action  $j = 2$ . Take this  $r > 1$  and notice that since  $\gamma_t \to 1$  as  $t \to +\infty$ , there must exist a  $T_0 > 0$  such that for any  $t \geq T_0$ , we have that  $\gamma_t > \frac{1}{r}$ .
Step 3: We show that the EBUCB algorithm always chooses action 2 when  $t \geq T_0$ , which thus leads to a linear regret.
We note that when  $t \geq T_0$ , by definition,
$$
\mathbb {P} _ {Q _ {t - 1, 1}} \left(x _ {1} \leq b _ {t - 1}\right) = \mathbb {P} _ {\Pi_ {t - 1, 1}} \left(x _ {1} \leq b _ {t - 1}\right) = \gamma_ {t},
$$
$$
\mathbb {P} _ {Q _ {t - 1, 2}} (x _ {2} \leq b _ {t - 1}) = \frac {1}{r} F _ {t - 1, 2} (b _ {t - 1}) \leq \frac {1}{r} <   \gamma_ {t},
$$
$$
Q u \left(\gamma_ {t}, Q _ {t - 1, 1}\right) = b _ {t - 1} <   Q u \left(\gamma_ {t}, Q _ {t - 1, 2}\right)
$$
Therefore after time step  $t \geq T_0$ , the EBUCB algorithm will always choose the action 2. We conclude that the lower bound of the average expected regret is given by
$$
R (T, \mathcal {A}) = \sum_ {j = 1} ^ {2} (\mu_ {1} - \mu_ {j}) \mathbb {E} [ N _ {j} (t) ] \geq (\mu_ {1} - \mu_ {2}) (T - T _ {0}) = \Omega (T)
$$
Appendix E Additional Experiments
In this section, we present additional experimental results. We enrich our experiments by studying an increasing number of arms as well as multiple new problem instances with different inference errors. These results further support our findings in Section 4 that EBUCB without the horizon-dependent term (i.e.,  $c = 0$ ) performs the best.
$$
(1 - w) * \operatorname {B e t a} (1 + S _ {j} (t), 1 + N _ {j} (t) - S _ {j} (t)) + w * \operatorname {B e t a} (\Gamma (1 + S _ {j} (t)), \Gamma (1 + N _ {j} (t) - S _ {j} (t)))
$$
where  $j \in [K]$ . We conduct two experiments: 1) The Bernoulli multi-armed bandit problem has  $K$  actions with the following mean rewards:
$$
K = 2: \text {m e a n r e w a r d s} = [ 0. 7, 0. 3 ]
$$
$$
K = 4: \text {m e a n r e w a r d s} = [ 0. 9, 0. 7, 0. 5, 0. 3 ]
$$
$$
K = 8: \text {m e a n r e w a r d s} = [ 0. 9, 0. 8, 0. 7, 0. 6, 0. 5, 0. 4, 0. 3, 0. 2 ]
$$
$$
\begin{array}{l} K = 1 6: \text {m e a n r e w a r d s} = [ 0. 9, 0. 8 5, 0. 8, 0. 7 5, 0. 7, 0. 6 5, 0. 6, 0. 5 5, 0. 5, 0. 4 5, 0. 4, 0. 3 5, 0. 3, 0. 2 5, 0. 2, \\ 0. 1 5 ] \end{array}
$$
Let  $\Gamma = 2$  or 0.5. Let  $w = 0.9$ . The results are shown in Figure 3 below.
2) We also study different  $\Gamma$  values in the appropriate distribution that lead to different inference errors. Consider  $\Gamma = 0.05, 0.1, 0.2, 0.5, 2, 5, 10, 15$  in the experiments. Let  $K = 2$  with mean rewards  $= [0.7, 0.3]$ . Let  $w = 0.9$ .
The results are shown in Figure 4 below.
Figure 3: Experiments with different  $K$  with  $\Gamma = 0.5, 2$ . The curve-algorithm correspondence is the same as in Figure 1.
Figure 4: Experiments with different  $\Gamma$ . The curve-algorithm correspondence is the same as in Figure 1.