SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap (k \in \operatorname {M i d} _ {t})\right) \\ = \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap (k \notin \mathcal {A} _ {t} \cup \mathcal {B} _ {t})\right) \\ \leqslant \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap \left(\mathrm {U C B} _ {k, t} > c\right)\right) \\ = \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap \left(\min  _ {m \in \mathcal {M}} \hat {\mu} _ {k, t} ^ {(m)} + \zeta^ {(m)} + \beta \left(N _ {k, t} ^ {(m)}, t\right) > c\right)\right) \\ \leqslant \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap \left(\hat {\mu} _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} + \zeta^ {\left(m _ {k} ^ {*}\right)} + \beta \left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)}, t\right) > c\right)\right) \\ \leqslant \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap \left(\hat {\mu} _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} - \mu_ {k} ^ {\left(m _ {k} ^ {*}\right)} > \left(c - \mu_ {k} ^ {\left(m _ {k} ^ {*}\right)} - \zeta^ {\left(m _ {k} ^ {*}\right)}\right) - \beta \left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)}, t\right)\right)\right) \\ \stackrel {(a)} {\leqslant} \mathbb {P} \left(\left(N _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 N _ {k, t} ^ {*}\right) \cap \left(\hat {\mu} _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} - \mu_ {k} ^ {\left(m _ {k} ^ {*}\right)} > \frac {\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{2} - \beta \left(N _ {k, t} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}, t\right)\right)\right) \\ \stackrel {(b)} {\leqslant} \sum_ {\tau > 1 6 N _ {k, t} ^ {*}} \mathbb {P} \left(\hat {\mu} _ {k, t (\tau)} ^ {(m _ {k} ^ {*})} - \mu_ {k} ^ {(m _ {k} ^ {*})} > \frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{4}\right) \\ \leqslant \sum_ {\tau > 1 6 N _ {k, t} ^ {*}} \exp \left(- \frac {\tau \left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}}{1 6}\right) \leqslant \int_ {\tau > 1 6 N _ {k, t} ^ {*}} \exp \left(- \frac {\tau \left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}}{1 6}\right) d \tau \leqslant \frac {1 6 \delta}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2} L t ^ {4}}, \\ \end{array}
$$
where inequality (a) is due to the definition of  $c$ , and inequality (b), denoting  $\hat{\mu}_{k,t(\tau)}^{(m_k^*)}$  as the empirical mean of  $\tau$  observations, is due to  $\beta (\tau ,t) < \frac{\Delta_k}{4}$  for  $\tau >16N_{k,t}^{*}$ .
From Step 3, we obtain that the following equation holds with high probability,
$$
N _ {k, t} ^ {\left(\tilde {m} _ {k} ^ {*}\right)} \leqslant \frac {1 6}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} \log \left(\frac {L t ^ {4}}{\delta}\right) \leqslant \frac {6 4}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} \log \left(\frac {L t}{\delta}\right). \tag {14}
$$
Next, we respectively present the cost complexity upper bounds for different fidelity selection procedures in Algorithm 2.
E.2.1 Proof for EXPLOR-E-A's Upper Bound
Step 4 for EXPLOR-E: prove that if the small probability events of Steps 2 and 3 do not happen, then the algorithm terminates with a high probability when  $\Lambda$  is large.
Lemma E.2. Give reward means  $\mu_1^{(M)}$  and  $\mu_2^{(M)}$ . For a fixed arm  $k$ , there exist  $\bar{N}_{k,t}$  and  $\alpha_{k} > 0$  such that when  $N_{k,t} > \bar{N}_{k,t}$ ,  $N_{k,t} < 2N_{k,t}^{(\tilde{m}_k^*)}$ , the number of times of pulling this arm  $k$  at fidelities  $m (\neq \tilde{m}_k^*)$  is  $O(\log (\log N_{k,t}))$ , or formally,
$$
N _ {k, t} ^ {(m)} \leqslant \frac {8}{\lambda^ {(m)}} \left(\frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \frac {\Delta_ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}\right) ^ {- 2} \log N _ {k, t}, \forall m \neq \tilde {m} _ {k} ^ {*}. \tag {15}
$$
Combine Lemma E.2 with Eq.(14) in Step 3, we have, for any arm  $k$  and fidelity  $m \neq \tilde{m}_k^*$ :
$$
N _ {k, t} ^ {(m)} \leqslant \frac {8}{\lambda^ {(m)}} \left(\frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \frac {\Delta_ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}\right) ^ {- 2} \log \left(\frac {1 2 8}{\left(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}\right) ^ {2}} \log \left(\frac {L t}{\delta}\right)\right) \tag {16}
$$
Next, we can upper bound the total cost of the LUCB algorithm (before it terminating) via Eq.(14) and Eq.(16). Specially, we show it is impossible for  $\Lambda = C\left(H\log \frac{L(G + H)}{\lambda^{(1)}\delta} +G\log \log \frac{L(G + H)}{\lambda^{(1)}\delta}\right)$  via contradiction. Suppose  $\Lambda = C\left(H\log \frac{L(G + H)}{\lambda^{(1)}\delta} +G\log \log \frac{L(G + H)}{\lambda^{(1)}\delta}\right)$ , we have the following,
$$
\mathbb {E} [ \Lambda ] \leqslant \sum_ {k \in \mathcal {K}} \sum_ {m \in \mathcal {M}} \lambda^ {(m)} N _ {k, t} ^ {(m)}
$$
$$
\leqslant \sum_ {k \in \mathcal {K}} \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)} N _ {k, t} ^ {\left(\tilde {m} _ {k} ^ {*}\right)} + \sum_ {k \in \mathcal {K}} \sum_ {m \neq \tilde {m} _ {k} ^ {*}} \lambda^ {(m)} N _ {k, t} ^ {(m)}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leqslant} 6 4 H \log \frac {L t}{\delta} + 8 G \log \log \frac {L t}{\delta} + G \log (1 2 8 H) \\ \stackrel {(b)} {\leqslant} 6 4 H \log \frac {L \Lambda}{\lambda^ {(1)} \delta} + 8 G \log \log \frac {L \Lambda}{\lambda^ {(1)} \delta} + G \log (1 2 8 H) \\ \stackrel {(c)} {=} 6 4 H \log \left(\frac {L}{\lambda^ {(1)} \delta} C \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right)\right) \\ + 8 G \log \log \left(\frac {L}{\lambda^ {(1)} \delta} C \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right)\right) + G \log (1 2 8 H) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d)} {\leqslant} 1 2 8 (2 + \log C) \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \\ \stackrel {(e)} {<  } C \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right), \\ \end{array}
$$
where the inequality (a) is due to Eq.(14) and Eq.(16), the inequality (b) is because  $t \leqslant \frac{\Lambda}{\lambda^{(1)}}$ , the inequality (c) is by the supposition, the inequality (d) is by separately bounding the above first two terms via Eq.(17) and Eq.(18) in the following, and the inequality (e) holds for  $C > 1200$ . This above inequality contradicts the supposition, and, therefore, we conclude the cost complexity upper bound proof for EXPLORE-A. Similar proof also holds for EXPLORE-B by replacing  $\tilde{m}_k^*$  with  $m_k^\dagger$ .
Next, we provide the upper bounds used in the inequality (d) above:
$$
\begin{array}{l} H \log \left(\frac {L}{\lambda^ {(1)} \delta} C \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right)\right) \\ \leqslant H \log \left(\frac {L}{\lambda^ {(1)} \delta} C H \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) + H \log \left(\frac {L}{\lambda^ {(1)} \delta} C G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \\ \leqslant H \log C + H \log \left(\frac {L H}{\lambda^ {(1)} \delta} \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) + H \log C + H \log \left(\frac {L G}{\lambda^ {(1)} \delta} \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \\ \leqslant 2 H \log C + 4 H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} \\ \leqslant (4 + 2 \log C) H \log \frac {L (G + H)}{\lambda^ {(1)} \delta}, \tag {17} \\ \end{array}
$$
$$
\begin{array}{l} G \log \log \left(\frac {L}{\lambda^ {(1)} \delta} C \left(H \log \frac {L (G + H)}{\lambda^ {(1)} \delta} + G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right)\right) \\ \leqslant G \log \log \left(\frac {L}{\lambda^ {(1)} \delta} C H \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) + G \log \log \left(\frac {L}{\lambda^ {(1)} \delta} C G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \\ \leqslant G \log \log C + G \log \log \left(\frac {L H}{\lambda^ {(1)} \delta} \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \tag {18} \\ + G \log \log C + G \log \log \left(\frac {L G}{\lambda^ {(1)} \delta} \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}\right) \\ \leqslant 2 G \log \log C + 4 G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta} \\ \leqslant (4 + 2 \log \log C) G \log \log \frac {L (G + H)}{\lambda^ {(1)} \delta}. \\ \end{array}
$$
Proof of Lemma E.2.
Claim 1. For any fixed  $m \neq \tilde{m}_k^*$ , if the following equation holds, then the algorithm will not pull arm  $k$  at fidelity  $m$  with high probability.
$$
N _ {k, t} ^ {(m)} > \frac {8}{\lambda^ {(m)}} \left(\frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda (\tilde {m} _ {k} ^ {*})}} - \frac {\Delta_ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}\right) ^ {- 2} \log N _ {k, t}.
$$
$$
\begin{array}{l} f - U C B _ {u _ {t}, t} ^ {(m)} \left(\mu_ {1} ^ {(M)}\right) = \frac {1}{\sqrt {\lambda^ {(m)}}} \left(\mu_ {1} ^ {(M)} - \hat {\mu} _ {u _ {t}, t} ^ {(m)} - \zeta^ {(m)} + \sqrt {\frac {2 \log N _ {u _ {t} , t}}{N _ {u _ {t} , t} ^ {(m)}}}\right) \\ \stackrel {(a)} {\leqslant} \frac {1}{\sqrt {\lambda^ {(m)}}} \left(\mu_ {1} ^ {(M)} - \mu_ {u _ {t}} ^ {(m)} - \zeta^ {(m)} + 2 \sqrt {\frac {2 \log N _ {u _ {t} , t}}{N _ {u _ {t} , t} ^ {(m)}}}\right) \\ \stackrel {(b)} {\leqslant} \frac {1}{\sqrt {\lambda^ {(m _ {u _ {t}} ^ {*})}}} \left(\mu_ {1} ^ {(M)} - \mu_ {u _ {t}} ^ {(m _ {u _ {t}} ^ {*})} - \zeta^ {(m _ {u _ {t}} ^ {*})}\right) \\ \stackrel {(c)} {\leqslant} \frac {1}{\sqrt {\lambda (\tilde {m} _ {k} ^ {*})}} \left(\mu_ {1} ^ {(M)} - \hat {\mu} _ {u _ {t}, t} ^ {(m _ {u _ {t}} ^ {*})} - \zeta^ {(m _ {u _ {t}} ^ {*})} + \sqrt {\frac {2 \log N _ {u _ {t} , t}}{N _ {u _ {t} , t} ^ {(\tilde {m} _ {k} ^ {*})}}}\right) \\ = \mathbf {f} - \mathrm {U C B} _ {u _ {t}, t} ^ {\left(m _ {u _ {t}} ^ {*}\right)} \left(\mu_ {1} ^ {(M)}\right), \\ \end{array}
$$
where the inequalities (a) and (c) hold with a probability at least  $1 - \frac{1}{(N_{k,t})^2}$  respectively (by Hoeffding's inequality), and the inequality (b) holds due to the equation in the claim.
E.2.2 Proof for EXPLOR-E-B's Upper Bound
As EXPLORE-B of Algorithm 2 also employs the LUCB framework, it shares the first three steps of the proof for Theorem 3.4 in Appendix E.2. Hence, in this part, we focus on the proof of the final cost complexity upper bound.
Lemma E.3. If the condition in Line 10 holds, then the committed fidelity  $\hat{m}_k^*$  fulfills the following inequality:
$$
2 \cdot \frac {\Delta_ {k} ^ {(\hat {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\hat {m} _ {k} ^ {*})}}} \geqslant \frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}}. \tag {19}
$$
Proof of Lemma E.3. Eq.(19) is proved as follows,
$$
\begin{array}{l} \frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})} / \sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}}{\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})} / \sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} \leqslant \frac {(\hat {\mu} _ {*} ^ {(M)} - (\hat {\mu} _ {k} ^ {(\tilde {m} _ {k} ^ {*})} + \zeta^ {(\tilde {m} _ {k} ^ {*})})) / \sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}} + \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}}{(\hat {\mu} _ {*} ^ {(M)} - (\hat {\mu} _ {k} ^ {(\tilde {m} _ {k} ^ {*})} + \zeta^ {(\tilde {m} _ {k} ^ {*})})) / \sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}} - \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}} \\ \stackrel {(a)} {\leqslant} \frac {(\hat {\mu} _ {*} ^ {(M)} - (\hat {\mu} _ {k} ^ {(\hat {m} _ {k} ^ {*})} + \zeta^ {(\hat {m} _ {k} ^ {*})})) / \sqrt {\lambda^ {(\hat {m} _ {k} ^ {*})}} + \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}}{(\hat {\mu} _ {*} ^ {(M)} - (\hat {\mu} _ {k} ^ {(\hat {m} _ {k} ^ {*})} + \zeta^ {(\hat {m} _ {k} ^ {*})})) / \sqrt {\lambda^ {(\hat {m} _ {k} ^ {*})}} - \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}} \\ \leqslant 1 + \frac {2 \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}}{(\hat {\mu} _ {*} ^ {(M)} - (\hat {\mu} _ {k} ^ {(\hat {m} _ {k} ^ {*})} + \zeta^ {(\hat {m} _ {k} ^ {*})})) / \sqrt {\lambda^ {(\hat {m} _ {k} ^ {*})}} - \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}} \\ \stackrel {(b)} {\leqslant} 1 + \frac {2 \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}}{2 \sqrt {\log (2 K M / \delta) / \lambda^ {(1)} N _ {k , t} ^ {(m)}}} = 2, \\ \end{array}
$$
where inequality (a) is due to the definition of  $\hat{m}_k^*$ , and inequality (b) is due to the condition in Line 10.
We next upper bound the number of times of  $N_{k,t}^{(m)}$  that guarantees that the condition in Line 10 is true. Let us consider the case of exploring arm  $u_{t}$ .
$$
\begin{array}{l} \max  _ {m \in \mathcal {M}} \frac {\hat {\Delta} _ {k , t} ^ {(m)}}{\sqrt {\lambda^ {(m)}}} \geqslant \frac {\hat {\mu} _ {k _ {*}} ^ {(M)} - \left(\hat {\mu} _ {k} ^ {(\tilde {m} _ {k} ^ {*})} + \zeta^ {(\tilde {m} _ {k} ^ {*})}\right)}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} \\ \stackrel {(a)} {\geqslant} \frac {\hat {\mu} _ {k _ {*}} ^ {(M)} - \left(\mu_ {k} ^ {(\tilde {m} _ {k} ^ {*})} + \zeta^ {(\tilde {m} _ {k} ^ {*})}\right)}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \sqrt {\frac {\log (2 K M / \delta)}{\lambda^ {(1)} N _ {k , t} ^ {(m)}}} \\ \stackrel {(b)} {\geqslant} \frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \sqrt {\frac {\log (2 K M / \delta)}{\lambda^ {(1)} N _ {k , t} ^ {(m)}}}, \\ \end{array}
$$
where inequality (a) is because that  $\hat{\mu}_k^{(\tilde{m}_k^*)} \leqslant \mu_k^{(\tilde{m}_k^*)} + \sqrt{\frac{\log(2KM / \delta)}{N_{k,t}^{(m)}}}$  with a probability of at least  $1 - \delta / 2KM$  (therefore, with the union bound over all arm-fidelity pairs, the total failure probability of EXPLORE is upper bounded by  $\delta / 2$ ), and inequality (b) is because  $\hat{\mu}_{k_*}^{(M)} - (\mu_k^{(\tilde{m}_k^*)} + \zeta^{(\tilde{m}_k^*)}) \geqslant \mu_{k_*}^{(M)} - (\mu_k^{(\tilde{m}_k^*)} + \zeta^{(\tilde{m}_k^*)}) = \Delta_k^{(\tilde{m}_k^*)}$ .
To make the condition in Line 10 hold, with the above inequality, we need
$$
\frac {\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \sqrt {\frac {\log (2 K M / \delta)}{\lambda^ {(1)} N _ {k , t} ^ {(m)}}} \geqslant 3 \sqrt {\frac {\log (2 K M / \delta)}{\lambda^ {(1)} N _ {k , t} ^ {(m)}}},
$$
which, after rearrangement, becomes
$$
N _ {k, t} ^ {(m)} > \frac {1 6 \lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2}} \frac {\log (2 K M / \delta)}{\lambda^ {(1)}}.
$$
It means that if the above inequality holds, than the condition in Line 10 must hold. That is, except for the committed fidelity  $\hat{m}_k^*$ , we have
$$
N _ {k, t} ^ {(m)} \leqslant \frac {1 6 \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} \frac {\log (2 K M / \delta)}{\lambda^ {(1)}}, \text {f o r a n y o t h e r f i d e l i t i e s} m \neq \hat {m} _ {k} ^ {*}.
$$
For another thing, Eq.(14) of LUCB's proof guarantees that for the selected fidelity  $\hat{m}_k^*$ , the number of pulling times is upper bounded as follows,
$$
N _ {k, t} ^ {(\hat {m} _ {k} ^ {*})} \leqslant \frac {6 4}{(\Delta_ {k} ^ {(\hat {m} _ {k} ^ {*})}) ^ {2}} \log \left(\frac {L t}{\delta}\right).
$$
Then, we upper bound the total budget of the algorithm as follows,
$$
\begin{array}{l} \Lambda = \sum_ {k \in \mathcal {K}} \sum_ {m \in \mathcal {M}} \lambda^ {(m)} N _ {k, t} ^ {(m)} \\ \leqslant \sum_ {k \in \mathcal {K}} \frac {6 4 \lambda^ {\left(\hat {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} \log \left(\frac {L t}{\delta}\right) + \sum_ {k \in \mathcal {K}} \sum_ {m \neq \hat {m} _ {k} ^ {*}} \frac {1 6 \lambda^ {(m)} \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2} \lambda^ {(1)}} \log \left(\frac {2 K M}{\delta}\right) \\ \stackrel {(a)} {\leqslant} \sum_ {k \in \mathcal {K}} \frac {2 5 6 \lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2}} \log \left(\frac {L t}{\delta}\right) + \sum_ {k \in \mathcal {K}} \sum_ {m \neq \hat {m} _ {k} ^ {*}} \frac {1 6 \lambda^ {(m)} \lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2} \lambda^ {(1)}} \log \left(\frac {2 K M}{\delta}\right) \\ \leqslant \left(\sum_ {k \in \mathcal {K}} \frac {2 5 6 \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} + \sum_ {k \in \mathcal {K}} \sum_ {m \neq \hat {m} _ {k} ^ {*}} \frac {1 6 \lambda^ {(m)} \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2} \lambda^ {(1)}}\right) \log \left(\frac {L t}{\delta}\right) \\ \leqslant \sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \sum_ {k \in \mathcal {K}} \frac {2 5 6 \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{\left(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}\right) ^ {2}} \log \left(\frac {L t}{\delta}\right) \\ \leqslant \sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \sum_ {k \in \mathcal {K}} \frac {2 5 6 \lambda^ {\left(\tilde {m} _ {k} ^ {*}\right)}}{(\Delta_ {k} ^ {\left(\tilde {m} _ {k} ^ {*}\right)}) ^ {2}} \log \left(\frac {L \Lambda}{\delta \lambda^ {(1)}}\right) \\ \stackrel {(b)} {\leqslant} \sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \sum_ {k \in \mathcal {K}} \frac {1 0 2 4 \lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2}} \log \left(\sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \sum_ {k \in \mathcal {K}} \frac {2 5 6 \lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\Delta_ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2}} \frac {L}{\lambda^ {(1)} \delta}\right) \\ \leqslant O \left(\sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \tilde {H} \log \left(\sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \cdot \tilde {H} \cdot \frac {L}{\lambda^ {(1)} \delta}\right)\right), \\ \end{array}
$$
where inequality (a) is due to Eq.(19), inequality (b) is due to that  $\Lambda \leqslant A\log (B\Lambda)\Rightarrow \Lambda \leqslant 4A\log (AB\Lambda)$ .
E.2.3 Proof for EXPLORE-C's Upper Bound in Theorem D.1
Step 4 for EXPLORE-C: Prove that if the events of Steps 2 and 3 do not happen, for  $\Lambda > O\left(Q\log \left(\frac{KM\sqrt{Q}}{\delta(\lambda^{(1)})^2}\right)\right)$ , the algorithm terminates with a probability at least  $O(1 - \delta /\Lambda^2)$ . Denote  $\bar{T} := \lceil \frac{\Lambda}{2\lambda^{(M)}}\rceil$  and two events  $E_{1},E_{2}$  as follows,
$$
\begin{array}{l} E _ {1} := \left\{\exists t \geqslant \bar {T}: \mathbf {C R O S} _ {t} \right\}, \\ E _ {2} := \{\exists t \geqslant \bar {T}, k \in \mathcal {K}: \left(n _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} > 1 6 n _ {k, t} ^ {*}\right) \cup \left(k \in \operatorname {M i d} _ {t}\right) \}. \\ \end{array}
$$
We first upper bound the number of rounds after  $\bar{T}$  as follows,
$$
\begin{array}{l} \sum_ {t \geqslant \bar {T}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\neg \operatorname {T E R M} _ {t} \right\} \stackrel {(a)} {=} \sum_ {t \geqslant \bar {T}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\neg \operatorname {T E R M} _ {t} \cap \neg \operatorname {C R O S} _ {t} \right\} \\ \stackrel {(b)} {\leqslant} \sum_ {t \geqslant T} \lambda^ {(m _ {t})} \mathbb {1} \left\{\left(\ell_ {t} \in \operatorname {M i d} _ {t}\right) \cup \left(u _ {t} \in \operatorname {M i d} _ {t}\right) \right\} \\ \leqslant \sum_ {t \geqslant \bar {T}} \sum_ {k \in \mathcal {K}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\left(\left(k = \ell_ {t}\right) \cup \left(k = u _ {t}\right)\right) \cap \left(k \in \operatorname {M i d} _ {t}\right) \right\} \\ \stackrel {(c)} {\leqslant} \sum_ {t \geqslant \bar {T}} \sum_ {k \in \mathcal {K}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\left(\left(k = \ell_ {t}\right) \cup \left(k = u _ {t}\right)\right) \cap \left(n _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} \leqslant 1 6 n _ {k, t} ^ {*}\right) \right\} \\ = \sum_ {k \in \mathcal {K}} \sum_ {t \geqslant T} \lambda^ {(m _ {t})} \mathbb {1} \left\{\left(\left(k = \ell_ {t}\right) \cup \left(k = u _ {t}\right)\right) \cap \left(n _ {k, t} ^ {\left(m _ {k} ^ {*}\right)} \leqslant 1 6 n _ {k, t} ^ {*}\right) \right\} \tag {20} \\ \leqslant \sum_ {k \in \mathcal {K}} \left(\sum_ {\ell = 1} ^ {m _ {k} ^ {*} - 1} \frac {\lambda^ {(\ell)} \log \left(L t ^ {4} / \delta\right)}{(\zeta^ {(\ell)}) ^ {2}} + 1 6 \lambda^ {(m _ {k} ^ {*})} n _ {k, t} ^ {*}\right) \\ \leqslant \sum_ {k \in \mathcal {K}} \left(\sum_ {\ell = 1} ^ {m _ {k} ^ {*} - 1} \frac {\lambda^ {(\ell)}}{(\zeta^ {(\ell)}) ^ {2}} + \frac {1 6 \lambda^ {(m _ {k} ^ {*})}}{\Delta_ {k} ^ {2}}\right) \log \left(\frac {L T ^ {4}}{\delta}\right) \\ \leqslant 4 \sum_ {k \in \mathcal {K}} \left(\sum_ {\ell = 1} ^ {m _ {k} ^ {*} - 1} \frac {\lambda^ {(\ell)}}{(\zeta^ {(\ell)}) ^ {2}} + \frac {1 6 \lambda^ {(m _ {k} ^ {*})}}{\Delta_ {k} ^ {2}}\right) \log \left(\frac {L T}{\delta}\right) \\ \leqslant 4 \sum_ {k \in \mathcal {K}} \left(\sum_ {\ell = 1} ^ {m _ {k} ^ {*} - 1} \frac {\lambda^ {(\ell)}}{(\zeta^ {(\ell)}) ^ {2}} + \frac {1 6 \lambda^ {(m _ {k} ^ {*})}}{\Delta_ {k} ^ {2}}\right) \log \left(\frac {L \Lambda}{\lambda^ {(1)} \delta}\right) \\ \end{array}
$$
where the equation (a) is due to  $\neg E_1$ , the inequality (b) is due to Step 1, and the inequality (c) is due to  $\neg E_2$ .
Also notice that
$$
\Lambda = \sum_ {t <   \bar {T}} \lambda^ {(m _ {t})} + \sum_ {t \geqslant \bar {T}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\neg \operatorname {T E R M} _ {t} \right\} \leqslant \frac {\Lambda}{2} + \sum_ {t \geqslant \bar {T}} \lambda^ {(m _ {t})} \mathbb {1} \left\{\neg \operatorname {T E R M} _ {t} \right\},
$$
and, combining with Eq.(20), we have,
$$
\Lambda \leqslant 8 \sum_ {k \in \mathcal {K}} \left(\sum_ {\ell = 1} ^ {m _ {k} ^ {*} - 1} \frac {\lambda^ {(\ell)}}{(\zeta^ {(\ell)}) ^ {2}} + \frac {1 6 \lambda^ {(m _ {k} ^ {*})}}{\Delta_ {k} ^ {2}}\right) \log \left(\frac {L \Lambda}{\lambda^ {(1)} \delta}\right).
$$
Solving the above inequality concludes the cost complexity upper bound for EXPLORE-C.
In the end of Step 4, we show that the LUCB algorithm fulfills the fixed confidence requirement. The probability that the algorithm does not terminate after spending  $\Lambda$  budget is upper bounded by  $\mathbb{P}(E_1 \cup E_2)$ . Based on Steps 2 and 3, it can be upper bounded as follows,
$$
\begin{array}{l} \mathbb {P} \left(E _ {1} \cup E _ {2}\right) \leqslant \sum_ {t > \bar {T}} \left(\frac {K M \delta}{L t ^ {3}} + \frac {1 6 \delta \sum_ {k \in \mathcal {K}} \Delta_ {k} ^ {- 2}}{L t ^ {4}}\right) \leqslant \Lambda \left(\frac {1}{\lambda^ {(1)}} - \frac {1}{2 \lambda^ {(M)}}\right) \left(\frac {\delta}{\left(\Lambda / \lambda^ {(1)}\right) ^ {3}}\right) \sum_ {k \in \mathcal {K}} \Delta_ {k} ^ {- 2} \\ \leqslant \frac {\delta}{\Lambda^ {2}} \sum_ {k \in \mathcal {K}} \Delta_ {k} ^ {- 2} \left(\left(\lambda^ {(1)}\right) ^ {2} - \frac {\left(\lambda^ {(1)}\right) ^ {3}}{2 \lambda^ {(M)}}\right) \\ \leqslant \delta . \\ \end{array}
$$
F Proofs for Regret Minimization Results
The two lower bound proofs utilize two different regret decomposition as follows,
$$
\begin{array}{l} R (\Lambda) \stackrel {(a)} {\geqslant} \sum_ {t = 1} ^ {N} \left(\frac {\lambda^ {(m _ {t})} - \lambda^ {(1)}}{\lambda^ {(1)}} \mu_ {1} + \Delta_ {I _ {t}} ^ {(M)}\right) \\ = \sum_ {m = 2} ^ {M} N _ {\forall k} ^ {(m)} (\Lambda) \frac {\lambda^ {(m)} - \lambda^ {(1)}}{\lambda^ {(1)}} \mu_ {1} + \sum_ {k \neq k _ {1} ^ {(M)}} N _ {k} ^ {(\forall m)} (\Lambda) \Delta_ {k} ^ {(M)}; \tag {21} \\ \end{array}
$$