SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {G} := \sum_ {k \in \mathcal {K}} \sum_ {m \neq \tilde {m} _ {k} ^ {*}} \left(\frac {\tilde {\Delta} _ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \frac {\tilde {\Delta} _ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}\right) ^ {- 2}.
$$
To give an intuition of when  $\tilde{G}$  is large or small, let us neglect the tilde for simplicity and fix a suboptimal arm  $k$ . The term  $\frac{\Delta_k^{(m)}}{\sqrt{\lambda^{(m)}}}$  corresponds to the effectiveness of pulling arm  $k$  at fidelity  $m$  for distinguishing suboptimal arm  $k$ . As EXPLORE-A aims to find out the optimal (most effective) fidelity  $m_k^*$  for distinguishing suboptimal arm  $k$ , it needs to compare the effectiveness of different fidelities, that is, their effectiveness gap  $\frac{\Delta_k^{(m_k^*)}}{\sqrt{\lambda^{(m_k^*)}}} - \frac{\Delta_k^{(m)}}{\sqrt{\lambda^{(m)}}}$ , which is similar to bandit's reward gap (reward mean difference)  $\mu^* - \mu_k$ . Therefore, for this suboptimal arm  $k$ , if all its effectiveness gaps are large, that is, the effectiveness of its optimal fidelity is much larger than other suboptimal fidelities, then the contribution of this arm  $k$  to the quantity  $G$  would be small, and vice versa. If for all arms  $k \in \mathcal{K}$ , all of their effectiveness gaps are large, then the  $G$  would be small; otherwise,  $G$  can be large.
For a more sensible example, one can check the simulation set up (deferred to Appendix A) of the empirical comparison of EXPLOR-E-A and EXPLOR-B in Figure 1. In Figure 1a's set up, the effectiveness gaps are large and thus  $G$  is small, hence EXPLOR-A outperforms EXPLOR-B, while in Figure 1b's set up, the effectiveness gaps are small and thus  $G$  is large, then EXPLOR-B outperforms EXPLOR-A.
Practically, one can just run both EXPLORE-A and -B in parallel for each arm  $k$ . If EXPLORE-A starts to converge on a fidelity, i.e., a large proportion of pull times on this arm is at this fidelity, we stop EXPLORE-B and continue EXPLORE-A. Otherwise, if EXPLORE-B commits to a fidelity at first, we stop EXPLORE-A and follow EXPLORE-B to commit this fidelity. With this heuristic approach, we can expect the empirical cost is close to the smaller one of running either EXPLORE-A or EXPLORE-B alone.
Remark A.2 (On the choice of "half" in EXPLORE-B as good fidelity). The selection of "half" is just for the simplicity of presentation, and any constant between  $(0,1)$  would work. If a constant  $C\in (0,1)$  is used in Eq.(6), then the factor 3 of right hand side in the condition of Line 10 of EXPLORE-B becomes  $\frac{1 + C}{1 - C}$ , and there would be an additional  $(C^{-2} + (1 - C)^{-2})$  multiplier appearing in the cost complexity upper bound of EXPLORE-B in Eq.(8). In the  $(C^{-2} + (1 - C)^{-2})$  multiplier, the first term  $C^{-2}$  factor is because a small  $C$  implies that EXPLORE-B may commit to a very inefficient suboptimal fidelity and thus suffer a high cost, while the second term  $(1 - C)^{-2}$  corresponds to that if  $C$  is close to 1, then it becomes very difficult for EXPLORE-B to find a good fidelity to commit, and thus EXPLORE-B has to spend a large cost in exploration.
A.2 Remarks on Regret Minimization
Remark A.3 (Relation to partial monitoring). Recall that, under our regret definition in Eq.(9), the optimal policy is to pull the optimal arm 1 at the lowest fidelity, and any exploration at the highest fidelity results in nonzero regret. This clear separation of exploration and exploitation is similar to the hard case of partial monitoring [2, 23] where the decision maker needs to choose the globally observable actions with nonzero regret cost to do exploration and then, when exploiting, choose locally observable actions. The regret of the hard case of partial monitoring is  $\Theta (T^{2 / 3})$  where  $T$  is the decision round, which is also similar to our  $\tilde{O} (\Lambda^{2 / 3})$  regret bound.
B Simulation Details of Figure 1
In Figure 1, we report the cost complexities of Algorithm 1 with EXPLORE-A and Algorithm 1 with EXPLORE-B (let  $\tilde{\mu}_1^{(M)} = 0.95$  and  $\tilde{\mu}_2^{(M)} = 0.75$ ). We set the confidence parameter  $\delta$  as 0.05, 0.1, 0.15, 0.2, 0.25 respectively in comparing the performance of both procedures. For each simulation, we run 100 trials, plot their cost complexities' mean as markers and their deviation as shaded regions. We present the parameters of MF-MAB instances of Figures 1a and 1b in Tables 1 and 2 respectively. We note that it is more difficult to find the optimal fidelity  $m_k^*$  in the MF-MAB instances for Figure 1b because (1) there are more fidelities choices in this instance than that of Figure 1a; (2) the value of  $\tilde{\Delta}_{k}^{(m)} / \sqrt{\lambda^{(m)}}$  are closer in the second instance than that of Figure 1a.
Table 1: Figure 1a's MF-MAB with  $K = 5$  arms and  $M = 3$  fidelities
Parameters	μ1(m)	μ2(m)	μ3(m)	μ4(m)	μ5(m)	ζ(m)	λ(m)
m = 1	0.70	0.75	0.50	0.50	0.30	0.30	1
m = 2	0.80	0.775	0.60	0.55	0.45	0.15	1.1
m = 3	0.90	0.80	0.70	0.60	0.50	0	1.2
Table 2: Figure 1b's MF-MAB with  $K = 5$  arms and  $M = 5$  fidelities
Parameters	μ1(m)	μ2(m)	μ3(m)	μ4(m)	μ5(m)	ζ(m)	λ(m)
m = 1	0.83	0.82	0.76	0.82	0.70	0.10	1
m = 2	0.84	0.83	0.80	0.80	0.72	0.08	1.1
m = 3	0.85	0.85	0.80	0.82	0.74	0.06	1.2
m = 4	0.85	0.86	0.80	0.80	0.76	0.04	1.3
m = 5	0.90	0.88	0.86	0.84	0.80	0	1.4
In this section, we present an additional numerical simulation to compare our two exploration policies, Explore-A and Explore-B, with a naive baseline that is always selecting the highest fidelity to explore arms. All three policies are employed under the LUCB algorithmic framework (Algorithm 1). This simulation aims to illustrate when our exploration policies Explore-A and Explore-B are better than the naive baseline.
Experiment set up. We consider a MF-MAB model consisting of  $K = 5$  arms and  $M = 5$  fidelities. The reward means are set as  $\mu_k^{(m)} = 0.1(k + 4) + 0.01(m - 5)$ , where, for example,  $\mu_5^{(5)} = 0.9$  is the true reward mean of optimal arm and its low-fidelity reward means are  $\{0.89, 0.88, 0.87, 0.86\}$  accordingly, etc. The error upper bound in different fidelities  $\zeta^{(m)}$  are  $\{0.04, 0.03, 0.02, 0.01, 0\}$ . We set the cost of fidelity  $m \in \{1, 2, 3, 4\}$  as  $\lambda^{(m)} = m$  respectively, and consider four different cases  $I \in \{1, 2, 3, 4\}$  of the cost of highest fidelity  $\lambda^{(5)} = 5I$ . That is, from Case 1 to Case 4, the only difference is that the cost of highest fidelity increases.
We set the confidence parameter  $\delta = 0.1$  and run each experiment for 100 trials. Averaged over these 100 trials of each experiment, we present the empirical cost complexity in the table as follows,
Table 3: Cost complexity under four different cases
Cost Complexity (×104)	Case 1	Case 2	Case 3	Case 4
Explore-A	20.5	20.0	17.4	17.4
Explore-B	36.1	47.9	59.5	69.4
Baseline	21.4	42.0	64.6	86.2
Discussion of additional simulation result. In all four cases, the cost complexity of Explore-A does not change too much. Especially, when the cost of highest fidelity becomes larger, it becomes
easier for Explore-A to find out the optimal fidelity (e.g., the highest fidelity is obviously not the optimal one) and hence enjoy a lower cost complexity. The cost complexity of both Explore-B and Baseline increases over these four cases.
When the cost of highest fidelity is expensive, e.g., Cases 3 and 4, both Explore-A and Explore-B outperform Baseline. This highlights the advantage of utilizing low fidelity to explore arms when the cost of high fidelity is large. When the cost of highest fidelity is cheap, e.g., Case 1, the cost complexity of Explore-A is similar to Baseline, while Explore-B is worse than Baseline. This suggests that when the cost of highest fidelity is similar to that of lower fidelity, the Baseline may be preferred.
In this comparison, Explore-A always outperforms Explore-B. For another scenario where Explore-B is better than Explore-B, please refer to Figure 1b.
D A Third Fidelity Selection Procedure: EXPLORE-C
Besides the EXPLOR - A and -B procedures, here we consider a third naive and conservative idea for fidelity selection that one should start from low risk (cost), gradually increase the risk (cost) as the learning task needs, and stop when finding the optimal arm. To decide when to increase the fidelity for exploring an arm  $k$ , we use the arm's confidence radius  $\beta(N_{k,t}^{(m)}, t, \delta)$  at fidelity  $m$  as a measure of the amount of information left in this fidelity, and when the fidelity  $m$ 's confidence radius is less than the error upper bound  $\zeta^{(m)}$  at this fidelity, we increase the fidelity by 1 for higher accuracy, or formally, the fidelity is selected as follows,
$$
m _ {k, t} \leftarrow \min  \left\{m \left| \beta \left(N _ {k, t} ^ {(m)}, t, \delta\right) \geqslant \zeta^ {(m)} \right. \right\}.
$$
Algorithm 4 EXPLORE-C Procedures
procedure EXPLORE-C(k)
$$
m _ {k, t} \leftarrow \min  \left\{m \left| \beta \left(N _ {k, t} ^ {(m)}, t, \delta\right) \geqslant \zeta^ {(m)} \right. \right\}
$$
Pull  $(k, m_{k,t})$ , observe reward, and update corresponding statistics
Theorem D.1 (Cost complexity upper bounds for Algorithm 1 with EXPLORE-C). Given  $L \geqslant 4KM$ , Algorithm 1 outputs the optimal arm with a probability at least  $1 - \delta$ . The cost complexity of Algorithm 1 with EXPLORE-C are upper bounded as follows,
$$
\mathbb {E} [ \Lambda ] = O \left(H ^ {\ddagger} \log \left(\frac {L \left(H ^ {\ddagger} + Q\right)}{\lambda^ {(1)} \delta}\right) + Q \log \left(\frac {L \left(H ^ {\ddagger} + Q\right)}{\lambda^ {(1)} \delta}\right)\right), \tag {12}
$$
where, letting  $m_{k}^{\ddagger}$  denote the smallest fidelity for arm  $k$  such that  $\Delta_k^{(m)} > 2\zeta^{(m)}$ , or formally,
$$
m _ {k} ^ {\ddagger} := \min  \{m: \Delta_ {k} ^ {(m)} > 2 \zeta^ {(m)} \},
$$
and we denote
$$
H ^ {\ddagger} := \sum_ {k \in \mathcal {K}} \frac {\lambda^ {(m _ {k} ^ {\ddagger})}}{(\Delta_ {k} ^ {(m _ {k} ^ {\ddagger})}) ^ {2}}, \quad Q := \sum_ {k \in \mathcal {K}} \sum_ {m = 1} ^ {m _ {k} ^ {\ddagger} - 1} \frac {\lambda^ {(m)}}{(\zeta^ {(m)}) ^ {2}}.
$$
Remark D.2 (EXPLORE-C vs. EXPLOR-E-A and -B). The EXPLOR-C procedure does not require additional knowledge as the other two. It is a one-size-fits-all option. If with some addition information of a specific scenario, e.g., the exact or approximated reward means of top two arms, one can use the EXPLOR-A or B.
E Proofs for Best Arm Identification with Fixed Confidence
E.1 Proof of Theorem 3.1
Lemma E.1 (Kaufmann et al. [21], Lemma 1). Let  $\nu$  and  $\nu'$  be two bandit models with  $K$  arms such that for all  $k$ , the distributions  $\nu_{k}$  and  $\nu_{k}^{\prime}$  are mutually absolutely continuous. For any almost-surely
finite stopping time  $\sigma$  with respect to the filtration  $\{\mathcal{F}_t\}_{t\in \mathbb{N}}$  where  $\mathcal{F}_t = \sigma (I_1,X_1,\ldots ,I_t,X_t)$
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {\nu} \left[ N _ {k} (\sigma) \right] \operatorname {K L} \left(\nu_ {k}, \nu_ {k} ^ {\prime}\right) \geqslant \sup  _ {\mathcal {E} \in \mathcal {F} _ {\sigma}} \operatorname {k l} \left(\mathbb {P} _ {\nu} (\mathcal {E}), \mathbb {P} _ {\nu^ {\prime}} (\mathcal {E})\right),
$$
where  $\operatorname {kl}(x,y)$  is the binary relative entropy.
In MF-MAB model, regarding each arm-fidelity  $(k,m)$ -pair as an individual arm, we can extend Lemma E.1 to multi-fidelity case as follows,
$$
\sum_ {k = 1} ^ {K} \sum_ {m = 1} ^ {M} \mathbb {E} _ {\nu} \left[ N _ {k} ^ {(m)} (\sigma) \right] \operatorname {K L} \left(\nu_ {k} ^ {(m)}, \nu_ {k} ^ {\prime (m)}\right) \geqslant \sup  _ {\mathcal {E} \in \mathcal {F} _ {\sigma}} \operatorname {k l} \left(\mathbb {P} _ {\nu} (\mathcal {E}), \mathbb {P} _ {\nu^ {\prime}} (\mathcal {E})\right). \tag {13}
$$
Next, we construct instances  $\nu$  and  $\nu'$ . We set the reward distributions  $\nu = (\nu_k^{(m)})_{(k,m) \in \mathcal{K} \times \mathcal{M}}$  as Bernoulli and the reward means fulfill  $\mu_1^{(M)} > \mu_2^{(M)} \geqslant \mu_3^{(M)} \geqslant \ldots \geqslant \mu_K^{(M)}$ , where  $\mu_k^{(m)} = \mathbb{E}_{X \sim \nu_k^{(m)}}[X]$ . We let  $\nu'_k^{(m)}$  be the same to  $\nu_k^{(m)}$  for all  $k$  and  $m$ , except for that an arm  $\ell \neq 1$ . We set arm  $\ell$ 's reward means on fidelities  $m \in \mathcal{M}_k$  to be  $\nu'_\ell^{(m)} = \nu_1^{(M)} - \zeta^{(m)} + \epsilon$ . So, in instance  $\nu'$ , the optimal arm is  $\ell$  and its true reward mean  $\mu'_\ell^{(M)}$  is slightly greater than  $\mu_1^{(M)}$ . This implies for the event  $\mathcal{E} = \{\text{output arm } 1\}$  and any algorithm  $\pi$  that can find the optimal arm with a confidence  $1 - \delta$ ,  $\mathbb{P}_{\nu,\pi}(\mathcal{E}) \geqslant 1 - \delta$  and  $\mathbb{P}_{\nu',\pi}(\mathcal{E}) \leqslant \delta$ . Then, from Eq.(13), we have
$$
\begin{array}{l} \sum_ {m \in \mathcal {M} _ {k}} \mathbb {E} _ {\nu} \left[ N _ {\ell} ^ {(m)} (\sigma) \right] \operatorname {K L} \left(\nu_ {\ell} ^ {(m)}, \nu_ {\ell} ^ {\prime (m)}\right) \geqslant \sup  _ {\mathcal {E} \in \mathcal {F} _ {\sigma}} \operatorname {k l} \left(\mathbb {P} _ {\nu , \pi} (\mathcal {E}), \mathbb {P} _ {\nu^ {\prime}, \pi} (\mathcal {E})\right) \\ \geqslant \operatorname {k l} (1 - \delta , \delta) \\ \geqslant \log \frac {1}{2 . 4 \delta}. \\ \end{array}
$$
We rewrite the above inequality as follows,
$$
\sum_ {m \in \mathcal {M} _ {k}} \lambda^ {(m)} \mathbb {E} _ {\nu} [ N _ {\ell} ^ {(m)} (\sigma) ] \cdot \frac {\mathrm {K L} (\nu_ {\ell} ^ {(m)} , \nu_ {\ell} ^ {\prime (m)})}{\lambda^ {(m)}} \geqslant \log \frac {1}{2 . 4 \delta}.
$$
Therefore, for the arm  $\ell$ , our aim is to minimize its cost complexity with a constraint as follows,
$$
\min  _ {\mathbb {E} \left[ N _ {\ell} ^ {(m)} \right], \forall m} \sum_ {m = 1} ^ {M} \lambda^ {(m)} \mathbb {E} _ {\nu} \left[ N _ {\ell} ^ {(m)} (\sigma) \right]
$$
$$
\text {s u c h} \sum_ {m \in \mathcal {M} _ {k}} \lambda^ {(m)} \mathbb {E} _ {\nu} \left[ N _ {\ell} ^ {(m)} (\sigma) \right] \cdot \frac {\mathrm {K L} \left(\nu_ {\ell} ^ {(m)} , \nu_ {\ell} ^ {\prime (m)}\right)}{\lambda^ {(m)}} \geqslant \log \frac {1}{2 . 4 \delta}.
$$
Note that the above is a linear programming (LP) and its optimum is reached at one of its polyhedron constraint's vertex—only one  $\mathbb{E}[N_{\ell}^{(m)}]$  is positive and all others are equal to zero.
$$
\begin{array}{l} \min  _ {\mathbb {E} \left[ N _ {\ell} ^ {(m)} \right], \forall m} \sum_ {m = 1} ^ {M} \lambda^ {(m)} \mathbb {E} _ {\nu} \left[ N _ {\ell} ^ {(m)} (\sigma) \right] \overset {(a)} {\geqslant} \min  _ {m \in \mathcal {M} _ {k}} \frac {\lambda^ {(m)}}{\operatorname {K L} \left(\nu_ {\ell} ^ {(m)} , \nu_ {\ell} ^ {\prime (m)}\right)} \log \frac {1}{2 . 4 \delta} \\ = \min  _ {m \in \mathcal {M} _ {k}} \frac {\lambda^ {(m)}}{\operatorname {K L} \left(\nu_ {\ell} ^ {(m)} , \nu_ {1} ^ {(M)} - \zeta^ {(m)} + \epsilon\right)} \log \frac {1}{2 . 4 \delta} \\ \stackrel {(b)} {\geqslant} \min  _ {m \in \mathcal {M} _ {k}} \frac {\lambda^ {(m)}}{\left(1 + \varepsilon\right) \operatorname {K L} \left(\nu_ {\ell} ^ {(m)} , \nu_ {1} ^ {(M)} - \zeta^ {(m)}\right)} \log \frac {1}{2 . 4 \delta} \\ \end{array}
$$
where the inequality (a) is due to the property of LP we mentioned above, and the inequality (b) is because of the continuity of KL-divergence.
To bound the optimal arm 1's cost complexity, we use the same  $\nu$  as above and construct another instance  $\nu''$ . The instance  $\nu''$ s reward means are the same to  $\nu$  except for arm 1 whose reward means
for fidelity  $m \in \mathcal{M}_1$  are set as  $\mu_{1}^{\prime \prime (m)} = \mu_2^{(m)} + \zeta^{(m)} - \epsilon$ . Then, with similar procedure as the above, we obtain
$$
\min _ {\mathbb {E} \left[ N _ {1} ^ {(m)} \right], \forall m} \sum_ {m = 1} ^ {M} \lambda^ {(m)} \mathbb {E} _ {\nu} \left[ N _ {1} ^ {(m)} (\sigma) \right] \geqslant \min _ {m \in \mathcal {M} _ {1}} \frac {\lambda^ {(m)}}{\left(1 + \varepsilon\right) \operatorname {K L} \left(\nu_ {1} ^ {(m)} , \nu_ {2} ^ {(M)} + \zeta^ {(m)}\right)} \log \frac {1}{2 . 4 \delta}.
$$
Summing up the above costs leads to the lower bound as follows, and letting the  $\epsilon$  goes to zeros concludes the proof.
$$
\mathbb {E} [ \Lambda ] \geqslant
$$
$$
\left(\min  _ {m \in \mathcal {M} _ {1}} \frac {\lambda^ {(m)}}{(1 + \varepsilon) \operatorname {K L} \left(\nu_ {1} ^ {(m)} , \nu_ {2} ^ {(M)} + \zeta^ {(m)}\right)} + \sum_ {k \neq 1} \min  _ {m \in \mathcal {M} _ {k}} \frac {\lambda^ {(m)}}{(1 + \varepsilon) \operatorname {K L} \left(\nu_ {k} ^ {(m)} , \nu_ {1} ^ {(M)} - \zeta^ {(m)}\right)}\right) \log \frac {1}{2 . 4 \delta}.
$$
E.2 Proof of Theorem 3.4
Notation. Denote the threshold  $c = \frac{\mu_1^{(M)} + \max_k(\mu_k^{(m_k^*)} + \zeta^{(m_k^*)})}{2}$  as the average between the optimal arm's reward mean and the highest reward mean upper bound of other arms  $k$  at fidelity  $m_{k}^{*}$ . Denote  $\mathcal{A}_t\coloneqq \{k\in \mathcal{K}:\mathrm{LCB}_{k,t} > c\}$  and  $\mathcal{B}_t\coloneqq \{k\in \mathcal{K}:\mathrm{UCB}_{k,t} < c\}$  as the above and below sets which respectively contain arms whose rewards are clearly higher or lower than the threshold with high probability, and let  $\mathcal{C}_t\coloneqq \mathcal{K}\setminus (\mathcal{A}_t\cup \mathcal{B}_t)$  as the complement of both sets' union. Then, we define two events as follows
$$
\operatorname {T E R M} _ {t} := \left\{\operatorname {L C B} _ {\ell_ {t}, t} > \operatorname {U C B} _ {u _ {t}, t} \right\},
$$
$$
\operatorname {C R O S} _ {t} := \{\exists k \neq 1: k \in \mathcal {A} _ {t} \} \cup \{1 \in \mathcal {B} _ {t} \}.
$$
The  $\mathrm{TERM}_t$  event corresponds to the complement of the main while loop condition in the LUCB algorithm. When the  $\mathrm{TERM}_t$  event happens, the LUCB algorithm terminates. The  $\mathrm{CROS}_t$  event means there exists a suboptimal arm whose  $\mathrm{LCB}_{k,t}$  is greater than  $c$  or that the optimal arm  $1$ 's  $\mathrm{UCB}_{1,t}$  is less than  $c$ , both of which means that at least one arm's reward mean confidence interval incorrectly crosses the threshold  $c$ .
Step 1. Prove  $\neg \mathsf{TERM}_t \cap \neg \mathsf{CROS}_t \Longrightarrow (\ell_t \in \mathcal{C}_t) \cup (u_t \in \mathcal{C}_t)$ . We show this statement by contradiction case by case. That is, the negation of  $(\ell_t \in \mathcal{C}_t) \cup (u_t \in \mathcal{C}_t)$  cannot happen when  $\neg \mathsf{TERM}_t \cap \neg \mathsf{CROS}_t$ .
Case 1:  $(\ell_t \in \mathcal{A}_t) \cap (u_t \in \mathcal{A}_t) \cap \neg \mathrm{TERM}_t$
$$
\Longrightarrow (\ell_ {t} \in \mathcal {A} _ {t}) \cap (u _ {t} \in \mathcal {A} _ {t}) \Longrightarrow | \mathcal {A} _ {t} | \geqslant 2 \Longrightarrow \exists k \neq 1: k \in \mathcal {A} _ {t} \Longrightarrow \operatorname {C R O S} _ {t},
$$
Case 2:  $(\ell_t\in \mathcal{B}_t)\cap (u_t\in \mathcal{A}_t)\cap \neg \mathrm{TERM}_t$
$$
\Longrightarrow \mathrm {U C B} _ {\ell_ {t}, t} <   c <   \mathrm {L C B} _ {u _ {t}, t} <   \mathrm {U C B} _ {u _ {t}, t} \Longrightarrow \emptyset (\text {c o n t r a d i c t s t h e s e l e c t i o n o f} \ell_ {t} \text {a n d} u _ {t}),
$$
Case 3:  $(\ell_t \in \mathcal{A}_t) \cap (u_t \in \mathcal{B}_t) \cap \neg \mathrm{TERM}_t$
$$
\Longrightarrow \left\{\mathrm {L C B} _ {\ell_ {t}, t} > c > \mathrm {U C B} _ {u _ {t}, t} \right\} \cap \neg \mathrm {T E R M} _ {t} \Longrightarrow \emptyset ,
$$
Case 4:  $(\ell_t\in \mathcal{B}_t)\cap (u_t\in \mathcal{B}_t)\cap \neg \mathrm{TERM}_t$
$$
\Longrightarrow (\ell_ {t} \in \mathcal {B} _ {t}) \cap (u _ {t} \in \mathcal {B} _ {t}) \Longrightarrow | \mathcal {B} _ {t} | = K \Longrightarrow 1 \in \mathcal {B} _ {t} \Longrightarrow \operatorname {C R O S} _ {t}.
$$
Step 2. Prove  $\mathbb{P}(\mathsf{CROS}_t) \leqslant \frac{KM\delta}{Lt^3}$ . For any suboptimal arm  $k \neq 1$ , we bound the probability that the arm  $k$  is in  $\mathcal{A}_t$  as follows,
$$
\begin{array}{l} \mathbb {P} (k \in \mathcal {A} _ {t}) = \mathbb {P} (\operatorname {L C B} _ {k, t} > c) = \mathbb {P} \left(\max  _ {m \in \mathcal {M}} \operatorname {L C B} _ {k, t} ^ {(m)} > c\right) \leqslant \sum_ {m \in \mathcal {M}} \mathbb {P} \left(\operatorname {L C B} _ {k, t} ^ {(m)} > c\right) \\ = \sum_ {m \in \mathcal {M}} \mathbb {P} \left(\hat {\mu} _ {k, t} ^ {(m)} - \zeta^ {(m)} - \beta \left(N _ {k, t} ^ {(m)}, t\right) > c\right) \\ = \sum_ {m \in \mathcal {M}} \mathbb {P} \left(\hat {\mu} _ {k, t} ^ {(m)} - \mu_ {k} ^ {(m)} + \left(\mu_ {k} ^ {(m)} - \zeta^ {(m)} - c\right) > \beta \left(N _ {k, t} ^ {(m)}, t\right)\right) \\ \stackrel {(a)} {\leqslant} \sum_ {m \in \mathcal {M}} \mathbb {P} \left(\hat {\mu} _ {k, t} ^ {(m)} - \mu_ {k} ^ {(m)} > \beta \left(N _ {k, t} ^ {(m)}, t\right)\right) \leqslant \sum_ {m \in \mathcal {M}} \sum_ {n = 1} ^ {t} \mathbb {P} \left(\hat {\mu} _ {k, t} ^ {(m)} - \mu_ {k} ^ {(m)} > \beta (n, t)\right) \\ \leqslant \sum_ {m \in \mathcal {M}} \sum_ {n = 1} ^ {t} \exp (- n (\beta (n, t)) ^ {2}) = \sum_ {m \in \mathcal {M}} \sum_ {n = 1} ^ {t} \frac {\delta}{L t ^ {4}} \\ \leqslant \frac {M \delta}{L t ^ {3}}, \\ \end{array}
$$
where the inequality (a) is due to that  $\mu_k^{(m)} - \zeta^{(m)} \leqslant \mu_k^{(M)} < c$ . With similar derivation, we have  $\mathbb{P}(1 \in \mathcal{B}_t) \leqslant \frac{M\delta}{Lt^3}$ . Noticing that  $\mathbb{P}(\mathsf{CROS}_t) \leqslant \sum_{k \neq 1} \mathbb{P}(k \in \mathcal{A}_t) + \mathbb{P}(1 \in \mathcal{B}_t)$ , we have  $\mathbb{P}(\mathsf{CROS}_t) \leqslant \frac{KM\delta}{Lt^3}$ .
Step 3. Prove  $\mathbb{P}\left(\exists k\in \mathcal{K}:(N_{k,t}^{(m_k^*)} > 16N_{k,t}^*)\cap (k\in \mathrm{Mid}_t)\right)\leqslant \frac{16\delta\sum_{k\in\mathcal{K}}\Delta_k^{-2}}{Lt^4}$ , where  $N_{k,t}^{*}\coloneqq \frac{\log(Lt^4 / \delta)}{(\Delta_k^{(\tilde{m}_k^*)})^2}$ . For any fixed suboptimal arm  $k\neq 1$  (with  $\mu_k^{(M)} < c$ ), we have