SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} P (| \tilde {\mu} _ {i} ^ {m} (t) - \mu_ {i} | > \sqrt {\frac {C _ {1} \ln T}{n _ {m , i} (t)}} | A _ {\epsilon , \delta}) \\ \leq 2 \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E \left[ \frac {1}{\exp \left\{\lambda (n) \sqrt {\frac {C _ {1} \ln T}{n _ {m , i} (t)}} \right\}} \cdot \exp \left\{\frac {\lambda^ {2} (n)}{2} \tilde {\sigma} ^ {2} \right\} \right] \tag {49} \\ \end{array}
$$
by a same argument from (43) to (47) replacing  $\frac{C_2 \ln T}{n_{m,i}(t)}$  with  $\sqrt{\frac{C_1 \ln T}{n_{m,i}(t)}}$ . When choosing  $\lambda(n) = \frac{\sqrt{\frac{C_1 \log T}{n_{m,i}(t)}}}{\hat{\sigma}^2}$  that meets the condition  $\lambda < \frac{1}{\alpha}$  under the assumption  $\frac{\sqrt{\frac{C_1 \log T}{n_{m,i}(t)}}}{\hat{\sigma}^2} < \frac{1}{\alpha}$  and noting that
$\frac{C_1}{2C\sigma^2} \geq 4$ , we obtain
$$
\begin{array}{l} (4 9) \leq 2 \frac {1}{P (A _ {\epsilon , \delta})} E [ \exp \{- \frac {C _ {1} \log T}{\tilde {\sigma} ^ {2} n _ {m , i} (t)} \} ] \\ = 2 \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E [ \exp \left\{- \frac {C _ {1} \log T}{n _ {m , i} (t)} \frac {1}{\frac {2 C \sigma^ {2}}{n _ {m , i} (t)}} \right\} ] \\ \leq 2 \frac {1}{P (A _ {\epsilon , \delta})} \exp \{- 4 \log T \} = \frac {2}{P (A _ {\epsilon , \delta}) T ^ {4}}. \\ \end{array}
$$
To conclude, by (41) and (42), we have
$$
P \left(\left| \tilde {\mu} _ {i} ^ {m} (t) - \mu_ {i} \right| > R a d \mid A _ {\epsilon , \delta}\right) \leq \frac {2}{P \left(A _ {\epsilon , \delta}\right) T ^ {4}}
$$
Proposition 13. Assume the parameter  $\delta$  satisfies that  $0 < \delta < c = f(\epsilon, M, T)$ . An arm  $k$  is said to be sub-optimal if  $k \neq i^{*}$  where  $i^{*}$  is the unique optimal arm in terms of the global reward, i.e.  $i^{*} = \arg \max \frac{1}{M} \sum_{j=1}^{M} \mu_{i}^{j}$ . Then in setting  $S_{1}, S_{2}$  and  $S_{3}$ , when the game ends, for every client  $m$ ,  $0 < \epsilon < 1$  and  $T > L$ , the expected numbers of pulling sub-optimal arm  $k$  after the burn-in period satisfy with  $P(A_{\epsilon, \delta}) = 1 - 7\epsilon$
$$
\begin{array}{l} E [ n _ {m, k} (T) | A _ {\epsilon , \delta} ] \\ \leq \max  \left(\left[ \frac {1 6 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], \left[ \frac {4 C _ {2} \log T}{\Delta_ {i}} \right], 2 (K ^ {2} + M K + M)\right) + \frac {4}{P (A _ {\epsilon , \delta}) T ^ {3}} + K ^ {2} + (2 M - 1) K \\ \leq O (\log T). \\ \end{array}
$$
Proof. Recall that  $\text{Rad} = \sqrt{\frac{C_1 \ln T}{n_{m,i}(t)}} + \frac{C_2 \ln T}{n_{m,i}(t)}$ . We again have  $a_t^m = i$  holds when any of the following conditions is met: Case 1:  $n_{m,i}(t) \leq \mathcal{N}_{m,i}(t) - K$ , Case 2:  $\tilde{\mu}_{m,i} - \mu_i > \sqrt{\frac{C_1 \ln T}{n_{m,i}(t)}} + \frac{C_2 \ln T}{n_{m,i}(t)}$ ,
Case 3:  $-\tilde{\mu}_{m,i^{*}} + \mu_{i^{*}} > \sqrt{\frac{C_{1} \ln T}{n_{m,i^{*}}(t)}} + \frac{C_{2} \ln T}{n_{m,i^{*}}(t)}$ , and Case 4:  $\mu_{i^{*}} - \mu_{i} < 2(\sqrt{\frac{C_{1} \ln T}{n_{m,i}(t)}} + \frac{C_{2} \ln T}{n_{m,i}(t)})$ .
By (30), the expected value of  $n_{m,i}(t)$  conditional on  $A_{\epsilon, \delta}$  reads as
$$
\begin{array}{l} E [ n _ {m, i} (T) | A _ {\epsilon , \delta} ] \\ = l + \sum_ {t = L + 1} ^ {T} P (C a s e 2, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) + \sum_ {t = L + 1} ^ {T} P (C a s e 3, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) \\ + \sum_ {t = L + 1} ^ {T} P \left(\text {C a s e 1}, a _ {t} ^ {m} = i, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) + \sum_ {t = L + 1} ^ {T} P \left(\text {C a s e 4}, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) \tag {50} \\ \end{array}
$$
where  $l$  is specified as  $l = \max \left\{\left[\frac{4C_1\log T}{\Delta_i^2}\right], 2(K^2 + MK + M)\right\}$  with  $\Delta_{i} = \mu_{i^{*}} - \mu_{i}$ .
For the last term in the above upper bound, we have
$$
\sum_ {t = L + 1} ^ {T} P \left(\text {C a s e} 4: \mu_ {i} + 2 \left(\sqrt {\frac {C _ {1} \ln T}{n _ {m , i} (t)}} + \frac {C _ {2} \ln T}{n _ {m , i} (t)}\right) > \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l\right) = 0 \tag {51}
$$
since the choice of  $l$  satisfies  $l \geq \max \left(\left[\frac{16C_1\log T}{\Delta_i^2}\right],\left[\frac{4C_2\log T}{\Delta_i}\right],2(K^2 + MK)\right)$ .
$$
\begin{array}{l} \sum_ {t = L + 1} ^ {T} P (C a s e 2, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) + \sum_ {t = 1} ^ {T} P (C a s e 3, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) \\ \leq \sum_ {t = L + 1} ^ {T} P \left(\tilde {\mu} _ {m, i} - \mu_ {i} > \sqrt {\frac {C _ {1} \ln T}{n _ {m , i} (t)}} + \frac {C _ {2} \ln T}{n _ {m , i} (t)} \mid A _ {\epsilon , \delta}\right) + \\ \sum_ {t = 1} ^ {T} P \left(- \tilde {\mu} _ {m, i ^ {*}} + \mu_ {i ^ {*}} > \sqrt {\frac {C _ {1} \ln T}{n _ {m , i ^ {*}} (t)}} + \frac {C _ {2} \ln T}{n _ {m , i ^ {*}} (t)} \mid A _ {\epsilon , \delta}\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{P \left(A _ {\epsilon , \delta}\right) T ^ {4}}\right) + \sum_ {t = 1} ^ {T} \left(\frac {1}{P \left(A _ {\epsilon , \delta}\right) T ^ {4}}\right) \leq \frac {2}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}} \tag {52} \\ \end{array}
$$
where the first inequality holds by the property of the probability measure when removing the event  $n_{m,i}(t - 1) \geq l$  and the second inequality holds by Proposition 12, which holds by the assumption that  $\delta < c$ .
$$
n _ {m, i} (t) > N _ {m, i} (t) - K (K + 2 M)
$$
At time step  $t$ , if Case 1 holds for client  $m$ , then  $n_{m,i}(t + 1)$  is increasing by 1 on the basis of  $n_{m,i}(t)$ . What follows characterizes the change of  $N_{m,i}(t + 1)$ . Client  $m$  satisfying  $n_{m,i}(t) \leq N_{m,i}(t) - K$  will not change the value of  $N_{m,i}(t + 1)$  by the definition  $N_{m,i}(t + 1) = \max \{n_{m,i}(t + 1), N_{j,i}(t), j \in \mathcal{N}_m(t)\}$ . Moreover, for client  $j \in \mathcal{N}_m(t)$  with  $n_{j,i}(t) < N_{j,i}(t) - K$ , i.e.  $N_{j,i}(t + 1)$  will not be affected by  $n_{j,i}(t + 1) \leq n_{j,i}(t) + 1$ . Thus, the value of  $N_{m,i}(t + 1) = \max \{n_{m,i}(t + 1), N_{j,i}(t), j \in \mathcal{N}_m(t)\}$  is independent of such clients. We observe that for client  $j \in \mathcal{N}_m(t)$  with  $n_{j,i}(t) > N_{j,i}(t) - K$ , the value  $N_{j,i}(t)$  will be the same if the client does not sample arm  $i$ , which leads to a decrease of 1 in the difference  $-n_{m,i}(t) + N_{m,i}(t)$ . Otherwise, if such a client samples arm  $i$  which brings an increment of 1 to  $N_{m,i}(t)$ , the difference between  $n_{m,i}(t)$  and  $N_{m,i}(t)$  will remain the same. However, the latter has just been discussed and must be the cases as in Case 2 and Case 3, the total length of which has already been upper bounded by  $\frac{2}{P(A_{\epsilon,\delta})T^3}$  as shown in (52).
Therefore, the gap is at most  $K(K + 2M) - K + \frac{2}{P(A_{\epsilon,\delta})T^3}$ , i.e.
$$
\sum_ {t = 1} ^ {T} P \left(\text {C a s e 1}, a _ {t} ^ {m} = i, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) \leq K (K + 2 M) - K + \frac {2}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}}. \tag {53}
$$
$$
\begin{array}{l} E [ n _ {m, i} (T) | A _ {\epsilon , \delta} ] \\ \leq l + \frac {2}{P (A _ {\epsilon , \delta}) T ^ {3}} + K (K + 2 M) - K + \frac {2}{P (A _ {\epsilon , \delta}) T ^ {3}} + 0 \\ = l + \frac {2 \pi^ {2}}{3} + K ^ {2} + (2 M - 1) K \\ = \max  \left(\left[ \frac {1 6 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], \left[ \frac {4 C _ {2} \log T}{\Delta_ {i}} \right], 2 \left(K ^ {2} + M K + M\right)\right) + \frac {4}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}} + K ^ {2} + (2 M - 1) K \\ \end{array}
$$
where the inequality results from (50), (51), (52) and (53).
F.2 Proof of Theorems
Proof. Recall that we define events
$$
A _ {1} = \{\forall t \geq L, | P _ {t} - c E | \leq \delta \},
$$
$$
A _ {2} = \{\forall t \geq L, \forall j, m, t + 1 - \min  _ {j} t _ {m, j} \leq t _ {0} \leq c _ {0} \min  _ {l} n _ {l, i} (t + 1) \},
$$
$$
A _ {3} = \{\forall t \geq L, G _ {t} \text {i s c o n n e c t e d} \}
$$
where  $A_{1}, A_{2}, A_{3}$  belong to the  $\sigma$ -algebra in the probability space since the time horizon is countable, i.e. for the probability space  $(\Omega, \Sigma, P)$ ,  $A_{1}, A_{2}, A_{3} \in \Sigma$ .
Meanwhile, we obtain
$$
\begin{array}{l} P \left(A _ {1}\right) = P \left(\left\{\forall t \geq L, \left| P _ {t} - c E \right| \leq \delta \right\}\right) \\ \geq P \left(\cap_ {i} \left\{\forall t \geq L _ {s _ {i}}, \left| P _ {t} - c E \right| \leq \delta \right\}\right) \\ \geq 1 - \sum_ {i} (1 - P (\cap_ {i} \{\forall t \geq L _ {s _ {i}}, | P _ {t} - c E | \leq \delta \})) \\ \geq 1 - \sum_ {i} (1 - (1 - \epsilon)) \\ = 1 - 3 \epsilon \tag {54} \\ \end{array}
$$
where the first inequality includes all settings and  $L \geq L_{s_i}$ , the second inequality results from the Bonferroni's inequality and the third inequality holds by Proposition 1 and Proposition 3.
At the same time, note that
$$
\begin{array}{l} P (A _ {2}) = P (\{\forall t \geq L, \forall j, m, t + 1 - \min  _ {j} t _ {m, j} \leq t _ {0} \leq c _ {0} \min  _ {l} n _ {l, i} (t + 1) \}) \\ \geq P \left(\cap_ {i} \{\forall t \geq L _ {s _ {i}}, \forall j, m, t + 1 - \min  _ {j} t _ {m, j} \leq t _ {0} \leq c _ {0} \min  _ {l} n _ {l, i} (t + 1) \}\right) \\ \geq 1 - \sum_ {i} (1 - P (\{\forall t \geq L _ {s _ {i}}, \forall j, m, t + 1 - \min  _ {j} t _ {m, j} \leq t _ {0} \leq c _ {0} \min  _ {l} n _ {l, i} (t + 1) \})) \\ \geq 1 - \sum_ {i} (1 - (1 - \epsilon)) = 1 - 3 \epsilon \tag {55} \\ \end{array}
$$
where the first inequality is by the definition of  $L$ , the second inequality again uses the Bonferroni's inequality, and the third inequality results from Proposition 6.
Moreover, we observer that
$$
\begin{array}{l} P \left(A _ {3}\right) = P \left(\left\{\forall t \geq L, G _ {t} \text {i s c o n n e c t e d} \right\}\right) \\ \geq P \left(\cap_ {i} \{\forall t \geq L _ {s _ {i}}, G _ {t} \text {i s c o n n e c t e d} \}\right) \\ \geq 1 - \sum_ {i} (1 - P (\{\forall t \geq L _ {s _ {i}}, G _ {t} \text {i s c o n n e c t e d} \})) \\ \geq 1 - (1 - (1 - \epsilon)) - 0 = 1 - \epsilon \tag {56} \\ \end{array}
$$
where the first inequality uses the definition of  $L$ , the second inequality is by the Bonferroni's inequality and the third inequality holds by Proposition 5 and the definition of  $s_2, s_3$  where all graphs are guaranteed to be connected.
$$
\begin{array}{l} P \left(A _ {\epsilon , \delta}\right) = P \left(A _ {1} \cap A _ {2} \cap A _ {3}\right) \\ = 1 - P \left(A _ {1} ^ {c} \cup A _ {2} ^ {c} \cup A _ {3} ^ {c}\right) \\ \geq 1 - \left(P \left(A _ {1} ^ {c}\right) + P \left(A _ {2} ^ {c}\right) + P \left(A _ {3} ^ {c}\right)\right) \\ \geq 1 - (3 \epsilon + 3 \epsilon + \epsilon) = 1 - 7 \epsilon \\ \end{array}
$$
where the first inequality utilizes the Bonferroni's inequality, the second inequality results from (54), (55), and (56).
This concludes the proof and shows the validity of the statement.
Theorem 2. Let  $f$  be a function specific to a setting and detailed later. For every  $0 < \epsilon < 1$  and  $0 < \delta < f(\epsilon, M, T)$ , in setting  $s_1$  with  $c \geq \frac{1}{2} + \frac{1}{2}\sqrt{1 - \left(\frac{\epsilon}{MT}\right)^{\frac{2}{M-1}}}$ ,  $s_2$  and  $s_3$ , with the time horizon  $T$  satisfying  $T \geq L$ , the regret of Algorithm 2 with  $F(m, i, t) = \sqrt{\frac{C_1\ln t}{n_{m,i}(t)}}$  satisfies that
$$
E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] \leq L + \sum_ {i \neq i ^ {*}} (\max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K + M) \right\} + \frac {2 \pi^ {2}}{3 P (A _ {\epsilon , \delta})} + K ^ {2} + (2 M - 1) K)
$$
where the length of the burn-in period is explicitly
$$
\begin{array}{l} L = \max  \left\{\underbrace {\frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {4 K \log_ {2} T}{c _ {0}}} _ {L _ {s _ {1}}}, \underbrace {\frac {\ln \frac {\delta}{1 0}}{\ln p ^ {*}} + 2 5 \frac {1 + \lambda}{1 - \lambda} \frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {4 K \log_ {2} T}{c _ {0}}} _ {L _ {s _ {2}}}, \right. \\ \left. \underbrace {\frac {\ln \frac {\delta}{1 0}}{\ln p ^ {*}} + 2 5 \frac {1 + \lambda}{1 - \lambda} \frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {\frac {K \ln (\frac {M T}{\epsilon})}{\ln (\frac {1}{1 - \frac {2 \log M}{M - 1}})}}{c _ {0}}} _ {L _ {s _ {3}}} \right\} \\ \end{array}
$$
Proof. The optimal arm is denoted as  $i^{*}$  satisfying
$$
i ^ {*} = \arg \max  _ {i} \sum_ {m = 1} ^ {M} \mu_ {i} ^ {m}.
$$
For the proposed regret, we have that for any constant  $L$ ,
$$
\begin{array}{l} R _ {T} = \frac {1}{M} \left(\max  _ {i} \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \mu_ {i} ^ {m} - \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m}\right) \\ = \sum_ {t = 1} ^ {T} \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {i ^ {*}} ^ {m} - \sum_ {t = 1} ^ {T} \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m} \\ \leq \sum_ {t = 1} ^ {L} \left| \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {i ^ {*}} ^ {m} - \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m} \right| + \sum_ {t = L + 1} ^ {T} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {i ^ {*}} ^ {m} - \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m}\right) \\ \leq L + \sum_ {t = L + 1} ^ {T} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {i ^ {*}} ^ {m} - \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m}\right) \\ = L + \sum_ {t = L + 1} ^ {T} \left(\mu_ {i ^ {*}} - \frac {1}{M} \sum_ {m = 1} ^ {M} \mu_ {a _ {t} ^ {m}} ^ {m}\right) \\ = L + \left((T - L) \cdot \mu_ {i ^ {*}} - \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {i = 1} ^ {K} n _ {m, i} (T) \mu_ {i} ^ {m}\right) \\ \end{array}
$$
where the first inequality is by taking the absolute value and the second inequality results from the assumption that  $0 < \mu_i^j < 1$  for any arm  $i$  and client  $j$ .
Note that  $\sum_{i=1}^{K}\sum_{m=1}^{M}n_{m,i}(T) = M(T - L)$  where by definition  $n_{m,i}(T)$  is the number of pulls of arm  $i$  at client  $m$  from time step  $L + 1$  to time step  $T$ , which yields that
$$
\begin{array}{l} R _ {T} \leq L + \sum_ {i = 1} ^ {K} \frac {1}{M} \sum_ {m = 1} ^ {M} n _ {m, i} (T) \mu_ {i ^ {*}} ^ {m} - \sum_ {i = 1} ^ {K} \frac {1}{M} \sum_ {m = 1} ^ {M} n _ {m, i} (T) \mu_ {i} ^ {m} \\ = L + \sum_ {i = 1} ^ {K} \frac {1}{M} \sum_ {m = 1} ^ {M} n _ {m, i} (T) \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \\ \leq L + \frac{1}{M}\sum_{i = 1}^{K}\sum_{m:\mu_{i^{*}}^{m} - \mu_{i}^{m} > 0}n_{m,i}(T)(\mu_{i^{*}}^{m} - \mu_{i}^{m}) \\ = L + \frac {1}{M} \sum_ {i \neq i *} \sum_ {m: \mu_ {i *} ^ {m} - \mu_ {i} ^ {m} > 0} n _ {m, i} (T) (\mu_ {i *} ^ {m} - \mu_ {i} ^ {m}). \\ \end{array}
$$
where the second inequality uses the fact that  $\sum_{m:\mu_{i^*}^m -\mu_i^m\leq 0}n_{m,i}(T)(\mu_{i^*}^m -\mu_i^m)\leq 0$  holds for any arm  $i$  and the last equality is true since  $n_{m,i}(T)(\mu_{i^*}^m -\mu_i^m) = 0$  for  $i = i^{*}$  and any  $m$ .
Meanwhile, by the choices of  $\delta$  such that  $\delta < c = f(\epsilon, M, T)$ , we apply Proposition 10 which leads to for any client  $m$  and arm  $i \neq i^*$ ,
$$
E [ n _ {m, i} (T) | A _ {\epsilon , \delta} ] \leq \max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K + M) \right\} + \frac {2 \pi^ {2}}{3} + K ^ {2} + (2 M - 1) K. (5 7)
$$
As a result, the upper bound on  $R_{T}$  can be derived as by taking the conditional expectation over  $R_{T}$  on  $A_{\epsilon,\delta}$
$$
\begin{array}{l} E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] \\ \leq L + \frac {1}{M} \sum_ {i \neq i ^ {*}} \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} E \left[ n _ {m, i} (T) \mid A _ {\epsilon , \delta} \right] \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \tag {58} \\ \leq L + \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{M} \sum_ {i \neq i ^ {*}} \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} (\max \{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K) \} + \frac {2 \pi^ {2}}{3} + K ^ {2} + (2 M - 1) K) (\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}) \\ = L + \\ \end{array}
$$
$$
\frac {1}{M} \sum_ {i \neq i ^ {*}} \left(\max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 \left(K ^ {2} + M K\right) \right\} + \frac {2 \pi^ {2}}{3} + K ^ {2} + (2 M - 1) K\right) \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \tag {59}
$$
where the second inequality holds by plugging in (57).
Meanwhile, we note that for any  $i \neq i^*$ ,
$$
\begin{array}{l} \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} (\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}) + \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} \leq 0} (\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}) \\ = \sum_ {m = 1} ^ {M} \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \\ = M \Delta_ {i} > 0 \\ \end{array}
$$
$$
\big|\sum_{m:\mu_{i^{*}}^{m} - \mu_{i}^{m}\leq 0}(\mu_{i^{*}}^{m} - \mu_{i}^{m})\big|\leq M
$$
which gives us that
$$
\begin{array}{l} \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} (\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}) \\ = M \Delta_ {i} - \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} \leq 0} \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \\ = M\Delta_{i} + \big|\sum_{m:\mu_{i^{*}}^{m} - \mu_{i}^{m}\leq 0}(\mu_{i^{*}}^{m} - \mu_{i}^{m})| \\ \leq M \Delta_ {i} + M = M \left(\Delta_ {i} + 1\right). \tag {60} \\ \end{array}
$$
Hence, the regret can be upper bounded by
$$
\begin{array}{l} \leq L + \sum_ {i \neq i ^ {*}} (\Delta_ {i} + 1) (\max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K + M) \right\} + \frac {2 \pi^ {2}}{3} + K ^ {2} + (2 M - 1) K) \\ = O \left(\max  \{L, \log T \}\right) \\ \end{array}
$$
where the inequality is derived from (60) and  $L$  is the same constant as in the definition of  $A_{\epsilon, \delta}$ .
Theorem 3. Let  $f$  be a function specific to a setting and defined in the above remark. For every  $0 < \epsilon < 1$  and  $0 < \delta < f(\epsilon, M, T)$ , in settings  $S_1$  with  $c \geq \frac{1}{2} + \frac{1}{2}\sqrt{1 - \left(\frac{\epsilon}{MT}\right)^{\frac{2}{M-1}}}, S_2, S_3$  with the time horizon  $T$  satisfying  $T \geq L$ , the regret of Algorithm 2 with  $F(m, i, t) = \sqrt{\frac{C_1\ln T}{n_{m,i}(t)}} + \frac{C_2\ln T}{n_{m,i}(t)}$  satisfies
$$
\begin{array}{l} E [ R _ {T} | A _ {\epsilon , \delta} ] \leq L + \sum_ {i \neq i ^ {*}} (\Delta_ {i} + 1) \cdot \left(\max  \left([ \frac {1 6 C _ {1} \log T}{\Delta_ {i} ^ {2}} ], [ \frac {4 C _ {2} \log T}{\Delta_ {i}} ], 2 (K ^ {2} + M K + M)\right) \right. \\ + \frac {4}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}} + K ^ {2} + (2 M - 1) K) \\ \end{array}
$$
where  $L, C_1$  are specified as in Theorem 2 and  $\frac{C_2}{C_1} \geq \frac{3}{2}$ .
Proof. By the regret decomposition as in (58), we obtain that
$$
E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] \leq L + \frac {1}{M} \sum_ {i \neq i ^ {*}} \sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} E \left[ n _ {m, i} (T) \mid A _ {\epsilon , \delta} \right] \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right). \tag {61}
$$
By Proposition 13, we have that with probability at least  $1 - 7\epsilon$
$$
\begin{array}{l} E [ n _ {m, i} (T) | A _ {\epsilon , \delta} ] \\ \leq \max  \left(\left[ \frac {1 6 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], \left[ \frac {4 C _ {2} \log T}{\Delta_ {i}} \right], 2 \left(K ^ {2} + M K + M\right)\right) + \frac {4}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}} + K ^ {2} + (2 M - 1) K. \tag {62} \\ \end{array}
$$
Following (60) gives us that
$$
\sum_ {m: \mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m} > 0} \left(\mu_ {i ^ {*}} ^ {m} - \mu_ {i} ^ {m}\right) \leq M \Delta_ {i} + M = M \left(\Delta_ {i} + 1\right). \tag {63}
$$
Therefore, we derive that with probability at least  $P(A_{\epsilon, \delta}) = 1 - 7\epsilon$
$$
\begin{array}{l} E [ R _ {T} | A _ {\epsilon , \delta} ] \leq L + \sum_ {i \neq i ^ {*}} (\Delta_ {i} + 1) \cdot \left(\max  \left([ \frac {1 6 C _ {1} \log T}{\Delta_ {i} ^ {2}} ], [ \frac {4 C _ {2} \log T}{\Delta_ {i}} ], 2 (K ^ {2} + M K + M)\right) \right. \\ + \frac {4}{P (A _ {\epsilon , \delta}) T ^ {3}} + K ^ {2} + (2 M - 1) K) \\ \end{array}
$$
Theorem 4. Assume the same conditions as in Theorems 2 and 3. The regret of Algorithm 2 satisfies that
$$
\begin{array}{l} E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] \leq L _ {1} + \frac {4}{P \left(A _ {\epsilon , \delta}\right) T ^ {3}} + \\ (1 + \max \{\sqrt {C _ {1} \ln T}, C _ {2} \ln T \}) (K (K + 2 M) - K + \frac {2}{P (A _ {\epsilon , \delta}) T ^ {3}}) + \\ K (C _ {2} (\ln T) ^ {2} + C _ {2} \ln T + \sqrt {C _ {1} \ln T} \sqrt {T (\ln T + 1)}) = O (\sqrt {T} \ln T). \\ \end{array}
$$
where  $L_{1} = \max (L, K(2(K^{2} + MK + M))), L, C_{1}$  is specified as in Theorem 2, and  $\frac{C_2}{C_1} \geq \frac{3}{2}$ . The involved constants depend on  $\sigma^2$  but not on  $\Delta_{i}$ .
Proof. Define  $U_{m}^{t}(i)$  and  $L_{m}^{t}(i)$  as  $\tilde{\mu}_{i}^{m}(t) + Rad(i,m,t)$  and  $\tilde{\mu}_{i}^{m}(t) - Rad(i,m,t)$ , respectively, where  $Rad$  is previously defined as  $Rad(i,m,t) = \sqrt{\frac{C_1\ln T}{n_{m,i}(t)}} +\frac{C_2\ln T}{n_{m,i}(t)}$ . We observe that by definition, the regret  $R_{T}$  can be written as
$$
\begin{array}{l} R _ {T} = \frac {1}{M} \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \left(\mu_ {i ^ {*}} - \mu_ {a _ {m} ^ {t}} ^ {t}\right) \\ = \frac {1}{M} \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \left(\mu_ {i ^ {*}} - U _ {m} ^ {t} \left(a _ {m} ^ {t}\right) + U _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) + L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - \mu_ {a _ {m} ^ {t}} ^ {t}\right). \\ \end{array}
$$
Subsequently, the conditional expectation of  $R_{T}$  has the following decomposition
$$
\begin{array}{l} E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] \\ = \frac {1}{M} \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \left(E \left[ \mu_ {i ^ {*}} - U _ {m} ^ {t} \left(a _ {m} ^ {t}\right) \mid A _ {\epsilon , \delta} \right] + E \left[ U _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) \mid A _ {\epsilon , \delta} \right] + E \left[ L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - \mu_ {a _ {m} ^ {t}} ^ {t} \mid A _ {\epsilon , \delta} \right]\right) \\ = L _ {1} + \frac {1}{M} \sum_ {t = L _ {1} + 1} ^ {T} \sum_ {m = 1} ^ {M} (E [ \mu_ {i ^ {*}} - U _ {m} ^ {t} (i ^ {*}) | A _ {\epsilon , \delta} ] + E [ U _ {m} ^ {t} (i ^ {*}) - U _ {m} ^ {t} (a _ {m} ^ {t}) | A _ {\epsilon , \delta} ] + \\ E \left[ U _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) \mid A _ {\epsilon , \delta} \right] + E \left[ L _ {m} ^ {t} \left(a _ {m} ^ {t}\right) - \mu_ {a _ {m} ^ {t}} ^ {t} \mid A _ {\epsilon , \delta} \right]) \tag {64} \\ \end{array}
$$
where  $L_{1} = \max (L,2(K^{2} + MK + M))$
For the first term, we derive its upper bound as follows.
$$
\begin{array}{l} E \left[ \mu_ {i ^ {*}} - U _ {m} ^ {t} \left(i ^ {*}\right) \mid A _ {\epsilon , \delta} \right] \\ \leq E \left[ \left(\mu_ {i ^ {*}} - U _ {m} ^ {t} \left(i ^ {*}\right)\right) 1 _ {\mu_ {i ^ {*}} - U _ {m} ^ {t} \left(i ^ {*}\right) > 0} \mid A _ {\epsilon , \delta} \right] \\ = E \left[ \mu_ {i ^ {*}} 1 _ {\mu_ {i ^ {*} -} U _ {m} ^ {t} (i ^ {*}) > 0} \mid A _ {\epsilon , \delta} \right] - E \left[ U _ {m} ^ {t} 1 _ {\mu_ {i ^ {*} -} U _ {m} ^ {t} (i ^ {*}) > 0} \mid A _ {\epsilon , \delta} \right] \\ \leq E \left[ \mu_ {i ^ {*}} 1 _ {\mu_ {i ^ {*}} - U _ {m} ^ {t} \left(i ^ {*}\right) > 0} \mid A _ {\epsilon , \delta} \right] \\ \leq E \left[ 1 _ {\mu_ {i} * - U _ {m} ^ {t} (i ^ {*}) > 0} | A _ {\epsilon , \delta} \right] \\ = P \left(\mu_ {i ^ {*}} - U _ {m} ^ {t} \left(i ^ {*}\right) > 0 \mid A _ {\epsilon , \delta}\right) \\ = P \left(\mu_ {i ^ {*}} - \tilde {\mu} _ {i ^ {*}} ^ {m} (t) > R a d | A _ {\epsilon , \delta}\right) \\ \leq P \left(\left| \mu_ {i ^ {*}} - \tilde {\mu} _ {i ^ {*}} ^ {m} (t) \right| > R a d \mid A _ {\epsilon , \delta}\right) \leq \frac {2}{P \left(A _ {\epsilon , \delta}\right) T ^ {4}} \tag {65} \\ \end{array}
$$
where the first inequality uses the monotone property of  $E[\cdot]$ , the second inequality omits the latter negative quantity, the third inequality holds by the fact that  $0 \leq \mu_i^* \leq 1$ , and the last inequality is by Proposition 12.