SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (2 2) = \Pi_ {s} E [ \exp \left\{\lambda \frac {\left(r _ {i} ^ {j} (s) - \mu_ {i} ^ {j}\right)}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \right\} 1 _ {A _ {\epsilon , \delta}} | \sigma (\left\{n _ {m, i} (t) \right\} _ {t, i, m}) ] \\ = \Pi_ {s} E [ \exp \left\{\lambda \frac {\left(r _ {i} ^ {j} (s) - \mu_ {i} ^ {j}\right)}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \right\} | \sigma (\left\{n _ {m, i} (t) \right\} _ {t, i, m}) ] \cdot E \left[ 1 _ {A _ {\epsilon , \delta}} | \sigma (\left\{n _ {m, i} (t) \right\} _ {t, i, m}) \right] \\ = \Pi_ {s} E _ {r} \left[ \exp \left\{\lambda \frac {\left(r _ {i} ^ {j} (s) - \mu_ {i} ^ {j}\right)}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \right\} \right] \cdot E \left[ 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \\ \leq \Pi_ {s} \exp \left\{\frac {\left(\frac {\lambda}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)}\right) ^ {2} \sigma^ {2}}{2} \right\} \cdot E \left[ 1 _ {A _ {\epsilon , \delta}} | \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \\ \leq \left(\exp \left\{\frac {\left(\frac {\lambda}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)}\right) ^ {2} \sigma^ {2}}{2} \right\}\right) ^ {n _ {j, i} \left(h _ {m, j} ^ {L}\right)} \\ = \exp \left\{\frac {\frac {\lambda^ {2}}{n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \sigma^ {2}}{2} \right\} \\ \leq \exp \left\{\frac {\lambda^ {2} \sigma^ {2}}{2 \min  _ {j} n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \right\} \tag {23} \\ \end{array}
$$
where the first inequality holds by the definition of sub-Gaussian random variables  $r_i^j (s) - \mu_i^j$  with an mean value 0, the second inequality results from  $1_{A_{\epsilon ,\delta}}\leq 1$ , and the last inequality uses  $n_{j,i}(h_{m,j}^{L})\geq \min_{j}n_{j,i}(h_{m,j}^{L})$  for any  $j$ .
Therefore, we obtain that by plugging (23) into (21)
$$
\begin{array}{l} (2 1) \leq \Pi_ {j = 1} ^ {M} (\exp \left\{\frac {\lambda^ {2} \sigma^ {2}}{2 \min _ {j} n _ {j , i} (h _ {m , j} ^ {L})} \right\}) ^ {\frac {1}{M}} \\ = \left(\left(\exp \left\{\frac {\lambda^ {2} \sigma^ {2}}{2 \min _ {j} n _ {j , i} \left(h _ {m , j} ^ {L}\right)} \right\}\right) ^ {\frac {1}{M}}\right) ^ {M} \\ = \exp \left\{\frac {\lambda^ {2} \sigma^ {2}}{2 \min _ {j} n _ {j , i} (h _ {m , j} ^ {L})} \right\} \\ \end{array}
$$
which completes the basis step.
Now we proceed to the induction step. Suppose that for any  $s < t + 1$  where  $t \geq L$ , we have
$$
\begin{array}{l} E \left[ \exp \left\{\lambda \left(\tilde {\mu} _ {i} ^ {m} (s) - \mu_ {i}\right) \right\} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (s) \right\} _ {s, i, m}\right) \right] \\ \leq \exp \left\{\frac {\lambda^ {2}}{2} \frac {C \sigma^ {2}}{\min  _ {j} n _ {j , i} (s)} \right\}. \tag {24} \\ \end{array}
$$
The update rule of  $\tilde{\mu}_i^m$  implies that
$$
E \left[ \exp \left\{\lambda \left(\tilde {\mu} _ {i} ^ {m} (t + 1) - \mu_ {i}\right) \right\} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (s) \right\} _ {s, i, m}\right) \right]
$$
$$
= E [ \exp \{\lambda (\sum_ {j = 1} ^ {M} P _ {t} ^ {\prime} (m, j) (\hat {\tilde {\mu}} _ {i, j} ^ {m} (t _ {m, j}) - \mu_ {i}) + d _ {m, t} \sum_ {j \in N _ {m} (t)} (\hat {\tilde {\mu}} _ {i, j} ^ {m} (t) - \mu_ {i} ^ {j})
$$
$$
+ d _ {m, t} \sum_ {j \not \in N _ {m} (t)} (\hat {\tilde {\mu}} _ {i, j} ^ {m} (t _ {m, j}) - \mu_ {i} ^ {j})) \} 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ]
$$
$$
= E [ \exp \{\lambda (\sum_ {j = 1} ^ {M} P _ {t} ^ {\prime} (m, j) (\tilde {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i}) + d _ {m, t} \sum_ {j \in N _ {m} (t)} (\bar {\mu} _ {i} ^ {j} (t) - \mu_ {i} ^ {j})
$$
$$
+ d _ {m, t} \sum_ {j \notin N _ {m} (t)} \left(\bar {\mu} _ {i} ^ {j} \left(t _ {m, j}\right) - \mu_ {i} ^ {j})\right) \} 1 _ {A _ {\epsilon , \delta}} \left| \sigma \left(\left\{n _ {m, i} (s) \right\} _ {s, i, m}\right) \right]
$$
$$
= E [ \Pi_ {j = 1} ^ {M} \exp \{\lambda P _ {t} ^ {\prime} (m, j) (\tilde {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i}) \} 1 _ {A _ {\epsilon , \delta}} \cdot \Pi_ {j \in N _ {m} (t)} \exp \{\lambda d _ {m, t} (\bar {\mu} _ {i} ^ {j} (t) - \mu_ {i} ^ {j}) \} 1 _ {A _ {\epsilon , \delta}}
$$
$$
\cdot \Pi_ {j \not \in N _ {m} (t)} \exp \left\{\lambda d _ {m, t} \left(\bar {\mu} _ {i} ^ {j} \left(t _ {m, j}\right) - \mu_ {i} ^ {j}\right) 1 _ {A _ {\epsilon , \delta}} \right\} \mid \sigma \left(\left\{n _ {m, i} (s) \right\} _ {s, i, m}\right) ]
$$
$$
\leq \Pi_ {j = 1} ^ {M} (E [ (\exp \{\lambda P _ {t} ^ {\prime} (m, j) (\tilde {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i}) \}) ^ {M + 2} 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ]) ^ {\frac {1}{M + 2}}.
$$
$$
E [ \Pi_ {j \in N _ {m} (t)} (\exp \{\lambda d _ {m, t} (\bar {\mu} _ {i} ^ {j} (t) - \mu_ {i} ^ {j}) \}) ^ {M + 2} 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ] ^ {\frac {1}{M + 2}}.
$$
$$
E [ \Pi_ {j \not \in N _ {m} (t)} (\exp \{\lambda d _ {m, t} (\bar {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i} ^ {j}) \}) ^ {M + 2} 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ] ^ {\frac {1}{M + 2}}
$$
$$
= \Pi_ {j = 1} ^ {M} (E [ (\exp \{\lambda P _ {t} ^ {\prime} (m, j) (M + 2) (\tilde {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i}) \}) 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ]) ^ {\frac {1}{M + 2}}.
$$
$$
E [ \Pi_ {j \in N _ {m} (t)} (\exp \{\lambda d _ {m, t} (M + 2) (\bar {\mu} _ {i} ^ {j} (t) - \mu_ {i} ^ {j}) \}) 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ] ^ {\frac {1}{M + 2}}.
$$
$$
E [ \Pi_ {j \not \in N _ {m} (t)} (\exp \{\lambda d _ {m, t} (M + 2) (\bar {\mu} _ {i} ^ {j} (t _ {m, j}) - \mu_ {i} ^ {j}) \}) 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (s) \} _ {s, i, m}) ] ^ {\frac {1}{M + 2}}
$$
$$
\leq \Pi_ {j = 1} ^ {M} (\exp \left\{\frac {\lambda^ {2} (P _ {t} ^ {\prime} (m , j)) ^ {2} (M + 2) ^ {2}}{2} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t _ {m , j})} \right\}) ^ {\frac {1}{M + 2}}.
$$
$$
\Pi_ {j \in N _ {m} (t)} \Pi_ {s} (E _ {r} [ \exp \{\lambda d _ {m, t} (M + 2) \frac {(r _ {i} ^ {j} (s) - \mu_ {i} ^ {j})}{n _ {j , i} (t)} \} ] \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ]) ^ {\frac {1}{M + 2}}.
$$
$$
\Pi_ {j \notin N _ {m} (t)} \Pi_ {s} \left(E _ {r} \left[ \exp \left\{\lambda d _ {m, t} (M + 2) \frac {\left(r _ {i} ^ {j} (s) - \mu_ {i} ^ {j}\right)}{n _ {j , i} \left(t _ {m , j}\right)} \right\} \right] \cdot E \left[ 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right]\right) ^ {\frac {1}{M + 2}} \tag {25}
$$
where the first inequality uses Lemma 3 and the second inequality applies (24) as the assumption for the induction step and holds by exchanging the expectations with the multiplication since again given  $s$  the reward  $(r_i^j (s) - \mu_i^j)$  is independent of other random variables.
We continue bounding the last two terms by using the definition of sub-Gaussian random variables  $(r_i^j (s) - \mu_i^j)$  and obtain
$$
(2 5) \leq (\exp \{\frac {\lambda^ {2} (P _ {t} ^ {\prime} (m , j)) ^ {2} (M + 2) ^ {2}}{2} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t _ {m , j})} \}) ^ {\frac {M}{M + 2}}.
$$
$$
\Pi_ {j \in N _ {m} (t)} \Pi_ {s} (\exp \frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) ^ {2} \sigma^ {2}}{2 n _ {j , i} ^ {2} (t)} \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ]) ^ {\frac {1}{M + 2}}.
$$
$$
\Pi_ {j \not \in N _ {m} (t)} \Pi_ {s} (\exp {\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) ^ {2} \sigma^ {2}}{2 n _ {j , i} ^ {2} (t _ {m , j})}} \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ]) ^ {\frac {1}{M + 2}}
$$
$$
= (\exp \left\{\frac {\lambda^ {2} (P _ {t} ^ {\prime} (m , j)) ^ {2} (M + 2) ^ {2}}{2} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t _ {m , j})} \right\}) ^ {\frac {M}{M + 2}}.
$$
$$
\Pi_ {j \in N _ {m} (t)} \exp \bigl \{\frac {n _ {j , i} (t)}{M + 2} \frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) ^ {2} \sigma^ {2}}{2 n _ {j , i} ^ {2} (t)} \bigr \} \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ].
$$
$$
\Pi_ {j \not \in N _ {m} (t)} \exp \left\{\frac {n _ {j , i} (t _ {m , j})}{M + 2} \frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) ^ {2} \sigma^ {2}}{2 n _ {j , i} ^ {2} (t _ {m , j})} \right\} \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ]
$$
Building on that, we establish
$$
\begin{array}{l} (2 5) \leq (\exp \{\frac {\lambda^ {2} (P _ {t} ^ {\prime} (m , j)) ^ {2} (M + 2) ^ {2}}{2} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t _ {m , j})} \}) ^ {\frac {M}{M + 2}}. \\ (\exp \{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) \sigma^ {2}}{2 \min _ {j} n _ {j , i} (t)} \}) ^ {| N _ {m} (t) |} \cdot E [ 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ]. \\ \left. \right.\left(\exp \left\{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) \sigma^ {2}}{2 \min  _ {j} n _ {j , i} \left(t _ {m , j}\right)} \right\}\right) ^ {| M - N _ {m} (t) |} \cdot E \left[ 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right)\right] \\ = E \left[ \right.\left(\exp \left\{\frac {\lambda^ {2} \left(P _ {t} ^ {\prime} (m , j)\right) ^ {2} M (M + 2)}{2} \frac {C \sigma^ {2}}{\min  _ {j} n _ {j , i} \left(t _ {m , j}\right)} \right\}\right) \cdot \left(\exp \left\{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) \left| N _ {m} (t) \right|}{2 \min  _ {j} n _ {j , i} (t)} \right\}\right) \\ \cdot \left(\exp \left\{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) \sigma^ {2} | M - N _ {m} (t) |}{2 \min  _ {j} n _ {j , i} (t _ {m , j})} \right\}\right) 1 _ {A _ {\epsilon , \delta}} \left| \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \\ \leq E [ (\exp \{\frac {\lambda^ {2} (P _ {t} ^ {\prime} (m , j)) ^ {2} M (M + 2)}{2 (1 - c _ {0})} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t + 1)} \}) \cdot (\exp \{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) | N _ {m} (t) | \sigma^ {2}}{2 \frac {L / K}{L / K + 1} \min _ {j} n _ {j , i} (t + 1)} \}) \\ \cdot \left(\exp \left\{\frac {\lambda^ {2} d _ {m , t} ^ {2} (M + 2) | M - N _ {m} (t) | \sigma^ {2}}{2 \left(1 - c _ {0}\right) \min  _ {j} n _ {j , i} (t + 1)} \right\}\right) 1 _ {A _ {\epsilon , \delta}} \left| \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \\ \end{array}
$$
where the first inequality uses the fact that for any  $j$ ,  $n_{j,i}(t) \geq \min_j n_{j,i}(t)$  and  $n_{j,i}(t_{m,j}) \geq \min_j n_{j,i}(t_{m,j})$ . For the second inequality, the first term is a result of  $\frac{\min_j n_{j,i}(t)}{\min_j n_{j,i}(t+1)} \geq \frac{\min_j n_{j,i}(t)}{\min_j n_{j,i}(t+1)} \geq \frac{L/K}{L/K+1}$  since  $n_{j,i}(t) > n_{j,i}(L) = L/K$  and the ratio is monotone increasing in  $n$ , and the second term is bounded based on the following derivations
$$
\begin{array}{l} \min  _ {j} n _ {j, i} \left(t _ {m, j}\right) \geq \min  _ {j} n _ {j, i} \left(t + 1 - t _ {0}\right) \\ \geq \min  _ {j} n _ {j, i} (t + 1) - t _ {0} \\ \geq \min  _ {j} n _ {j, i} (t + 1) - c _ {0} \min  _ {j} n _ {j, i} (t + 1) \\ = (1 - c _ {0}) \min  _ {j} n _ {j, i} (t + 1) \\ \end{array}
$$
where the last inequality holds by applying Proposition 6 that holds on event  $A_{\epsilon, \delta}$ .
Therefore, we can rewrite the above expression as
$$
\begin{array}{l} (2 5) = E [ (\exp \{\frac {\lambda^ {2} \sigma^ {2}}{2 \min  _ {j} n _ {j , i} (t + 1)} \cdot \left(\frac {C \left(P _ {t} ^ {\prime} (m , j)\right) ^ {2} M (M + 2)}{2 (1 - c _ {0})} + \right. \\ \frac {d _ {m , t} ^ {2} (M + 2) | N _ {m} (t) |}{\frac {L / K}{L / K + 1}} + \frac {d _ {m , t} ^ {2} (M + 2) | M - N _ {m} (t) |}{(1 - c _ {0})}) \} 1 _ {A _ {\epsilon , \delta}} | \sigma (\{n _ {m, i} (t) \} _ {t, i, m}) ] \\ \leq E \left[ \exp \left\{\frac {C \lambda^ {2} \sigma^ {2}}{2 \min  _ {j} n _ {j , i} (t + 1)} \right\} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \\ \leq \exp \left\{\frac {C \lambda^ {2} \sigma^ {2}}{2 \min  _ {j} n _ {j , i} (t + 1)} \right\} \\ \end{array}
$$
where the first inequality holds by the choice of  $P_{t}^{\prime}(m,j),d_{m,t},L,c_{0}$  and  $C$  and the second inequality uses the fact that  $1_{A_{\epsilon ,\delta}}\leq 1$  and  $\min_jn_{j,i}(t + 1)\in \sigma (\{n_{m,i}(t)\}_{t,i,m})$
This completes the induction step and subsequently concludes the proof.
Proposition 9. Assume the parameter  $\delta$  satisfies that  $0 < \delta < c = f(\epsilon, M, T)$ . In setting  $s_1, s_2$ , and  $s_3$ , for any  $m, i$  and  $t > L$  where  $L$  is the length of the burn-in period,  $\tilde{\mu}_{m,i}(t)$  satisfies that if if
$$
n _ {m, i} (t) \geq 2 \left(K ^ {2} + K M + M\right), \text {t h e n w i t h} P \left(A _ {\epsilon , \delta}\right) = 1 - 7 \epsilon ,
$$
$$
P (\tilde {\mu} _ {m, i} (t) - \mu_ {i} \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} | A _ {\epsilon , \delta}) \leq \frac {1}{P (A _ {\epsilon , \delta})} \frac {1}{t ^ {2}},
$$
$$
P (\mu_ {i} - \tilde {\mu} _ {m, i} (t) \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} | A _ {\epsilon , \delta}) \leq \frac {1}{P (A _ {\epsilon , \delta}) t ^ {2}}.
$$
Proof. By Proposition 7, we have  $E[\tilde{\mu}_{m,i}(t) - \mu_i|A_{\epsilon ,\delta}] = 0$ , which allows us to consider the tail bound of the global estimator  $\tilde{\mu}_i^m (t)$  conditional on event  $A_{\epsilon ,\delta}$  as follows.
$$
\begin{array}{l} P (\tilde {\mu} _ {m, i} (t) - \mu_ {i} \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} | A _ {\epsilon , \delta}) \\ = E \left[ 1 _ {\hat {\mu} _ {m, i} (t) - \mu_ {i} \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}}} | A _ {\epsilon , \delta} \right] \\ = \frac {1}{P (A _ {\epsilon , \delta})} E [ 1 _ {\tilde {\mu} _ {m, i} (t) - \mu_ {i} \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}}} 1 _ {A _ {\epsilon , \delta}} ] \\ = \frac {1}{P (A _ {\epsilon , \delta})} E [ 1 _ {\exp \{\lambda (n) (\tilde {\mu} _ {m, i} (t) - \mu_ {i}) \} \geq \exp \{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \}} 1 _ {A _ {\epsilon , \delta}} ] \\ \leq \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E \left[ \frac {\exp \left\{\lambda (n) \left(\tilde {\mu} _ {m , i} (t) - \mu_ {i}\right) \right\}}{\exp \left\{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \right\}} 1 _ {A _ {\epsilon , \delta}} \right] \tag {26} \\ \end{array}
$$
where the last inequality is by the fact that 1  $\exp \{\lambda (n)(\tilde{\mu}_{m,i}(t) - \mu_i)\} \geq \exp \{\lambda (n)\sqrt{\frac{C_1\log t}{n_{m,i}(t)}}\}$ $\frac{\exp\{\lambda(n)(\tilde{\mu}_{m,i}(t) - \mu_i)\}}{\exp\{\lambda(n)\sqrt{\frac{C_1\log t}{n_{m,i}(t)}}\}}.$
By the assumption that  $\delta < c$ , we have Proposition 8 holds. Subsequently, by Proposition 8 and Lemma 2 which holds since  $n_{m,i}(t) \geq 2(K^2 + KM + M)$ , we have for any  $\lambda$
$$
\begin{array}{l} E \left[ \exp \left\{\lambda \left(\tilde {\mu} _ {i} ^ {m} (t) - \mu_ {i}\right) \right\} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {t, i, m}\right) \right] \leq \exp \left\{\frac {\lambda^ {2}}{2} \frac {C \sigma^ {2}}{\min _ {j} n _ {j , i} (t)} \right\} \\ \leq \exp \left\{\frac {\lambda^ {2}}{1} \frac {C \sigma^ {2}}{n _ {m , i} (t)} \right\}. \tag {27} \\ \end{array}
$$
Again, we utilize the law of total expectation and further obtain
$$
\begin{array}{l} (2 6) = \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E \left[ E \left[ \frac {\exp \left\{\lambda (n) \left(\tilde {\mu} _ {m , i} (t) - \mu_ {i}\right) \right\}}{\exp \left\{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \right\}} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {m, i, t}\right) \right] \right. \\ = \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E \left[ E \left[ \frac {\exp \left\{\lambda (n) \left(\tilde {\mu} _ {m , i} (t) - \mu_ {i}\right) \right\}}{\exp \left\{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \right\}} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {m, i, t}\right) \right] \right] \\ = \frac {1}{P \left(A _ {\epsilon , \delta}\right)} E \left[ \frac {1}{\exp \left\{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \right\}} E \left[ \exp \left\{\lambda (n) \left(\tilde {\mu} _ {m, i} (t) - \mu_ {i}\right) \right\} 1 _ {A _ {\epsilon , \delta}} \mid \sigma \left(\left\{n _ {m, i} (t) \right\} _ {m, i, t}\right) \right] \right. \\ \leq \frac {1}{P (A _ {\epsilon , \delta})} E [ \frac {1}{\exp \{\lambda (n) \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \}} \cdot \exp \{\frac {\lambda^ {2} (n)}{1} \frac {C \sigma^ {2}}{n _ {m , i} (t)} \} ] \\ \leq \frac {1}{P \left(A _ {\epsilon , \delta}\right)} \exp \{- 2 \log t \} = \frac {1}{P \left(A _ {\epsilon , \delta}\right) t ^ {2}} \tag {28} \\ \end{array}
$$
where the first inequality holds by (44) and the second inequality holds by choosing  $\lambda(n) = \frac{\sqrt{\frac{C_1 \log t}{n_{m,i}(t)}}}{2 \frac{C \sigma^2}{n_{m,i}(t)}}$  and by the choice of parameter  $C_1$  such that  $\frac{C_1}{4C \sigma^2} \geq 2$  or equivalently  $C_1 \geq 8C \sigma^2$ .
In like manner, we obtain that by repeating the above steps with  $\mu_{i} - \tilde{\mu}_{m,i}(t)$ , we have
$$
P \left(\mu_ {i} - \tilde {\mu} _ {m, i} (t) \geq \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t)}} \mid A _ {\epsilon , \delta}\right) \leq \frac {1}{P \left(A _ {\epsilon , \delta}\right) t ^ {2}} \tag {29}
$$
Proposition 10. Assume the parameter  $\delta$  satisfies that  $0 < \delta < c = f(\epsilon, M, T)$ . An arm  $k$  is said to be sub-optimal if  $k \neq i^{*}$  where  $i^{*}$  is the unique optimal arm in terms of the global reward, i.e.  $i^{*} = \arg \max \frac{1}{M} \sum_{j=1}^{M} \mu_{i}^{j}$ . Then in setting  $s_{1}, s_{2}$  and  $s_{3}$ , when the game ends, for every client  $m$ ,  $0 < \epsilon < 1$  and  $T > L$ , the expected numbers of pulling sub-optimal arm  $k$  after the burn-in period satisfy with  $P(A_{\epsilon, \delta}) = 1 - 7\epsilon$
$$
\begin{array}{l} E [ n _ {m, k} (T) \mid A _ {\epsilon , \delta} ] \\ \leq \max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K + M) \right\} + \frac {2 \pi^ {2}}{3 P (A _ {\epsilon , \delta})} + K ^ {2} + (2 M - 1) K \\ \leq O (\log T). \\ \end{array}
$$
Proof of Proposition 10. We claim that what lead to pulling an sub-optimal arm  $i$  are explicit by the decision rule of Algorithm 2, meaning that the result  $a_{t}^{m} = i$  holds when any of the following conditions is met:
- Case 1:  $n_{m,i}(t) \leq \mathcal{N}_{m,i}(t) - K$ ,
Case 2:  $\tilde{\mu}_{m,i} - \mu_i > \sqrt{\frac{C_1\log t}{n_{m,i}(t - 1)}},$
Case 3:  $-\tilde{\mu}_{m,i^{*}} + \mu_{i^{*}} > \sqrt{\frac{C_{1}\log t}{n_{m,i^{*}}(t - 1)}},$
Case 4:  $\mu_{i^*} - \mu_i < 2\sqrt{\frac{C_1\log t}{n_{m,i}(t - 1)}}$
Then we formally consider the number of pulling arms  $n_{m,i}(T)$  starting from  $L + 1$ . For any  $l > 1$ , we have that based on the above listed conditions
$$
\begin{array}{l} n _ {m, i} (T) \leq l + \sum_ {t = L + 1} ^ {T} 1 _ {\{a _ {t} ^ {m} = i, n _ {m, i} (t) > l \}} \\ \leq l + \sum_ {t = L + 1} ^ {T} 1 _ {\left\{\tilde {\mu} _ {i} ^ {m} - \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} > \mu_ {i}, n _ {m, i} (t - 1) \geq l \right\}} \\ + \sum_ {t = L + 1} ^ {T} 1 _ {\left\{\tilde {\mu} _ {i ^ {*}} ^ {m} + \sqrt {\frac {C _ {1} \log t}{n _ {m , i ^ {*}} (t - 1)}} <   \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l \right\}} \\ + \sum_ {t = L + 1} ^ {T} 1 _ {\{n _ {m, i} (t) <   \mathcal {N} _ {m, i} (t) - K, a _ {t} ^ {m} = i, n _ {m, i} (t - 1) \geq l \}} \\ + \sum_ {t = L + 1} ^ {T} 1 _ {\left\{\mu_ {i} + 2 \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} > \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l \right\}}. \\ \end{array}
$$
Consequently, the expected value of  $n_{m,i}(t)$  conditional on  $A_{\epsilon ,\delta}$  reads as
$$
E \left[ n _ {m, i} (T) \mid A _ {\epsilon , \delta} \right]
$$
$$
\begin{array}{l} = l + \sum_ {t = L + 1} ^ {T} P \left(\tilde {\mu} _ {i} ^ {m} - \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} > \mu_ {i}, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) \\ + \sum_ {t = L + 1} ^ {T} P \left(\tilde {\mu} _ {i ^ {*}} ^ {m} + \sqrt {\frac {C _ {1} \log t}{n _ {m , i ^ {*}} (t - 1)}} <   \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}\right) \\ + \sum_ {t = L + 1} ^ {T} P \left(n _ {m, i} (t) <   \mathcal {N} _ {m, i} (t) - K, a _ {t} ^ {m} = i, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) \\ + \sum_ {t = L + 1} ^ {T} P (\mu_ {i} + 2 \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} > \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) \\ \end{array}
$$
$$
\begin{array}{l} = l + \sum_ {t = L + 1} ^ {T} P (C a s e 2, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) + \sum_ {t = L + 1} ^ {T} P (C a s e 3, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) \\ + \sum_ {t = L + 1} ^ {T} P \left(\text {C a s e 1}, a _ {t} ^ {m} = i, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) + \sum_ {t = L + 1} ^ {T} P \left(\text {C a s e 4}, n _ {m, i} (t - 1) \geq l \mid A _ {\epsilon , \delta}\right) \tag {30} \\ \end{array}
$$
where  $l = \max \left\{\left[\frac{4C_1\log T}{\Delta_i^2}\right], 2(K^2 + MK + M)\right\}$ .
For the last term in (30), we have
$$
\sum_ {t = L + 1} ^ {T} P \left(\text {C a s e 4}: \mu_ {i} + 2 \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} > \mu_ {i ^ {*}}, n _ {m, i} (t - 1) \geq l\right) = 0 \tag {31}
$$
since the choice of  $l$  satisfies  $l \geq \left[\frac{4C_1\log T}{\Delta_i^2}\right]$  with  $\Delta_{i} = \mu_{i^{*}} - \mu_{i}$ .
For the first two terms, we have on event  $A_{\epsilon ,\delta}$
$$
\begin{array}{l} \sum_ {t = L + 1} ^ {T} P (C a s e 2, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) + \sum_ {t = 1} ^ {T} P (C a s e 3, n _ {m, i} (t - 1) \geq l | A _ {\epsilon , \delta}) \\ \leq \sum_ {t = L + 1} ^ {T} P (\tilde {\mu} _ {m, i} - \mu_ {i} > \sqrt {\frac {C _ {1} \log t}{n _ {m , i} (t - 1)}} | A _ {\epsilon , \delta}) + \sum_ {t = 1} ^ {T} P (- \tilde {\mu} _ {m, i ^ {*}} + \mu_ {i ^ {*}} > \sqrt {\frac {C _ {1} \log t}{n _ {m , i ^ {*}} (t - 1)}} | A _ {\epsilon , \delta}) \\ \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{t ^ {2}}\right) + \sum_ {t = 1} ^ {T} \left(\frac {1}{t ^ {2}}\right) \leq \frac {\pi^ {2}}{3} \tag {32} \\ \end{array}
$$
where the first inequality holds by the property of the probability measure when removing the event  $n_{m,i}(t - 1) \geq l$  and the second inequality holds by (47) and (29) as stated in Proposition 9, which holds by the assumption that  $\delta < c$ .
For Case 1, we note that Lemma 1 implies that
$$
n _ {m, i} (t) > \mathcal {N} _ {m, i} (t) - K (K + 2 M)
$$
with the definition of  $N_{m,i}(t + 1) = \max \{n_{m,i}(t + 1),N_{j,i}(t),j\in \mathcal{N}_m(t)\}$
Departing from the result that the difference between  $N_{m,i}(t)$  and  $n_{m,i}(t)$  is at most  $K(K + 2M)$ , we then present the following analysis on how long it takes for the value  $-n_{m,i}(t) + N_{m,i}(t)$  to be smaller than  $K$ .
At time step  $t$ , if Case 1 holds for client  $m$ , then  $n_{m,i}(t + 1)$  is increasing by 1 on the basis of  $n_{m,i}(t)$ . What follows characterizes the change of  $N_{m,i}(t + 1)$ . Client  $m$  satisfying  $n_{m,i}(t) \leq \mathcal{N}_{m,i}(t) - K$  will not change the value of  $N_{m,i}(t + 1)$  by the definition  $N_{m,i}(t + 1) = \max \{n_{m,i}(t + 1), N_{j,i}(t), j \in$