SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the work [23], they give a lower bound for misspecified linear bandits with a single user. The lower bound of  $R(T)$  is given by:  $R_{3}(T) \geq \epsilon_{*} T \sqrt{d}$ . Therefore, suppose our problem with multiple users and  $m$  underlying clusters where the arrival times are  $T_{i}$  for each cluster, then for any algorithms, even if they know the underlying clustering structure and keep  $m$  independent linear bandit algorithms to leverage the common information of clusters, the best they can get is  $R(T) = \sum_{i \in [m]} R_{3}(T_{i}) \geq \epsilon_{*} \sum_{i \in [m]} T_{i} \sqrt{d} = \epsilon_{*} T \sqrt{d}$ , which gives a lower bound of  $O(\epsilon_{*} T \sqrt{d})$  for the CBMUM problem. Recall that the regret upper bound of our algorithms is of  $O(\epsilon_{*} T \sqrt{md \log T} + d \sqrt{mT} \log T)$ , asymptotically matching this lower bound with respect to  $T$  up to logarithmic factors and with respect to  $m$  up to  $O(\sqrt{m})$  factors, showing the tightness of our theoretical results (where  $m$  are typically very small for real applications).
We conjecture that the gap for the  $m$  factor is due to the strong assumption that cluster structures are known to prove our lower bound, and whether there exists a tighter lower bound will be left for future work.
G Proof of the key Lemma 5.7
In Lemma 5.7, we want to bound the term  $\left|\pmb{x}_a^\top \overline{\pmb{M}}_{\overline{V}_t,t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_s\in V_t}}\pmb{x}_{a_s}\pmb{x}_{a_s}^\top (\pmb{\theta}_{i_s} - \pmb{\theta}_{i_t})\right|$ . By the definition of "good partition", we have  $\| \pmb{\theta}_{i_s} - \pmb{\theta}_{i_t}\| _2\leq \zeta ,\forall i_s\in \overline{V}_t$ . It is an easy-to-be-made mistake to directly drag  $\| \pmb{\theta}_{i_s} - \pmb{\theta}_{i_t}\| _2$  out to upper bound it by  $\left\| \pmb{x}_a^\top \overline{\pmb{M}}_{\overline{V}_t,t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_s\in V_t}}\pmb{x}_{a_s}\pmb{x}_{a_s}^\top \right\| _2\times \zeta$  and then proceed. We need more careful analysis.
We first prove the following general lemma.
Lemma G.1. For vectors  $\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_k \in \mathbb{R}^d, \| \pmb{x}_i \|_2 \leq 1, \forall i \in [k]$ , and vectors  $\pmb{\theta}_1, \pmb{\theta}_2, \dots, \pmb{\theta}_k \in \mathbb{R}^d, \| \pmb{\theta}_i \|_2 \leq C, \forall i \in [k]$ , where  $C > 0$  is a constant, we have:
$$
\left\| \sum_ {i = 1} ^ {k} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {i} \right\| _ {2} \leq C \sqrt {d} \left\| \sum_ {i = 1} ^ {k} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\top} \right\| _ {2}.
$$
Proof. Let  $\mathbf{X} \in \mathbb{R}^{d \times k}$  be a matrix such that it has  $\mathbf{x}_i$  as its columns, i.e.,  $\mathbf{X} = [\mathbf{x}_1, \ldots, \mathbf{x}_k] =$
Let  $\pmb{y} \in \mathbb{R}^{k \times 1}$  be a vector that has  $\pmb{x}_i^\top \pmb{\theta}_i$  as its elements, i.e.,  $\pmb{y} = [\pmb{x}_1^\top \pmb{\theta}_1, \dots, \pmb{x}_k^\top \pmb{\theta}_k]^\top$ . Then we have:
$$
\begin{array}{l} \left[ \begin{array}{c c c c} \boldsymbol {x} _ {1 1} & x _ {2 1} & \dots & \boldsymbol {x} _ {k 1} \\ \boldsymbol {x} _ {1 2} & x _ {2 2} & \dots & \boldsymbol {x} _ {k 2} \\ \vdots & \vdots & \ddots & \vdots \\ \boldsymbol {x} _ {1 d} & x _ {2 d} & \dots & \boldsymbol {x} _ {k d} \end{array} \right]. \\ \left\| \sum_ {i = 1} ^ {k} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {i} \right\| _ {2} ^ {2} = \| \boldsymbol {X} \boldsymbol {y} \| _ {2} ^ {2} \leq \| \boldsymbol {X} \| _ {2} ^ {2} \| \boldsymbol {y} \| _ {2} ^ {2} (27) \\ = \| \boldsymbol {X} \| _ {2} ^ {2} \sum_ {i = 1} ^ {k} \left(\boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\theta} _ {i}\right) ^ {2} \\ \leq \| \boldsymbol {X} \| _ {2} ^ {2} \sum_ {i = 1} ^ {k} \| \boldsymbol {x} _ {i} \| _ {2} ^ {2} \| \boldsymbol {\theta} _ {i} \| _ {2} ^ {2} (28) \\ \leq C ^ {2} \| \boldsymbol {X} \| _ {2} ^ {2} \sum_ {i = 1} ^ {k} \| \boldsymbol {x} _ {i} \| _ {2} ^ {2} \\ = C ^ {2} \| \boldsymbol {X} \| _ {2} ^ {2} \| \boldsymbol {X} \| _ {F} ^ {2} \\ \leq C ^ {2} d \| \boldsymbol {X} \| _ {2} ^ {4} (29) \\ = C ^ {2} d \left\| \boldsymbol {X} \boldsymbol {X} ^ {\top} \right\| _ {2} ^ {2} (30) \\ = C ^ {2} d \left\| \sum_ {i = 1} ^ {k} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\top} \right\| _ {2} ^ {2}, (31) \\ \end{array}
$$
where Eq. (27) follows by the matrix operator norm inequality, Eq. (28) follows by the Cauchy-Schwarz inequality, Eq. (29) follows by  $\| X\| _F\leq \sqrt{d}\| X\| _2$  , Eq. (30) follows from  $\| X\| _2^2 = \left\| XX^\top \right\| _2$
The above result is tight. We can show that the lower bound of  $\left\| \sum_{i=1}^{k} \boldsymbol{x}_i \boldsymbol{x}_i^\top \boldsymbol{\theta}_i \right\|_2$  under the conditions in the lemma is exactly  $C \sqrt{d} \left\| \sum_{i=1}^{k} \boldsymbol{x}_i \boldsymbol{x}_i^\top \right\|_2$ . Specifically, let  $k = 2$ ,  $C = 1$ ,  $d = 2$ ,  $\boldsymbol{x}_1 = [0,1]^\top$ ,  $\boldsymbol{x}_2 = [1,0]^\top$ ,  $\boldsymbol{\theta}_1 = [1,0]^\top$ ,  $\boldsymbol{\theta}_2 = [0,1]^\top$ , then we have  $\left\| \sum_{i=1}^{2} \boldsymbol{x}_i \boldsymbol{x}_i^\top \boldsymbol{\theta}_i \right\|_2 = \left\| [1,1]^\top \right\|_2 = \sqrt{2}$ , and  $C \sqrt{d} \left\| \sum_{i=1}^{2} \boldsymbol{x}_i \boldsymbol{x}_i^\top \right\|_2 = 1 \times \sqrt{2} \times \left\| \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \right\|_2 = \sqrt{2}$ . Therefore, we have that the upper bound given in Lemma G.1 matches the lower bound.
We are now ready to prove the key Lemma 5.7 with the above Lemma G.1.
At any  $t \geq T_0$ , if the current partition is a "good partition", and  $\overline{V}_t \notin \mathcal{V}$ , then for all  $\boldsymbol{x}_a \in \mathbb{R}^d$ ,  $\| \boldsymbol{x}_a \|_2 \leq 1$ , with probability at least  $1 - \delta$ :
$$
\begin{array}{l} \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{M}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} \left(\boldsymbol {\theta} _ {i _ {s}} - \boldsymbol {\theta} _ {i _ {t}}\right) \right| \leq \| \boldsymbol {x} _ {a} \| _ {2} \left\| \overline {{M}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} \left(\boldsymbol {\theta} \mathbf {i} _ {s} - \boldsymbol {\theta} _ {i _ {t}}\right) \right\| _ {2} (32) \\ \leq \left\| \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \right\| _ {2} \left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} \left(\boldsymbol {\theta} _ {i _ {s}} - \boldsymbol {\theta} _ {i _ {t}}\right) \right\| _ {2} (33) \\ \leq 2 \epsilon_ {*} \sqrt {\frac {2 d}{\bar {\lambda} _ {x}}} \times \left\| \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \right\| _ {2} \left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \bar {V} _ {t}}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} \right\| _ {2} (34) \\ \leq 2\epsilon_{*}\sqrt{\frac{2d}{\tilde{\lambda}_{x}}}\times \frac{\lambda_{max}(\sum_{\substack{s\in[t - 1]\\ i_{s}\in\overline{V}_{t}}} \boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}})}{\lambda_{\min}(\overline{M}_{\overline{V}_{t},t - 1})} \\ \leq 2 \epsilon_ {*} \sqrt {\frac {2 d}{\tilde {\lambda} _ {x}}} \times \frac {T _ {\bar {V} _ {t} , t - 1}}{2 T _ {\bar {V} _ {t} , t - 1} \tilde {\lambda} _ {x} + \lambda} (35) \\ \leq \frac {\epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}}, \\ \end{array}
$$
where Eq.(32) follows by the Cauchy-Schwarz inequality, Eq.(33) follows from the inequality of matrix's operator norm, Eq.(34) follows from the fact that in a "good partition",  $\| \pmb{\theta}_{i_t} - \pmb{\theta}_l\|_2 \leq 2\epsilon_*\sqrt{\frac{2}{\lambda_x}}, \forall l \in \overline{V}_t$  and Lemma G.1, Eq.(35) follows by Eq.(47) with probability  $\geq 1 - \delta$ .
H Lemma H.1 of the sufficient time  $T_0$  and its proof
The following lemma gives a sufficient time  $T_0$  for the algorithm to get a "good partition".
Lemma H.1. With the carefully designed edge deletion rule, after
$$
\begin{array}{l} T _ {0} \triangleq 1 6 u \log (\frac {u}{\delta}) + 4 u \max \max \{\frac {8 d}{\tilde {\lambda} _ {x} (\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}) ^ {2}} \log (\frac {u}{\delta}), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log (\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}) \} \\ = O \left(u \left(\frac {d}{\tilde {\lambda} _ {x} (\gamma_ {1} - \zeta) ^ {2}} + \frac {1}{\tilde {\lambda} _ {x} ^ {2}}\right) \log \frac {1}{\delta}\right) \\ \end{array}
$$
rounds, with probability at least  $1 - 3\delta$  for some  $\delta \in (0,\frac{1}{3})$ , RCLUMB can always get a "good partition".
Below is the detailed proof of Lemma H.1.
Proof. We first prove the following result:
With probability at least  $1 - \delta$  for some  $\delta \in (0,1)$ , at any  $t\in [T]$ :
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right) + \epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {\lambda + \lambda_ {\operatorname* {m i n}} (\boldsymbol {M} _ {i , t})}}, \forall i \in \mathcal {U}, \tag {36}
$$
where  $\beta (T_{i,t},\frac{\delta}{u})\triangleq \sqrt{\lambda} +\sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{i,t}}{\lambda d})}$
$$
\begin{array}{l} \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} = \left(\sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} + \lambda I\right) ^ {- 1} \left(\sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \left(\boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta} ^ {j (i)} + \boldsymbol {\epsilon} _ {a _ {s}} ^ {i _ {s}, s} + \eta_ {s}\right)\right) - \boldsymbol {\theta} ^ {j (i)} \tag{37} \\ = (\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top} + \lambda \boldsymbol {I})^{-1}[(\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top} + \lambda \boldsymbol {I})\boldsymbol{\theta}^{j(i)} - \lambda \boldsymbol{\theta}^{j(i)} + \sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\eta_{s}] - \boldsymbol{\theta}^{j(i)} \\ = -\lambda \tilde{M}_{i,t}^{-1}\boldsymbol{\theta}^{j(i)} + \tilde{M}_{i,t}^{-1}\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \tilde{M}_{i,t}^{-1}\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\eta_{s}  , \\ \end{array}
$$
where we denote  $\tilde{M}_{i,t} = M_{i,t} + \lambda I$ , and Eq.(37) holds by definition.
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \lambda \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} + \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {\epsilon} _ {a _ {s}} ^ {i _ {s}, s} \right\| _ {2} + \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2}. \tag{38}
$$
We then bound the three terms in Eq.(38) one by one. For the first term:
$$
\lambda \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \lambda \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- \frac {1}{2}} \right\| _ {2} ^ {2} \left\| \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\sqrt {\lambda}}{\sqrt {\lambda_ {\min} (\tilde {\boldsymbol {M}} _ {i , t})}}, \tag {39}
$$
where we use the Cauchy-Schwarz inequality, the inequality for the operator norm of matrices, and the fact that  $\lambda_{\min}(\tilde{M}_{i,t})\geq \lambda$
For the second term in Eq.(38):
$$
\begin{array}{l} \left\| \tilde{M}_{i,t}\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\epsilon_{a_{s}}^{i_{s},s}\right\|_{2} = \max_{\boldsymbol {x}\in S^{d - 1}}\sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}^{\top}\tilde{M}_{i,t}^{-1}\boldsymbol{x}_{a_{s}}\epsilon_{a_{s}}^{i_{s},s} \\ \leq \max_{\boldsymbol {x}\in S^{d - 1}}\sum_{\substack{s\in \{t\} \\ i_{s} = i}}\left|\boldsymbol{x}^{\top}\tilde{\boldsymbol{M}}_{i,t}^{-1}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},\boldsymbol{s}}\right| \\ \leq \max  _ {\boldsymbol {x} \in S ^ {d - 1}} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \left| \boldsymbol {x} ^ {\top} \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right| \| \boldsymbol {\epsilon} _ {a _ {s}} ^ {i _ {s}, s} \| _ {\infty} (40) \\ \leq \epsilon_{*}\max_{\boldsymbol {x}\in S^{d - 1}}\sum_{\substack{s\in [t]\\ s_{s} = i}}\left|\boldsymbol{x}^{\top}\tilde{M}_{i,t}^{-1}\boldsymbol{x}_{a_{s}}\right| \\ \leq \epsilon_ {*} \max  _ {\boldsymbol {x} \in S ^ {d - 1}} \sqrt {\sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} 1 \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \left| \boldsymbol {x} ^ {\top} \tilde {\boldsymbol {M}} _ {i , t} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right| ^ {2}} (41) \\ \leq \epsilon_ {*} \sqrt {T _ {i , t}} \sqrt {\max  _ {\boldsymbol {x} \in S ^ {d - 1}} \boldsymbol {x} ^ {\top} \tilde {\boldsymbol {M}} _ {i , t} ^ {- 1} \boldsymbol {x}} (42) \\ = \frac {\epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {\lambda_ {\operatorname* {m i n}} (\tilde {M} _ {i , t})}}, (43) \\ \end{array}
$$
where we denote  $S^{d - 1} = \{\pmb{x} \in \mathbb{R}^d : \| \pmb{x} \|_2 = 1\}$ , Eq.(40) follows from Holder's inequality, Eq.(41) follows by the Cauchy-Schwarz inequality, Eq.(42) holds because  $\tilde{M}_{i,t} \succeq \sum_{\substack{s \in [t] \\ i_s = i}} x_{a_s} x_{a_s}^\top$ , Eq.(43) follows from the Courant-Fischer theorem.
For the last term in Eq.(38)
$$
\begin{array}{l} \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2} \leq \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- \frac {1}{2}} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2} \left\| \tilde {\boldsymbol {M}} _ {i, t} ^ {- \frac {1}{2}} \right\| _ {2} (44) \\ = \frac {\left\| \sum_ {s \in [ t ]} x _ {a _ {s}} \eta_ {s} \right\| _ {\tilde {M} _ {i , t} ^ {- 1}}}{\sqrt {\lambda_ {\operatorname* {m i n}} (\tilde {M} _ {i , t})}}, (45) \\ \end{array}
$$
where Eq.(44) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, and Eq.(45) follows by the Courant-Fischer theorem.
Following Theorem 1 in [1], with probability at least  $1 - \delta$  for some  $\delta \in (0,1)$ , for any  $i\in \mathcal{U}$ , we have:
$$
\begin{array}{l} \left\| \sum_{\substack{s\in [t]\\ i_{s} = i}}\boldsymbol{x}_{a_{s}}\eta_{s}\right\|_{\tilde{M}_{i,t}^{-1}}\leq \sqrt{2\log(\frac{u}{\delta}) + \log(\frac{\det(\tilde{M}_{i,t})}{\det(\lambda I)})} \\ \leq \sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}, \tag {46} \\ \end{array}
$$
where  $\operatorname{det}(M)$  denotes the determinant of matrix  $M$ , Eq.(46) is because  $\operatorname{det}(\tilde{M}_{i,t}) \leq \left(\frac{\operatorname{trace}(\lambda I + \sum_{s \in [t]} x_{a_s} x_{a_s}^\top)}{d}\right)^d \leq \left(\frac{\lambda d + T_{i,t}}{d}\right)^d$ , and  $\operatorname{det}(\lambda I) = \lambda^d$ .
Plugging Eq.(46) into Eq. (45), then plugging Eq. (39), Eq.(43) and Eq.(45) into Eq.(38), we can get that Eq.(73) holds with probability  $\geq 1 - \delta$ .
Then, with the item regularity assumption stated in Assumption 3.3, the technical Lemma J.1, together with Lemma 7 in [25], with probability at least  $1 - \delta$ , for a particular user  $i$ , at any  $t$  such that  $T_{i,t} \geq \frac{16}{\tilde{\lambda}_x^2} \log \left( \frac{8d}{\tilde{\lambda}_x^2 \delta} \right)$ , we have:
$$
\lambda_ {\min } \left(\tilde {M} _ {i, t}\right) \geq 2 \tilde {\lambda} _ {x} T _ {i, t} + \lambda . \tag {47}
$$
Based on the above reasoning, we have: if  $T_{i,t} \geq \frac{16}{\bar{\lambda}_x^2} \log \left( \frac{8d}{\bar{\lambda}_x^2 \delta} \right)$ , then with probability  $\geq 1 - 2\delta$ , we have:
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right) + \epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {\lambda_ {\operatorname* {m i n}} (\tilde {\boldsymbol {M}} _ {i , t})}} \\ \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right) + \epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {2 \tilde {\lambda} _ {x} T _ {i , t} + \lambda}} \\ \leq \frac {\sqrt {\lambda} + \sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}}{\sqrt {2 \tilde {\lambda} _ {x} T _ {i , t} + \lambda}} + \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}, \tag {48} \\ \end{array}
$$
for any  $i\in \mathcal{U}$
$$
\frac {\sqrt {\lambda} + \sqrt {2 \log \left(\frac {u}{\delta}\right) + d \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right)}}{\sqrt {2 \tilde {\lambda} _ {x} T _ {i , t} + \lambda}} + \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} <   \frac {\gamma_ {1}}{4}, \tag {49}
$$
$$
\frac {\sqrt {\lambda} + \sqrt {2 \log \left(\frac {u}{\delta}\right) + d \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right)}}{\sqrt {2 \tilde {\lambda} _ {x} T _ {i , t} + \lambda}} <   \frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}, \tag {50}
$$
where  $\gamma_{1}$  is given in Definition 5.1.
Assume  $\lambda \leq 2\log \left(\frac{u}{\delta}\right) + d\log \left(1 + \frac{T_{i,t}}{\lambda d}\right)$ , which is typically held, then a sufficient condition for Eq. (50) is:
$$
\frac {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}{2 \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {1}{4} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}. \tag {51}
$$
To satisfy the condition in Eq.(51), it is sufficient to show
$$
\frac {2 \log (\frac {u}{\delta})}{2 \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {1}{8} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2} \tag {52}
$$
$$
\frac {d \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right)}{2 \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {1}{8} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}. \tag {53}
$$
From Eq.(52), we can get:
$$
T _ {i, t} \geq \frac {8 \log \left(\frac {u}{\delta}\right)}{\tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}}. \tag {54}
$$
Following Lemma 9 in [25], we can get the following sufficient condition for Eq.(53):
$$
T _ {i, t} \geq \frac {8 d \log \left(\frac {4}{\lambda \tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}}\right)}{\tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}}. \tag {55}
$$
Assume  $\frac{u}{\delta} \geq \frac{4}{\lambda \tilde{\lambda}_x \left( \frac{\gamma_1}{4} - \epsilon_* \sqrt{\frac{1}{2\tilde{\lambda}_x}} \right)^2}$ , which is typically held, we can get that
$$
T _ {i, t} \geq \frac {8 d}{\tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}} \log \left(\frac {u}{\delta}\right) \tag {56}
$$
is a sufficient condition for Eq.(49). Together with the condition that  $T_{i,t} \geq \frac{16}{\tilde{\lambda}_x^2} \log \left( \frac{8d}{\tilde{\lambda}_x^2 \delta} \right)$ , we can get that if
$$
T _ {i, t} \geq \max  \left\{\frac {8 d}{\tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}} \log \left(\frac {u}{\delta}\right), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log \left(\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}\right) \right\}, \forall i \in \mathcal {U}, \tag {57}
$$
then with probability  $\geq 1 - 2\delta$
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} <   \frac {\gamma_ {1}}{4}, \forall i \in \mathcal {U}.
$$
By Lemma 8 in [25], and Assumption 3.2 of user arrival uniformness, we have that for all
$$
t \geq T _ {0} \triangleq 1 6 u \log \left(\frac {u}{\delta}\right) + 4 u \max  \left\{\frac {8 d}{\tilde {\lambda} _ {x} \left(\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2}} \log \left(\frac {u}{\delta}\right), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log \left(\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}\right) \right\}, \tag {58}
$$
with probability at least  $1 - \delta$ , condition in Eq.(57) is satisfied.
Therefore we have that for all  $t \geq T_0$ , with probability  $\geq 1 - 3\delta$ :
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} <   \frac {\gamma_ {1}}{4}, \forall i \in \mathcal {U}. \tag {59}
$$
Next, we show that with Eq.(59), we can get that the RCLUMB keeps a "good partition". First, if we delete the edge  $(i,l)$ , then user  $i$  and user  $j$  belong to different ground-truth clusters, i.e.,  $\|\pmb{\theta}_i - \pmb{\theta}_l\|_2 > 0$ . This is because by the deletion rule of the algorithm, the concentration bound, and triangle inequality,  $\|\pmb{\theta}_i - \pmb{\theta}_l\|_2 = \left\|\pmb{\theta}^{j(i)} - \pmb{\theta}^{j(l)}\right\|_2 \geq \left\|\hat{\pmb{\theta}}_{i,t} - \hat{\pmb{\theta}}_{l,t}\right\|_2 - \left\|\pmb{\theta}^{j(l)} - \pmb{\theta}_{l,t}\right\|_2 -$
$\left\| \pmb{\theta}^{j(i)} - \pmb{\theta}_{i,t}\right\|_2 > 0$ . Second, we show that if  $\|\pmb{\theta}_i - \pmb{\theta}_l\| \geq \gamma_1 > 2\epsilon_*\sqrt{\frac{2}{\tilde{\lambda}_x}}$ , the RCLUMB algorithm will delete the edge  $(i,l)$ . This is because if  $\|\pmb{\theta}_i - \pmb{\theta}_l\| \geq \gamma_1$ , then by the triangle inequality, and  $\left\| \hat{\pmb{\theta}}_{i,t} - \pmb{\theta}^{j(i)}\right\|_2 < \frac{\gamma_1}{4}$ ,  $\left\| \hat{\pmb{\theta}}_{l,t} - \pmb{\theta}^{j(l)}\right\|_2 < \frac{\gamma_1}{4}$ ,  $\pmb{\theta}_i = \pmb{\theta}^{j(i)}$ ,  $\pmb{\theta}_l = \pmb{\theta}^{j(l)}$ , we have  $\left\| \hat{\pmb{\theta}}_{i,t} - \hat{\pmb{\theta}}_{l,t}\right\|_2 \geq \left\| \pmb{\theta}_i - \pmb{\theta}_l\right\| - \left\| \hat{\pmb{\theta}}_{i,t} - \pmb{\theta}^{j(i)}\right\|_2 - \left\| \hat{\pmb{\theta}}_{l,t} - \pmb{\theta}^{j(l)}\right\|_2 > \gamma_1 - \frac{\gamma_1}{4} - \frac{\gamma_1}{4} = \frac{\gamma_1}{2} > \frac{\sqrt{\lambda} + \sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{i,t}}{\lambda d})}}{\sqrt{\lambda + 2\tilde{\lambda}_xT_{i,t}}} + \epsilon_*\sqrt{\frac{1}{2\tilde{\lambda}_x}} + \frac{\sqrt{\lambda} + \sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{l,t}}{\lambda d})}}{\sqrt{\lambda + 2\tilde{\lambda}_xT_{l,t}}} + \epsilon_*\sqrt{\frac{1}{2\tilde{\lambda}_x}}$  which will trigger the deletion condition Line 10 in Algo.1.
From the above reasoning, we can get that at round  $t$ , any user within  $\overline{V}_t$  is  $\zeta$ -close to  $i_t$ , and all the users belonging to  $V_{j(i)}$  are contained in  $\overline{V}_t$ , which means the algorithm has done a "good partition" at  $t$  by Definition 5.5.
I Proof of Lemma 5.6
We prove the result in two situations: when  $\overline{V}_t\in \mathcal{V}$  and when  $\overline{V}_t\notin \mathcal{V}$ .
(1) Situation 1: for any  $t \geq T_0$  and  $\overline{V}_t \in \mathcal{V}$ , which means that the current user  $i_t$  is clustered completely correctly, i.e.,  $\overline{V}_t = V_{j(i_t)}$ , therefore  $\pmb{\theta}_l = \pmb{\theta}_{i_t}, \forall l \in \overline{V}_t$ , then we have: