SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \lambda \left\| M _ {i, t} ^ {\prime - 1} \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} + \left\| M _ {i, t} ^ {\prime - 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2} + \left\| M _ {i, t} ^ {\prime - 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {2}. \tag{9}
$$
We then bound the three terms in Eq.(9) one by one. For the first term:
$$
\lambda \left\| M _ {i, t} ^ {\prime - 1} \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \lambda \left\| M _ {i, t} ^ {\prime - \frac {1}{2}} \right\| _ {2} ^ {2} \left\| \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\sqrt {\lambda}}{\sqrt {\lambda_ {\operatorname* {m i n}} \left(M _ {i , t} ^ {\prime}\right)}}, \tag {10}
$$
where we use the Cauchy-Schwarz inequality, the inequality for the operator norm of matrices, and the fact that  $\lambda_{\min}(M_{i,t}^{\prime})\geq \lambda$
For the second term in Eq.(9), we have
$$
\begin{array}{l} \left\| M _ {i, t} ^ {\prime - 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2} \leq \left\| M _ {i, t} ^ {\prime - \frac {1}{2}} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {2} \left\| M _ {i, t} ^ {\prime - \frac {1}{2}} \right\| _ {2} (11) \\ = \frac {\left\| \sum_ {s \in [ t ]} w _ {i _ {s} , s} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {\boldsymbol {M} _ {i , t} ^ {\prime - 1}}}{\sqrt {\lambda_ {\operatorname* {m i n}} \left(\boldsymbol {M} _ {i , t} ^ {\prime}\right)}}, (12) \\ \end{array}
$$
where Eq.(11) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, and Eq.(12) follows by the Courant-Fischer theorem.
Let  $\tilde{\boldsymbol{x}}_s\triangleq \sqrt{w_{i_s,s}}\boldsymbol{x}_{a_s}$ ,  $\tilde{\eta}_s\triangleq \sqrt{w_{i_s,s}}\eta_s$ , then we have:  $\| \tilde{\boldsymbol{x}}_s\| _2\leq \left\| \sqrt{w_{i_s,s}}\right\| _2\| \boldsymbol{x}_{a_s}\| _2\leq 1$ ,  $\tilde{\eta}_s$  is still 1-sub-gaussian (since  $\eta_{s}$  is 1-sub-gaussian and  $\sqrt{w_{i_s,s}}\leq 1$ ),  $M_{i,t}^{\prime} = \lambda I + \sum_{\substack{s\in [t]\\ i_s = i}}\tilde{\boldsymbol{x}}_s\tilde{\boldsymbol{x}}_s^\top$ , and the nominator in Eq.(12) becomes  $\left\| \sum_{\substack{s\in [t]\\ i_s = i}}\tilde{\boldsymbol{x}}_s\tilde{\eta}_s\right\|_{M_{i,t}^{\prime -1}}$ . Then, following Theorem 1 in [1] and by union bound, with probability at least  $1 - \delta$  for some  $\delta \in (0,1)$ , for any  $i\in \mathcal{U}$ , we have:
$$
\begin{array}{l} \left\| \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s}\right\|_{M^{\prime -1}_{i,t}} = \left\| \sum_{\substack{s\in [t]\\ i_{s} = i}}\tilde{\boldsymbol{x}}_{s}\tilde{\eta}_{s}\right\|_{M^{\prime -1}_{i,t}} \\ \leq \sqrt {2 \log (\frac {u}{\delta}) + \log (\frac {\det  (M _ {i , t} ^ {\prime})}{\det  (\lambda I)})} \\ \leq \sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}, \tag {13} \\ \end{array}
$$
where  $\operatorname{det}(\cdot)$  denotes the determinant of matrix argument, Eq.(13) is because  $\operatorname{det}(M_{i,t}^{\prime}) \leq \left(\frac{\operatorname{trace}(\lambda I + \sum_{s\in[t]}\omega_{i,s}\boldsymbol{x}_{a_s}\boldsymbol{x}_{a_s}^\top)}{d}\right)^d \leq \left(\frac{\lambda d + T_{i,t}}{d}\right)^d$ , and  $\operatorname{det}(\lambda I) = \lambda^d$ .
For the third term in Eq.(9), we have
$$
\begin{array}{l} \left\| M _ {i, t} ^ {\prime - 1} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {2} \leq \left\| M _ {i, t} ^ {\prime - \frac {1}{2}} \sum_ {\substack {s \in [ t ] \\ i _ {s} = i}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {2} \left\| M _ {i, t} ^ {\prime - \frac {1}{2}} \right\| _ {2} (14) \\ = \frac {\left\| \sum_ {s \in [ t ]} w _ {i _ {s} , s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {\boldsymbol {M} _ {i , t} ^ {\prime - 1}}}{\sqrt {\lambda_ {\operatorname* {m i n}} \left(\boldsymbol {M} _ {i , t} ^ {\prime}\right)}} (15) \\ \leq \frac{\sum_{\substack{s\in[t]\\ i_{s} = i}}|c_{s}|w_{i,s}\left\|\boldsymbol{x}_{a_{s}}\right\|_{M^{' - 1}_{i,t}}}{\sqrt{\lambda_{\min}(\boldsymbol{M^{'}}_{i,t})}} \\ \leq \frac {\alpha C}{\sqrt {\lambda_ {\min} \left(M _ {i , t} ^ {\prime}\right)}} (16) \\ \end{array}
$$
where Eq.(14) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, Eq.(15) follows by the Courant-Fischer theorem, and Eq.(16) is because by definition  $w_{i,s} \leq \frac{\alpha}{\|\pmb{x}_{a_s}\|_{M_{i,s}^{\prime - 1}}} \leq \frac{\alpha}{\|\pmb{x}_{a_s}\|_{M_{i,t}^{\prime - 1}}}$  (since  $M_{i,t}^{\prime} \succeq M_{i,s}^{\prime}$ ,  $M_{i,s}^{\prime - 1} \succeq M_{i,t}^{\prime - 1}$ ,  $\| \pmb{x}_{a_s}\|_{M_{i,s}^{\prime - 1}} \geq \| \pmb{x}_{a_s}\|_{M_{i,t}^{\prime - 1}}$ ),  $\sum_{t = 1}^{T}|c_t| \leq C$ ).
Combining the above bounds of these three terms, we can get that Eq.(8) holds.
We then prove the following technical lemma.
Lemma 6. Under Assumption 3, at any time  $t$ , for any fixed unit vector  $\pmb{\theta} \in \mathbb{R}^d$
$$
\mathbb {E} _ {t} \left[ \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {a _ {t}}\right) ^ {2} \mid \left| \mathcal {A} _ {t} \right| \right] \geq \tilde {\lambda} _ {x} \triangleq \int_ {0} ^ {\lambda_ {x}} \left(1 - e ^ {- \frac {\left(\lambda_ {x} - x\right) ^ {2}}{2 \sigma^ {2}}}\right) ^ {K} d x, \tag {17}
$$
where  $K$  is the upper bound of  $|\mathcal{A}_t|$  for any  $t$ .
Proof. The proof of this lemma mainly follows the proof of Claim 1 in [8], but with more careful analysis, since their assumption on the arm generation distribution is more stringent than our Assumption 3 by putting more restrictions on the variance upper bound  $\sigma^2$  (specifically, they require  $\sigma^2 \leq \frac{\lambda^2}{8\log(4K)}$ ).
Denote the feasible arms at round  $t$  by  $\mathcal{A}_t = \{\pmb{x}_{t,1},\pmb{x}_{t,2},\dots,\pmb{x}_{t,|\mathcal{A}_t|}\}$ . Consider the corresponding i.i.d. random variables  $\theta_{i} = (\pmb{\theta}^{\top}\pmb{x}_{t,i})^{2} - \mathbb{E}_{t}[(\pmb{\theta}^{\top}\pmb{x}_{t,i})^{2}||\mathcal{A}_{t}|], i = 1,2,\dots,|\mathcal{A}_{t}|$ . By Assumption 3,  $\theta_{i}$  s are sub-Gaussian random variables with variance bounded by  $\sigma^2$ . Therefore, for any  $\alpha >0$  and any  $i\in ||\mathcal{A}_t||$ , we have:
where we use  $\mathbb{P}_t(\cdot)$  to be the shorthand for the conditional probability  $\mathbb{P}(\cdot | (i_1, \mathcal{A}_1, r_1), \ldots, (i_{t-1}, \mathcal{A}_{t-1}, r_{t-1}), i_t)$ .
By Assumption 3, we can also get that  $\mathbb{E}_t[(\pmb{\theta}^\top \pmb{x}_{t,i})^2 ||\mathcal{A}_t| = \mathbb{E}_t[\pmb{\theta}^\top \pmb{x}_{t,i}\pmb{x}_{t,i}^\top \pmb{\theta}||\mathcal{A}_t|]\geq$ $\lambda_{\min}(\mathbb{E}_{\pmb{x}\sim \rho}[\pmb {x}\pmb{x}^{\top}])\geq \lambda_{x}$  . With these inequalities above, we can get
$$
\mathbb {P} _ {t} (\min _ {i = 1, \dots , | \mathcal {A} _ {t} |} (\pmb {\theta} ^ {\top} \pmb {x} _ {t, i}) ^ {2} \geq \lambda_ {x} - \alpha | | \mathcal {A} _ {t} |) \geq (1 - e ^ {- \frac {\alpha^ {2}}{2 \sigma^ {2}}}) ^ {K}.
$$
Therefore, we can get
$$
\begin{array}{l} \mathbb {E} _ {t} [ (\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {a _ {t}}) ^ {2} | | \mathcal {A} _ {t} | ] \geq \mathbb {E} _ {t} [ \min  _ {i = 1, \dots , | \mathcal {A} _ {t} |} (\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {t, i}) ^ {2} | | \mathcal {A} _ {t} | ] \\ \geq \int_ {0} ^ {\infty} \mathbb {P} _ {t} \left(\min  _ {i = 1, \dots , | \mathcal {A} _ {t} |} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {t, i}\right) ^ {2} \geq x \mid | \mathcal {A} _ {t} |\right) d x \\ \geq \int_ {0} ^ {\lambda_ {x}} (1 - e ^ {- \frac {(\lambda_ {x} - x) ^ {2}}{2 \sigma^ {2}}}) ^ {K} d x \triangleq \tilde {\lambda} _ {x} \\ \end{array}
$$
Note that  $w_{i,s} = \min \{1,\frac{\alpha}{\|\pmb{x}_{a_s}\|_{M_{i,t}^{' - 1}}}\}$ , and we have
$$
\frac {\alpha}{\| \boldsymbol {x} _ {a _ {s}} \| _ {\boldsymbol {M} _ {i , t} ^ {\prime - 1}}} = \frac {\alpha}{\sqrt {\boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {M} _ {i , t} ^ {\prime - 1} \boldsymbol {x} _ {a _ {s}}}} \geq \frac {\alpha}{\sqrt {\lambda_ {\min} (\boldsymbol {M} _ {i , t} ^ {\prime - 1})}} = \alpha \sqrt {\lambda_ {\min} (\boldsymbol {M} _ {i , t} ^ {\prime})} \geq \alpha \sqrt {\lambda}.
$$
Since  $\alpha \sqrt{\lambda} < 1$  typically holds, we have  $w_{i,s}\geq \alpha \sqrt{\lambda}$
Then, with the item regularity assumption stated in Assumption 3, the technical Lemma 6, together with Lemma 7 in [19], with probability at least  $1 - \delta$ , for a particular user  $i$ , at any  $t$  such that  $T_{i,t} \geq \frac{16}{\lambda_x^2} \log \left(\frac{8d}{\lambda_x^2 \delta}\right)$ , we have:
$$
\lambda_ {\min } \left(M _ {i, t} ^ {\prime}\right) \geq 2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i, t} + \lambda . \tag {18}
$$
With this result, together with Eq.(8), we can get that for any  $t$  such that  $T_{i,t} \geq \frac{16}{\lambda_x^2} \log \left(\frac{8d}{\lambda_x^2 \delta}\right)$ , with probability at least  $1 - \delta$  for some  $\delta \in (0,1)$ ,  $\forall i \in \mathcal{U}$ , we have:
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right)}{\sqrt {\lambda_ {\min} \left(\boldsymbol {M} _ {i , t} ^ {\prime}\right)}} \\ \leq \frac {\beta (T _ {i , t} , \frac {\delta}{u})}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t} + \lambda}} \\ \leq \frac {\beta (T _ {i , t} , \frac {\delta}{u})}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}}} \\ = \frac {\sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})} + \sqrt {\lambda} + \alpha C}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}}}. \tag {19} \\ \end{array}
$$
Then, we want to find a sufficient time  $T_{i,t}$  for a fixed user  $i$  such that
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} <   \frac {\gamma}{4}. \tag {20}
$$
To do this, with Eq.(19), we can get it by letting
$$
\frac {\sqrt {\lambda}}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}}} <   \frac {\gamma}{1 2}, \tag {21}
$$
$$
\frac {\alpha C}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}}} <   \frac {\gamma}{1 2}, \tag {22}
$$
$$
\frac {\sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}}{\sqrt {2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}}} <   \frac {\gamma}{1 2}. \tag {23}
$$
For Eq.(21), we can get
$$
T _ {i, t} > \frac {7 2 \sqrt {\lambda}}{\alpha \gamma^ {2} \tilde {\lambda} _ {x}}. \tag {24}
$$
For Eq.(22), we can get
$$
T _ {i, t} > \frac {7 2 \alpha C ^ {2}}{\gamma^ {2} \sqrt {\lambda} \tilde {\lambda} _ {x}}. \tag {25}
$$
For Eq.(23), we have
$$
\frac {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}{2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {\gamma^ {2}}{1 4 4}. \tag {26}
$$
Then it is sufficient to get Eq.(26) if the following holds
$$
\frac {2 \log (\frac {u}{\delta})}{2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {\gamma^ {2}}{2 8 8}, \tag {27}
$$
$$
\frac {d \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right)}{2 \alpha \sqrt {\lambda} \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {\gamma^ {2}}{2 8 8}. \tag {28}
$$
For Eq.(27), we can get
$$
T _ {i, t} > \frac {2 8 8 \log \left(\frac {u}{\delta}\right)}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \tag {29}
$$
For Eq.(28), we can get
$$
T _ {i, t} > \frac {1 4 4 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right). \tag {30}
$$
Following Lemma 9 in [19], we can get the following sufficient condition for Eq.(30):
$$
T _ {i, t} > \frac {2 8 8 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log \left(\frac {2 8 8}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}}\right). \tag {31}
$$
Then, since typically  $\frac{u}{\delta} > \frac{288}{\gamma^2 \alpha \sqrt{\lambda} \tilde{\lambda}_x}$ , we can get the following sufficient condition for Eq.(29) and Eq.(31)
$$
T _ {i, t} > \frac {2 8 8 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log \left(\frac {u}{\delta}\right). \tag {32}
$$
Together with Eq.(24), Eq.(25), and the condition for Eq.(18) we can get the following sufficient condition for Eq.(20) to hold
$$
T _ {i, t} > \max  \left\{\frac {2 8 8 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log (\frac {u}{\delta}), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log (\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}), \frac {7 2 \sqrt {\lambda}}{\alpha \gamma^ {2} \tilde {\lambda} _ {x}}, \frac {7 2 \alpha C ^ {2}}{\gamma^ {2} \sqrt {\lambda} \tilde {\lambda} _ {x}} \right\}. \tag {33}
$$
Then, with Assumption 2 on the uniform arrival of users, following Lemma 8 in [19], and by union bound, we can get that with probability at least  $1 - \delta$ , for all
$$
t \geq T _ {0} \triangleq 1 6 u \log \left(\frac {u}{\delta}\right) + 4 u \max  \left\{\frac {2 8 8 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log \left(\frac {u}{\delta}\right), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log \left(\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}\right), \frac {7 2 \sqrt {\lambda}}{\alpha \gamma^ {2} \tilde {\lambda} _ {x}}, \frac {7 2 \alpha C ^ {2}}{\gamma^ {2} \sqrt {\lambda} \tilde {\lambda} _ {x}} \right\}, \tag {34}
$$
Eq.(32) holds for all  $i \in \mathcal{U}$ , and therefore Eq.(20) holds for all  $i \in \mathcal{U}$ . With this, we can show that RCLUB-WCU will cluster all the users correctly after  $T_0$ . First, if RCLUB-WCU deletes the edge  $(i,l)$ , then user  $i$  and user  $j$  belong to different ground-truth clusters, i.e.,  $\|\pmb{\theta}_i - \pmb{\theta}_l\|_2 > 0$ . This is because by the deletion rule of the algorithm, the concentration bound, and triangle inequality,  $\|\pmb{\theta}_i - \pmb{\theta}_l\|_2 = \left\|\pmb{\theta}^{j(i)} - \pmb{\theta}^{j(l)}\right\|_2 \geq \left\|\hat{\pmb{\theta}}_{i,t} - \hat{\pmb{\theta}}_{l,t}\right\|_2 - \left\|\pmb{\theta}^{j(l)} - \pmb{\theta}_{l,t}\right\|_2 - \left\|\pmb{\theta}^{j(i)} - \pmb{\theta}_{i,t}\right\|_2 > 0$ . Second, we show that if  $\|\pmb{\theta}_i - \pmb{\theta}_l\| \geq \gamma$ , RCLUB-WCU will delete the edge  $(i,l)$ . This is because if  $\|\pmb{\theta}_i - \pmb{\theta}_l\| \geq \gamma$ , then by the triangle inequality, and  $\left\|\hat{\pmb{\theta}}_{i,t} - \pmb{\theta}^{j(i)}\right\|_2 < \frac{\gamma}{4}$ ,  $\left\|\hat{\pmb{\theta}}_{l,t} - \pmb{\theta}^{j(l)}\right\|_2 < \frac{\gamma}{4}$ ,  $\pmb{\theta}_i = \pmb{\theta}^{j(i)}, \pmb{\theta}_l = \pmb{\theta}^{j(l)}$ , we have  $\left\|\hat{\pmb{\theta}}_{i,t} - \hat{\pmb{\theta}}_{l,t}\right\|_2 \geq \left\|\pmb{\theta}_i - \pmb{\theta}_l\right\| - \left\|\hat{\pmb{\theta}}_{i,t} - \pmb{\theta}^{j(i)}\right\|_2 - \left\|\hat{\pmb{\theta}}_{l,t} - \pmb{\theta}^{j(l)}\right\|_2 > \gamma - \frac{\gamma}{4} - \frac{\gamma}{4} = \frac{\gamma}{2} > \frac{\sqrt{\lambda} + \sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{i,t}}{\lambda d})}}{\sqrt{\lambda + 2\tilde{\lambda}_x T_{i,t}}} + \frac{\sqrt{\lambda} + \sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{l,t}}{\lambda d})}}{\sqrt{\lambda + 2\tilde{\lambda}_x T_{l,t}}}$ , which will trigger the deletion condition Line 10 in Algo.1.
B Proof of Lemma 2
After  $T_{0}$ , if the clustering structure is correct, i.e.,  $V_{t} = V_{j(i_{t})}$ , then we have
$$
\begin{array}{l} \hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}} = M _ {V _ {t}, t - 1} ^ {- 1} \boldsymbol {b} _ {V _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}} \\ = (\lambda \boldsymbol {I} + \sum_ {\substack {s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}})^{-1}(\sum_ {\substack {s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}r_{s}) - \boldsymbol{\theta}_{i_{t}} \\ = \left(\lambda I + \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top}\right) ^ {- 1} \left(\sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \left(\boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} + \eta_ {s} + c _ {s}\right)\right) - \boldsymbol {\theta} _ {i _ {t}} \tag{35} \\ = (\lambda \boldsymbol {I} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}})^{-1}\Bigg(\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}} (w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})\boldsymbol{\theta}_{i_{t}} - \lambda \boldsymbol{\theta}_{i_{t}} \\ \left. + \sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}c_{s})\right) - \boldsymbol{\theta}_{i_{t}} \\ = -\lambda M^{\prime -1}_{V_{t},t - 1}\boldsymbol{\theta}_{i_{t}} - M^{\prime -1}_{V_{t},t - 1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s} + M^{\prime -1}_{V_{t},t - 1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}c_{s}  , \\ \end{array}
$$
where we denote  $M_{V_t,t - 1}' = M_{V_t,t - 1} + \lambda I$ , and Eq.(35) is because  $V_{t} = V_{j(i_{t})}$  thus  $\theta_{i_s} = \theta_{i_t}, \forall i_s \in V_t$ .
$$
\begin{array}{l} \left|\boldsymbol{x}_{a}^{\top}(\hat{\boldsymbol{\theta}}_{V_{t},t - 1} - \boldsymbol{\theta}_{i_{t}})\right|\leq \lambda \left|\boldsymbol{x}_{a}^{\top}\boldsymbol{M}_{V_{t},t - 1}^{\prime -1}\boldsymbol{\theta}_{i_{t}}\right| + \left|\boldsymbol{x}_{a}^{\top}\boldsymbol{M}_{V_{t},t - 1}^{\prime -1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s}\right| + \left|\boldsymbol{x}_{a}^{\top}\boldsymbol{M}_{V_{t},t - 1}^{\prime -1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}c_{s}\right| \\ \leq \left\| \boldsymbol {x} _ {a} \right\| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {\prime - 1}} \left(\sqrt {\lambda} + \left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {\prime - 1}} + \left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {\prime - 1}}\right), \\ \end{array}
$$
where Eq.(36) is by Cauchy-Schwarz inequality, matrix operator inequality, and  $\left|\pmb{x}_a^\top M_{V_t,t-1}^{\prime-1}\pmb{\theta}_{i_t}\right| \leq \lambda\left\|\pmb{M}_{V_t,t-1}^{\prime-\frac{1}{2}}\right\|_2\|\pmb{\theta}_{i_t}\|_2 = \lambda\frac{1}{\sqrt{\lambda_{\min}(M_{V_t,t-1})}}\|\pmb{\theta}_{i_t}\|_2 \leq \sqrt{\lambda}$  since  $\lambda_{\min}(M_{V_t,t-1}) \geq \lambda$  and  $\|\pmb{\theta}_{i_t}\|_2 \leq 1$ .
Let  $\tilde{\pmb{x}}_s\triangleq \sqrt{w_{i_s,s}}\pmb{x}_{a_s}$ ,  $\tilde{\eta}_s\triangleq \sqrt{w_{i_s,s}}\eta_s$ , then we have:  $\| \tilde{\pmb{x}}_s\| _2\leq \| \sqrt{w_{i_s,s}}\| _2\| \pmb{x}_{a_s}\| _2\leq 1$ ,  $\tilde{\eta}_s$  is still 1-sub-gaussian (since  $\eta_{s}$  is 1-sub-gaussian and  $\sqrt{w_{i_s,s}}\leq 1$ ),  $M_{i,t}^{\prime} = \lambda I + \sum_{\substack{s\in [t]\\ i_s = i}}\tilde{\pmb{x}}_s\tilde{\pmb{x}}_s^\top$
and  $\left\| \sum_{\substack{s\in [t - 1]\\ i_s\in V_t}}w_{i_s,s}\boldsymbol{x}_{a_s}\eta_s\right\|_{M^{\prime -1}_{V_t,t - 1}}$  becomes  $\left\| \sum_{s\in [t]\atop i_s = i}\tilde{\boldsymbol{x}}_s\tilde{\eta}_s\right\|_{M^{\prime -1}_{V_t,t - 1}}$  . Then, following Theorem 1 in [1], with probability at least  $1 - \delta$  for some  $\delta \in (0,1)$ , we have:
$$
\begin{array}{l} \left\| \sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s}\right\|_{M^{\prime}_{V_{t},t - 1}^{-1}} = \left\| \sum_{\substack{s\in [t]\\ i_{s} = i}}\tilde{\boldsymbol{x}}_{s}\tilde{\eta}_{s}\right\|_{M^{\prime}_{V_{t},t - 1}^{-1}} \\ \leq \sqrt {2 \log (\frac {u}{\delta}) + \log (\frac {\det  (M _ {V _ {t} , t - 1} ^ {\prime})}{\det  (\lambda I)})} \\ \leq \sqrt {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T}{\lambda d})}, \tag {37} \\ \end{array}
$$
And for  $\left\| \sum_{\substack{s\in [t - 1]\\ i_s\in V_t}}w_{i_s,s}\pmb{x}_{a_s}c_s\right\|_{M_{V_t,t - 1}^{\prime -1}}$  , we have
$$
\left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \boldsymbol {x} _ {a _ {s}} c _ {s} \right\| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {\prime - 1}} \leq \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} | c _ {s} | \| \boldsymbol {x} _ {a _ {s}} \| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {\prime - 1}} \leq \alpha C, \tag{38}
$$
where we use  $\sum_{t=1}^{T}|c_t| \leq C$ ,  $w_{i_s,s} \leq \frac{\alpha}{\|\pmb{x}_{a_s}\|_{M_{i_s,t-1}'^{-1}}^{'-1}} \leq \frac{\alpha}{\|\pmb{x}_{a_s}\|_{M_{V_t,t-1}'^{-1}}^{'-1}}$ .
Plugging Eq.(38) and Eq.(37) into Eq.(36), together with Lemma 1, we can complete the proof of Lemma 2.
After  $T_{0}$ , we define event
$$
\mathcal {E} = \{\text {t h e a l g o r i t h m c l u s t e r s a l l t h e u s e r s c o r r e c t l y f o r a l l} t \geq T _ {0} \}. \tag {39}
$$
Then, with Lemma 1 and picking  $\delta = \frac{1}{T}$ , we have
$$
\begin{array}{l} R (T) = \mathbb {P} (\mathcal {E}) \mathbb {I} \{\mathcal {E} \} R (T) + \mathbb {P} (\overline {{\mathcal {E}}}) \mathbb {I} \{\overline {{\mathcal {E}}} \} R (T) \\ \leq \mathbb {I} \{\mathcal {E} \} R (T) + 4 \times \frac {1}{T} \times T \tag {40} \\ = \mathbb {I} \{\mathcal {E} \} R (T) + 4. \\ \end{array}
$$
Then it remains to bound  $\mathbb{I}\{\mathcal{E}\} R(T)$ . For the first  $T_0$  rounds, we can upper bound the regret in the first  $T_0$  rounds by  $T_0$ . After  $T_0$ , under event  $\mathcal{E}$  and by Lemma 2, we have that with probability at least  $1 - \delta$ , for any  $\boldsymbol{x}_a$ :
$$
\left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) \right| \leq \beta \| \boldsymbol {x} _ {\boldsymbol {a}} \| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {- 1}} \triangleq C _ {a, t}. \tag {41}
$$
Therefore, for the instantaneous regret  $R_{t}$  at round  $t$ , with  $\mathcal{E}$ , with probability at least  $1 - \delta$ , at  $\forall t \geq T_0$ :
$$
\begin{array}{l} R _ {t} = \boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} - \boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} \\ = \boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \left(\boldsymbol {\theta} _ {i _ {t}} - \hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1}\right) + \left(\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} + C _ {a _ {t} ^ {*}, t}\right) - \left(\boldsymbol {x} _ {a _ {t}} ^ {\top} \hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} + C _ {a _ {t}, t}\right) \tag {42} \\ + \boldsymbol {x} _ {a _ {t}} ^ {\top} (\hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}) + C _ {a _ {t}, t} - C _ {a _ {t} ^ {*}, t} \\ \leq 2 C _ {a _ {t}, t}, \\ \end{array}
$$
where the last inequality holds by the UCB arm selection strategy in Eq.(3) and Eq.(41).
Therefore, for  $\mathbb{I}\{\mathcal{E}\} R(T)$
$$
\begin{array}{l} \mathbb {I} \{\mathcal {E} \} R (T) \leq R \left(T _ {0}\right) + \mathbb {E} [ \mathbb {I} \{\mathcal {E} \} \sum_ {t = T _ {0} + 1} ^ {T} R _ {t} ] \\ \leq T _ {0} + 2 \mathbb {E} \left[ \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} C _ {a _ {t}, t} \right]. \tag {43} \\ \end{array}
$$
Then it remains to bound  $\mathbb{E}[\mathbb{I}\{\mathcal{E}\} \sum_{t = T_0 + 1}^T C_{a_t,t}]$ . For  $\sum_{t = T_0 + 1}^{T}C_{a_t,t}$ , we can distinguish it into two cases: