SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[37] Zhi Wang, Chicheng Zhang, and Kamalika Chaudhuri. Thompson sampling for robust transfer in multi-task bandits. arXiv preprint arXiv:2206.08556, 2022.
[38] Zhiyong Wang, Xutong Liu, Shuai Li, and John CS Lui. Efficient explorative key-term selection strategies for conversational contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10288-10295, 2023.
[39] Zhiyong Wang, Jize Xie, Tong Yu, Shuai Li, and John Lui. Online corrupted user detection and regret minimization. arXiv preprint arXiv:2310.04768, 2023.
[40] Junda Wu, Canzhe Zhao, Tong Yu, Jingyang Li, and Shuai Li. Clustering of conversational bandits for user preference learning and elicitation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2129-2139, 2021.
[41] Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 529-538, 2016.
[42] Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. arXiv preprint arXiv:1603.05359, 2016.
A More Discussions on Related Work
In this section, we will give more comparisons and discussions on some previous works that are related to our work to some extent.
There are some other works on bandits leveraging user (or task) relations, which have some relations with the clustering of bandits (CB) works to some extent, but are in different lines of research from CB, and are quite different from our work. First, besides CB, the work [41] also leverages user relations. Specifically, it utilizes a known user adjacency graph to share context and payoffs among neighbors, whereas in CB, the user relations are unknown and need to be learnt, thus the setting differs a lot from CB. Second, there are lines of works on multi-task learning [6, 10, 33, 8, 37, 36], meta-learning [35, 18, 7] and federated learning [32, 19], where multiple different tasks are solved jointly and share information. Note that all of these works do not assume an underlying unknown user clustering structure which needs to be inferred by the agent to speed up learning. For works on multi-task learning [6, 10, 33, 8, 37, 36], they assume the tasks are related but no user clustering structures, and to the best of our knowledge, none of them consider model misspecifications, thus differing a lot from ours. For some recent works on meta-learning [35, 18, 34], they propose general Bayesian hierarchical models to share knowledge across tasks, and design Thompson-Sampling-based algorithms to optimize the Bayes regret, which are quite different from the line of CB works, and differ a lot from ours. And additionally, as supported by the discussions in the works [7, 36], multi-task learning and meta-learning are different lines of research from CB. For the works on federated learning [32, 19], they consider the privacy and communication costs among multiple servers, whose setting is also very different from the previous CB works and our work.
Remark. Again, we emphasize that the goal of this work is to initialize the study of the important CBMUM problem, and propose general design ideas for dealing with model misspecifications in CB problems. Therefore, our study is based on fundamental models on CB [12, 27] and MLB [23], and the algorithm design ideas and theoretical analysis are pretty general. We leave incorporating the more recent model selection methods [31, 11] into our framework to address the unknown exact maximum model misspecification level as an interesting future work. It would also be interesting to consider incorporating our methods and ideas of tackling model misspecifications into the studies of multi-task learning, meta learning and federated learning.
B More Discussions on Assumptions
All the assumptions (Assumptions 3.1,3.2,3.3,3.4) in this work are natural and basically follow (or less stringent than) previous works on CB and MLB [12, 25, 27, 28, 23].
B.1 Less Strigent Assumption on on the Generating Distribution of Arm Vectors
We also make some contributions to relax a widely-used but stringent assumption on the generating distribution of arm vectors. Specifically, our Assumption 3.3 on item regularity relaxes the previous one used in previous CB works [12, 25, 27, 28] by removing the condition that the variance should be upper bounded by  $\frac{\lambda^2}{8\log(4|\mathcal{A}_t|)}$ . For technical details on this, please refer to the theoretical analysis and discussions in Appendix J.
B.2 Discussions on Assumption 3.4 about Bounded Misspecification Level
This assumption follows [23]. Note that this  $\epsilon_{*}$  can be an upper bound on the maximum misspecification level, not the exact maximum itself. In real-world applications, the deviations are usually small [14], and we can set a relatively big  $\epsilon_{*}$  (e.g., 0.2) to be the upper bound. Our experimental results support this claim. As shown in our experimental results on real-data case 2, even when  $\epsilon_{*}$  is unknown, our algorithms still perform well by setting  $\epsilon_{*} = 0.2$ . Some recent studies [31, 11] use model selection methods to theoretically deal with unknown exact maximum misspecification level in the single-user case, which is not the emphasis of this work. Additionally, the work [11] assumes that the learning agent has access to a regression oracle. And for the work [31], though their regret bound
is dependent on the exact maximum misspecification level that needs not to be known by the agent, an upper bound of the exact maximum misspecification level is still needed. We leave incorporating their methods to deal with unknown exact maximum misspecification level as an interesting future work.
B.3 Discussions on Assumption 3.2 about the Theoretical Results under General User Arrival Distributions
The uniform arrival in Assumption 3.2 follows previous CB works [12, 25, 28], it only affects the  $T_0$  term, which is the time after which the algorithm maintains a "good partition" and is of  $O(u\log T)$ . For an arbitrary arrival distribution,  $T_0$  becomes  $O(1 / p_{min}\log T)$ , where  $p_{min}$  is the minimal arrival probability of a user. And since it is a lower-order term (of  $O(\log T)$ ), it will not affect the main order of our regret upper bound which is of  $O(\epsilon_*T\sqrt{md\log T} +d\sqrt{mT}\log T)$ . The work [27] studies arbitrary arrivals and aims to remove the  $1 / p_{min}$  factor in this term, but their setting is different. They make an additional assumption that users in the same cluster not only have the same preference vector, but also the same arrival probability, which is different from our setting and other classic CB works [12, 25, 28] where we only assume users in the same cluster share the same preference vector.
C Highlight of the Theoretical Analysis
Our proof flow and methodologies are novel in clustering of bandits (CB), which are expected to inspire future works on model misspecifications and CB. The main challenge of the regret analysis in CBMUM is that due to the estimation inaccuracy caused by misspecifications, it is impossible to cluster all users exactly correctly, and it is highly non-trivial to bound the regret caused by "misclustering"  $\zeta$ -close users.
To the best of our knowledge, the common proof flow of previous CB works (e.g., [12, 25, 28]) can be summarized in two steps: The first is to prove a sufficient time  $T_0'$  after which the algorithms can cluster all users exactly correctly with high probability. Note that the inferred clustering structure remains static after  $T_0'$ , making the analysis easy. Second, after the correct static clustering, the regret can be trivially bounded by bounding  $m$  (number of underlying clusters) independent linear bandit algorithms, resulting in a  $O(d\sqrt{mT}\log T)$  regret.
The above common proof flow is straightforward in CB with perfectly linear models, but it would fail to get a non-vacuous regret bound for CBMUM. In CBMUM, it is impossible to learn an exactly correct static clustering structure with model misspecifications. In particular, we prove that we can only expect the algorithm to cluster  $\zeta$ -close users together rather than cluster all users exactly correctly. Therefore, the previous flow can not be applied to the more challenging CBMUM problem.
We do the following to address the challenges in obtaining a tight regret bound for CBMUM. With the carefully-designed novel key components of RCLUMB, we can prove a sufficient time  $T_0$  after which RCLUMB can get a "good partition" (Definition 5.5) with high probability, which means the cluster  $\overline{V_t}$  assigned to  $i_t$  contains all users in the same ground-truth cluster as  $i_t$ , and possibly some other  $i_t$ 's  $\zeta$ -close users. Intuitively, after  $T_0$ , the algorithm can leverage all the information from the users' ground-truth clusters but may misuse some information from other  $\zeta$ -close users with preference gaps up to  $\zeta$ , causing a regret of "misclustering"  $\zeta$ -close users. It is highly non-trivial to bound this part of regret, and the proof methods would be beneficial for future studies in CB in challenging cases when it is impossible to cluster all users exactly correctly. For details, please refer to the discussions "(ii) Bounding the term of misclustering it's  $\zeta$ -close users" in Section 5, the key Lemma 5.7 (Bound of error caused by misclustering), its proof and tightness discussion in Appendix G. Also, a more subtle analysis is needed to handle the time-varying inferred clustering structure since the "good partition" may change over time, whereas in the previous CB works, the clustering structure remains static after  $T_0'$ . For theoretical details on this, please refer to Appendix E.
D Discussions on why Trivially Combining Existing CB and MLB Works Could Not Achieve a Non-vacuous Regret Upper Bound
We consider discussing regret upper bounds for CB without considering misspecifications for three cases: (1) neither the clustering process nor the decision process considers misspecifications (previous
CB algorithms); (2) the decision process does not consider misspecifications; (3) the clustering process does not consider misspecifications.
For cases (1) and (2), the decision process could contribute to the leading regret. We consider the case where there are  $m$  underlying clusters, with each cluster's arrival being  $T / m$ , and the agent knows the underlying clustering structure. For this case, there exist some instances where the regret upper bound  $R(T)$  is strictly larger than  $\epsilon_{*}T\sqrt{m\log T}$  asymptotically in  $T$ . Formally, in the discussion of "Failure of unmodified algorithm" in Appendix E in [23], they give an example to show that in the single-user case, the regret  $R_{1}(T)$  of the classic linear bandit algorithms without considering misspecifications will have:  $\lim_{T\to +\infty}\frac{R_1(T)}{\epsilon_*T\sqrt{m\log T}} = +\infty$ . In our problem with multiple users and  $m$  underlying clusters, even if we know the underlying clustering structure and keep  $m$  independent linear bandit algorithms with  $T_{i}$  for the cluster  $i\in [m]$  to leverage the common information of clusters, the best we can get is  $R_{2}(T) = \sum_{i\in [m]}R_{1}(T_{i})$ . By the above results, if the decision process does not consider misspecifications, we have  $\lim_{T\to +\infty}\frac{R_2(T)}{\epsilon_*T\sqrt{m\log T}} = \lim_{T\to +\infty}\frac{mR_1(T / m)}{\epsilon_*T\sqrt{m\log T}} = +\infty$ . Recall that the regret upper bound  $R(T)$  of our proposed algorithms is of  $O(\epsilon_{*}T\sqrt{md\log T} +d\sqrt{mT}\log T)$  (thus, we have  $\lim_{T\to +\infty}\frac{R(T)}{\epsilon_*T\sqrt{m\log T}} < + \infty$ ), which gives a proof that the regret upper bound of our proposed algorithms is asymptotically much better than CB algorithms in cases (1)(2).
For case (3), if the clustering process does not use the more tolerant deletion rule in Line 10 of Algo.1, the gap between users linked by edges would possibly exceed  $\zeta$  ( $\zeta = 2\epsilon_{*}\sqrt{\frac{2}{\tilde{\lambda}_{x}}}$ ) even after  $T_{0}$ , which will result in a regret upper bound no better than  $O(\epsilon_{*}u\sqrt{dT})$ . As the number of users  $u$  is usually huge in practice, this result is vacuous. The reasons for getting the above claim are as follows. Even if the clustering process further uses our deletion rule considering misspecifications, and the users linked by edges are within  $\zeta$  distance, failing to extract 1-hop users (Line 5 in Algo.1) would cause the leading  $O(\epsilon_{*}u\sqrt{dT})$  regret term, as in the worst case, the preference vector  $\theta$  of the user in  $\tilde{V}_t$  who is  $h$ -hop away from user  $i_t$  could deviate by  $h\zeta$  from  $\theta_{i_t}$ , where  $h$  can be as large as  $u$ , and it would make the second term in Eq.(8) a  $O(\epsilon_{*}u\sqrt{dT})$  term. If we completely do not consider the misspecifications in the clustering process, the above user gap between users linked by edges would possibly exceed  $\zeta$ , which will cause a regret upper bound worse than  $O(\epsilon_{*}u\sqrt{dT})$ .
E Proof of Theorem 5.3
We first prove the result in the case when  $\gamma_{1}$  defined in Definition 5.1 is not infinity, i.e.,  $4\epsilon_*\sqrt{\frac{2}{\lambda_x}} < \gamma_1 < \infty$ . The proof of the special case when  $\gamma_{1} = \infty$  will directly follow the proof of this case.
For the instantaneous regret  $R_{t}$  at round  $t$ , with probability at least  $1 - 5\delta$  for some  $\delta \in (0, \frac{1}{5})$ , at  $\forall t \geq T_0$ :
$$
\begin{array}{l} R _ {t} = \left(\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} + \epsilon_ {a _ {t} ^ {*}} ^ {i _ {t}, t}\right) - \left(\boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} + \epsilon_ {a _ {t}} ^ {i _ {t}, t}\right) \\ = \boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} (\boldsymbol {\theta} _ {i _ {t}} - \hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t - 1}) + (\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t - 1} + C _ {a _ {t} ^ {*}, t}) - (\boldsymbol {x} _ {a _ {t}} ^ {\top} \hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t - 1} + C _ {a _ {t}, t}) \\ + \boldsymbol {x} _ {a _ {t}} ^ {\top} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) + C _ {a _ {t}, t} - C _ {a _ {t} ^ {*}, t} + \left(\boldsymbol {\epsilon} _ {a _ {t} ^ {*}} ^ {i _ {t}, t} - \boldsymbol {\epsilon} _ {a _ {t}} ^ {i _ {t}, t}\right) \tag {10} \\ \leq 2 C _ {a _ {t}, t} + \frac {2 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \mathbb {I} \{\overline {{V}} _ {t} \notin \mathcal {V} \} + 2 \epsilon_ {*}, \\ \end{array}
$$
where the last inequality holds by the UCB arm selection strategy in Eq.(5), the concentration bound given in Lemma 5.6, and the fact that  $\left\| \boldsymbol{\epsilon}^{i,t}\right\|_{\infty} \leq \epsilon_{*}, \forall i \in \mathcal{U}, \forall t$ .
We define the following events. Let
$\mathcal{E}_0 = \{R_t\leq 2C_{a_t,t} + \frac{2\epsilon_*\sqrt{2d}}{\tilde{\lambda}_x^{\frac{3}{2}}}\mathbb{I}\{\overline{V}_t\notin \mathcal{V}\} +2\epsilon_*,$  for all  $\{t:t\geq T_0,$  and the algorithm maintains a "good partition" at  $t\}\}$
$\mathcal{E}_1 = \{\text{the algorithm maintains a "good partition" for all } t \geq T_0\}$ ,
$$
\mathcal {E} = \mathcal {E} _ {0} \cap \mathcal {E} _ {1}.
$$
$\mathbb{P}(\mathcal{E}_0) \geq 1 - 2\delta$ . According to Lemma H.1,  $\mathbb{P}(\mathcal{E}_1) \geq 1 - 3\delta$ . Thus,  $\mathbb{P}(\mathcal{E}) \geq 1 - 5\delta$  for some  $\delta \in (0, \frac{1}{5})$ . Take  $\delta = \frac{1}{T}$ , we can get that
$$
\begin{array}{l} \mathbb {E} [ R (T) ] = \mathbb {P} (\mathcal {E}) \mathbb {I} \{\mathcal {E} \} R (T) + \mathbb {P} (\overline {{\mathcal {E}}}) \mathbb {I} \{\overline {{\mathcal {E}}} \} R (T) \\ \leq \mathbb {I} \left\{\mathcal {E} \right\} R (T) + 5 \times \frac {1}{T} \times T \tag {11} \\ = \mathbb {I} \left\{\mathcal {E} \right\} R (T) + 5, \\ \end{array}
$$
where  $\overline{\mathcal{E}}$  denotes the complementary event of  $\mathcal{E}$ ,  $\mathbb{I}\{\mathcal{E}\} R(T)$  denotes  $R(T)$  under event  $\mathcal{E}$ ,  $\mathbb{I}\{\overline{\mathcal{E}}\} R(T)$  denotes  $R(T)$  under event  $\overline{\mathcal{E}}$ , and we use  $R(T) \leq T$  to bound  $R(T)$  under event  $\overline{\mathcal{E}}$ .
Then it remains to bound  $\mathbb{I}\{\mathcal{E}\} R(T)$
$$
\begin{array}{l} \mathbb {I} \{\mathcal {E} \} R (T) \leq R \left(T _ {0}\right) + \mathbb {E} \left[ \mathbb {I} \{\mathcal {E} \} \sum_ {t = T _ {0} + 1} ^ {T} R _ {t} \right] \\ \leq T _ {0} + 2 \mathbb {E} \left[ \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} C _ {a _ {t}, t} \right] + \frac {2 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \sum_ {t = T _ {0} + 1} ^ {T} \mathbb {E} \left[ \mathbb {I} \left\{\mathcal {E}, \bar {V} _ {t} \notin \mathcal {V} \right\} \right] + 2 \epsilon_ {*} T (12) \\ = T _ {0} + 2 \mathbb {E} \left[ \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} C _ {a _ {t}, t} \right] + \frac {2 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \sum_ {t = T _ {0} + 1} ^ {T} \mathbb {P} \left(\mathbb {I} \left\{\mathcal {E}, \bar {V} _ {t} \notin \mathcal {V} \right\}\right) + 2 \epsilon_ {*} T \\ \leq T _ {0} + 2 \mathbb {E} \left[ \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} C _ {a _ {t}, t} \right] + \frac {2 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \times \frac {\tilde {u}}{u} T + 2 \epsilon_ {*} T, (13) \\ \end{array}
$$
where Eq.(12) follows from Eq.(10). Eq.(13) holds since under Assumption 3.2 about user arrival uniformness and by Definition 5.5 of "good partition",  $\mathbb{P}(\mathbb{I}\{\mathcal{E},\overline{V}_t\notin \mathcal{V}\})\leq \frac{\tilde{u}}{u},\forall t\geq T_0$  where  $\tilde{u}$  is defined in Definition 5.2.
Then we need to bound  $\mathbb{E}[\mathbb{I}\{\mathcal{E}\} \sum_{t = T_0 + 1}^T C_{a_t,t}]$ :
$$
\begin{array}{l} \mathbb {I} \{\mathcal {E} \} \sum_ {t = T _ {0} + 1} ^ {T} C _ {a _ {t}, t} = \left(\sqrt {\lambda} + \sqrt {2 \log (\frac {1}{\delta}) + d \log (1 + \frac {T}{\lambda d})}\right) \mathbb {I} \{\mathcal {E} \} \sum_ {t = T _ {0} + 1} ^ {T} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1}} \\ + \mathbb {I} \{\mathcal {E} \} \epsilon_ {*} \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} \left| \boldsymbol {x} _ {a _ {t}} ^ {\top} \overline {{M}} _ {V _ {t}, t - 1} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right|. \tag{14} \\ \end{array}
$$
Next, we bound the  $\mathbb{I}\{\mathcal{E}\} \sum_{t = T_0 + 1}^{T}\| \pmb{x}_{a_t}\|_{\overline{M}_{V_t,t - 1}^{-1}}$  term in Eq.(14):
$$
\begin{array}{l} \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {V _ {t}, t - 1} ^ {- 1}} = \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {k = 1} ^ {m _ {t}} \mathbb {I} \left\{i _ {t} \in \tilde {V} _ {t, k} ^ {\prime} \right\} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {V _ {t, k} ^ {\prime}, t - 1} ^ {- 1}} \\ \leq \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {t = T _ {0} + 1} ^ {T} \sum_ {j = 1} ^ {m} \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {V _ {j}, t - 1} ^ {- 1}} (15) \\ \leq \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {j = 1} ^ {m} \sqrt {\sum_ {t = T _ {0} + 1} ^ {T} \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \sum_ {t = T _ {0} + 1} ^ {T} \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {V _ {j}, t - 1}} ^ {2}} (16) \\ \leq \mathbb {I} \left\{\mathcal {E} \right\} \sum_ {j = 1} ^ {m} \sqrt {2 T _ {V _ {j} , T} d \log \left(1 + \frac {T}{\lambda d}\right)} (17) \\ \leq \mathbb {I} \left\{\mathcal {E} \right\} \sqrt {2 \sum_ {j = 1} ^ {m} 1 \sum_ {j = 1} ^ {m} T _ {V _ {j} , T} d \log \left(1 + \frac {T}{\lambda d}\right)} = \mathbb {I} \left\{\mathcal {E} \right\} \sqrt {2 m d T \log \left(1 + \frac {T}{\lambda d}\right)}, (18) \\ \end{array}
$$
where we use  $m_t$  to denote the number of connected components partitioned by the algorithm at  $t$ ,  $\tilde{V}_{t,k}^{\prime}, k \in [m_t]$  to denote the connected components partitioned by the algorithm at  $t$ ,  $\overline{V}_{t,k}^{\prime} \subseteq \tilde{V}_{t,k}^{\prime}$  to denote the subset extracted to be the cluster  $\overline{V}_t$  for  $i_t$  from  $\tilde{V}_{t,k}^{\prime}$  conditioned on  $i_t \in \tilde{V}_{t,k}^{\prime}$ , and  $T_{V_j,T}$  to denote the number of times that the served users lie in the ground-truth cluster  $V_j$  up to time  $T$ , i.e.,  $T_{V_j,T} = \sum_{t \in [T]} \mathbb{I}\{i_t \in V_j\}$ .
The reasons for having Eq.(15) are as follows. Under event  $\mathcal{E}$ , the algorithm will always have a "good partition" after  $T_0$ . By Definition 5.5 and the proof process of Lemma H.1 about the edge deletion conditions, we can get  $m_t \leq m$  and if  $i_t \in \tilde{V}_{t,k}'$ ,  $i_t \in V_j$ , then  $V_j \subseteq \overline{V}_{t,k}'$  since  $\overline{V}_{t,k}'$  contains  $V_j$  and possibly other ground-truth clusters  $V_n$ ,  $n \in [m]$ , whose preference vectors are  $\zeta$ -close to  $\theta^j$ . Therefore, by the definition of the regularized Gramian matrix, we can get  $M_{\overline{V}_{t,k}', t-1} \succeq M_{V_j, t-1}, \forall t \geq T_0 + 1$ . Thus by the above reasoning,  $\sum_{k=1}^{m_t} \mathbb{I}\{i_t \in \tilde{V}_{t,k}'\} \| \boldsymbol{x}_{a_t} \|_{\overline{M}_{V_{t,k}', t-1}^{-1}} \leq \sum_{j=1}^{m} \mathbb{I}\{i_t \in V_j\} \| \boldsymbol{x}_{a_t} \|_{\overline{M}_{V_{j,t-1}}^{-1}}, \forall t \geq T_0 + 1$ . Eq.(16) holds by the Cauchy-Schwarz inequality; Eq.(17) follows by the following technical Lemma J.2. Eq.(18) is from the Cauchy-Schwarz inequality and the fact that  $\sum_{j=1}^{m} T_{V_j, T} = T$ .
We then bound the last term in Eq.(14):
where Eq.(19), Eq.(21) and Eq.(23) hold because of the Cauchy-Schwarz inequality, Eq.(20) holds since  $\overline{M}_{\overline{V}_{t,k}^{\prime},t - 1}\succeq \sum_{\substack{s\in [t - 1]\\ i_s\in \overline{V}_{t,k}^{\prime}}}\boldsymbol{x}_{a_s}\boldsymbol{x}_{a_s}^\top$ , Eq.(22) is because  $T_{\overline{V}_{t,k}^{\prime},t - 1}\leq T$ , Eq. (24) follows from the same reasoning as Eq.(15), and Eq.(25) comes from the following technical Lemma J.2.
Finally, plugging Eq.(18) and Eq.(25) into Eq.(14), take expectation and plug it into Eq.(13), we can get:
$$
\begin{array}{l} R (T) \leq 5 + T _ {0} + \frac {\tilde {u}}{u} \times \frac {2 \epsilon_ {*} \sqrt {2 d} T}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} + 2 \epsilon_ {*} T \left(1 + \sqrt {2 m d \log \left(1 + \frac {T}{\lambda d}\right)}\right) \\ + 2 \left(\sqrt {\lambda} + \sqrt {2 \log (T) + d \log \left(1 + \frac {T}{\lambda d}\right)}\right) \times \sqrt {2 m d T \log \left(1 + \frac {T}{\lambda d}\right)}, \tag {26} \\ \end{array}
$$
$$
T _ {0} = 1 6 u \log (\frac {u}{\delta}) + 4 u \max \max \{\frac {8 d}{\tilde {\lambda} _ {x} (\frac {\gamma_ {1}}{4} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}) ^ {2}} \log (\frac {u}{\delta}), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log (\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}) \}
$$
is given in the following Lemma H.1 in Appendix H.
F Proof and Discussions of Theorem 5.4