SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat{\boldsymbol{\theta}}_{\overline{V}_{t},t - 1} - \boldsymbol{\theta}_{i_{t}} = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top} + \lambda \boldsymbol {I})^{-1}(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}r_{s}) - \boldsymbol{\theta}_{i_{t}} \\ = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}\Bigg(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}(\boldsymbol{x}^{\top}_{a_{s}}\boldsymbol{\theta}_{i_{s}} + \boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \eta_{s})\Bigg) - \boldsymbol{\theta}_{i_{t}} \\ = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}\Bigg(\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}\boldsymbol{x}_{a_{s}}(\boldsymbol{x}^{\top}_{a_{s}}\boldsymbol{\theta}_{i_{t}} + \boldsymbol{\epsilon}^{i_{s},s}_{a_{s}} + \eta_{s})\Bigg) - \boldsymbol{\theta}_{i_{t}} \\ = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}[(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})\boldsymbol{\theta}_{i_{t}} - \lambda \boldsymbol{\theta}_{i_{t}} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}^{i_{s},s}_{a_{s}} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s}] - \boldsymbol{\theta}_{i_{t}} \\ = -\lambda \overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{\theta}_{i_{t}} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\eta_{s}. \\ \end{array}
$$
$$
\left| \boldsymbol {x} _ {a} ^ {\top} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) \right| \leq \lambda \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {i _ {t}} \right| + \left| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \bar {V} _ {t}}} \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \boldsymbol {\epsilon} _ {a _ {s}} ^ {i _ {s}, s} \right| + \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \bar {V} _ {t}}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right|.
$$
Next, we bound the three terms in Eq.(60). For the first term:
$$
\lambda \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {i _ {t}} \right| \leq \lambda \| \boldsymbol {x} _ {a} \| _ {\overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1}} \sqrt {\lambda_ {\max } (\overline {{\boldsymbol {M}}} _ {\bar {V} _ {t} , t - 1} ^ {- 1})} \| \boldsymbol {\theta} _ {i _ {t}} \| _ {2} \leq \sqrt {\lambda} \| \boldsymbol {x} _ {a} \| _ {\overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1}}, \tag {61}
$$
where we use the inequality of matrix norm, the Cauchy-Schwarz inequality,  $\| \pmb{\theta}_{i_t}\| _2\leq 1$  , and the fact that  $\lambda_{\mathrm{max}}(\overline{M}_{V_t,t - 1}^{-1}) = \frac{1}{\lambda_{\mathrm{min}}(\overline{M}_{V_t,t - 1})}\leq \frac{1}{\lambda}$
For the second term in Eq.(60):
$$
\begin{array}{l} \left|\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a}^{\top}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s}\right|\\ \leq \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\left|\boldsymbol{x}_{a}^{\top}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s}\right| \\ \leq \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\left\| \boldsymbol{\epsilon}_{a_{s}}^{i_{s},\mathbf{s}}\right\|_{\infty}\left|\boldsymbol{x}_{a}^{\top}\overline{\boldsymbol{M}}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\right| \\ \leq \epsilon_ {*} \sum_ { \begin{array}{c} s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t} \end{array} } \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{M}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right|, \tag {62} \\ \end{array}
$$
where in the second inequality we use the Holder's inequality.
For the last term, with probability at least  $1 - \delta$ :
$$
\begin{array}{l} \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right| \leq \| \boldsymbol {x} _ {a} \| _ {\overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1}} \left\| \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \boldsymbol {x} _ {a _ {s}} \eta_ {s} \right\| _ {\overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1}} (63) \\ \leq \| \boldsymbol {x} _ {a} \| _ {\overline {{M}} _ {\bar {V} _ {t}, t - 1} ^ {- 1}} \sqrt {2 \log (\frac {1}{\delta}) + \log (\frac {\det  (\overline {{M}} _ {\bar {V} _ {t} , t - 1})}{\det  (\lambda I)})} \\ \leq \left\| \boldsymbol {x} _ {a} \right\| _ {\overline {{M}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1}} \sqrt {2 \log (\frac {1}{\delta}) + d \log (1 + \frac {T}{\lambda d})}, (64) \\ \end{array}
$$
where the second inequality follows by Theorem 1 in [1], Eq.(64) is because  $\operatorname*{det}(\overline{M}_{\overline{V}_t,t - 1})\leq$
$$
\left(\frac {\operatorname {t r a c e} (\lambda I + \sum_ {s \in [ t ]} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top})}{d}\right) ^ {d} \leq \left(\frac {\lambda d + T _ {\overline {{V}} _ {t} , t}}{d}\right) ^ {d} \leq \left(\frac {\lambda d + T}{d}\right) ^ {d}, \text {a n d} \det  (\lambda I) = \lambda^ {d}.
$$
Plugging Eq.(61), Eq.(62) and Eq.(64) into Eq.(60), we can prove Lemma 5.6 in situation 1, i.e., for any  $t \geq T_0$  and  $\overline{V}_t \in V$ , with probability at least  $1 - \delta$ :
$$
\left| \boldsymbol {x} _ {a} ^ {\top} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) \right| \leq \epsilon_ {*} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \bar {V} _ {t}}} \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right| + \| \boldsymbol {x} _ {a} \| _ {\overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1}} \left(\sqrt {\lambda} + \sqrt {2 \log (\frac {1}{\delta}) + d \log (1 + \frac {T}{\lambda d})}\right). \tag{65}
$$
(2) Situation 2: for any  $t \geq T_0$  and  $\overline{V}_t \notin \mathcal{V}$ , which means that the current user is misclustered by the algorithm, i.e.,  $\overline{V}_t \neq V_{j(i_t)}$ , but with Lemma H.1, with probability at least  $1 - 3\delta$ , the current
partition is a "good partition", i.e.,  $\| \pmb{\theta}_l - \pmb{\theta}_{i_t}\| _2\leq 2\epsilon_*\sqrt{\frac{2}{\tilde{\lambda}_x}},\forall l\in \overline{V}_t$  , we have:
$$
\begin{array}{l} \hat{\boldsymbol{\theta}}_{\overline{V}_{t},t - 1} - \boldsymbol{\theta}_{i_{t}} = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top} + \lambda \boldsymbol {I})^{-1}(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}r_{s}) - \boldsymbol{\theta}_{i_{t}} \\ = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}\Bigg(\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}\boldsymbol{x}_{a_{s}}(\boldsymbol{x}^{\top}_{a_{s}}\boldsymbol{\theta}_{i_{s}} + \boldsymbol{\epsilon}^{i_{s},s}_{a_{s}} + \eta_{s})\Bigg) - \boldsymbol{\theta}_{i_{t}} \\ = \overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{i_{t}} \\ = \overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}(\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{i_{t}}) \\ + \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} (\sum_ {s \in [ t - 1 ] \atop i _ {s} \in \overline {{V}} _ {t}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} + \lambda \boldsymbol {I}) \boldsymbol {\theta} _ {i _ {t}} - \lambda \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {i _ {t}} - \boldsymbol {\theta} _ {i _ {t}} \\ = \overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in V_{t}}}\boldsymbol{x}_{a_{s}}\epsilon_{a_{s}}^{i_{s},s} + \overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}(\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{i_{t}}) \\ - \lambda \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {i _ {t}}. \\ \end{array}
$$
Thus, with Lemma 5.7 and with the previous reasoning, with probability at least  $1 - 5\delta$ , we have:
$$
\begin{array}{l} \left|\boldsymbol{x}_{a}^{\top}(\hat{\boldsymbol{\theta}}_{\overline{V}_{t},t - 1} - \boldsymbol{\theta}_{i_{t}})\right|\leq \lambda \left|\boldsymbol{x}_{a}^{\top}\overline{M}_{V_{t},t - 1}^{-1}\boldsymbol{\theta}_{i_{t}}\right| + \left|\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a}^{\top}\overline{M}_{V_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s}\right|\\ +\left|\boldsymbol{x}_{a}^{\top}\overline{M}_{V_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s}\right| \\ +\left|\boldsymbol{x}_{a}^{\top}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}(\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{i_{t}})\right| \\ \leq \epsilon_{*}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\left|\boldsymbol{x}_{a}^{\top}\overline{\boldsymbol{M}}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\right| + \| \boldsymbol{x}_{a}\|_{\overline{\boldsymbol{M}}_{\overline{V}_{t},t - 1}^{-1}}\Bigg(\sqrt{\lambda} +\sqrt{2\log(\frac{1}{\delta}) + d\log(1 + \frac{T}{\lambda d})}\Bigg) \\ + \frac {\epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}}. \\ \end{array}
$$
Therefore, combining situation 1 and situation 2, the result of Lemma 5.6 then follows.
J Technical Lemmas and Their Proofs
We first prove the following technical lemma which is used to prove Lemma H.1.
Lemma J.1. Under Assumption 3.3, at any time  $t$ , for any fixed unit vector  $\pmb{\theta} \in \mathbb{R}^d$
$$
\mathbb {E} _ {t} \left[ \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {a _ {t}}\right) ^ {2} \mid \left| \mathcal {A} _ {t} \right| \right] \geq \tilde {\lambda} _ {x} \triangleq \int_ {0} ^ {\lambda_ {x}} \left(1 - e ^ {- \frac {\left(\lambda_ {x} - x\right) ^ {2}}{2 \sigma^ {2}}}\right) ^ {C} d x. \tag {66}
$$
Proof. The proof of this lemma mainly follows the proof of Claim 1 in [12], but with more careful analysis, since their assumption is more stringent than ours.
Denote the feasible arms at round  $t$  by  $\mathcal{A}_t = \{\pmb{x}_{t,1},\pmb{x}_{t,2},\dots,\pmb{x}_{t,|\mathcal{A}_t|}\}$ . Consider the corresponding i.i.d. random variables  $\theta_{i} = (\pmb{\theta}^{\top}\pmb{x}_{t,i})^{2} - \mathbb{E}_{t}[(\pmb{\theta}^{\top}\pmb{x}_{t,i})^{2}||\mathcal{A}_{t}|]$ ,  $i = 1,2,\ldots,|\mathcal{A}_t|$ . By Assumption 3.3,  $\theta_{i}$  s are sub-Gaussian random variables with variance bounded by  $\sigma^2$ . Therefore, we have that
for any  $\alpha > 0$  and any  $i \in [|\mathcal{A}_t|]$ :
$$
\mathbb {P} _ {t} (\theta_ {i} <   - \alpha | | \mathcal {A} _ {t} |) \leq e ^ {- \frac {\alpha^ {2}}{2 \sigma^ {2}}},
$$
where  $\mathbb{P}_t(\cdot)$  is the shorthand for the conditional probability  $\mathbb{P}(\cdot | (i_1, \mathcal{A}_1, r_1), \ldots, (i_{t-1}, \mathcal{A}_{t-1}, r_{t-1}), i_t)$ .
We also have that  $\mathbb{E}_t[(\pmb{\theta}^\top \pmb{x}_{t,i})^2 ||\mathcal{A}_t| = \mathbb{E}_t[\pmb{\theta}^\top \pmb{x}_{t,i}\pmb{x}_{t,i}^\top \pmb{\theta}||\mathcal{A}_t] \geq \lambda_{\min}(\mathbb{E}_{\pmb{x}\sim \rho}[\pmb{x}\pmb{x}^\top]) \geq \lambda_x$  by Assumption 3.3. With the above inequalities, we can get
$$
\mathbb {P} _ {t} (\min  _ {i = 1, \dots , | \mathcal {A} _ {t} |} (\pmb {\theta} ^ {\top} \pmb {x} _ {t, i}) ^ {2} \geq \lambda_ {x} - \alpha | | \mathcal {A} _ {t} |) \geq (1 - e ^ {- \frac {\alpha^ {2}}{2 \sigma^ {2}}}) ^ {C},
$$
where  $C$  is the upper bound of  $|\mathcal{A}_t|$ .
$$
\begin{array}{l} \mathbb {E} _ {t} [ (\pmb {\theta} ^ {\top} \pmb {x} _ {a _ {t}}) ^ {2} | | \mathcal {A} _ {t} | ] \geq \mathbb {E} _ {t} [ \min _ {i = 1, \dots , | \mathcal {A} _ {t} |} (\pmb {\theta} ^ {\top} \pmb {x} _ {t, i}) ^ {2} | | \mathcal {A} _ {t} | ] \\ \geq \int_ {0} ^ {\infty} \mathbb {P} _ {t} \left(\min  _ {i = 1, \dots , | \mathcal {A} _ {t} |} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {x} _ {t, i}\right) ^ {2} \geq x \mid | \mathcal {A} _ {t} |\right) d x \\ \geq \int_ {0} ^ {\lambda_ {x}} (1 - e ^ {- \frac {(\lambda_ {x} - x) ^ {2}}{2 \sigma^ {2}}}) ^ {C} d x \triangleq \tilde {\lambda} _ {x} \\ \end{array}
$$
Finally, we prove the following lemma which is used in the proof of Theorem 5.3.
Lemma J.2.
$$
\sum_ {t = T _ {0} + 1} ^ {T} \min  \left\{\mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {j}, t - 1}} ^ {2}, 1 \right\} \leq 2 d \log \left(1 + \frac {T}{\lambda d}\right), \forall j \in [ m ]. \tag {67}
$$
$$
\begin{array}{l} \det  (\overline {{M}} _ {V _ {j}, T}) = \det  \left(\overline {{M}} _ {V _ {j}, T - 1} + \mathbb {I} \{i _ {T} \in V _ {j} \} \boldsymbol {x} _ {a _ {T}} \boldsymbol {x} _ {a _ {T}} ^ {\top}\right) \\ = \det  (\overline {{M}} _ {V _ {j}, T - 1}) \det  \left(\boldsymbol {I} + \mathbb {I} \{i _ {T} \in V _ {j} \} \overline {{M}} _ {V _ {j}, T - 1} ^ {- \frac {1}{2}} \boldsymbol {x} _ {a _ {T}} \boldsymbol {x} _ {a _ {T}} ^ {\top} \overline {{M}} _ {V _ {j}, T - 1} ^ {- \frac {1}{2}}\right) \\ = \det  (\overline {{\boldsymbol {M}}} _ {V _ {j}, T - 1}) \left(1 + \mathbb {I} \{i _ {T} \in V _ {j} \} \| \boldsymbol {x} _ {a _ {T}} \| _ {\overline {{\boldsymbol {M}}} _ {V _ {j}, T - 1}} ^ {2}\right) \\ = \det  (\overline {{\boldsymbol {M}}} _ {V _ {j}, T _ {0}}) \prod_ {t = T _ {0} + 1} ^ {T} \left(1 + \mathbb {I} \{i _ {t} \in V _ {j} \} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{\boldsymbol {M}}} _ {V _ {j}, t - 1} ^ {- 1}} ^ {2}\right) \\ \geq \det (\lambda I) \prod_ {t = T _ {0} + 1} ^ {T} \left(1 + \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {j}, t - 1}} ^ {2}\right). \tag {68} \\ \end{array}
$$
$\forall x \in [0,1]$ , we have  $x \leq 2\log(1 + x)$ . Therefore
$$
\begin{array}{l} \sum_ {t = T _ {0} + 1} ^ {T} \min  \left\{\mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {j}, t - 1} ^ {- 1}} ^ {2}, 1 \right\} \leq 2 \sum_ {t = T _ {0} + 1} ^ {T} \log \left(1 + \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {j}, t - 1} ^ {- 1}} ^ {2}\right) \\ = 2 \log \left(\prod_ {t = T _ {0} + 1} ^ {T} \left(1 + \mathbb {I} \{i _ {t} \in V _ {j} \} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {j}, t - 1}} ^ {2}\right)\right) \\ \leq 2 \left[ \log \left(\det \left(\overline {{M}} _ {V _ {j}, T}\right)\right) - \log \left(\det (\lambda I)\right) \right] \\ \leq 2 \log \left(\frac {\operatorname {t r a c e} \left(\lambda \boldsymbol {I} + \sum_ {t = 1} ^ {T} \mathbb {I} \left\{i _ {t} \in V _ {j} \right\} \boldsymbol {x} _ {a _ {t}} \boldsymbol {x} _ {a _ {t}} ^ {\top}\right)}{\lambda d}\right) ^ {d} \\ \leq 2 d \log \left(1 + \frac {T}{\lambda d}\right). \tag {69} \\ \end{array}
$$
K Algorithms of RSCLUMB
This section introduces the Robust Set-based Clustering of Misspecified Bandits Algorithm (RSCLUMB). Unlike RCLUMB, which maintains a graph-based clustering structure, RSCLUMB maintains a set-based clustering structure. Besides, RCLUMB only splits clusters during the learning process, while RSCLUMB allows both split and merge operations. A brief illustration is that the agent will split a user out of its current set(cluster) if it finds an inconsistency between the user and its set, and if there are two clusters whose estimated preferences are close enough, the agent will merge them. A detailed discussion of the connection between the graph structure and the set structure can be found in [27].
Now we introduce the details of RSLUMB. The algorithm first initializes a single set  $S_{1}$  containing all users and updates it during the learning process. The whole learning process consists of phases (Algo. 2 Line 3), where the  $s - th$  phase contains  $2^{s}$  rounds. At the beginning of each phase, the agent marks all users as "unchecked", and if a user comes later, it will be marked as "checked". If all users in a cluster are checked, then this cluster will be marked as "checked" meaning it is an accurate cluster in the current phase. With this mechanism, every phase can maintain an accuracy level, and the agent can put the accurate clusters aside and focus on exploring the inaccurate ones. For each cluster  $V_{j}$ , the algorithm maintains two estimated vectors  $\hat{\theta}_{V_{j}}$  and  $\tilde{\theta}_{V_{j}}$ , where the  $\hat{\theta}_{V_{j}}$  is similar to the  $\hat{\theta}_{\overline{V}_j}$  in RCLUMB and is used for the recommendation, while the  $\tilde{\theta}_{V_j}$  is the average of all the estimated user preference vectors in this cluster and is used for the split and merge operations.
At time  $t$  in phase  $s$ , the user  $i_{\tau}$  comes with the item set  $\mathcal{D}_{\tau}$ , where  $\tau$  represents the index of total time steps. Then the algorithm determines the cluster and makes a cluster-based recommendation. This process is similar to RCLUMB. After updating the information (Algo. 2 Line12), the agent checks if a split or a merge is possible (Algo. 2 Line13-17).
By our assumption, users in the same cluster have the same vectors. So a cluster can be regarded as a good cluster only when all the estimated user vectors are close to the estimated cluster vector. We call a user is consistent with the cluster if their estimated vectors are close enough. If a user is inconsistent with its current cluster, the agent will split it out. Two clusters are consistent when their estimated vectors are close, and the agent will merge them.
RSCLUMB maintains two sets of estimated cluster vectors: (i) cluster-level estimation with integrated user information, which is for recommendations (Line 12 and Line 10 in Algo.2); (ii) the average of estimated user vectors, which is used for robust clustering (Line 3 in Algo.3 and Line 2 in Algo.4). The previous set-based CB work [27] only uses (i) for both recommendations and clustering, which would lead to erroneous clustering under misspecifications, and cannot get any non-vacuous regret bound in CBMUM.
L Main Theorem and Lemmas of RSLUMB
Theorem L.1 (main result on regret bound for RSCLUMB). With the same assumptions in Theorem 5.3, the expected regret of the RSCLUMB algorithm for  $T$  rounds satisfies:
$$
\begin{array}{l} R (T) \leq O \left(u \left(\frac {d}{\tilde {\lambda} _ {x} \left(\gamma_ {1} - \zeta_ {1}\right) ^ {2}} + \frac {1}{\tilde {\lambda} _ {x} ^ {2}}\right) \log T + \frac {\epsilon_ {*} \sqrt {d} T}{\tilde {\lambda} _ {x} ^ {1 . 5}} + \epsilon_ {*} T \sqrt {m d \log T} + d \sqrt {m T} \log T + \epsilon_ {*} \sqrt {\frac {1}{\tilde {\lambda} _ {x}}} T\right) \\ \leq O \left(\epsilon_ {*} T \sqrt {m d \log T} + d \sqrt {m T} \log T\right) \tag {70} \\ \end{array}
$$
Lemma L.2. For RSLUMB, we use  $T_{1}$  to represent the corresponding  $T_{0}$  of RCLUMB. Then:
$$
\begin{array}{l} T _ {1} \triangleq 1 6 u \log (\frac {u}{\delta}) + 4 u \max \{\frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log (\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}), \frac {8 d}{\tilde {\lambda} _ {x} (\frac {\gamma_ {1}}{6} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}) ^ {2}} \log (\frac {u}{\delta}) \} \\ = O \left(u \left(\frac {d}{\tilde {\lambda} _ {x} \left(\gamma_ {1} - \zeta_ {1}\right) ^ {2}} + \frac {1}{\tilde {\lambda} _ {x} ^ {2}}\right) \log \frac {1}{\delta}\right) \\ \end{array}
$$
Algorithm 2 Robust Set-based Clustering of Misspecified Bandits Algorithm (RSCLUMB)
1: Input: Deletion parameter  $\alpha_{1},\alpha_{2} > 0,f(T) = \sqrt{\frac{1 + \ln(1 + T)}{1 + T}},\lambda ,\beta ,\epsilon_{*} > 0.$
2: Initialization:
-  $M_{i,0} = 0_{d \times d}, b_{i,0} = 0_{d \times 1}, T_{i,0} = 0, \forall i \in \mathcal{U}$ ;
- Initialize the set of cluster indexes by  $J = \{1\}$  and the single cluster  $S_{1}$  by  $M_{1} = 0_{d\times d}$ ,  $b_{1} = 0_{d\times 1}$ ,  $T_{1} = 0$ ,  $C_{1} = \mathcal{U}$ ,  $j(i) = 1, \forall i$ .
3: for all  $s = 1,2,\ldots$  do
4: Mark every user unchecked for each cluster.
5: For each cluster  $V_{j}$ , compute  $\tilde{T}_{V_j} = T_{V_j},\hat{\pmb{\theta}}_{V_j} = (\lambda \pmb {I} + \pmb{M}_{V_j})^{-1}\pmb {b}_{V_j},\tilde{\pmb{\theta}}_{V_j} = \frac{\sum_{i\in V_j}\hat{\pmb{\theta}}_i}{[V_j]}$
6: for all  $t = 1,2,\ldots ,T$  do
7: Compute  $\tau = 2^s - 2 + t$
8: Receive the user  $i_{\tau}$  and the decision set  $\mathcal{D}_{\tau}$
9: Determine the cluster index  $j = j(i_{\tau})$
10: Recommend item  $a_{\tau}$  with the largest UCB index as shown in Eq. (5)
11: Received the feedback  $r_{\tau}$
12: Update the information:
$$
\begin{array}{l} \boldsymbol {M} _ {i _ {\tau}, \tau} = \boldsymbol {M} _ {i _ {\tau}, \tau - 1} + \boldsymbol {x} _ {a _ {\tau}} \boldsymbol {x} _ {a _ {\tau}} ^ {\mathrm {T}}, \boldsymbol {b} _ {i _ {\tau}, \tau} = \boldsymbol {b} _ {i _ {\tau}, \tau - 1} + r _ {\tau} \boldsymbol {x} _ {a _ {\tau}}, \\ T _ {i _ {\tau}, \tau} = T _ {i _ {\tau}, \tau - 1} + 1, \hat {\boldsymbol {\theta}} _ {i _ {\tau}, \tau} = (\lambda \boldsymbol {I} + \boldsymbol {M} _ {i _ {\tau}, \tau}) ^ {- 1} \boldsymbol {b} _ {i _ {\tau}, \tau} \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {M} _ {V _ {j}, \tau} = \boldsymbol {M} _ {V _ {j}, \tau - 1} + \boldsymbol {x} _ {a _ {\tau}} \boldsymbol {x} _ {a _ {\tau}} ^ {\mathrm {T}}, \boldsymbol {b} _ {V _ {j}, \tau} = \boldsymbol {b} _ {V _ {j}, \tau - 1} + r _ {\tau} \boldsymbol {x} _ {\tau}, \\ T _ {V _ {j}, \tau} = T _ {V _ {j}, \tau - 1} + 1, \hat {\boldsymbol {\theta}} _ {V _ {j}, \tau} = (\lambda \boldsymbol {I} + \boldsymbol {M} _ {V _ {j}, \tau}) ^ {- 1} \boldsymbol {b} _ {V _ {j}, \tau}, \\ \tilde {\boldsymbol {\theta}} _ {V _ {j}, \tau} = \frac {\sum_ {i \in V _ {j}} \hat {\boldsymbol {\theta}} _ {i} , \tau}{[ V _ {j} ]} \\ \end{array}
$$
13: if  $i_{\tau}$  is unchecked then
14: Run Split
15: Mark user  $i_{\tau}$  has been checked
16: Run Merge
Algorithm 3 Split