SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Define  $F(T) = \sqrt{\frac{1 + \ln(1 + T)}{1 + T}}$
2: if  $\left\| \hat{\pmb{\theta}}_{i_{\tau},\tau} - \tilde{\pmb{\theta}}_{V_j,\tau}\right\| >\alpha_1(F(T_{i_{\tau},\tau}) + F(T_{V_j,\tau})) + \alpha_2\epsilon_*$  then
3: Split user  $i_{\tau}$  from cluster  $V_{j}$  and form a new cluster  $V_{j}^{\prime}$  of user  $i_{\tau}$
$$
\begin{array}{l} \boldsymbol {M} _ {V _ {j}, \tau} = \boldsymbol {M} _ {V _ {j}, \tau} - \boldsymbol {M} _ {i _ {\tau}, \tau}, \boldsymbol {b} _ {V _ {j}} = \boldsymbol {b} _ {V _ {j}} - \boldsymbol {b} _ {i _ {\tau}, \tau}, \\ T _ {V _ {j}, \tau} = T _ {V _ {j}, \tau} - T _ {i _ {\tau}, \tau}, C _ {j, \tau} = C _ {j, \tau} - \left\{i _ {\tau} \right\}, \\ \boldsymbol {M} _ {V _ {j} ^ {\prime}, \tau} = \boldsymbol {M} _ {i _ {\tau}, \tau}, \boldsymbol {b} _ {V _ {j} ^ {\prime}, \tau} = \boldsymbol {b} _ {i _ {\tau}, \tau}, \\ T _ {V _ {j} ^ {\prime}, \tau} = T _ {i _ {\tau}, \tau}, C _ {j ^ {\prime}, \tau} = \{i _ {\tau} \} \\ \end{array}
$$
Lemma L.3. For RSCLUMB, after  $2T_{1} + 1$  rounds: in each phase, after the first  $u$  rounds, with probability at least  $1 - 5\delta$ :
$$
\begin{array}{l} \left|\boldsymbol{x}_{a}^{\top}(\boldsymbol{\theta}_{i_{t}} - \hat{\boldsymbol{\theta}}_{\overline{V}_{t},t - 1})\right|\leq (\frac{3\epsilon_{*}\sqrt{2d}}{2\tilde{\lambda}_{x}^{\frac{3}{2}}} +6\epsilon_{*}\sqrt{\frac{1}{2\tilde{\lambda}_{x}}})\mathbb{I}\{\overline{V}_{t}\notin V\} +\beta   \| \boldsymbol{x}_{a}\|_{\overline{M}_{\overline{V}_{t},t - 1}^{-1}} + \epsilon_{*}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\left|\boldsymbol{x}_{a}^{\top}\overline{M}_{\overline{V}_{t},t - 1}^{-1}\boldsymbol{x}_{a_{s}}\right| \\ \triangleq (\frac {3 \epsilon_ {*} \sqrt {2 d}}{2 \tilde {\lambda} _ {x} ^ {\frac {3}{2}}} + 6 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}) \mathbb {I} \{\overline {{V}} _ {t} \notin V \} + C _ {a, t} \\ \end{array}
$$
Algorithm 4 Merge
1: for any two checked clusters  $V_{j_1}, V_{j_2}$  satisfying
$$
\left\| \tilde {\boldsymbol {\theta}} _ {j _ {1}} - \tilde {\boldsymbol {\theta}} _ {j _ {2}} \right\| <   \frac {\alpha_ {1}}{2} \left(F \left(T _ {V _ {j _ {1}}}\right) + F \left(T _ {V _ {j _ {2}}}\right)\right) + \frac {\alpha_ {2}}{2} \epsilon_ {*}
$$
2: Merge them:
$$
\boldsymbol {M} _ {V _ {j _ {1}}} = \boldsymbol {M} _ {j _ {1}} + \boldsymbol {M} _ {j _ {2}}, \boldsymbol {b} _ {V _ {j _ {1}}} = \boldsymbol {b} _ {V _ {j _ {1}}} + \boldsymbol {b} _ {V _ {j _ {2}}},
$$
$$
T _ {V _ {j _ {1}}} = T _ {V _ {j _ {1}}} + T _ {V _ {j _ {2}}}, C _ {V _ {j _ {1}}} = C _ {V _ {j _ {1}}} \cup C _ {V _ {j _ {2}}}
$$
3: Set  $j(i) = j_{1}, \forall i \in j_{2}$ , delete  $V_{j_{2}}$
M Proof of Lemma L.3
$$
\begin{array}{l} \left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\boldsymbol {\theta} _ {i} - \hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t}\right) \right| = \left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {V _ {t}}\right) \right| + \left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t} - \boldsymbol {\theta} _ {V _ {t}}\right) \right| \\ \leq \left\| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \right\| \underline {{\left\| \boldsymbol {\theta} _ {i} - \boldsymbol {\theta} _ {V _ {t}} \right\| + \left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t} , t} - \boldsymbol {\theta} _ {V _ {t}}\right) \right|}} \tag {71} \\ \leq 6 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} + | \boldsymbol {x} _ {a} ^ {\mathrm {T}} (\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t} - \boldsymbol {\theta} _ {V _ {t}}) | \\ \end{array}
$$
where the last inequality holds due to the fact  $\| \pmb{x}_a\| \leq 1$  and the condition of "split" and "merge".
For \(|\pmb{x}_a^{\mathrm{T}}(\hat{\pmb{\theta}}_{\overline{V}_t,t} - \pmb{\theta}_{V_t})|:
$$
\begin{array}{l} \hat{\boldsymbol{\theta}}_{\overline{V}_{t},t - 1} - \boldsymbol{\theta}_{V_{t}} = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}r_{s}) - \boldsymbol{\theta}_{V_{t}} \\ = (\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}} + \lambda \boldsymbol {I})^{-1}\Bigg(\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}(\boldsymbol{x}^{\top}_{a_{s}}\boldsymbol {\theta}_{i_{s}} + \boldsymbol{\epsilon}^{i_{s},s}_{a_{s}} + \eta_{s})\Bigg) - \boldsymbol{\theta}_{V_{t}} \\ = \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{V_{t}} \\ = \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\mathbf{x}}_{a_{s}}^{\top}(\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{V_{t}}) \\ + \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} (\sum_ {s \in [ t - 1 ] \atop i _ {s} \in \overline {{V}} _ {t}} \boldsymbol {x} _ {a _ {s}} \boldsymbol {x} _ {a _ {s}} ^ {\top} + \lambda \boldsymbol {I}) \boldsymbol {\theta} _ {V _ {t}} - \lambda \overline {{\boldsymbol {M}}} _ {\overline {{V}} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {V _ {t}} - \boldsymbol {\theta} _ {V _ {t}} \\ = \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{\epsilon}_{a_{s}}^{i_{s},s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\eta_{s} + \overline{M}_{\overline{V}_{t},t - 1}^{-1}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top}(\boldsymbol{\theta}_{i_{s}} - \boldsymbol{\theta}_{V_{t}}) \\ - \lambda \overline {{\boldsymbol {M}}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \boldsymbol {\theta} _ {V _ {t}}. \\ \end{array}
$$
Thus, with the same method in Lemma 5.7 but replace  $\zeta = 4\epsilon_{*}\sqrt{\frac{1}{2\tilde{\lambda}_{x}}}$  with  $\zeta_{1} = 6\epsilon_{*}\sqrt{\frac{1}{2\tilde{\lambda}_{x}}}$ , and with the previous reasoning, with probability at least  $1 - 5\delta$ , we have:
$$
\left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t} - \boldsymbol {\theta} _ {V _ {t}}\right) \right| \leq C _ {a _ {t}} + \frac {3 \epsilon_ {*} \sqrt {2 d}}{2 \tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \tag {72}
$$
The lemma can be concluded.
N Proof of Lemma L.2
With the analysis in the proof of Lemma H.1, with probability at least  $1 - \delta$ :
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right) + \epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {\lambda + \lambda_ {\operatorname* {m i n}} (\boldsymbol {M} _ {i , t})}}, \forall i \in \mathcal {U}, \tag {73}
$$
and the estimated error of the current cluster  $\left\| \tilde{\pmb{\theta}}^{j(i)} - \pmb{\theta}^{j(i)}\right\|$  also satisfies this inequality. For set-based clustering structure, to ensure for each user there is only one  $\zeta$ -close cluster, we let:
$$
\frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right) + \epsilon_ {*} \sqrt {T _ {i , t}}}{\sqrt {\lambda + \lambda_ {\min } \left(\boldsymbol {M} _ {i , t}\right)}} \leq \frac {\gamma_ {1}}{6} \tag {74}
$$
By assuming  $\lambda < 2\log \left(\frac{u}{\delta}\right) + d\log \left(1 + \frac{T_{i,t}}{\lambda d}\right)$ , we can simplify it to
$$
\frac {2 \log (\frac {u}{\delta}) + d \log (1 + \frac {T _ {i , t}}{\lambda d})}{2 \tilde {\lambda} _ {x} T _ {i , t}} <   \frac {1}{4} \left(\frac {\gamma_ {1}}{6} - \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}}\right) ^ {2} \tag {75}
$$
which can be proved by  $\frac{2\log(\frac{u}{\delta})}{2\tilde{\lambda}_xT_{i,t}}\leq \frac{1}{8} (\frac{\gamma_1}{6} -\epsilon_*\sqrt{\frac{1}{2\tilde{\lambda}_x}})^2$  and  $\frac{d\log(1 + \frac{T_{i,t}}{\lambda d})}{2\tilde{\lambda}_xT_{i,t}}\leq \frac{1}{8} (\frac{\gamma_1}{6} -\epsilon_*\sqrt{\frac{1}{2\tilde{\lambda}_x}})^2.$  It's obvious that the former one can be satisfied by  $T_{i,t}\geq \frac{8\log(u / \delta)}{\tilde{\lambda}_x(\frac{\gamma_1}{6} - \epsilon_*\sqrt{1 / 2\tilde{\lambda}_x})^2}.$  As for the latter one, by [25] Lemma 9, we can get  $T_{i,t}\geq \frac{8d\log(\frac{16}{\tilde{\lambda}_x\lambda(\frac{\gamma_1}{6} - \epsilon_*\sqrt{1 / 2\tilde{\lambda}_x})^2})}{4\tilde{\lambda}_x(\frac{\gamma_1}{6} - \epsilon_*\sqrt{1 / 2\tilde{\lambda}_x})^2}.$  By assuming  $\frac{u}{\delta}\geq \frac{16}{4\tilde{\lambda}_x\lambda(\frac{\gamma_1}{6} - \epsilon_*\sqrt{2 / 4\tilde{\lambda}_x})^2},$  the lemma is proved.
O Proof of Theorem L.1
After  $2T_{1}$  rounds, in each phase, at most  $u$  times split operations will happen, we use  $u\log (T)$  to bound the regret generated in these rounds. Then in the remained rounds the cluster num will be no more than  $m$ .
For the instantaneous regret  $R_{t}$  at round  $t$ , with probability at least  $1 - 2\delta$  for some  $\delta \in (0, \frac{1}{2})$ :
$$
\begin{array}{l} R _ {t} = \left(\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\mathrm {T}} \boldsymbol {\theta} _ {i _ {t}} + \boldsymbol {\epsilon} _ {a _ {t} ^ {*}} ^ {i _ {t}, t}\right) - \left(\boldsymbol {x} _ {a _ {t}} ^ {\mathrm {T}} \boldsymbol {\theta} _ {i _ {t}} + \boldsymbol {\epsilon} _ {a _ {t}} ^ {i _ {t}, t}\right) \\ = \pmb {x} _ {a _ {t} ^ {*}} ^ {\top} (\pmb {\theta} _ {i _ {t}} - \hat {\pmb {\theta}} _ {\overline {{V}} _ {t}, t - 1}) + (\pmb {x} _ {a _ {t} ^ {*}} ^ {\top} \hat {\pmb {\theta}} _ {\overline {{V}} _ {t}, t - 1} + C _ {a _ {t} ^ {*}, t}) - (\pmb {x} _ {a _ {t}} ^ {\top} \hat {\pmb {\theta}} _ {\overline {{V}} _ {t}, t - 1} + C _ {a _ {t}, t}) \\ + \boldsymbol {x} _ {a _ {t}} ^ {\top} \left(\hat {\boldsymbol {\theta}} _ {\bar {V} _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) + C _ {a _ {t}, t} - C _ {a _ {t} ^ {*}, t} + \left(\boldsymbol {\epsilon} _ {a _ {t} ^ {*}} ^ {i _ {t}, t} - \boldsymbol {\epsilon} _ {a _ {t}} ^ {i _ {t}, t}\right) \tag {76} \\ \leq 2 C _ {a _ {t}} + 2 \epsilon_ {*} + (1 2 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} + \frac {3 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}}) \mathbb {I} (\bar {V} _ {t} \notin V) \\ \end{array}
$$
where the last inequality holds due to the UCB arm selection strategy, the concentration bound given in LemmaL.3 and the fact that  $\left\| \epsilon^{i,t}\right\|_{\infty} \leq \epsilon_*$ .
Define such events. Let:
$$
\mathcal {E} _ {2} = \left\{\text {A l l c l u s t e r s} \bar {V} _ {t} \text {o n l y c o n t a i n u s e r s w h o s a s s i f y} \left\| \tilde {\boldsymbol {\theta}} _ {i} - \tilde {\boldsymbol {\theta}} _ {\bar {V} _ {t}} \right\| \leq \alpha_ {1} (\sqrt {\frac {1 + \log (1 + T _ {i , t})}{1 + T _ {i , t}}} + \sqrt {\frac {1 + \log (1 + T _ {\bar {V} _ {t} , t})}{1 + T _ {\bar {V} _ {t} , t}}}) + \alpha_ {2} \epsilon_ {*} \right\}
$$
$$
\mathcal {E} _ {3} = \left\{r _ {t} \leq 2 C _ {a _ {t}} + 2 \epsilon_ {*} + 1 2 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} + \frac {3 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} \right\}
$$
$$
\mathcal {E} ^ {\prime} = \mathcal {E} _ {2} \cap \mathcal {E} _ {3}
$$
From previous analysis, we can know that  $\mathbb{P}(\mathcal{E}_2) \geq 1 - 3\delta$  and  $\mathbb{P}(\mathcal{E}_3) \geq 1 - 2\delta$ , thus  $\mathbb{P}(\mathcal{E}' \geq 1 - 5\delta)$ . By taking  $\delta = \frac{1}{T}$ , we can get:
$$
\begin{array}{l} E (R _ {t}) = P (\mathcal {E}) \mathbb {I} \{\mathcal {E} \} R _ {t} + P (\bar {\mathcal {E}}) \mathbb {I} \{\bar {\mathcal {E}} \} R _ {t} \\ \leq \mathbb {I} \{\mathcal {E} \} R _ {t} + 5 \\ \leq 2 T _ {1} + 2 \epsilon_ {*} T + \left(1 2 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} + \frac {3 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}}\right) T + 2 \sum_ {2 T _ {1}} ^ {T} C _ {a _ {t}} + 5 \tag {77} \\ \end{array}
$$
Now we need to bound  $2\sum_{2T_1}^T C_{a_t}$ . We already know that after  $2T_{1}$  rounds, in each phase  $k$  after the first  $u$  rounds, there will be at most  $m$  clusters
Consider phase  $k$ , for simplicity, ignore the fist  $u$  rounds. For the first term in  $C_{a_t}$ :
$$
\begin{array}{l} \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {\bar {V} _ {t}, t - 1}} ^ {- 1} = \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \sum_ {j = 1} ^ {m _ {t}} \mathbb {I} \{i \in \bar {V} _ {t, j} \} \left\| \boldsymbol {x} _ {a _ {t}} \right\| _ {\overline {{M}} _ {\bar {V} _ {t}, j}} ^ {- 1} \\ \leq \sum_ {j = 1} ^ {m _ {t}} \sqrt {\sum_ {t = T _ {k - 1}} ^ {T _ {k}} \mathbb {I} \{i \in V _ {t , j} \} \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \mathbb {I} \{i \in V _ {t , j} \} \| \boldsymbol {x} _ {a _ {t}} \| _ {M _ {V _ {t , j}} ^ {- 1}} ^ {2}} \tag {78} \\ \leq \sum_ {j = 1} ^ {m _ {t}} \sqrt {2 T _ {k , j} d \log (1 + \frac {T}{\lambda d})} \\ \leq \sqrt {2 m (T _ {k} - T _ {k - 1}) d \log \left(1 + \frac {T}{\lambda d}\right)} \\ \end{array}
$$
For all phases:
$$
\begin{array}{l} \sum_ {k = 1} ^ {s} \sqrt {2 m \left(T _ {k + 1} - T _ {k}\right) d \log \left(1 + \frac {T}{\lambda d}\right)} \leq \sqrt {2 \sum_ {k = 1} ^ {s} 1 \sum_ {k = 1} ^ {s} \left(T _ {k + 1} - T _ {k}\right) m d \log \left(1 + \frac {T}{\lambda d}\right)} \tag {79} \\ \leq \sqrt {2 m d T \log (T) \log \left(1 + \frac {T}{\lambda d}\right)} \\ \end{array}
$$
Similarly, for the second term in  $C_{a_t}$ :
$$
\begin{array}{l} \sum_{t = T_{k - 1}}^{T_{k}}\sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t}}}\epsilon_{*}|\boldsymbol{x}_{a_{t}}^{\mathrm{T}}\overline{M}_{\overline{V}_{t,t - 1}}^{-1}\boldsymbol{x}_{a_{s}}| = \sum_{t = T_{k - 1}}^{T_{k}}\sum_{j = 1}^{m_{t}}\mathbb{I}\{i\in \overline{V}_{t,j}\} \sum_{\substack{s\in [t - 1]\\ i_{s}\in \overline{V}_{t,j}}}\epsilon_{*}|\boldsymbol{x}_{a_{t}}^{\mathrm{T}}\overline{M}_{\overline{V}_{t,j}^{-1}}\boldsymbol{x}_{a_{s}}| \\ \leq \epsilon_{*}\sum_{t = T_{k - 1}}^{T_{k}}\sum_{j = 1}^{m_{t}}\mathbb{I}\{i\in \overline{V}_{t,j}\} \\ \sqrt{\sum_{\substack{s\in[t - 1]\\ i_{s}\in\overline{V}_{t,j}}}1\sum_{\substack{s\in[t - 1]\\ i_{s}\in\overline{V}_{t,j}}}|\boldsymbol{x}_{a_{t}}^{T}\overline{M}_{\overline{V}_{t,j}^{-1}}\boldsymbol{x}_{a_{s}}|^{2}} \\ \leq \epsilon_ {*} \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \sum_ {j = 1} ^ {m _ {t}} \mathbb {I} \{i \in \overline {{V}} _ {t, j} \} \sqrt {T _ {k , j} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {V _ {t , j}} ^ {- 1}} ^ {2}} \\ \leq \epsilon_ {*} \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \sqrt {\sum_ {j = 1} ^ {m _ {t}} \mathbb {I} \{i \in \bar {V} _ {t , j} \} \sum_ {j = 1} ^ {m _ {t}} \mathbb {I} \{i \in \bar {V} _ {t , j} \} T _ {k , j} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {\bar {V} _ {t , j}} ^ {- 1}} ^ {2}} \\ \leq \epsilon_ {*} \sqrt {\left(T _ {k} - T _ {k - 1}\right)} \sum_ {t = T _ {k - 1}} ^ {T _ {k}} \sqrt {\sum_ {j = 1} ^ {m _ {t}} \mathbb {I} \{i \in \bar {V} _ {t , j} \} \| \boldsymbol {x} _ {a _ {t}} \| _ {\overline {{M}} _ {\bar {V} _ {t , j}} ^ {- 1}} ^ {2}} \\ \leq \epsilon_ {*} \left(T _ {k} - T _ {k - 1}\right) \sqrt {2 m d \log \left(1 + \frac {T}{\lambda d}\right)} \tag {80} \\ \end{array}
$$
Then for all phases this term can be bounded by  $\epsilon_{*}T\sqrt{2md\log(1 + \frac{T}{\lambda d})}$ .
Thus the total regret can be bounded by:
$$
\begin{array}{l} R _ {t} \leq 2 \sqrt {2 m T d \log (T) \log \left(1 + \frac {T}{\lambda d}\right)} \left(\sqrt {2 \log (T) + d \log \left(1 + \frac {T}{\lambda d}\right)} + 2 \sqrt {\lambda}\right) \\ + 2 \epsilon_ {*} T \sqrt {2 m d \log (1 + \frac {T}{\lambda d})} + 2 \epsilon_ {*} T + 1 2 \epsilon_ {*} \sqrt {\frac {1}{2 \tilde {\lambda} _ {x}}} T + \frac {3 \epsilon_ {*} \sqrt {2 d}}{\tilde {\lambda} _ {x} ^ {\frac {3}{2}}} T + 2 T _ {1} + u \log (T) + 5 \\ \end{array}
$$
where  $T_{1} = 16u\log (\frac{u}{\delta}) + 4u\max \{\frac{16}{\tilde{\lambda}_{x}^{2}}\log (\frac{8d}{\tilde{\lambda}_{x}^{2}\delta}),\frac{8d}{\tilde{\lambda}_{x}(\frac{\gamma_{1}}{6} - \epsilon_{*}\sqrt{\frac{1}{2\tilde{\lambda}_{x}}})^{2}}\log (\frac{u}{\delta})\}$
(a) Known Misspecification Level
(b) Unknown Misspecification Level
Figure 2: The cumulative regret of the algorithms under different scales of misspecification level.
P More Experiments
For ablation study, we test our algorithms' performance under different scales of deviation. We test RCLUMB and RSCLUMB when  $\epsilon^{*} = 0.05$ , 0.1, 0.2, 0.3 and 0.4 in both misspecification level known and unknown cases. In the known case, we set  $\epsilon^{*}$  according to the real misspecification level, and we compare our algorithms' performance to the baselines except LinUCB and CW-OFUL which perform worst; in the unknown case, we keep  $\epsilon^{*} = 0.2$ , and we compare our algorithms to RLinUCB-Ind as only it has the pre-spicified parameter  $\epsilon^{*}$  among the baselines. The results are shown in Fig.2. We plot each algorithm's final cumulative regret under different misspecification levels. All the algorithms' performance get worse when the deviation gets larger, and our two algorithms always perform better than the baselines. Besides, the regrets in the unknown case are only slightly larger than the known case. These results can match our theoretical results and again show our algorithms' effectiveness, as well as verify that our algorithm can handle the unknown misspecification level.