SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Update the robust estimation of user  $i_t$ . After receiving  $r_t$ , the algorithm updates the estimation statistics of user  $i_t$ , while keeping the statistics of others unchanged (Line 8 and Line 9). Specifically, RCLUB-WCU estimates the preference vector of user  $i_t$  by solving a weighted ridge regression
$$
\hat {\boldsymbol {\theta}} _ {i _ {t}, t} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\arg \min } \sum_ {\substack {s \in [ t ] \\ i _ {s} = i _ {t}}} w _ {i _ {s}, s} \left(r _ {s} - \boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta}\right) ^ {2} + \lambda \| \boldsymbol {\theta} \| _ {2} ^ {2} \tag{4}
$$
with closed-form solution  $\hat{\pmb{\theta}}_{i_t,t} = (\lambda \pmb {I} + \pmb{M}_{i_t,t})^{-1}\pmb{b}_{i_t,t}$ , where  $M_{i_t,t} = \sum_{\substack{s\in [t]\\ i_s = i_t}}w_{i_s,s}\pmb{x}_{a_s}\pmb{x}_{a_s}^\top$ $\pmb{b}_{i_t,t} = \sum_{\substack{s\in [t]\\ i_s = i_t}}w_{i_s,s}r_{a_s}\pmb{x}_{a_s}$ , and we design the weights in the same way by the same reasoning. Update the dynamic graph. Finally, with the updated statistics of user  $i_t$ , RCLUB-WCU checks
Algorithm 2 OCCUD (At round  $t$ , used in Line 11 in Algo.1)
1: Initialize  $\tilde{\mathcal{U}}_t = \emptyset$ ; input probability parameter  $\delta$ .
2: Update the statistics for non-robust estimation of user  $i_t$
$$
\tilde {\boldsymbol {M}} _ {i _ {t}, t} = \tilde {\boldsymbol {M}} _ {i _ {t}, t - 1} + \boldsymbol {x} _ {a _ {t}} \boldsymbol {x} _ {a _ {t}} ^ {\top}, \tilde {\boldsymbol {b}} _ {i _ {t}, t} = \tilde {\boldsymbol {b}} _ {i _ {t}, t - 1} + r _ {t} \boldsymbol {x} _ {a _ {t}}, \tilde {\boldsymbol {\theta}} _ {i _ {t}, t} = (\lambda \boldsymbol {I} + \tilde {\boldsymbol {M}} _ {i _ {t}, t}) ^ {- 1} \tilde {\boldsymbol {b}} _ {i _ {t}, t},
$$
3: Keep non-robust estimation statistics of other users unchanged
$$
\tilde {M} _ {\ell , t} = \tilde {M} _ {\ell , t - 1}, \tilde {b} _ {\ell , t} = \tilde {b} _ {\ell , t - 1}, \tilde {\theta} _ {\ell , t} = \tilde {\theta} _ {\ell , t - 1}, \text {f o r a l l} \ell \in \mathcal {U}, \ell \neq i _ {t}.
$$
4: for all connected component  $V_{j,t} \in G_t$  do
5: Calculate the robust estimation statistics for the cluster  $V_{j,i}$ :
$$
\boldsymbol {M} _ {V _ {j, t}, t} = \lambda \boldsymbol {I} + \sum_ {\ell \in V _ {j, t}} \boldsymbol {M} _ {\ell , t}, T _ {V _ {j, t}, t} = \sum_ {\ell \in V _ {j, t}} T _ {\ell , t},
$$
$$
\boldsymbol {b} _ {V _ {j, t}, t} = \sum_ {\ell \in V _ {j, t}} \boldsymbol {b} _ {\ell , t}, \boldsymbol {\theta} _ {V _ {j, t}, t} = \boldsymbol {M} _ {V _ {j, t}, t} ^ {- 1} \boldsymbol {b} _ {V _ {j, t}, t};
$$
6: for all user  $i \in \tilde{V}_{j,t}$  do
7: Detect user  $i$  to be a corrupted user and add user  $i$  to the set  $\mathcal{U}_t$  if the following holds:
$$
\left\| \tilde {\boldsymbol {\theta}} _ {i, t} - \hat {\boldsymbol {\theta}} _ {V _ {i, t}, t} \right\| _ {2} > \frac {\sqrt {d \log \left(1 + \frac {T _ {i , t}}{\lambda d}\right) + 2 \log \left(\frac {1}{\delta}\right)} + \sqrt {\lambda}}{\sqrt {\lambda_ {\min } \left(\tilde {M} _ {i , t}\right) + \lambda}} + \frac {\sqrt {d \log \left(1 + \frac {T _ {V _ {i , t} , t}}{\lambda d}\right) + 2 \log \left(\frac {1}{\delta}\right)} + \sqrt {\lambda} + \alpha C}{\sqrt {\lambda_ {\min } \left(M _ {V _ {i , t} , t}\right)}}, \tag {5}
$$
where  $\lambda_{\mathrm{min}}(\cdot)$  denotes the minimum eigenvalue of the matrix argument.
8: end for
9: end for
whether the inferred  $i_{t}$ 's preference similarities with other users are still true, and updates the graph accordingly. Precisely, if gap between the updated estimation  $\hat{\theta}_{i_t,t}$  of  $i_{t}$  and the estimation  $\hat{\theta}_{\ell ,t}$  of user  $\ell$  exceeds a threshold in Line 10, RCLUB-WCU will delete the edge  $(i_t,\ell)$  in  $G_{t - 1}$  to split them apart. The threshold is carefully designed to handle the estimation uncertainty from both stochastic noises and potential corruptions. The updated graph  $G_{t} = (\mathcal{U},E_{t})$  will be used in the next round.
4.2 OCCUD
Based on the inferred clustering structure of RCLUB-WCU, we devise a novel online detection algorithm OCCUD (Algo.2). The design ideas and process of OCCUD are as follows.
Besides the robust preference estimations (with weighted regression) of users and clusters kept by RCLUB-WCU, OCCUD also maintains the non-robust estimations for each user by online ridge regression without weights (Line 2 and Line 3). Specifically, at round  $t$ , OCCUD updates the non-robust estimation of user  $i_t$  by solving the following online ridge regression:
$$
\tilde {\boldsymbol {\theta}} _ {i _ {t}, t} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\arg \min } \sum_ {\substack {s \in [ t ] \\ i _ {s} = i _ {t}}} (r _ {s} - \boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta}) ^ {2} + \lambda \| \boldsymbol {\theta} \| _ {2} ^ {2}, \tag{6}
$$
with solution  $\tilde{\pmb{\theta}}_{i_t,t} = (\lambda \pmb {I} + \tilde{\pmb{M}}_{i_t,t})^{-1}\tilde{\pmb{b}}_{i_t,t}$ , where  $\tilde{M}_{i_t,t} = \sum_{\substack{s\in [t]\\ i_s = i_t}}\pmb{x}_{a_s}\pmb{x}_{a_s}^\top ,\tilde{\pmb{b}}_{i_t,t} = \sum_{\substack{s\in [t]\\ i_s = i_t}}r_{a_s}\pmb{x}_{a_s}$ .
With the robust and non-robust preference estimations, OCCUD does the following to detect corrupted users based on the clustering structure inferred by RCLUB-WCU. First, OCCUD finds the connected components in the graph kept by RCLUB-WCU, which represent the inferred clusters. Then, for each inferred cluster  $V_{j,t} \in G_t$ : (1) OCCUD computes its robustly estimated preferences vector  $\hat{\theta}_{V_{i,t},t}$  (Line 5). (2) For each user  $i$  whose inferred cluster is  $V_{j,t}$  ( $i.e., i \in V_{j,t}$ ), OCCUD computes the gap between user  $i$ 's non-robustly estimated preference vector  $\hat{\theta}_{i,t}$  and the robust estimation  $\hat{\theta}_{V_{i,t},t}$  for user  $i$ 's inferred cluster  $V_{j,t}$ . If the gap exceeds a carefully-designed threshold, OCCUD will detect user  $i$  as corrupted and add  $i$  to the detected corrupted user set  $\tilde{\mathcal{U}}_t$  (Line 7).
Figure 2: Algorithm illustrations. Users 6 and 8 are corrupted users (orange), and the others are normal (green). (a) illustrates RCLUB-WCU, which starts with a complete user graph, and adaptively deletes edges between users (dashed lines) with dissimilar robustly learned preferences. The corrupted behaviors of users 6 and 8 may cause inaccurate preference estimations, leading to erroneous relation inference. In this case, how to delete edges correctly is non-trivial, and RCLUB-WCU addresses this challenge (detailed in Section 4.1). (b) illustrates OCCUD at some round  $t$ , where person icons with triangle hats represent the non-robust user preference estimations. The gap between the non-robust estimation of user 6 and the robust estimation of user 6's inferred cluster ( $\text{circle } C_1$ ) exceeds the threshold  $r_6$  at this round (Line 7 in Algo.2), so OCCUD detects user 6 to be corrupted.
The intuitions of OCCUD are as follows. On the one hand, after some interactions, RCLUB-WCU will infer the user clustering structure accurately. Thus, at round  $t$ , the robust estimation  $\hat{\theta}_{V_{i,t},t}$  for user  $i$ 's inferred cluster should be pretty close to user  $i$ 's ground-truth preference vector  $\theta_i$ . On the other hand, since the feedback of normal users are always regular, at round  $t$ , if user  $i$  is a normal user, the non-robust estimation  $\tilde{\theta}_{i,t}$  should also be close to the ground-truth  $\theta_i$ . However, the non-robust estimation of a corrupted user should be quite far from the ground truth due to corruptions. Based on this reasoning, OCCUD compares each user's non-robust estimation and the robust estimation of the user's inferred cluster to detect the corrupted users. For technical details, please refer to Section 5.2 and Appendix. Simple illustrations of our proposed algorithms can be found in Fig.2.
In this section, we theoretically analyze the performances of our proposed algorithms, RCLUB-WCU and OCCUD. Due to the page limit, we put the proofs in the Appendix.
5.1 Regret Analysis of RCLUB-WCU
This section gives an upper bound of the expected regret (defined in Eq.(1)) for RCLUB-WCU.
The following lemma provides a sufficient time  $T_0(\delta)$ , after which RCLUB-WCU can cluster all the users correctly with high probability.
Lemma 1. With probability at least  $1 - 3\delta$ , RCLUB-WCU will cluster all the users correctly after
$$
T _ {0} (\delta) \triangleq 1 6 u \log (\frac {u}{\delta}) + 4 u \max \{\frac {2 8 8 d}{\gamma^ {2} \alpha \sqrt {\lambda} \tilde {\lambda} _ {x}} \log (\frac {u}{\delta}), \frac {1 6}{\tilde {\lambda} _ {x} ^ {2}} \log (\frac {8 d}{\tilde {\lambda} _ {x} ^ {2} \delta}), \frac {7 2 \sqrt {\lambda}}{\alpha \gamma^ {2} \tilde {\lambda} _ {x}}, \frac {7 2 \alpha C ^ {2}}{\gamma^ {2} \sqrt {\lambda} \tilde {\lambda} _ {x}} \}
$$
for some  $\delta \in (0,\frac{1}{3})$ , where  $\tilde{\lambda}_x \triangleq \int_0^{\lambda_x} (1 - e^{-\frac{(\lambda_x - x)^2}{2\sigma^2}})^K dx$ ,  $|\mathcal{A}_t| \leq K, \forall t \in [T]$ .
After  $T_0(\delta)$ , the following lemma gives a bound of the gap between  $\hat{\theta}_{V_t,t - 1}$  and the ground-truth  $\pmb{\theta}_{i_t}$  in direction of action vector  $\pmb{x}_a$  for RCLUB-WCU, which supports the design in Eq.(3).
Lemma 2. With probability at least  $1 - 4\delta$  for some  $\delta \in (0,\frac{1}{4}),\forall t\geq T_0(\delta)$  we have:
$$
\left| \boldsymbol {x} _ {a} ^ {\mathrm {T}} \left(\hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} - \boldsymbol {\theta} _ {i _ {t}}\right) \right| \leq \beta \| \boldsymbol {x} _ {\boldsymbol {a}} \| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {- 1}} \triangleq C _ {a, t}.
$$
With Lemma 1 and 2, we prove the following theorem on the regret upper bound of RCLUB-WCU.
Theorem 3 (Regret Upper Bound of RCLUB-WCU). With the assumptions in Section 3, and picking  $\alpha = \frac{\sqrt{d} + \sqrt{\lambda}}{C}$ , the expected regret of the RCLUB-WCU algorithm for  $T$  rounds satisfies
$$
R (T) \leq O \left(\left(\frac {C \sqrt {d}}{\gamma^ {2} \tilde {\lambda} _ {x}} + \frac {1}{\tilde {\lambda} _ {x} ^ {2}}\right) u \log (T)\right) + O \left(d \sqrt {m T} \log (T)\right) + O \left(m C d \log^ {1. 5} (T)\right). \tag {7}
$$
Discussion and Comparison. The regret bound in Eq.(7) has three terms. The first term is the time needed to get enough information for accurate robust estimations such that RCLUB-WCU could cluster all users correctly afterward with high probability. This term is related to the corruption level  $C$ , which is inevitable since, if there are more corrupted user feedback, it will be harder for the algorithm to learn the clustering structure correctly. The last two terms correspond to the regret after  $T_{0}$  with the correct clustering. Specifically, the second term is caused by stochastic noises when leveraging the aggregated information within clusters to make recommendations; the third term associated with the corruption level  $C$  is the regret caused by the disruption of corrupted behaviors.
When the corruption level  $C$  is unknown, we can use its estimated upper bound  $\hat{C} \triangleq \sqrt{T}$  to replace  $C$  in the algorithm. In this way, if  $C \leq \hat{C}$ , the bound will be replacing  $C$  with  $\hat{C}$  in Eq.(7); when  $C > \sqrt{T}$ ,  $R(T) = O(T)$ , which is already optimal for a large class of bandit algorithms [12].
The following theorem gives a regret lower bound of the LOCUD problem.
Theorem 4 (Regret lower bound for LOCUD). There exists a problem instance for the LOCUD problem such that for any algorithm
$$
R (T) \geq \Omega (d \sqrt {m T} + d C).
$$
Its proof and discussions can be found in Appendix D. The upper bound in Theorem 3 asymptotically matches this lower bound in  $T$  up to logarithmic factors, showing our regret bound is nearly optimal.
We then compare our regret upper bound with several degenerated cases. First, when  $C = 0$ , i.e., all users are normal, our setting degenerates to the classic CB problem [8]. In this case the bound in Theorem 3 becomes  $O(1 / \tilde{\lambda}_x^2\cdot u\log (T)) + O(d\sqrt{mT}\log (T))$ , perfectly matching the state-of-the-art results in CB [8, 19, 21]. Second, when  $m = 1$  and  $u = 1$ , i.e., there is only one user, our setting degenerates to linear bandits with adversarial corruptions [22, 12], and the bound in Theorem 3 becomes  $O(d\sqrt{T}\log (T)) + O(Cd\log^{1.5}(T))$ , it also perfectly matches the nearly optimal result in [12]. The above comparisons also show the tightness of the regret bound of RCLUB-WCU.
5.2 Theoretical Performance Guarantee for OCCUD
The following theorem gives a performance guarantee of the online detection algorithm OCCUD.
Theorem 5 (Theoretical Guarantee for OCCUD). With assumptions in Section 3, at  $\forall t \geq T_0(\delta)$ , for any detected corrupted user  $i \in \tilde{\mathcal{U}}_t$ , with probability at least  $1 - 5\delta$ ,  $i$  is indeed a corrupted user.
This theorem guarantees that after RCLUB-WCU learns the clustering structure accurately, with high probability, the corrupted users detected by OCCUD are indeed corrupted, showing the high detection accuracy of OCCUD. The proof of Theorem 5 can be found in Appendix D.
This section shows experimental results on synthetic and real data to evaluate RCLUB-WCU's recommendation quality and OCCUD's detection accuracy. We compare RCLUB-WCU to LinUCB [1] with a single non-robust estimated vector for all users, LinUCB-Ind with separate non-robust estimated vectors for each user, CW-OFUL [12] with a single robust estimated vector for all users, CW-OFUL-Ind with separate robust estimated vectors for each user, CLUB[8], and SCLUB[21]. More description of these baselines are in Appendix F. To show that the design of OCCUD is nontrivial, we develop a straightforward detection algorithm GCUD, which leverages the same cluster structure as OCCUD but detects corrupted users by selecting users with highest  $\left\| \hat{\pmb{\theta}}_{i,t} - \hat{\pmb{\theta}}_{V_{i,t},t - 1}\right\|_2$  in each inferred cluster. GCUD selects users according to the underlying percentage of corrupted users, which is unrealistic in practice, but OCCUD still performs better in this unfair condition.
Remark. The offline detection methods [39, 6, 18, 32] need to know all the user information in advance to derive the user embedding for classification, so they cannot be directly applied in online detection with bandit feedback thus cannot be directly compared to OCCUD. However, we observe the AUC achieved by OCCUD on Amazon and Yelp (in Tab.1) is similar to recent offline methods [18, 32]. Additionally, OCCUD has rigorous theoretical performance guarantee (Section 5.2).
6.1 Experiments on Synthetic Dataset
We use  $u = 1,000$  users and  $m = 10$  clusters, where each cluster contains 100 users. We randomly select 100 users as the corrupted users. The preference and arm (item) vectors are drawn in  $d - 1$  ( $d = 50$ ) dimensions with each entry a standard Gaussian variable and then normalized, added one more dimension with constant 1, and divided by  $\sqrt{2}$  [21]. We fix an arm set with  $|\mathcal{A}| = 1000$  items, at each round, 20 items are randomly selected to form a set  $\mathcal{A}_t$  to choose from. Following [40, 3], in the first  $k$  rounds, we always flip the reward of corrupted users by setting  $r_t = -\pmb{x}_{a_t}^{\mathrm{T}}\pmb{\theta}_{i_t,t} + \eta_t$ . And we leave the remaining  $T - k$  rounds intact. Here we set  $T = 1,000,000$  and  $k = 20,000$ .
(b) Movielens
(c) Amazon
Figure 3: Recommendation results on the synthetic and real-world datasets
(d) Yelp
Fig.3(a) shows the recommendation results. RCLUB-WCU outperforms all baselines and achieves a sub-linear regret. LinUCB and CW-OFUL perform worst as they ignore the preference differences among users. CW-OFUL-Ind outperforms LinUCB-Ind because it considers the corruption, but worse than RCLUB-WCU since it does not consider leveraging user relations to speed up learning.
The detection results are shown in Tab.1. We test the AUC of OCCUD and GCUD in every 200,000 rounds. OCCUD's performance improves over time with more interactions, while GCUD's performance is much worse as it detects corrupted users only relying on the robust estimations. OCCUD finally achieves an AUC of 0.855, indicating it can identify most of the corrupted users.
We use three real-world data Movielens [11], Amazon[31], and Yelp [33]. The Movielens data does not have the corrupted users' labels, so following [24], we manually select the corrupted users. On Amazon data, following [39], we label the users with more than  $80\%$  helpful votes as normal users, and label users with less than  $20\%$  helpful votes as corrupted users. The Yelp data contains users and their comments on restaurants with true labels of the normal users and corrupted users.
We select 1,000 users and 1,000 items for Movielens; 1,400 users and 800 items for Amazon; 2,000 users and 2,000 items for Yelp. The ratios of corrupted users on these data are  $10\%$ ,  $3.5\%$ , and
$30.9\%$ , respectively. We generate the preference and item vectors following [37, 21]. We first construct the binary feedback matrix through the users' ratings: if the rating is greater than 3, the feedback is 1; otherwise, the feedback is 0. Then we use SVD to decompose the extracted binary feedback matrix  $R_{u\times m} = \theta SX^{\mathrm{T}}$ , where  $\pmb {\theta} = (\pmb {\theta}_i)$ $i\in [u]$  and  $X = (x_{j}),j\in [m]$ , and select  $d =$  50 dimensions. We have 10 clusters on Movielens and Amazon, and 20 clusters on Yelp. We use the same corruption mechanism as the synthetic data with  $T = 1,000,000$  and  $k = 20,000$ . We conduct more experiments in different environments to show our algorithms' robustness in Appendix.G.
The recommendation results are shown in
Fig.3(b)-(d). RCLUB-WCU outperforms all baselines. On the Amazon dataset, the percentage of corrupted users is lowest, RCLUB-WCU's advantages over baselines decrease because of the weakened corruption. The corrupted user detection results are provided in Tab.1. OCCUD's performance improves over time and is much better than GCUD. On the MovieLens dataset, OCCUD achieves an AUC of 0.85; on the Amazon dataset, OCCUD
Dataset	Time Alg	0.2M	0.4M	0.6M	0.8M	1M
Synthetic	OCCUD	0.599	0.651	0.777	0.812	0.855
GCUD	0.477	0.478	0.483	0.484	0.502
Movielens	OCCUD	0.65	0.750	0.785	0.83	0.85
GCUD	0.450	0.474	0.485	0.489	0.492
Amazon	OCCUD	0.639	0.735	0.761	0.802	0.840
GCUD	0.480	0.480	0.486	0.500	0.518
Yelp	OCCUD	0.452	0.489	0.502	0.578	0.628
GCUD	0.473	0.481	0.496	0.500	0.510
Table 1: Detection results on synthetic and real datasets
achieves an AUC of 0.84; and on the Yelp dataset, OCCUD achieves an AUC of 0.628. According to recent works on offline settings [18, 32], our results are relatively high.
In this paper, we are the first to propose the novel LOCUD problem, where there are many users with unknown preferences and unknown relations, and some corrupted users can occasionally perform disrupted actions to fool the agent. Hence, the agent not only needs to learn the unknown user preferences and relations robustly from potentially disrupted bandit feedback, balance the exploration-exploitation trade-off to minimize regret, but also needs to detect the corrupted users over time. To robustly learn and leverage the unknown user preferences and relations from corrupted behaviors, we propose a novel bandit algorithm RCLUB-WCU. To detect the corrupted users in the online bandit setting, based on the learned user relations of RCLUB-WCU, we propose a novel detection algorithm OCCUD. We prove a regret upper bound for RCLUB-WCU, which matches the lower bound asymptotically in  $T$  up to logarithmic factors and matches the state-of-the-art results in degenerate cases. We also give a theoretical guarantee for the detection accuracy of OCCUD. Extensive experiments show that our proposed algorithms achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.
The corresponding author Shuai Li is supported by National Key Research and Development Program of China (2022ZD0114804) and National Natural Science Foundation of China (62376154, 62006151, 62076161). The work of John C.S. Lui was supported in part by the RGC's SRFS2122-4S02.
[35] Zhiyong Wang, Xutong Liu, Shuai Li, and John CS Lui. Efficient explorative key-term selection strategies for conversational contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10288-10295, 2023.
[36] Zhiyong Wang, Jize Xie, Xutong Liu, Shuai Li, and John Lui. Online clustering of bandits with misspecified user models. arXiv preprint arXiv:2310.02717, 2023.
[37] Junda Wu, Canzhe Zhao, Tong Yu, Jingyang Li, and Shuai Li. Clustering of conversational bandits for user preference learning and elicitation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2129-2139, 2021.
[38] Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 529-538, 2016.
[39] Ge Zhang, Jia Wu, Jian Yang, Amin Beheshti, Shan Xue, Chuan Zhou, and Quan Z Sheng. Fraud: fraud detection dual-resistant to graph inconsistency and imbalance. In 2021 IEEE International Conference on Data Mining (ICDM), pages 867-876. IEEE, 2021.
[40] Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615, 2021.
[41] Jinhang Zuo, Zhiyao Zhang, Zhiyong Wang, Shuai Li, Mohammad Hajiesmaili, and Adam Wierman. Adversarial attacks on online learning to rank with click feedback. arXiv preprint arXiv:2305.17071, 2023.
We first prove the following result:
$$
\left\| \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \right\| _ {2} \leq \frac {\beta \left(T _ {i , t} , \frac {\delta}{u}\right)}{\sqrt {\lambda + \lambda_ {\min } \left(\boldsymbol {M} _ {i , t}\right)}}, \forall i \in \mathcal {U}, \tag {8}
$$
where  $\beta (T_{i,t},\frac{\delta}{u})\triangleq \sqrt{2\log(\frac{u}{\delta}) + d\log(1 + \frac{T_{i,t}}{\lambda d})} +\sqrt{\lambda} +\alpha C.$
$$
\begin{array}{l} \hat {\boldsymbol {\theta}} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} = (\lambda \boldsymbol {I} + \boldsymbol {M} _ {i, t}) ^ {- 1} \boldsymbol {b} _ {i, t} - \boldsymbol {\theta} ^ {j (i)} \\ = (\lambda I + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}})^{-1}\sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}r_{s} - \boldsymbol{\theta}^{j(i)} \\ = (\lambda \boldsymbol {I} + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}^{\top}_{a_{s}})^{-1}\Bigg(\sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}(\boldsymbol{x}^{\top}_{a_{s}}\boldsymbol{\theta}_{i_{s}} + \eta_{s} + c_{s})\Bigg) - \boldsymbol{\theta}^{j(i)} \\ = (\lambda I + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top})^{-1}\Bigg[(\lambda I + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\boldsymbol{x}_{a_{s}}^{\top})\boldsymbol{\theta}^{j(i)} - \lambda \boldsymbol{\theta}^{j(i)} + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s} \\ \left. + \sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}c_{s}\right] - \boldsymbol{\theta}^{j(i)} \\ = -\lambda M^{\prime -1}_{i,t}\boldsymbol{\theta}^{j(i)} + M^{\prime -1}_{i,t}\sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}\eta_{s} + M^{\prime -1}_{i,t}\sum_{\substack{s\in [t]\\ i_{s} = i}}w_{i_{s},s}\boldsymbol{x}_{a_{s}}c_{s}  , \\ \end{array}
$$
where we denote  $M_{i,t}^{\prime} = M_{i,t} + \lambda I$ , and the above equations hold by definition.