SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The weight  $\min(1, \alpha / \|x_{m,t}\|_{V_{m,t}^{-1}})$  is a truncation of the inverse bonus, where  $\alpha > 0$  is a threshold parameter that shall be optimized later. When  $x_{m,t}$  is not explored much, we have a large exploration bonus  $\|x_{m,t}\|_{V_{m,t}^{-1}}$  (low confidence). Hence, the algorithm will put a small weight on it to avoid a large regret due to stochastic noise and misspecification. When  $\|x_{m,t}\|_{V_{m,t}^{-1}}$  is small (high confidence), H-LINUCB puts a large weight on it, and it can be as large as one [He et al., 2022].<sup>3</sup> We note that using this weighting could have a significant negative impact on the performance if we are not careful. Several recent studies show that we can incur a regret of  $\tilde{\mathcal{O}}(d\sqrt{T} + \varepsilon\sqrt{dT})$  and  $\tilde{\mathcal{O}}(d\sqrt{T} + \varepsilon dT)$  for misspecified and corrupted linear bandits, respectively [Lattimore and Csaba, 2020, Takemura et al., 2021, Foster et al., 2020, He et al., 2022].<sup>4</sup> However, a direct application of an algorithm designed for misspecified/corrupted linear bandits to our setting can lead to linear regret when  $\varepsilon = \Theta(1)$ . This is significantly worse as compared to naive independent learning, which always achieves sub-linear  $\tilde{\mathcal{O}}(dM\sqrt{T})$  regret.
We emphasize that the condition  $(t < \tau)$  in Line 18 is crucial to avoid linear regret for H-LINUCB in the regime of large  $\varepsilon$ . For example, for  $\varepsilon = \Theta(1)$ ,  $\tau = T$ , Algorithm 1 incurs  $\tilde{\mathcal{O}}(dMT)$  regret, which is linear in term of  $MT$ . Furthermore, Theorem 4.3 indicates that there exists an instance of  $\varepsilon$ -MALCB such that any algorithm incurs at least  $\Omega(dM\sqrt{T})$  regret. Then, each agent playing OFUL independently would be enough to achieve an optimal  $\tilde{\mathcal{O}}(dM\sqrt{T})$  regret. This suggests that we get a tighter upper bound if we cease collaboration; we discuss the stopping criterion and the choice of  $\tau$  in Theorem 4.2.
Communication protocol. We use a star-shaped communication network where  $M$  agents can interact with a central server [Wang et al., 2020, Dubey and Pentland, 2020]. Each agent communicates with the server by uploading and downloading its data but does not communicate directly with each other. The communication will be triggered only if any agent has enough new data since the last synchronization round. Finally, we assume no latency, or error in the communication between the central server and agents.
3The technique of using the exploration bonus to control misspecification is also used in Zhang et al. [2023].
$^4$ Takemura et al. [2021], Foster et al. [2020] give guarantees for when the misspecification level is unknown. The CW-OFUL algorithm of He et al. [2022] has  $\tilde{\mathcal{O}}(d\sqrt{T} + dC)$  regret, where  $C$  is the total amount of corruption. Setting  $C = \varepsilon T$ , where  $\varepsilon$  is the level of misspecification at each round, gives us  $\tilde{\mathcal{O}}(d\sqrt{T} + \varepsilon dT)$  regret.
Remark 4.1. Wang et al. [2020] show that DISLINUCB can rely on old data and produce nearly optimal policy without much communication, only incurring logarithmic factors in the final regret. H-LINUCB has the same communication cost of  $\mathcal{O}(M^{1.5}d^3)$ , as DISLINUCB, which does not depend on the horizon  $T$ .
When an agent uses data from other agents, due to the dissimilarity, we need to adjust our confidence bound to make sure the true linear parameter  $\theta_{m}$  lies in the defined ellipsoid with high probability. The next result shows how to construct such a Confidence Ellipsoid.
Lemma 4.1. (Confidence Ellipsoid). With probability at least  $1 - M\delta_{1} - M\delta_{2}$ , for each agent  $m \in [M]$ ,  $\theta_{m}$  lies in the confidence set,
$$
\mathcal {C} _ {m, t} = \left\{\theta \in \mathbb {R} ^ {d}: \left\| \hat {\theta} _ {m, t} - \theta \right\| _ {V _ {m, t}} \leq \beta_ {t} \right\},
$$
$$
\beta_ {t} = \left\{ \begin{array}{l} \sqrt {\lambda} + \alpha \varepsilon M t + \sqrt {d \log \left(\frac {1 + M t / (\lambda d)}{\delta_ {1}}\right)}, f o r t <   \tau , \\ \sqrt {\lambda} + \sqrt {d \log \left(\frac {1 + t / (\lambda d)}{\delta_ {2}}\right)}, f o r t \geq \tau . \end{array} \right.
$$
Note that the result above provides two separate confidence bounds for  $\theta_{m}$ , one for the period before round  $\tau$  and the other one for after round  $\tau$ . Before round  $\tau$ , agent  $m$  will use all the data from other agents to construct  $\mathcal{C}_{m,t}$ . The proof follows by first bounding  $\left\| \hat{\theta}_{m,t} - \theta_{m} \right\|_{V_{m,t}(\lambda)}$  as
$$
\left\| \hat {\theta} _ {m, t} - \theta_ {m} \right\| _ {V _ {m, t} (\lambda)} \leq \underbrace {I _ {1}} _ {\text {R e g u l a r i z a t i o n t e r m}} + \underbrace {\left\| \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i , k} x _ {i , k} x _ {i , k} ^ {\top} (\theta_ {i} - \theta_ {m}) \right\| _ {V _ {m , t} ^ {- 1} (\lambda)}} _ {I _ {2}: \text {D i s s i m i l a r i t y t e r m}} + \underbrace {I _ {3}} _ {\text {N o i s e t e r m}}.
$$
Here,  $I_{1}$  is a bounded regularization term,  $I_{3}$  is a noise term that can be bounded by self-normalization lemma (Lemma C.4). Finally, the dissimilarity term  $I_{2}$  is bounded from above by  $\alpha \varepsilon Mt$  using the definition of dissimilarity  $\| \theta_{i} - \theta_{m}\| \leq \varepsilon$ , and applying a similar argument as He et al. [2022] and the choice of the weight  $w_{m,t} = \min(1, \alpha / \| x_{m,t}\|_{V_{m,t}^{-1}})$ . For the phase after round  $\tau$ , we use the same argument as Abbasi-Yadkori et al. [2011] to construct the confidence bound.
Next, we present the group regret upper bound of Algorithm 1 up to round  $\tau$ .
Theorem 4.1. Given Assumption 3.1,  $T \geq 1$ , any  $\tau \leq T$ , and  $\delta_1 > 0$ , setting  $\lambda = 1$  and  $\alpha = \frac{\sqrt{d}}{\varepsilon M\tau}$  in the upper bound of  $\beta_t$  on the confidence interval according to Lemma 4.1  $\forall t \in [T]$ , and setting the synchronization threshold  $D = \tau \log (M\tau) / (dM)$ , we have that with probability at least  $1 - M\delta_1$ , the group regret of Algorithm 1 up to round  $\tau$ , is bounded as
$$
\mathcal {R} (M, \tau) \leq 2 0 \sqrt {2} \left(d \sqrt {M \tau} \xi_ {\tau} ^ {2} + \varepsilon d M \tau \xi_ {\tau} ^ {1. 5}\right),
$$
where  $\xi_{t} = \log \left(\frac{1 + Mt / (\lambda d)}{\delta_{1}}\right)$
Theorem 4.1 shows that Algorithm 1 incurs  $\tilde{\mathcal{O}}(d\sqrt{M\tau})$  regret in the first term (which is the same order as a single agent playing for  $M\tau$  rounds) plus a penalty of using the data from other agents in the order of  $\tilde{\mathcal{O}}(\varepsilon dM\tau)$ . The regret of  $\tilde{\mathcal{O}}(d\sqrt{M\tau})$  is unavoidable for any regime of  $\varepsilon$ , and this rate is known to be optimal in the case of homogeneous multi-agent. Since we use the technique from He et al. [2022] to handle the dissimilarity, the corruption amount of each round is  $\varepsilon$ , and a total corruption of  $\varepsilon dM\tau$  when the central server allows to collaborate up to round  $\tau$ . To the best of our knowledge, we are not aware of any UCB-based algorithm for misspecified linear bandits in the setting of infinite arms. We expect that employing a misspecified linear bandits algorithm would achieve a regret of  $\tilde{\mathcal{O}}(d\sqrt{MT} + \varepsilon \sqrt{dMT})$ , which is tighter by a factor of  $\sqrt{d}$ . It is worth noting that the CW-OFUL algorithm in He et al. [2022] is designed for handling corruption, whether it can achieve  $\tilde{\mathcal{O}}(d\sqrt{T} + \varepsilon \sqrt{dT})$  in a misspecified setting remains an open question.
We now present our main result giving an upper bound on the group regret of H-LINUCB.
Theorem 4.2. Given Assumption 3.1,  $T \geq 1$ , let  $\lambda = 1, \alpha = \frac{\sqrt{d}}{\varepsilon M\tau}$ ,  $\delta_{1} = \delta_{2} = \frac{1}{M^{2}T}$  in the upper bound of  $\beta_{t}$  on the confidence interval (see Lemma 4.1)  $\forall t \in [T]$ . Let  $\tau = \min(\lfloor \frac{1}{2\varepsilon^{2}} \rfloor, T)$ , and let the synchronization threshold  $D = \tau \log(M\tau) / (dM)$ . Then, the expected group regret of Algorithm 1 is bounded as
$$
\mathbb {E} \left[ \mathcal {R} (M, T) \right] \leq 3 2 0 \sqrt {2} \left(d \sqrt {M T} + 2 \min  \left\{\varepsilon d M T, d M \sqrt {T} \right\}\right) \log^ {2} (M T).
$$
Here,  $\tau$  is the maximum round that the central server allows communication. After that, all agents switch to independent learning. By choosing  $\tau = \min (\lfloor \frac{1}{2\varepsilon^2}\rfloor ,T)$ , agents fully cooperate in the regime  $\varepsilon \in [0,\frac{1}{\sqrt{2T}} ]$  if  $T\leq \frac{1}{2\varepsilon^2}$ , and gradually reduce  $\tau$  as  $\varepsilon$  increases from  $\frac{1}{\sqrt{2T}}$  to  $+\infty$ . This is important for avoiding a linear regret since when  $\varepsilon$  is large, most of the regret comes from the  $\varepsilon dMT$  term and dominates the  $d\sqrt{MT}$  term. The condition on Line 6 also discards all of the synchronized data. In the extreme case, when  $\varepsilon >1$ , there is no collaboration happening due to the condition in Line 18 failing at every round. In other words, H-LINUCB behaves like IND-OFUL.
In the other extreme case, when  $\varepsilon = 0$ , all agents solve identical linear bandits. The weight in Line 15 always evaluates to its minimum value of 1 for all  $t \in [T]$  since  $\alpha = \frac{\sqrt{d}}{\varepsilon MT} \to +\infty$ . We have  $t < \varepsilon^2$  for all rounds. Therefore, the reset condition in Line 6 is never triggered and H-LINUCB behaves exactly like DISLINUCB, achieving a regret of  $\tilde{\mathcal{O}}(d\sqrt{MT})$ , which is optimal up to some logarithmic factors.
Theorem 4.2 suggests that the upper bound of H-LINUCB is tighter than IND-OFUL for all  $\varepsilon$ .
Remark 4.2. Ghosh et al. [2023] also propose a personalized algorithm (PMLB) for the heterogeneous multi-agent linear bandits; however, our problem setting is fundamentally different than that of Ghosh et al. [2023]. They consider a finite action set and impose a strong distributional assumption on how contexts are generated, i.e., the stochastic context  $x_{i,t}$  for each action  $i$  and at each round  $t$  is zero-mean and forms a positive-definite covariance matrix. In stark contrast, we consider the adversarial setting where the context set is adversarially generated at each round (and thus, the associated action set can be infinite and arbitrary). This renders the algorithm and guarantees of Ghosh et al. [2023] inapplicable in our setting and, thus, requires a completely different treatment. Those assumptions of Ghosh et al. [2023] are crucial for them to obtain the  $\tilde{\mathcal{O}}(T^{1/4})$  bound (for  $\varepsilon = 0$ ). We note that this bound is not information-theoretically possible in our adversarial setting; the minimax lower bound in such settings is  $\Omega(\sqrt{T})$  (Theorem 4.3).
4.2 Lower bound
In this section, we present a lower bound result for the  $\varepsilon$ -MALCB problem. We denote  $\mathcal{R}_{\mathcal{A},\mathcal{I}}(M,T)$  as a regret of algorithm  $\mathcal{A}$  on a problem instance  $\mathcal{I}$  of  $M$  agents run for  $T$  rounds.
Theorem 4.3. Let  $\mathcal{I}(\varepsilon)$  denote the class of  $\varepsilon$ -MALCB problem instances that satisfy the Assumption 3.1. Then for any  $d, M, T \in \mathbb{Z}^+$  with  $\frac{d}{2} \leq T, \frac{d^2}{48} \leq T, \forall \varepsilon \geq 0$ , we have the following,
$$
\inf  _ {\mathcal {A}} \sup  _ {\mathcal {I} \in \mathcal {I} (\varepsilon)} \mathcal {R} _ {\mathcal {A}, \mathcal {I}} (M, T) = \Omega \left(d \sqrt {M T} + \min  \left\{\varepsilon M T, d M \sqrt {T} \right\}\right).
$$
The first term is a straightforward observation that solving an  $\varepsilon$ -MALCB is at least as hard as solving a single linear bandits for  $MT$  rounds, or  $M$  agents solving identical bandits for  $T$  rounds. The second term suggests that we pay an additional regret of  $\varepsilon MT$  for a small  $\varepsilon \in [0, \frac{d}{\sqrt{T}}]$ , and  $\Omega(dM\sqrt{T})$  for a large  $\varepsilon \geq \frac{d}{\sqrt{T}}$ . We note that  $\Omega(dM\sqrt{T})$  is also the lower bound of IND-OFUL when each agent incurs a regret of at least  $\Omega(d\sqrt{T})$ . We believe that the analysis of the lower bound could be tightened by using the arguments from misspecified bandits literature, achieving a lower bound of  $\Omega(d\sqrt{MT} + \min\{\varepsilon\sqrt{dMT}, dM\sqrt{T}\})$ .
The lower bound suggests that our upper bound is tight up to logarithmic factors in the following extreme regimes, (i)  $\varepsilon \in [0,\frac{1}{\sqrt{MT}} ]$  , where  $\mathcal{R}(M,T) = \Theta (d\sqrt{MT})$  ; (ii)  $\varepsilon \in [\frac{d}{\sqrt{T}}, + \infty ]$  , where  $\mathcal{R}(M,T) = \Theta (dM\sqrt{T})$  . In regime (i), all agents solve tasks that are similar to one another, yielding the highest collaborative gain. In regime (ii), tasks are highly dissimilar, H-LINUCB turns off the collaboration and lets agents solve their own tasks individually.
Figure 1: Simulation on synthetic data with  $M = 60$ ,  $d = 30$ ,  $T = 10000$ .
Finally, in the regime that  $\varepsilon \in (\frac{1}{\sqrt{MT}}, \frac{d}{\sqrt{T}})$ , our results illustrate the interpolation between two extremes. In this regime, our upper bound presents a gap of  $d$  in the dissimilarity term  $\varepsilon dMT$  compared to  $\varepsilon MT$  in the lower bound.
The key idea in the proof of Theorem 4.3 is based on an information-theoretic lower bound of Lemma C.1, wherein we extend the result from (single-agent) linear bandits to heterogeneous multi-agent linear bandits. Here, we give the lower bound result without any constraints on the communication, hence, this is also the lower bound of the H-LINUCB algorithm.
5 Numerical Simulations
In this section, we provide some numerical simulations to support our theory. Our goal is to address the following question: how does H-LINUCB perform in three different regimes of dissimilarity: (i)  $\varepsilon \in [0,\frac{1}{\sqrt{MT}} ]$  , (ii)  $\varepsilon \in (\frac{1}{\sqrt{MT}},\frac{d}{\sqrt{T}})$  , (iii)  $\varepsilon \in [\frac{d}{\sqrt{T}}, + \infty ]?$
We compare the performance of H-LINUCB with that of the following two algorithms: (a) Independent Learners (IND-OFUL), wherein each agent independently runs OFUL algorithm of Abbasi-Yadkori et al. [2011], and there is no communication between agents (zero collaboration), and (b) DISLINUCB, for which we use the implementation of Wang et al. [2020] without any modification.
Simulation setup. We generate the  $\varepsilon$ -MALCB problem for  $M = 60$ ,  $d = 30$ ,  $T = 10000$  via the following procedure. We first choose a value of  $\varepsilon$  in each of the three dissimilarity regimes. Then we create the linear parameters  $\{\theta_m\}_{m=1}^M$  as follows. Let  $u, \{v_m\}_{m=1}^M$  be random vectors with unit norm. We set  $\theta_m = c \cdot u + \frac{\varepsilon}{2} v_m$ , where  $c$  is a constant in the range  $[0, 1 - \varepsilon]$ . This guarantees  $\|\theta_m\| \leq 1$  and  $\|\theta_i - \theta_j\| \leq \varepsilon$  for any two agents  $i, j$ . At each round, for each agent, we create a new decision set with a size of 50, each action is random and normalized to 1. The random noise is sampled from the standard normal distribution,  $\eta \sim \mathcal{N}(0, 1)$ . We run each experiment 10 times, then report the
group regret averaged over the runs and the confidence intervals in Figure 1. Our code is available here: https://github.com/anhddo/hlinUCB.
Results and discussions. In regime (i), where the level of dissimilarity is small, plots (a) and (b) show that H-LINUCB retains a regret comparable with DISLINUCB.
In regime (ii), plots (c) and (d) illustrate the interpolation between the two extreme regimes.
In regime (iii), plots (e) and (f), DISLINUCB incurs linear regret, H-LINUCB has the same rate with IND-OFUL. This illustrates that collaboration brings no benefit when the dissimilarity is high.
In this paper, we studied the heterogeneous multi-agent linear contextual bandit problem. We formulated the problem under the notion of  $\varepsilon$ -MALCB, and provided the upper and lower bounds when  $\varepsilon$  is known. We showed that our results are provably optimal in the regime where tasks are highly similar or highly dissimilar. Finally, we validated our theoretical results with numerical simulations on synthetic data.
A natural avenue for future work would be to close the gap in the regime  $\varepsilon \in \left(\frac{1}{\sqrt{MT}}, \frac{d}{\sqrt{T}}\right)$ . Another research direction pertains to designing an adaptive algorithm when  $\varepsilon$  is unknown. Such an algorithm would be practical and flexible enough to apply to a wide range of heterogeneous multi-agent bandit problems. We are also interested in extending this work to a more challenging setting such as Reinforcement Learning.
This research was supported, in part, by DARPA GARD award HR00112020004, NSF CAREER award IIS-1943251, funding from the Institute for Assured Autonomy (IAA) at JHU, and the Spring'22 workshop on "Learning and Games" at the Simons Institute for the Theory of Computing.
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011. 2, 3, 4, 6, 7, 9, 14, 22, 23
Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC), pages 1-8. IEEE, 2020. 2
Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning. In Uncertainty in Artificial Intelligence, page 122. Citeseer, 2013. 1
Leonardo Cella, Karim Lounici, Grégoire Pacreau, and Massimiliano Pontil. Multi-task representation learning with stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics, pages 4822-4847. PMLR, 2023. 3
Niladri Chatterji, Vidya Muthukumar, and Peter Bartlett. Osom: A simultaneously optimal algorithm for multi-armed and linear contextual bandits. In International Conference on Artificial Intelligence and Statistics, pages 1844–1854. PMLR, 2020. 2
Ronshee Chawla, Abishek Sankararaman, and Sanjay Shakkottai. Multi-agent low-dimensional linear bandits. IEEE Transactions on Automatic Control, 2022. 2
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208-214. JMLR Workshop and Conference Proceedings, 2011. 2
Yihan Du, Longbo Huang, and Wen Sun. Multi-task representation learning for pure exploration in linear bandits. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on
David Martínez-Rubio, Varun Kanade, and Patrick Rebeschini. Decentralized cooperative stochastic bandits. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 2
Aritra Mitra, Arman Adibi, George J. Pappas, and Hamed Hassani. Collaborative linear bandits with adversarial agents: Near-optimal regret bounds. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 2
Ahmadreza Moradipari, Mohammad Ghavamzadeh, and Mahnoosh Alizadeh. Collaborative multiagent stochastic linear bandits. In 2022 American Control Conference (ACC), pages 2761-2766. IEEE, 2022. 2
Michael T. Rosenstein. To transfer or not to transfer. In NIPS 2005, 2005. 1
Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395-411, 2010. 2
Marta Soare, Ouais Alsharif, Alessandro Lazaric, and Joelle Pineau. Multi-task linear bandits. In NIPS2014 workshop on transfer and multi-task learning: theory meets practice, 2014. 2
Kei Takemura, Shinji Ito, Daisuke Hatano, Hanna Sumita, Takuro Fukunaga, Naonori Kakimura, and Ken-ichi Kawarabayashi. A parameter-free algorithm for misspecified linear contextual bandits. In International Conference on Artificial Intelligence and Statistics, pages 3367-3375. PMLR, 2021. 3, 6
Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations, 2020. 2, 6, 7, 9, 14, 15
Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1531-1539. PMLR, 13-15 Apr 2021. 1, 2, 3, 4
Jiaqi Yang, Wei Hu, Jason D. Lee, and Simon Shaolei Du. Impact of representation learning in linear bandits. In International Conference on Learning Representations, 2021. 1, 3
Jiaqi Yang, Qi Lei, Jason D Lee, and Simon S Du. Nearly minimax algorithms for linear bandits with shared representation. arXiv preprint arXiv:2203.15664, 2022. 3
Weitong Zhang, Jiafan He, Zhiyuan Fan, and Quanquan Gu. On the interplay between misspecification and sub-optimality gap in linear contextual bandits. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 41111-41132. PMLR, 23-29 Jul 2023. 6
A Proof of Upper Bound
A.1 Proof of Lemma 4.1
Proof. We prove that the true parameter of each agent lies in the Confidence Ellipsoid with high probability. In particular, we prove such results for two periods, for round  $t < \tau$  and  $t \geq \tau$ . We denote  $\delta_1, \delta_2$  as the probabilities such that true linear parameters lie outside the confidence ellipsoid for the period of  $t < \tau$ , and  $t \geq \tau$ , respectively.
For the round  $t < \tau$ , let  $t_s$  be the last round that synchronization occurs, where  $t_s < t$ . Then each agent's statistics can be divided into two parts, its own data and the pooled data from other agents since round  $t_s$ . Specifically, for an agent  $m$ , at round  $t$ , we have the following for  $V_{m,t}, b_{m,t}$ ,
$$
V _ {m, t} = \lambda I + \sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} x _ {m, k} ^ {\top} + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} x _ {i, k} ^ {\top},
$$
$$
b _ {m, t} = \sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} y _ {m, k} + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} y _ {i, k}.
$$
We then have the following for  $\hat{\theta}_{m,t}$
$$
\begin{array}{l} \hat {\theta} _ {m, t} = V _ {m, t} ^ {- 1} b _ {m, t} \\ = V _ {m, t} ^ {- 1} \left[ \sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} y _ {m, k} + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} y _ {i, k} \right] \\ = V _ {m, t} ^ {- 1} \left[ \sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} \left(x _ {m, k} ^ {\top} \theta_ {m} + \eta_ {m, k}\right) + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} \left(x _ {i, k} ^ {\top} \theta_ {i} + \eta_ {i, k}\right) \right] \\ = V _ {m, t} ^ {- 1} \left[ \left(\sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} x _ {m, k} ^ {\top} + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} x _ {i, k} ^ {\top}\right) \theta_ {m} \right] \\ + V _ {m, t} ^ {- 1} \left(\sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} x _ {i, k} ^ {\top} \left(\theta_ {i} - \theta_ {m}\right)\right) \\ + V _ {m, t} ^ {- 1} \left[ \sum_ {k = 1} ^ {t - 1} w _ {m, k} x _ {m, k} \eta_ {m, k} + \sum_ {i \neq m} \sum_ {k = 1} ^ {t _ {s}} w _ {i, k} x _ {i, k} \eta_ {i, k} \right]. \\ \end{array}
$$