SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\bar {\mu} _ {i} ^ {m} (t + 1) = \frac {\bar {\mu} _ {i} ^ {m} (t) \cdot n _ {m , i} (t) + r _ {m , i} (t) \cdot \mathbb {1} _ {a _ {m} ^ {t} = i}}{n _ {m , i} (t + 1)}, P _ {t} ^ {\prime} (m, j) = \frac {M - 1}{M ^ {2}} \text {i f} P _ {t} (m, j) > 0 \text {a n d} 0 \text {o t h e r w i s e}
$$
$$
\tilde {\mu} _ {i} ^ {m} (t + 1) = \sum_ {j = 1} ^ {M} P _ {t} ^ {\prime} (m, j) \hat {\tilde {\mu}} _ {i, j} ^ {m} (t _ {m, j}) + d _ {m, t} \sum_ {j \in N _ {m} (t)} \hat {\tilde {\mu}} _ {i, j} ^ {m} (t) + d _ {m, t} \sum_ {j \notin N _ {m} (t)} \hat {\tilde {\mu}} _ {i, j} ^ {m} (t _ {m, j})
$$
$$
\mathrm {w i t h} d _ {m, t} = \left(1 - \sum_ {j = 1} ^ {M} P _ {t} ^ {\prime} (m, j)\right) / M
$$
Similar to [Zhu et al., 2021b, Zhu and Liu, 2023], the algorithm balances between exploration and exploitation by the upper confidence bound and a criterion on  $n_{m,i}(t)$  and  $N_{m,i}(t)$  that ensures that all clients explore arms at similar rates and thereby "staying on the same page." After interacting
Algorithm 1: DrFed-UCB: Burn-in period
Initialization: The length of the burn-in period is  $L$  and we are also given  $\tau_{1} < L$ ; In the time step  $t = 0$ , the estimates are initialized as  $\bar{\mu}_i^m(0) = 0$ ,  $n_{m,i}(0) = 0$ ,  $\hat{\bar{\mu}}_i^m(0) = 0$ , and  $P_0(m,j) =$  for any arm  $i$  and clients  $m,j$ ;  
for  $1 \leq t \leq \tau_1$  do  
The environment generates a sample graph  $G_t = (V, E_t)$  based on either E-R or Algorithm 3;  
end  
for  $1 \leq t \leq \tau_1$  do  
for each client  $m$  do  
Sample arm  $a_t^m = (t \mod K)$ ;  
Receive reward  $r_{a_t^m}^m(t)$  and update  $n_{m,i}(t) = n_{m,i}(t-1) + \mathbb{1}_{a_m^t = i}$ ;  
Update the local estimate for any arm  $i$ :  $\bar{\mu}_i^m(t) = \frac{n_{m,i}(t-1)\bar{\mu}_i^m(t-1) + r_{a_t^m}(t)\cdot 1_{a_t^m = i}}{n_{m,i}(t-1) + 1_{a_t^m = i}}$ ;  
end  
for  $\tau_1 < t \leq L$  do  
The environment generates a sample graph  $G_t = (V, E_t)$  based on either E-R or Algorithm 3;  
for each client  $m$  do  
Sample arm  $a_t^m = (t \mod K)$ ;  
Receive rewards  $r_{a_t^m}^m(t)$  and update  $n_{m,i}(t) = n_{m,i}(t-1) + \mathbb{1}_{a_m^t = i}$ ;  
Update the local estimates for any arm  $i$ :  $\bar{\mu}_i^m(t) = \frac{n_{m,i}(t-1)\bar{\mu}_i^m(t-1) + r_{a_t^m}(t)\cdot 1_{a_t^m = i}}{n_{m,i}(t-1) + 1_{a_t^m = i}}$ ;  
Update the maintained matrix  $P_t(m,j) = \frac{(t-1)P_{t-1}(m,j) + X_m^t}{t}$  for each  $j \in V$ ;  
Send  $\{\bar{\mu}_i^m(t)\}_{i=1}^{i-K}$  to all clients in  $\mathcal{N}_m(t)$ ;  
Receive  $\{\bar{\mu}_i^j(t)\}_{i=1}^{i-K}$  from all clients  $j \in \mathcal{N}_m(t)$  and store them as  $\hat{\bar{\mu}}_i^m(t)$ .  
end  
for each client  $m$  and arm  $i$  do  
For client  $1 \leq j \leq M$ , set  $h^L(m,j) = \max_{s \geq \tau_1} \{(m,j) \in E_s\}$  or 0 if such  $s$  does not exist  
 $\tilde{\mu}_i^m(L+1) = \sum_{j=1}^{M} P_{m,j}^l(L) \hat{\bar{\mu}}_i^m(h_m^L)$  where  $P_{m,j}^l(L) = \begin{cases} \frac{1}{M} & \text{if } P_L(m,j) > 0 \\ 0 & \text{otherwise} \end{cases}$ ;  
end
with the environment, clients move to the transmission stage, where they share information with the neighbors on  $G_{t}$ , as a preparation for the update stage.
Different from the upper confidence bound approach in [Zhu and Liu, 2023], which has an extra term of  $\frac{1}{t}$  in the UCB criterion, our proposal is aligned with the conventional UCB algorithm. Meanwhile, our update rule differs from that in [Zhu et al., 2021b, Zhu and Liu, 2023] in three key aspects: (1) maintaining a stopping time  $t_{m,j}$  that tracks the most recent communication to client  $j$ , and (2) updating  $\tilde{\mu}_i^m$  based on both  $\tilde{\mu}_i^j$  and  $\bar{\mu}_i^j$  for  $j \in \mathcal{N}_m(t)$ , and (3) using a weight matrix based on  $P_t'$  and  $P_t$  computed from the trajectory  $\{G_s\}_{s \leq t}$  in the previous steps. The first point ensures that the latest information from other clients is leveraged, in case there is no communication at the current time step. The second point ensures proper integration of both network-wide and local information, smoothing out biases from local estimators and reducing variances through averaging. The third point distills the information carried by the time-varying graphs and determines the weights of the available local and network-wide estimators, removing the need for the doubly stochasticity assumption. The algorithm assumes that the clients know  $M$  and  $\sigma^2$ .
We note that  $t_{m,j}$  is the stopping time by definition and that  $\bar{\mu}_i^m$  is an unbiased estimator for  $\mu_i^m$  with a decaying variance proxy. Meanwhile, the matrices  $P_t'$  and  $P_t$  are not doubly stochastic and keep track of the history of the random graphs. By introducing  $t_{m,j}$  and  $P_t$ , we can show that the global estimator  $\tilde{\mu}_i^m(t)$  behaves similarly to a sub-Gaussian/sub-exponential random variable with an expectation of  $\mu_i$  and a time-decaying variance proxy proportional to  $\frac{1}{\min_j n_{j,i}(t)}$ . This ensures that the concentration inequality holds for  $\tilde{\mu}_i^m(t)$  with respect to  $\mu_i$  and that client  $m$  tends to identify the
Algorithm 2: DrFed-UCB: Learning period
Initialization: For each client  $m$  and arm  $i\in \{1,2,\dots ,K\}$  we have  $\tilde{\mu}_i^m (L + 1)$ $N_{m,i}(L + 1) = n_{m,i}(L)$  ; all other values at  $L + 1$  are initialized as O;   
for  $t = L + 1,L + 2,\ldots ,T$  do   
for each client m do //UCB if there is no arm i such that  $n_{m,i}(t)\leq N_{m,i}(t) - K$  then  $a_{m}^{t} = \arg \max_{i}\tilde{\mu}_{m,i}(t) + F(m,i,t)$  else Randomly sample an arm  $a_{m}^{t}$  end Pull arm  $a_{m}^{t}$  and receive reward  $r_{m,a_m^t}(t)$    
end The environment generates a sample graph  $G_{t} = (V,E_{t})$  based on E-R or Algorithm 3; //Env Each client m sends  $\mu_i^m (t),N_{j,i}(t),\bar{\mu}_i^m (t),\tilde{\mu}_i^m (t)$  to each client in  $\mathcal{N}_m(t)$  .   
Each client m receives  $\mu_j^j (t),N_{j,i}(t),\bar{\mu}_i^j (t),\tilde{\mu}_i^j (t)$  from all clients  $j\in \mathcal{N}_m(t)$  and stores them as  $\hat{\mu}_{i,j}^{m}(t),\hat{N}_{i,j}^{m}(t),\hat{\mu}_{i,j}^{m}(t),\hat{\mu}_{i,j}^{m}(t);$  //Transmission for each client m do for  $i = 1,\dots ,K$  do Update  $P_{t}$  for  $1\le j\le M$  by  $P_{t}(m,j) = \frac{(t - 1)P_{t - 1}(m,j) + X_{m,j}^{t}}{t};$  Update  $P_{t}^{\prime}$  for  $1\le j\le M$  by  $P_{t}'(m,j) = \left\{ \begin{array}{ll}1 & \text{if} P_t(m,j) > 0\\ 0 & \text{if} P_t(m,j) = 0 \end{array} \right.$  Update nm,i(t),Nm,i(t) and  $\tilde{\mu}_i^m (t)$  based on equations (2);   
end   
end
global optimal arms with high probability, which plays an important role in minimizing regret. The formal statements are presented in the next section.
3 Regret Analyses
In this section, we show the theoretical guarantees of the proposed algorithm, assuming mild conditions on the environment. Specifically, we consider various settings with different model assumptions. We prove that the regret of Algorithm 2 has different instance-dependent upper bounds of order  $\log T$  for settings with sub-Gaussian and sub-exponential distributed rewards, and a mean-gap independent upper bound of order  $\sqrt{T}\log T$  across settings. Many researchers call such a bound instance independent but we believe such a terminology is misleading and thus we prefer to call it man-gap independent, given that it still has dependency on parameters pertaining to the problem instance. The results are consistent with the regret bounds in prior works.
3.1 Model Assumptions
By definition, the environment is determined by how the graphs (E-R or uniform) and rewards are generated. For reward we consider two cases.
Sub-g At time step  $t$ , the reward of arm  $i$  at client  $m$  has bounded support  $[0,1]$ , and is drawn from a sub-Gaussian distribution with mean  $0 \leq \mu_i^m \leq 1$  and variance proxy  $0 \leq (\sigma_i^m)^2 \leq \sigma^2$ .
Sub-e At time step  $t$ , the reward of arm  $i$  at client  $m$  has bounded support  $[0,1]$ , and follows a sub-exponential distribution with mean  $0 \leq \mu_i^m \leq 1$  and parameters  $0 \leq (\sigma_i^m)^2 \leq \sigma^2$ ,  $0 \leq \alpha_i^m \leq \alpha$ .
With these considerations, we investigate four different environments (settings) based on the two graph assumptions and the two reward assumptions: Setting 1.1 corresponds to E-R and Sub-g, Setting 1.2 to Uniform and Sub-g, Setting 2.1 to E-R and Sub-e, and Setting 2.2 to Uniform and Sub-e. For each setting, we derive upper bounds on the regret in the next section.
3.2 Regret Analyses
In this section, we establish the regret bounds formally when clients adhere to Algorithm 2 in various settings. We denote Setting 1.1, Setting 1.2 with  $M < 11$ , and Setting 1.2 with  $M \geq 11$  as  $s_1, s_2$  and  $s_3$ , respectively. Likewise, we denote Setting 2.1, Setting 2.2 with  $M < 11$ , and Setting 2.2 with  $M \geq 11$  as  $S_1, S_2$  and  $S_3$ , respectively. See Table 1 in Appendix for a tabular view of the various settings.
Note that the randomness of  $R_{T}$  arises from both the reward and graph observations. Considering  $S_{1}, S_{2}, S_{3}$  differ in the reward assumptions compared to  $s_{1}, s_{2}, s_{3}$ , we define an event  $A$  that preserves the properties of the variables with respect to the random graphs. Given the length of the burn-in period  $L_{i}$  for  $i \in \{s_{1}, s_{2}, s_{3}\}$  and the fact that  $L_{s_{i}} = L_{S_{i}}$  since it only relies on the graph assumptions, we use  $L$  to denote  $\max_{i} L_{s_{i}}$ . Parameters  $0 < \delta, \epsilon < 1$  are any constants, and the parameter  $c = c(M)$  represents the mean value of the Bernoulli distribution in  $s_{1}, S_{1}$  and the probability of an edge in  $s_{2}, S_{2}, s_{3}$ , and  $S_{3}$  among all connected graphs (see (1)). We define events  $A_{1} = \{\forall t \geq L, ||P_{t} - cE||_{\infty} \leq \delta\}$ ,  $A_{2} = \{\exists t_{0}, \forall t \geq L, \forall j, \forall m, t + 1 - \min_{j} t_{m,j} \leq t_{0} \leq c_{0} \min_{l} n_{l,i}(t + 1)\}$ , and  $A_{3} = \{\forall t \geq L, G_{t} \text{ is connected}\}$ . Here  $E$  is the matrix with all values of 1. Constant  $c_{0} = c_{0}(K, \min_{i \neq i^{*}} \Delta_{i}, M, \epsilon, \delta)$  is defined later. Since  $c = c(M)$  this implies that  $G_{t}$  depends on  $M$ . We define  $A = A_{\epsilon, \delta} = A_{1} \cap A_{2} \cap A_{3}$ , which yields  $A \in \Sigma$  with  $\Sigma$  being the sub- $\sigma$ -algebra formed by  $\{\Omega, \emptyset, A, A^{c}\}$ . This implies  $E[\cdot | A_{\epsilon, \delta}]$  and  $P[\cdot | A_{\epsilon, \delta}]$  are well-defined, since  $A$  only relies on the graphs and removes the differences among  $s_{1}, s_{2}, s_{3} (S_{1}, S_{2}, S_{3})$ , enabling universal regret upper bounds.
Next, we demonstrate that event  $A$  holds with high probability.
Theorem 1. For event  $A_{\epsilon, \delta}$  and any  $1 > \epsilon, \delta > 0$ , we have  $P(A_{\epsilon, \delta}) \geq 1 - 7\epsilon$ .
Proof Sketch. The complete proof is deferred to Appendix; we discuss the main logic here. The proof relies on bounding the probabilities of  $A_{1}, A_{2}, A_{3}$  separately. For  $A_{1}$ , its upper bound holds by the analysis of the mixing time of the Markov chain underlying  $G_{t}$  and on the matrix-form Hoeffding inequality. We obtain an upper bound on  $P(A_{2})$  by analyzing the stopping time  $t_{m,j}$  and the counter  $n_{m,i}(t)$ . For the last term  $P(A_{3})$ , we show that the minimum degree of  $G_{t}$  has a high probability lower bound that is sufficient for claiming the connectivity of  $G_{t}$ . To put all together, we use the Bonferroni's inequality and reach the lower bound of  $P(A_{\epsilon,\delta})$ .
Subsequently, we have the following general upper bound on the regret  $R_{T}$  of Algorithm 2 in the high probability sense, which holds on  $A$  in any of the settings  $s_1, s_2, s_3$  with sub-Gaussian rewards.
Theorem 2. Let  $f$  be a function specific to a setting and detailed later. For every  $0 < \epsilon < 1$  and  $0 < \delta < f(\epsilon, M, T)$ , in setting  $s_1$  with  $c \geq \frac{1}{2} + \frac{1}{2}\sqrt{1 - \left(\frac{\epsilon}{MT}\right)^{\frac{2}{M-1}}}$ ,  $s_2$  and  $s_3$ , with the time horizon  $T$  satisfying  $T \geq L$ , the regret of Algorithm 2 with  $F(m, i, t) = \sqrt{\frac{C_1\ln t}{n_{m,i}(t)}}$  satisfies that  $E[R_T|A_{\epsilon,\delta}] \leq L + \sum_{i \neq i^*} (\max \{[\frac{4C_1\log T}{\Delta_i^2}], 2(K^2 + MK)\} + \frac{2\pi^2}{3P(A_{\epsilon,\delta})} + K^2 + (2M - 1)K) = O(\ln T)$  where the length of the burn-in period is explicitly  $L =$
$$
\max \left\{\underbrace {\frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {4 K \log_ {2} T}{c _ {0}}} _ {L _ {s _ {1}}}, \underbrace {\frac {\ln \frac {\delta}{1 0}}{\ln p ^ {*}} + 2 5 \frac {1 + \lambda}{1 - \lambda} \frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {4 K \log_ {2} T}{c _ {0}}} _ {L _ {s _ {2}}}, \underbrace {\frac {\ln \frac {\delta}{1 0}}{\ln p ^ {*}} + 2 5 \frac {1 + \lambda}{1 - \lambda} \frac {\ln \frac {T}{2 \epsilon}}{2 \delta^ {2}} , \frac {\frac {K \ln (\frac {M T}{\epsilon})}{\ln (\frac {1}{1 - 2 \log M})}}{c _ {0}}} _ {L _ {s _ {3}}} \right\}
$$
with  $\lambda$  being the spectral gap of the Markov chain in  $s_2, s_3$  that satisfies  $1 - \lambda \geq \frac{1}{2\frac{\ln 2}{\ln 2p^*}\ln 4 + 1}$ ,  $p^* = p^*(M) < 1$  and  $c_0 = c_0(K, \min_{i \neq i^*} \Delta_i, M, \epsilon, \delta)$ , and the instance-dependent constant  $C_1 = 8\sigma^2 \max \{12\frac{M(M + 2)}{M^4}\}$ .
Proof Sketch. The proof is carried out in Appendix; here we describe the main ideas as follows. We note that the regret is proportional to the total number of pulling global sub-optimal arms by the end of round  $T$ . We fix client  $m$  for illustration without loss of generality. We tackle all the possible cases when clients pull such a sub-optimal arm - (i) the condition  $n_{m,i}(t) \leq N_{m,i}(t) - K$  is met, (ii) the upper confidence bounds of global sub-optimal arms deviate from the true means, (iii) the upper confidence bounds of global optimal arms deviate from the true means, and (iv) the mean values of global sub-optimal arms are greater than the mean values of global optimal arms. The technical novelty of our proof is in that 1) we deduce that the total number of events (ii) and (iii) occurring can
be bounded by some constants using newly derived conditional concentration inequalities that hold by our upper bounds on the conditional moment generating functions and by the unbiasedness of the network-wide estimators and 2) we control (i) by analyzing the scenarios where the criteria are met, which do not occur frequently.
Remark (Specification of the parameters). Note that the choice of  $f$  depends on the problem settings. Specifically, in setting  $s_1$ , we set  $f(\epsilon, M, T) = \frac{1}{2} + \frac{1}{4}\sqrt{1 - \left(\frac{\epsilon}{MT}\right)^{\frac{2}{M-1}}}$ . By the definition of  $c$ , we have  $f(\epsilon, M, T) < c$ . In setting  $s_2$  with  $M < 11$ , we specify  $f(\epsilon, M, T) = \frac{1}{2}$  which meets  $f < c$  due to (1). Lastly, in setting  $s_3$  with  $M \geq 11$ , we choose  $f(\epsilon, M, T) = \frac{1}{2}\frac{2\log M}{M-1}$  and again we have  $f < c$  due to (1). We observe that the regret bound is dependent on the transition kernel  $\pi$  and the spectral gap  $\lambda$  of the underlying Markov chain associated with  $\pi$ . This indicates the significance of graph complexities and distributions within the framework of the random graph model when deriving the regret bounds, in a similar manner as the role of graph conductance in the regret bounds established in [Sankararaman et al., 2019, Chawla et al., 2020] for time-invariant graphs.
To proceed, we show a high probability upper bound on the regret  $E[R_T|A_{\epsilon,\delta}]$  of Algorithm 2 for settings  $S_1, S_2, S_3$  with sub-exponential rewards.
Theorem 3. Let  $f$  be a function specific to a setting and defined in the above remark. For every  $0 < \epsilon < 1$  and  $0 < \delta < f(\epsilon, M, T)$ , in settings  $S_{1}$  with  $c \geq \frac{1}{2} + \frac{1}{2}\sqrt{1 - \left(\frac{\epsilon}{MT}\right)^{\frac{2}{M - 1}}}$ ,  $S_{2}$ ,  $S_{3}$  with the time horizon  $T$  satisfying  $T \geq L$ , the regret of Algorithm 2 with  $F(m, i, t) = \sqrt{\frac{C_{1} \ln T}{n_{m, i}(t)}} + \frac{C_{2} \ln T}{n_{m, i}(t)}$  satisfies  $E[R_{T}|A_{\epsilon, \delta}] \leq L + \sum_{i \neq i^{*}} (\Delta_{i} + 1) \cdot (\max([\frac{16C_{1} \log T}{\Delta_{i}^{2}}], [\frac{4C_{2} \log T}{\Delta_{i}}], 2(K^{2} + MK)) + \frac{4}{P(A_{\epsilon, \delta})T^{3}} + K^{2} + (2M - 1)K) = O(\ln T)$  where  $L, C_{1}$  are specified as in Theorem 2 and  $\frac{C_{2}}{C_{1}} \geq \frac{3}{2}$ .
Proof Sketch. The proof is detailed in Appendix. The proof logic is similar to that of Theorem 2. However, the main differences lie in the upper confidence bounds, which require proving new concentration inequalities and moment generating functions for the network-wide estimators.  $\square$
In addition to the instance-dependent regret bounds of order  $O\left(\frac{\log T}{\Delta_i}\right)$  that depend on the suboptimality gap  $\Delta_i$  which may be arbitrarily small and thereby leading to large regret, we also establish a universal, mean-gap independent regret bound that applies to settings with sub-exponential and sub-Gaussian rewards. A formal proof is deferred to Appendix.
Theorem 4. Assume the same conditions as in Theorems 2 and 3. The regret of Algorithm 2 satisfies that  $E[R_{T}|A_{\epsilon,\delta}] \leq L_{1} + \frac{4}{P(A_{\epsilon,\delta})T^{3}} + (\sqrt{\max(C_{1},C_{2})\ln T} + 1)\frac{4M}{P(A_{\epsilon,\delta})T^{3}} + K(C_{2}(\ln T)^{2} + C_{2}\ln T + \sqrt{C_{1}\ln T}\sqrt{T(\ln T + 1)}) = O(\sqrt{T}\ln T)$ . where  $L_{1} = \max(L, K(2(K^{2} + MK))), L, C_{1}$  is specified as in Theorem 2, and  $\frac{C_2}{C_1} \geq \frac{3}{2}$ . The involved constants depend on  $\sigma^2$  but not on  $\Delta_{i}$ .
3.3 Other Performance Measures
Communication cost Assuming a constant cost of establishing a communication link, as defined in [Wang et al., 2020, Li and Song, 2022], denoted as  $c_{1}$ , the communication cost  $C_T$  can be calculated as  $C_T = c_1 \cdot \sum_{t=1}^{T} |E_t|$ , which is proportional to  $\sum_{t=1}^{T} |E_t|$ . Alternatively, following the framework proposed in [Wang et al., 2022, Li and Wang, 2022, Sankararaman et al., 2019, Chawla et al., 2020], the communication cost can be defined as the total number of communications among clients, which can be represented as  $C_T = \sum_{t=1}^{T} |E_t|$ , similar to the previous definition. Regarding the quantity  $C_T = \sum_{t=1}^{T} |E_t|$ , the number of edges  $E_t$  could be  $O(M)$  for sparse graphs, and at most  $O(M^2)$ . In the random graph model, the expected number of edges is  $\frac{M(M-1)}{2} c$ , which implies  $O(M^2)$  in the worst case scenario and the total communication cost, in a worst-case scenario, is of order  $O(TM^2)$ . This analysis holds also for the random connected graph case where  $c$  represents the probability of having an edge. This cost aligns with the existing work of research on decentralized distributed multi-agent MAB problems without a focus on the communication cost, where edge-wise communication is a standard practice. Optimizing communication costs to achieve sublinearity, as discussed in [Sankararaman et al., 2019, Chawla et al., 2020], is a subject for future research.
Complexity and privacy guarantee At each time step, the time complexity of the graph generation algorithm is  $O(M^2 + M + |E_t|) \leq O(M^2)$ , where the first term accounts for edge selection, and the second and third terms are for graph connectivity verification using Algorithm 1 (benefitting from the use of Markov chains for graph generation). The main algorithm comprises various stages involving
multiple agents. The overall time complexity is calculated as  $O(MK + M^2 + MK + M^2) = O(M^2 + MK)$ , consistent with that of [Zhu et al., 2021b, Dubey et al., 2020]. It is noteworthy that most procedures can operate in parallel (synchronously), such as arm pulling, broadcasting, estimator updating, and E-R model generation, with the exception being random connected graph generation due to the Markovian property. Privacy guarantees are also important. Here, we note that clients only communicate aggregated values of raw rewards. Differential privacy is not the focus of this work but may be considered in future research.
In this section, we present a numerical study of the proposed algorithm. Specifically, we first demonstrate the regret performance of Algorithms 2 and 3, in comparison with existing benchmark methods from the literature, in a setting with time-invariant graphs. Moreover, we conduct a numerical experiment with respect to time-varying graphs, comparing the proposed algorithm with the most recent work [Zhu and Liu, 2023]. Furthermore, the theoretical regret bounds of the proposed algorithm, as discussed in the previous section, exhibit different dependencies on the parameters that determine the underlying problem settings. Therefore, we examine these dependencies through simulations to gain insights into the exact regret incurred by the algorithm in practice. The benchmark algorithms include GoSInE [Chawla et al., 2020], Gossip_UCB [Zhu et al., 2021b], and Dist_UCB [Zhu and Liu, 2023]. Notably, GoSInE and Gossip_UCB are designed for time-invariant graphs, while Dist_UCB and the proposed algorithm (referred to as DrFed-UCB) are tailored for time-varying graphs. Details about numerical experiments are referred to Appendix.
Benchmark comparison results The visualizations for both time-invariant and time-varying graphs are in Appendix. We evaluated regret by averaging over 50 runs, along with the corresponding  $95\%$  confidence intervals. In time-invariant graphs, DrFed-UCB consistently demonstrates the lowest average regret, showcasing significant improvements. Dist_UCB and DrFed-UCB exhibit larger variances (Dist_UCB having the largest variances), which might have resulted from the communication mechanisms designed for time-varying graphs. In time-varying graphs, our regret is substantially lower compared to that of Dist_UCB. In terms of time complexity, DrFed-UCB and GoSInE are approximately six times faster than Dist_UCB.
Regret dependency results Additionally, we illustrate how the regret of DrFed-UCB depends on several factors, including the number of clients  $M$ , the number of arms  $K$ , the Bernoulli parameter  $c$  for the E-R model, and heterogeneity measured by  $\max_{i,j,k}|\mu_i^k -\mu_j^k |$ . The visualizations are available in Appendix. We observe that regret monotonically increases with the level of heterogeneity and the number of arms, while it monotonically decreases with connectivity, which is equivalent to an increase in graph complexity. However, this monotonic dependency does not hold with respect to  $M$  due to the accumulation of the information gain.