SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 2. Without any communication, each client would execute SupLinUCB (Chu et al., 2011) for  $T_{i}$  rounds locally, and each client can achieve regret of order  $\tilde{O}(\sqrt{dT_i})$ . Therefore, the total regret of  $M$  clients is upper bound by  $R_{T} \leq \sum_{i=1}^{M} \sqrt{dT_{i}}$  polylog(T)  $\leq \sqrt{dM \sum_{i=1}^{M} T_{i}}$  polylog(T), where the last inequality becomes equality when  $T_{i} = T_{j}, \forall i, j \in [M]$ . Compared with conducting  $M$  independent SupLinUCB algorithms locally, Async-FedSupLinUCB yields an average per-client gain of  $1/\sqrt{M}$ , demonstrating that communications in the federated system can speed up local linear bandits decision-making at clients.
Remark 3. Most previous federated linear bandits consider the infinite action setting, based on the LinUCB principle (Abbasi-Yadkori et al., 2011). Async-FedSupLinUCB considers a finite adversarial action setting and has a  $\sqrt{d}$  reduction on the regret bound. Fed-PE proposed in Huang et al. (2021) also considers the finite action setting. However, their action sets are fixed. We generalize their formulation and take into account a more challenging scenario, where the finite action set can be chosen adversarily. The regret order is the same as Fed-PE (ignoring the ploylog term), indicating
that it is the finite actions as opposed to fixed actions that fundamentally leads to the  $\sqrt{d}$  regret improvement in the federated linear bandits setting.
Communication cost analysis of FedSupLinUCB. We sketch the proof for the communication cost bound in Theorem 5.1 in the following, while deferring the detailed proofs for the regret and the communication cost to Appendix C.
We first study the communication cost triggered by some layer  $s$ . Denote by  $A_{t,s}^{ser}$  the gram matrix in the server aggregated by the gram matrices uploaded by all clients up to round  $t$ . Define  $T_{n,s} = \min \{t \in [T] | \det(A_{t,s}^{ser}) \geq 2^n\}$ , for each  $n \geq 0$ . We then divide rounds into epochs  $\{T_{n,s}, T_{n,s} + 1, \dots, \min(T_{n+1,s} - 1, T)\}$  for each  $n \geq 0$ . The number of communications triggered by layer  $s$  within any epoch can be upper bounded by  $2(M + 1/C)$  (see Lemma C.1), and the number of non-empty epochs is at most  $d \log(1 + T/d)$  by Lemma A.1. Since there are  $S = \lceil \log d \rceil$  layers and synchronization among all layers is performed once communication is triggered by any layer (Line 8 in Algorithm 3), the total communication cost is thus upper-bounded by  $O(d(M + 1/C) \log d \log T)$ . Plugging  $C = 1/M^2$  proves the result.
We note that although choosing a larger  $C$  would trigger fewer communications, the final choice of  $C = 1 / M^2$  takes into consideration both the regret and the communication cost, i.e., to achieve a small communication cost while maintaining an order-optimal regret.
6 Synchronous FedSupLinUCB
In the synchronous setting, all clients are active and make decisions at each round. Though it can be viewed as a special case of the asynchronous scenario (clients are active and pulling arms in a round-robin manner), the information update is broadcast to all clients. In other words, the key difference from the asynchronous scenario besides that all clients are active at each round is that when a client meets the communication criterion, all clients will upload local information to the server and download the updated matrices. This leads to a higher communication cost per communication round, but in this synchronous scenario, knowing all clients are participating allows the communicated information to be well utilized by other clients. This is in sharp contrast to the asynchronous setting, where if many other clients are active in the current round, uploading local information to the clients seems unworthy. To mitigate the total communication cost, we use a more refined communication criterion to enable time-independent communication cost.
6.1 The Algorithm
The Sync-FedSupLinUCB algorithm allows each client to make decisions by the S-LUCB subroutine. Note that the decision-making is based on all available local information instead of the lazy update in the Async-FedSupLinUCB algorithm. The communication criterion involves the count of rounds since the last communication, which forces the communication to prevent the local data from being obsolete. Some layers may trigger the communication criterion either because the local client has gathered enough new data or due to having no communication with the server for too long. We categorize these layers in the CommLayers and synchronize all the clients with the server.
6.2 Performance Analysis
Theorem 6.1. For any  $0 < \delta < 1$ , if we run Algorithm 4 with  $D = \frac{T_c \log T_c}{d^2 M}$ , with probability at least  $1 - \delta$ , the regret of the algorithm is bounded as  $R_T \leq \tilde{O}(\sqrt{d M T_c})$  where  $T_c$  is the total per-client arm pulls. Moreover, the corresponding communication cost is bounded by  $O(\sqrt{d^3 M^3} \log d)$ .
Remark 4. Theorem 6.1 demonstrates Sync-FedSupLinUCB also achieves the minimax regret lower bound while the communication cost is independent of  $T_{c}$ . It is particularly beneficial for large  $T_{c}$ . Especially, the number of total rounds in the synchronous scenario is  $T = MT_{c}$ , while in the asynchronous setting, we have  $T = \sum_{i=1}^{M} T_{i}$  rounds.
Communication cost analysis of Sync-FedSupLinUCB. We sketch the proof for the communication cost bound in Theorem 6.1 below, while deferring the detailed proofs for the regret and the communication cost to Appendix D.
Algorithm 4 Sync-FedSupLinUCB
1: Initialization:  $T_{c},D,S = [\log d],t_{last}^{s}\gets 0,\forall s\in [0:S]$  CommLayers  $\leftarrow \emptyset$    
2:  $\{A_s^{\text{ser}}\gets I_d,b_s^{\text{ser}}\gets 0\mid s\in [0:S]\}$  ▷Server initialization   
3:  $\{A_s^i\gets I_d,\Delta A_s^i,b_s^i,\Delta b_s^i\gets 0\mid s\in [0:S],i\in [M]\}$  ▷Clients initialization   
4: for  $t = 1,2,\dots ,T_c$  do   
5: for  $i = 1,2,\dots ,M$  do   
6: Client  $i_t = i$  is active, and observes  $K$  contexts  $\{x_{t,1}^i,x_{t,2}^i,\dots ,x_{t,K}^i\}$    
7:  $s\gets S$  -LUCB (client  $i,\{x_{t,1}^i,x_{t,2}^i,\dots ,x_{t,K}^i\}$    
8: if  $(t - t_{last}^{s})\log \frac{\operatorname*{det}(A_{s}^{i} + \Delta A_{s}^{i})}{\operatorname*{det}(A_{s}^{i})} >D$  then   
9: Add  $s$  to CommLayers   
10: end if   
11: end for   
12: end for   
13: for  $s\in$  CommLayers do   
14: Sync(s, server, clients  $[M])$  .  $t_{last}^{s}\gets t$  CommLayers  $\leftarrow \emptyset$    
15: end for
We call the chunk of consecutive rounds without communicating information in layer  $s$  (except the last round) an epoch. Information in layer  $s$  is collected locally by each client and synchronized at the end of the epoch, following which the next epoch starts. Denoted by  $A_{p,s}^{all}$  the synchronized gram matrix at the end of the  $p$ -th epoch. For any value  $\beta > 0$ , there are at most  $\lceil \frac{T_c}{\beta} \rceil$  epochs that contain more than  $\beta$  rounds by pigeonhole principle. If the  $p$ -th epoch contains less than  $\beta$  rounds, then  $\log \left( \frac{\operatorname*{det}(A_{p,s}^{all})}{\operatorname*{det}(A_{p - 1,s}^{all})} \right) > \frac{D}{\beta}$  based on the communication criterion and the fact that  $\sum_{p=1}^{P} \log \frac{\operatorname*{det}(A_{p,s}^{all})}{\operatorname*{det}(A_{p - 1,s}^{all})} \leq R_s = O(d \log (T_c))$  (see Equation (6)). The number of epochs containing rounds fewer than  $\beta$  is at most  $O(\lceil \frac{R_s}{D / \beta} \rceil)$ . Noting that  $D = \frac{T_c \log (T_c)}{d^2 M}$ , the total number of epochs for layer  $s$  is at most  $\lceil \frac{T_c}{\beta} \rceil + \lceil \frac{R_s \beta}{D} \rceil = O(\sqrt{\frac{T_c R_s}{D}}) = O(\sqrt{d^3 M})$  by taking  $\beta = \sqrt{\frac{DT_c}{R_s}}$ . The total communication cost is thus upper bounded by  $O(SM \sqrt{d^3 M}) = O(\log (d) \sqrt{d^3 M^3})$ .
7 Extensions of FedSupLinUCB
In this section, we extend the FedSupLinUCB algorithm to address two distinct settings in federated systems: scenarios characterized by heterogeneous variances, and those affected by adversarial corruptions.
7.1 Federated Heteroscedastic Linear Bandits
We have so far focused on the federated linear bandits with 1-sub-Gaussian reward noises. In this section, we adapt Async-FedSupLinUCB to the case where the reward noises have heterogeneous variances, which extends the heteroscedastic linear bandits as studied in Zhou et al. (2021); Zhou and Gu (2022) to the asynchronous federated setting, where one client is active at a time. Specifically, the reward noises  $\{\epsilon_t\}_{t\in [T]}$  are independent with  $|\epsilon_t| \leq R, \mathbb{E}[\epsilon_t] = 0$  and  $\mathbb{E}[\epsilon_t^2] \leq \sigma_t^2$ , where  $\sigma_t$  is known to the active client.
We propose a variance-adaptive Asyc-FedSupLinUCB and analyze its regret and the communication cost in the theorem below, with the algorithm and the proof details in Appendix E due to space constraint. The regret is significantly less than that of the Async-FedSupLinUCB when the variances  $\{\sigma_t^2\}$  are small.
Theorem 7.1. For any  $0 < \delta < 1$ , if we run the variance-adaptive Async-FedSupLinUCB algorithm in Appendix  $E$  with  $C = 1 / M^2$ , with probability at least  $1 - \delta$ , the regret is bounded as  $R_T \leq \tilde{O}(\sqrt{d\sum_{t=1}^{T}\sigma_t^2})$ , and the communication cost is bounded by  $O(dM^2\log^2 T)$ .
7.2 Federated Linear Bandits with Corruption
We further explore asynchronous federated linear bandits with adversarial corruptions, where an adversary inserts a corruption  $c_t$  to the reward  $r_t$  of the active client at round  $t$ . The total corruption is bounded by  $\sum_{t=1}^{T} |c_t| \leq C_p$ . We incorporate the idea of linear bandits with adversarial corruption studied in He et al. (2022b) to the proposed FedSupLinUCB framework and propose the Robust Async-FedSupLinUCB algorithm, with details in Appendix F. Robust Async-FedSupLinUCB can achieve the optimal minimax regret (matching the lower bound in He et al. (2022b)) while incurring a low communication cost.
Theorem 7.2. For any  $0 < \delta < 1$ , if we run the Robust Async-FedSupLinUCB algorithm in Appendix  $F$  with  $C = 1 / M^2$ , with probability at least  $1 - \delta$ , the regret is bounded as  $R_T \leq \tilde{O}(\sqrt{dT} + dC_p)$ , and the communication cost is bounded by  $O(dM^2 \log d \log T)$ .
We have experimentally evaluated FedSupLinUCB in the asynchronous and synchronous settings on both synthetic and real-world datasets. We report the results in this section.
8.1 Experiment Results Using Synthetic Dataset
We simulate the federated linear bandits environment specified in Section 3. With  $T = 40000$ ,  $M = 20$ ,  $d = 25$ ,  $\mathcal{A} = 20$ , contexts are uniformly randomly sampled from an  $l_{2}$  unit sphere, and reward  $r_{t,a} = \theta^{\top}x_{t,a} + \epsilon_{t}$ , where  $\epsilon_{t}$  is Gaussian distributed with zero mean and variance  $\sigma = 0.01$ . It should be noted that while  $M$  clients participate in each round in the synchronous scenario, only one client is active in the asynchronous case. In the plots, the  $x$ -axis coordinate denotes the number of arm pulls, which flattens the actions in the synchronous setting.
(a) Regret: arrival patterns.
(b) Communication: arrival patterns.
(c) Regret: client numbers.
Figure 1: Experimental results with the synthetic dataset.
(d) Regret vs communications.
Arrival pattern. We first investigate the impact of different arrival patterns (the sequence of activating clients): (1) Random, which randomly allocates  $T / M$  arm pulls in  $[T]$  for each client. (2) Round-robin, i.e.  $[1, 2, 3, \dots, M, 1, 2, 3, \dots, M, \dots]$ . (3) Click-leave, i.e.  $[1, 1, \dots, 2, 2, \dots, \dots, M, M, \dots]$ . The regret and the communication cost of these three arrival patterns in the synthetic experiment are reported in Figure 1(a) and Figure 1(b), respectively. We note that although the upper bound analysis in our proof is for the worst-case instance, the numerical results suggest that different arrival patterns result in diverse regret performances. Round-robin and random patterns are more challenging since both local bandit learning and each client's policy updates happen relatively slowly. The click-leave pattern, which is the closest to the centralized setting, achieves the best regret. In addition, compared with Async-FedSupLinUCB, Sync-FedSupLinUCB achieves better cumulative regrets with a higher communication cost.
Amount of clients. The per-client cumulative regret as a function of  $T_{c} = T / M$  with different amounts of clients is plotted in Figure 1(c). In comparison to the baseline SupLinUCB, FedSupLinUCB algorithms achieve better regret via communication between clients and the server. We can see from the experiment that FedSupLinUCB significantly reduces the per-client regret compared with SupLinUCB, and achieves a better regret as  $M$  increases in both asynchronous and synchronous settings.
Trade-off between regrets and communications. We evaluate the tradeoff between communication and regret by running FedSupLinUCB with different communication threshold values  $C$  and  $D$  in asynchronous and synchronous settings respectively. The results are reported in Figure 1(d), where each scattered dot represents the communication cost and the cumulative regret that FedSupLinUCB has achieved with a given threshold value at round  $T = 40000$ . We see a clear tradeoff between the regret and the communication. More importantly, Sync-FedSupLinUCB achieves a better tradeoff than Async-FedSupLinUCB.
8.2 Experiment Results Using Real-world Dataset
We further investigate how efficiently the federated linear bandits algorithm performs in a more realistic and difficult environment. We have carried out experiments utilizing the real-world recommendation dataset MovieLens 20M (Harper and Konstan, 2015). Following the steps in Li and Wang (2022b), we first filter the data by maintaining users with above 2500 movie ratings and treating rating points greater than 3 as positive, ending up with  $N = 37$  users and 121934 total movie rating interactions. Then, we follow the process described in Cesa-Bianchi et al. (2013) to generate the contexts set, using the TF-IDF feature  $d = 25$  and the arm set  $K = 20$ . We plot the per-client normalized rewards of the FedSupLinUCB algorithm with different client numbers  $M$  in synchronous and asynchronous cases respectively. Note that the per-client cumulative rewards here are normalized by a random strategy. From Figure 2(a) and Figure 2(b), we can see that in both synchronous and asynchronous experiments, FedSupLinUCB has better rewards than SupLinUCB, and the advantage becomes more significant as the number of users increases.
(a)Async-FedSupLinUCB.
Figure 2: Experimental results with the real-world MovieLens-20M dataset.
(b) Sync-FedSupLinUCB.
We studied federated linear bandits with finite adversarial actions, a model that has not been investigated before. We proposed FedSupLinUCB that extends the SupLinUCB and OFUL principles to the federated setting in both asynchronous and synchronous scenarios, and analyzed their regret and communication cost, respectively. The theoretical results proved that FedSupLinUCB is capable of approaching the minimal regret lower bound (up to polylog terms) while only incurring sublinear communication costs, suggesting that it is the finite actions that fundamentally determines the regret behavior in the federated linear bandits setting. Furthermore, we examined the extensions of the algorithm design to the variance-adaptive and adversarial corruption scenarios.
The work of LF and CS was supported in part by the U.S. National Science Foundation (NSF) under grants 2143559, 2029978, and 2132700.
Shi, C., Shen, C., and Yang, J. (2021). Federated multi-armed bandits with personalization. In International Conference on Artificial Intelligence and Statistics, pages 2917–2925. PMLR.
Wang, Y., Hu, J., Chen, X., and Wang, L. (2019). Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations.
Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507.
Zhou, D., Gu, Q., and Szepesvari, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR.