Title: Federated Linear Bandits with Finite Adversarial Actions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated bandits

contextual bandits

regret analysis
Abstract: We study a federated linear bandits model, where $M$ clients communicate with a central server to solve a linear contextual bandits problem with finite adversarial action sets that may be different across clients. To address the unique challenges of **adversarial finite** action sets, we propose the FedSupLinUCB algorithm, which extends the principles of SupLinUCB and OFUL algorithms in linear contextual bandits. We prove that FedSupLinUCB achieves a total regret of $\tilde{O}(\sqrt{d T})$, where $T$ is the total number of arm pulls from all clients, and $d$ is the ambient dimension of the linear model. This matches the minimax lower bound and thus is order-optimal (up to polylog terms). We study both asynchronous and synchronous cases and show that the communication cost can be controlled as $O(d M^2 \log(d)\log(T))$ and $O(\sqrt{d^3 M^3} \log(d))$, respectively. The FedSupLinUCB design is further extended to two scenarios: (1) variance-adaptive, where a total regret of $\tilde{O} (\sqrt{d \sum \nolimits_{t=1}^{T} \sigma_t^2})$ can be achieved with $\sigma_t^2$ being the noise variance of round $t$; and (2) adversarial corruption, where a total regret of $\tilde{O}(\sqrt{dT} + d C_p)$ can be achieved with $C_p$ being the total corruption budget. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of \alg on both synthetic and real-world datasets.

Federated Linear Bandits with Finite Adversarial Actions

Abstract

We study a federated linear bandits model, where M clients communicate with a central server to solve a linear contextual bandits problem with finite adversarial action sets that may be different across clients. To address the unique challenges of adversarial finite action sets, we propose the FedSupLinUCB algorithm, which extends the principles of SupLinUCB and OFUL algorithms in linear contextual bandits. We prove that FedSupLinUCB achieves a total regret of ˜ O ( √ dT ) , where T is the total number of arm pulls from all clients, and d is the ambient dimension of the linear model. This matches the minimax lower bound and thus is order-optimal (up to polylog terms). We study both asynchronous and synchronous cases and show that the communication cost can be controlled as O ( dM 2 log( d ) log( T )) and O ( √ d 3 M 3 log( d )) , respectively. The FedSupLinUCB design is further extended √

to two scenarios: (1) variance-adaptive, where a total regret of ˜ O ( d ∑ T t =1 σ 2 t ) can be achieved with σ 2 t being the noise variance of round t ; and (2) adversarial corruption, where a total regret of ˜ O ( √ dT + dC p ) can be achieved with C p being the total corruption budget. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of FedSupLinUCB on both synthetic and realworld datasets.

1 Introduction

In the canonical formulation of contextual bandits, a single player would repeatedly make arm-pulling decisions based on contextual information with the goal of maximizing the long-term reward. With the emerging federated learning paradigm (McMahan et al., 2017) where multiple clients and a server jointly learn a global model with each client locally updating the model with its own data and server only aggregating the local models periodically, researchers have started exploring contextual bandits algorithms in such federated learning setting (Dubey and Pentland, 2020; Huang et al., 2021; Li and Wang, 2022a,b). This federated contextual bandits framework broadens the applicability of contextual bandits to practical scenarios such as recommender systems, clinical trials, and cognitive radio. In these applications, although the goal is still to maximize the cumulative reward for the overall system, decision-making and observations are naturally distributed at the participating clients.

Several intrinsic challenges arise with the federated contextual bandit formulation. One important issue is that besides regret, we should also take into account the communication cost, which is usually the system bottleneck. To reduce the communication cost while maintaining the same regret guarantee, the clients should transmit the necessary information to the server only when the local

information has accumulated to the extent that it would affect the decision-making. Compared with the centralized contextual bandits, which have a linearly growing communication cost, algorithms for federated contextual bandits attempt to achieve a comparable regret with sub-linear communication cost.

Second, most existing studies on federated contextual bandits focus on the synchronous communication scenario (Huang et al., 2021; Li and Wang, 2022b), in which all participating clients first upload local information and then download updated global information from the server in each communication round. This stringent communication requirement is often not met in practice. A recent work of Li and Wang (2022a) studies the asynchronous federated linear bandit problem. However, communications for different clients are not independent in their approach because the upload from one client may trigger the server to perform downloads for all clients. To address this issue, He et al. (2022a) proposes FedLinUCB, which enables independent synchronizations between clients and the server.

Third, the majority of prior studies on federated linear bandits focused on the infinite-arm setting (Li and Wang, 2022b,a; He et al., 2022a) (see Section 2 for a detailed literature review). From a methodology point of view, these papers largely build on the OFUL principle (Abbasi-Yadkori et al., 2011). One notable exception is Huang et al. (2021), which studies synchronous communication with fixed contexts and proposes the Fed-PE algorithm based on the phased elimination G-optimal design (Lattimore and Szepesvári, 2020). To the best of our knowledge, no prior result exists for federated linear bandits with finite arms and time-evolving adversarial contexts, which is the focus of our work.

d : the dimension of the unknown parameter, M : the number of clients, K : the number of finite actions, T : the total arm pulls from all clients.

Main contributions. Our main contributions are summarized as follows.

We develop a general federated bandits framework, termed FedSupLinUCB , for solving the problem of federated linear contextual bandits with finite adversarial actions. FedSupLinUCB extends SupLinUCB (Chu et al., 2011; Ruan et al., 2021) and OFUL (Abbasi-Yadkori et al., 2011), two important principles in (single-player, centralized) linear bandits, to the federated bandits setting with a carefully designed layered successive screening.

We instantiate FedSupLinUCB with both asynchronous and synchronous client activities. For the former setting, we propose AsyncFedSupLinUCB where communication is triggered only when the cumulative local information impacts the exploration uncertainty to a certain extent. We prove that AsyncFedSupLinUCB achieves ˜ O ( √ dT ) regret with O ( dM 2 log d log T ) communication cost, which not only reduces the regret by √ d compared with previous results on asynchronous federated linear bandits with infinite arms, but also matches the minimax lower bound up to polylog terms, indicating that AsyncFedSupLinUCB achieves order-optimal regret.

For synchronous communications, we propose SyncFedSupLinUCB , which has a refined communication design where only certain layers are communicated, as opposed to the complete information. SyncFedSupLinUCB achieves order-optimal regret of ˜ O ( √ dT ) with horizon-independent communication cost O ( √ d 3 M 3 log d ) . Compared with the best previous result (Huang et al., 2021) which achieves the same order-optimal regret but only for fixed actions, we show that it is the finite actions that fundamentally determines the regret behavior in the federated linear bandits setting.

We further develop two extensions of FedSupLinUCB : (1) Variance-adaptive FedSupLinUCB , for which a total regret of ˜ O ( √ d ∑ T t =1 σ 2 t ) is achieved, where σ 2 t is the noise variance at round t . (2)

Adversarial corruption FedSupLinUCB , for which a total regret of ˜ O ( √ dT + dC p ) is achieved, where C p is the total corruption budget.

2 Related Works

The linear bandit model, as a generalization of finite armed bandits with linear contextual information, has been extensively studied. The setting of infinite arm sets solved by LinUCB was analyzed in (Dani et al., 2008; Abbasi-Yadkori et al., 2011), which achieves regret ˜ O ( d √ T ) with appropriate confidence width (Abbasi-Yadkori et al., 2011) and matches the lower bound (Dani et al., 2008) up to logarithmic factors. In contrast, algorithms like SupLinRel (Auer, 2002) and SupLinUCB (Chu et al., 2011) achieve ˜ O ( √ dT ) in the setting of finite time-varying adversarial arm sets under K ≪ 2 d , with a lower bound Ω( √ dT ) (Chu et al., 2011). The SupLinUCB algorithm was later optimized and matches the lower bound up to iterated logarithmic factors in Li et al. (2019). As a special case of the finite arm setting, if the arm set is time-invariant, an elimination-based algorithm (Lattimore and Szepesvári, 2020) via G-optimal design can be applied to achieve similar optimal performance.

The federated linear bandits problems were studied under the settings of infinite arm set (Dubey and Pentland, 2020; Li et al., 2020; Li and Wang, 2022a) and time-invariant finite arm set (Huang et al., 2021), while the time-varying finite arm set setting has not been well explored. A finite time-varying arm set has many meaningful practical applications such as recommendation system (Li et al., 2010; Chu et al., 2011), and the distributed (federated) nature of the applications naturally falls in the federated linear bandits problem with finite time-varying arms. The paper fills this gap by generalizing the SupLinUCB algorithm to the federated setting.

We study both the asynchronous setting (Li and Wang, 2022a) (He et al., 2022a), where clients are active on their own and full participation is not required, and the synchronous setting (Shi et al., 2021; Dubey and Pentland, 2020), where all the clients make decisions at each round and the communication round requires all the clients to upload new information to the server and download the updated information. We design algorithms so as to reduce the communication cost while maintaining optimal regret. Technically, the communication cost is associated with the algorithmic adaptivity, since less adaptivity requires fewer updates and thus fewer communication rounds. The algorithmic adaptivity of linear bandits algorithms was studied in the single-player setting (Han et al., 2020) (Ruan et al., 2021). It was also considered in the federated setting (Wang et al., 2019; Huang et al., 2021; Salgia and Zhao, 2023).

3 System Model and Preliminaries

3.1 Problem Formulation

We consider a federated linear contextual bandits model with K finite but possibly time-varying arms. The model consists of M clients and one server in a star-shaped communication network. Clients jointly solve a linear bandit problem by collecting local information and communicating with the central server through the star-shaped network in a federated manner, with no direct communications among clients. The only function of the server is to aggregate received client information and to send back updated information to clients. It cannot directly play the bandits game.

Specifically, some clients I t ⊆ [ M ] are active at round t . Client i ∈ I t receives K arms (actions to take) associated with contexts { x i t,a } a ∈ [ K ] ⊂ R d with ∥ x i t,a ∥ 2 ≤ 1 . Here we adopt the oblivious adversarial setting, where all contexts are chosen beforehand, and not dependent on previous game observation. Client i then pulls an arm a i t ∈ [ K ] based on the information collected locally as well as previously communicated from the server. A reward r i t,a i t = θ ⊤ x i t,a i t + ϵ t is revealed privately to client i , where θ ∈ R d is an unknown weight vector with ∥ θ ∥ 2 ≤ 1 and ϵ t is an independent 1 -sub-Gaussian noise. At the end of round t , depending on the communication protocol, client i may exchange the collected local information with the server so that it can update the global information.

We aim to design algorithms to guide the clients' decision-making and overall communication behaviors. We analyze two patterns of client activity. 1) Synchronous : all M clients are active at each round. 2) Asynchronous : one client is active at each round. For the latter case, we further assume that client activity is independent of data and history. Denote by T i the number of times client

i is active. In the former case, T i = T j , ∀ i, j ∈ [ M ] , while in the latter case, T i may be different among clients. We define T = ∑ M i =1 T i as the total number of arm pulls from all clients.

The performance is measured under two metrics total regret and communication cost , which concern the decision-making effectiveness and the communication efficiency respectively. Denote by P i T = { t ∈ [ T ] | i ∈ I t } the set of time indices at which client i is active, with | P i T | = T i . The total regret is defined as

where a i, ∗ t = arg max a ∈ [ K ] θ ⊤ x i t,a . We define the communication cost as the total number of communication rounds between clients and the server.

3.2 Preliminaries

Information encoding. In the linear bandits setting (federated or not), the information a client acquires is usually encoded by the gram matrix and the action-reward vector. Specifically, when the client has observed n action-reward pairs { ( x t , r t ) } n t =1 , the information is encoded by matrix A n = ∑ n t =1 x t x ⊤ t and vector b n = ∑ n t =1 r t x t . Denote by Encoder( · ) this encoding function, i.e., A n , b n ← Encoder( { x t , r t } n t =1 ) .

Communication criterion. Communication in our proposed framework is data-dependent, in the same spirit as the 'doubling trick' introduced in Abbasi-Yadkori et al. (2011) to reduce the computation complexity in single-player linear bandits. The key idea is that communication is triggered only when the cumulative local information, represented by the determinant of the gram matrix A n , affects the exploration uncertainty to a great extent and hence the client needs to communicate with the server. Detailed communication protocols will be presented in each algorithm design.

Synchronization procedure. Denote by Sync() a routine that n clients (client 1, . . . , client n ) first communicate their local gram matrices and action-reward vectors to the server, and the server then aggregates the matrices (and vectors) into one gram matrix (and action-reward vector) and transmits them back to the n clients. Specifically, each client i holds newly observed local information (∆ A i , ∆ b i ) , which is the difference between the client's current information ( A i , b i ) and the information after the last synchronization. In other words, (∆ A i , ∆ b i ) is the information that has not been communicated to the server. The server, after receiving the local information { (∆ A i , ∆ b i ) } n i =1 , updates the server-side information ( A ser , b ser ) by A ser ← A ser + ∑ n i =1 ∆ A i , b ser ← b ser + ∑ n i =1 ∆ b i and sends them back to each of the n clients. Each client i will then update the local information by A i ← A ser , b i ← b ser . The procedure is formally presented in Algorithm 1.

Algorithm 1 Sync ( s , server, client 1 , . . . client n )

1: for i = 1 , 2 , . . . , n do

▷ Client-side local information upload

2: Client i sends the local new layer s information (∆ A i s , ∆ b i s ) to the server

3: end for

4: Update server's layer s information: ▷ Server-side information aggregation and distribution

5: Send server information A ser s , b ser s back to all clients

6: for i = 1 , 2 , . . . , n do

7: A i s ← A ser s , b i s ← b ser s , ∆ A i s ← 0 , ∆ b i s ← 0 ▷ Client i updates the local information 8: end for

4 The FedSupLinUCB Framework

In this section, we present a general framework of federated bandits for linear bandits with finite oblivious adversarial actions. Two instances (asynchronous and synchronous) of this general framework will be discussed in subsequent sections.

Building block: SupLinUCB. As the name suggests, the proposed FedSupLinUCB framework is built upon the principle of SupLinUCB (Chu et al., 2011; Ruan et al., 2021). The information ( A,b ) is useful in the sense that the reward corresponding to an action x can be estimated within confidence interval x ⊤ ˆ θ ± α ∥ x ∥ A -1 , where ˆ θ = A -1 b . It is shown in Abbasi-Yadkori et al. (2011) that in linear bandits (even with an infinite number of actions) with α = ˜ O ( √ d ) , the true reward is within the confidence interval with high probability. Moreover, if the rewards in the action-reward vector b are mutually independent, α can be reduced to O (1) . The former choice of α naturally guarantees ˜ O ( d √ T ) regret. However, to achieve regret ˜ O ( √ dT ) , it is critical to keep α = O (1) . This is fulfilled by the SupLinUCB algorithm (Chu et al., 2011) and then recently improved by Ruan et al. (2021). The key intuition is to successively refine an action set that contains the optimal action, where the estimation precision of sets is geometrically strengthened. Specifically, the algorithm maintains ( S +1) layers of information pairs { ( A s , b s ) } S s =0 , and the rewards in the action-reward vectors are mutually independent, except for layer 0 . The confidence radius for each layer s is w s = 2 -s d 1 . 5 / √ T .

Algorithm 2 S-LUCB

FedSupLinUCB. S-LUCB , presented in Algorithm 2, combines the principles of SupLinUCB and OFUL (Abbasi-Yadkori et al., 2011) and is the core subroutine for FedSupLinUCB . We maintain S = ⌈ log d ⌉ information layers, and the estimation accuracy starts from d 1 . 5 / √ T of layer 0 and halves as the layer index increases. Finally, it takes Θ(log d ) layers to reach the sufficient accuracy of √ d/T and achieves the minimax-optimal regret.

When client i is active, the input parameters ( A i , b i ) contain information received from the server at the last communication round, and (∆ A i , ∆ b i ) is the new local information collected between two consecutive communication rounds. { x i t, 1 , . . . , x i t,K } is the set of contexts observed in this round. Client i can estimate the unknown parameter θ either with all available information or just making a lazy update. This choice depends on the communication protocol and will be elaborated later. During the decision-making process, client i first makes arm elimination at layer 0 to help bootstrap the accuracy parameters. Then, it goes into the layered successive screening in the same manner as the SupLinUCB algorithm, where we sequentially eliminate suboptimal arms depending on their empirical means and confidence widths. After taking action a i t and receiving the corresponding reward r i t,a i t , client i updates its local information set (∆ A i s , ∆ b i s ) by aggregating the context into layer s in which we take the action, before returning layer s .

5 Asynchronous FedSupLinUCB

In the asynchronous setting, only one client is active in each round. Note that global synchronization and coordination are not required, and all inactive clients are idle.

5.1 Algorithm

We first initialize the information for all clients and the server (gram matrix and action-reward vector) in each layer s ∈ [0 : S ] . We assume only one client i t is active at round t . It is without loss of generality since if multiple clients are active, we can queue them up and activate them in turn. More discussion of this equivalence can be found in He et al. (2022a); Li and Wang (2022a). The active client chooses the action, receives a reward, updates local information matrices of layer s with a lazy update according to S-LUCB , and decides whether communication with the server is needed by the criterion in Line 7 of Algorithm 3. If communication is triggered, we synchronize client i t with the server by Algorithm 1.

5.2 Performance Analysis

Theorem 5.1. For any 0 < δ < 1 , if we run Algorithm 3 with C = 1 /M 2 , then with probability at least 1 -δ , the regret of the algorithm is bounded as R T ≤ ˜ O (√ d ∑ M i =1 T i ) = ˜ O ( √ dT ) . Moreover, the corresponding communication cost is bounded by O ( dM 2 log d log T ) .

Remark 1. The minimax lower bound of the expected regret for linear contextual bandits with K adversarial actions is Ω( √ dT ) , given in Chu et al. (2011). Theorem 5.1 indicates that AsyncFedSupLinUCB achieves order-optimal regret (up to polylog term) with O ( dM 2 log d log T ) communication cost. To the best of our knowledge, this is the first algorithm that achieves the (near) optimal regret in federated linear bandits with finite adversarial actions.

Remark 2. Without any communication, each client would execute SupLinUCB (Chu et al., 2011) for T rounds locally, and each client can achieve regret of order ˜ O ( √ dT ) . Therefore, the total regret of

M clients is upper bound by R T ≤ ∑ i =1 dT i polylog ( T ) ≤ dM ∑ i =1 T i polylog ( T ) , where the last inequality becomes equality when T i = T j , ∀ i, j ∈ [ M ] . Compared with conducting M independent SupLinUCB algorithms locally, AsyncFedSupLinUCB yields an average per-client gain of 1 / √ M , demonstrating that communications in the federated system can speed up local linear i i M √ √ M bandits decision-making at clients.

Remark 3. Most previous federated linear bandits consider the infinite action setting, based on the LinUCB principle (Abbasi-Yadkori et al., 2011). AsyncFedSupLinUCB considers a finite adversarial action setting and has a √ d reduction on the regret bound. Fed-PE proposed in Huang et al. (2021) also considers the finite action setting. However, their action sets are fixed. We generalize their formulation and take into account a more challenging scenario, where the finite action set can be chosen adversarially. The regret order is the same as Fed-PE (ignoring the ploylog term), indicating

that it is the finite actions as opposed to fixed actions that fundamentally leads to the √ d regret improvement in the federated linear bandits setting.

Communication cost analysis of FedSupLinUCB . We sketch the proof for the communication cost bound in Theorem 5.1 in the following, while deferring the detailed proofs for the regret and the communication cost to Appendix C.

We first study the communication cost triggered by some layer s . Denote by A ser t,s the gram matrix in the server aggregated by the gram matrices uploaded by all clients up to round t . Define T n,s = min { t ∈ [ T ] | det ( A ser t,s ) ≥ 2 n } , for each n ≥ 0 . We then divide rounds into epochs { T n,s , T n,s + 1 , · · · , min( T n +1 ,s -1 , T ) } for each n ≥ 0 . The number of communications triggered by layer s within any epoch can be upper bounded by 2( M +1 /C ) (see Lemma C.1), and the number of non-empty epochs is at most d log(1 + T/d ) by Lemma A.1. Since there are S = ⌈ log d ⌉ layers and synchronization among all layers is performed once communication is triggered by any layer (Line 8 in Algorithm 3), the total communication cost is thus upper-bounded by O ( d ( M +1 /C ) log d log T ) . Plugging C = 1 /M 2 proves the result.

We note that although choosing a larger C would trigger fewer communications, the final choice of C = 1 /M 2 takes into consideration both the regret and the communication cost, i.e., to achieve a small communication cost while maintaining an order-optimal regret.

6 Synchronous FedSupLinUCB

In the synchronous setting, all clients are active and make decisions at each round. Though it can be viewed as a special case of the asynchronous scenario (clients are active and pulling arms in a round-robin manner), the information update is broadcast to all clients. In other words, the key difference from the asynchronous scenario besides that all clients are active at each round is that when a client meets the communication criterion, all clients will upload local information to the server and download the updated matrices. This leads to a higher communication cost per communication round, but in this synchronous scenario, knowing all clients are participating allows the communicated information to be well utilized by other clients. This is in sharp contrast to the asynchronous setting, where if many other clients are active in the current round, uploading local information to the clients seems unworthy. To mitigate the total communication cost, we use a more refined communication criterion to enable time-independent communication cost.

6.1 The Algorithm

The SyncFedSupLinUCB algorithm allows each client to make decisions by the S-LUCB subroutine. Note that the decision-making is based on all available local information instead of the lazy update in the AsyncFedSupLinUCB algorithm. The communication criterion involves the count of rounds since the last communication, which forces the communication to prevent the local data from being obsolete. Some layers may trigger the communication criterion either because the local client has gathered enough new data or due to having no communication with the server for too long. We categorize these layers in the CommLayers and synchronize all the clients with the server.

6.2 Performance Analysis

Theorem 6.1. For any 0 < δ < 1 , if we run Algorithm 4 with D = T c log T c d 2 M , with probability at least 1 -δ , the regret of the algorithm is bounded as R T ≤ ˜ O ( √ dMT c ) where T c is the total per-client arm pulls. Moreover, the corresponding communication cost is bounded by O ( √ d 3 M 3 log d ) .

Remark 4. Theorem 6.1 demonstrates SyncFedSupLinUCB also achieves the minimax regret lower bound while the communication cost is independent of T c . It is particularly beneficial for large T c . Especially, the number of total rounds in the synchronous scenario is T = MT c , while in the asynchronous setting, we have T = ∑ M i =1 T i rounds.

Communication cost analysis of Sync-FedSupLinUCB . We sketch the proof for the communication cost bound in Theorem 6.1 below, while deferring the detailed proofs for the regret and the communication cost to Appendix D.

Algorithm 4 SyncFedSupLinUCB

We call the chunk of consecutive rounds without communicating information in layer s (except the last round) an epoch . Information in layer s is collected locally by each client and synchronized at the end of the epoch, following which the next epoch starts. Denoted by A all p,s the synchronized gram matrix at the end of the p -th epoch. For any value β > 0 , there are at most ⌈ T c β ⌉ epochs that contain more than β rounds by pigeonhole principle. If the p -th epoch contains less than

β rounds, then log ( det ( A all p,s ) det ( A all p -1 ,s ) ) > D β based on the communication criterion and the fact that

∑ P p =1 log det ( A all p,s ) det ( A all p -1 ,s ) ≤ R s = O ( d log( T c )) (see Equation (6)). The number of epochs containing rounds fewer than β is at most O ( ⌈ R s D/β ⌉ ) . Noting that D = T c log( T c ) d 2 M , the total number of epochs for layer s is at most ⌈ T c β ⌉ + ⌈ R s β D ⌉ = O ( √ T c R s D ) = O ( √ d 3 M ) by taking β = √ DT c R s . The total communication cost is thus upper bounded by O ( SM √ d 3 M ) = O (log( d ) √ d 3 M 3 ) .

7 Extensions of FedSupLinUCB

In this section, we extend the FedSupLinUCB algorithm to address two distinct settings in federated systems: scenarios characterized by heterogeneous variances, and those affected by adversarial corruptions.

7.1 Federated Heteroscedastic Linear Bandits

We have so far focused on the federated linear bandits with 1-sub-Gaussian reward noises. In this section, we adapt AsyncFedSupLinUCB to the case where the reward noises have heterogeneous variances, which extends the heteroscedastic linear bandits as studied in Zhou et al. (2021); Zhou and Gu (2022) to the asynchronous federated setting, where one client is active at a time. Specifically, the reward noises { ϵ t } t ∈ [ T ] are independent with | ϵ t | ≤ R, E [ ϵ t ] = 0 and E [ ϵ 2 t ] ≤ σ 2 t , where σ t is known to the active client.

We propose a variance-adaptive AsycFedSupLinUCB and analyze its regret and the communication cost in the theorem below, with the algorithm and the proof details in Appendix E due to space constraint. The regret is significantly less than that of the AsyncFedSupLinUCB when the variances { σ 2 t } are small.

Theorem 7.1. For any 0 < δ < 1 , if we run the variance-adaptive AsyncFedSupLinUCB algorithm in Appendix E with C = 1 /M 2 , with probability at least 1 -δ , the regret is bounded as R T ≤ ˜ O ( √ d ∑ T t =1 σ 2 t ) , and the communication cost is bounded by O ( dM 2 log 2 T ) .

7.2 Federated Linear Bandits with Corruption

We further explore asynchronous federated linear bandits with adversarial corruptions, where an adversary inserts a corruption c t to the reward r t of the active client at round t . The total corruption is bounded by ∑ T t =1 | c t | ≤ C p . We incorporate the idea of linear bandits with adversarial corruption studied in He et al. (2022b) to the proposed FedSupLinUCB framework and propose the Robust AsyncFedSupLinUCB algorithm, with details in Appendix F. Robust AsyncFedSupLinUCB can achieve the optimal minimax regret (matching the lower bound in He et al. (2022b)) while incurring a low communication cost.

Theorem 7.2. For any 0 < δ < 1 , if we run the Robust AsyncFedSupLinUCB algorithm in Appendix F with C = 1 /M 2 , with probability at least 1 -δ , the regret is bounded as R T ≤ ˜ O ( √ dT + dC p ) , and the communication cost is bounded by O ( dM 2 log d log T ) .

8 Experiments

We have experimentally evaluated FedSupLinUCB in the asynchronous and synchronous settings on both synthetic and real-world datasets. We report the results in this section.

8.1 Experiment Results Using Synthetic Dataset

We simulate the federated linear bandits environment specified in Section 3. With T = 40000 , M = 20 , d = 25 , A = 20 , contexts are uniformly randomly sampled from an l 2 unit sphere, and reward r t,a = θ ⊤ x t,a + ϵ t , where ϵ t is Gaussian distributed with zero mean and variance σ = 0 . 01 . It should be noted that while M clients participate in each round in the synchronous scenario, only one client is active in the asynchronous case. In the plots, the x -axis coordinate denotes the number of arm pulls, which flattens the actions in the synchronous setting.

(a) Regret: arrival patterns.

(b) Communication: arrival pat- terns.

(c) Regret: client numbers.

(d)

Regret vs communications.

500

3500

Async-round-robin

Async-random

70

1000

Sync

Async

400

3000

Async-click-leave

Sync-round-robin

60

900

Cumulative regret

000

Communication

2500

2000

Cumulative regret

OS

40

SupLinUCB

20 clients async

800

200

1500

30

40 clients async

80 clients async

700

1000

20

600

100

Async-random

Async-round-robin

Async-click-leave

500

10

500

Sync-round-robin

Q

0

50001000015000 20000 25000300003500040000

Number of arm pulling

0

000000005000000 0005200002 00050 000000005

Number of arm pulling

0

250

500

Number of arm pulling

750

10001250

1500

17502000

2000

4000

Communication cost

6000

8000100001200014000

Arrival pattern. We first investigate the impact of different arrival patterns (the sequence of activating clients): (1) Random , which randomly allocates T/M arm pulls in [ T ] for each client. (2) Round-robin , i.e. [1 , 2 , 3 , · · · , M, 1 , 2 , 3 , · · · M, · · · ] . (3) Click-leave , i.e. [1 , 1 , · · · , 2 , 2 , · · · , · · · , M, M, · · · ] . The regret and the communication cost of these three arrival patterns in the synthetic experiment are reported in Figure 1(a) and Figure 1(b), respectively. We note that although the upper bound analysis in our proof is for the worst-case instance, the numerical results suggest that different arrival patterns result in diverse regret performances. Round-robin and random patterns are more challenging since both local bandit learning and each client's policy updates happen relatively slowly. The click-leave pattern, which is the closest to the centralized setting, achieves the best regret. In addition, compared with AsyncFedSupLinUCB , SyncFedSupLinUCB achieves better cumulative regrets with a higher communication cost.

Amount of clients. The per-client cumulative regret as a function of T c = T/M with different amounts of clients is plotted in Figure 1(c). In comparison to the baseline SupLinUCB, FedSupLinUCB algorithms achieve better regret via communication between clients and the server. We can see from the experiment that FedSupLinUCB significantly reduces the per-client regret compared with SupLinUCB, and achieves a better regret as M increases in both asynchronous and synchronous settings.

Trade-off between regrets and communications. Weevaluate the tradeoff between communication and regret by running FedSupLinUCB with different communication threshold values C and D in asynchronous and synchronous settings respectively. The results are reported in Figure 1(d), where each scattered dot represents the communication cost and the cumulative regret that FedSupLinUCB has achieved with a given threshold value at round T = 40000 . We see a clear tradeoff between the regret and the communication. More importantly, SyncFedSupLinUCB achieves a better tradeoff than AsyncFedSupLinUCB .

8.2 Experiment Results Using Real-world Dataset

We further investigate how efficiently the federated linear bandits algorithm performs in a more realistic and difficult environment. We have carried out experiments utilizing the real-world recommendation dataset MovieLens 20M (Harper and Konstan, 2015). Following the steps in Li and Wang (2022b), we first filter the data by maintaining users with above 2500 movie ratings and treating rating points greater than 3 as positive, ending up with N = 37 users and 121934 total movie rating interactions. Then, we follow the process described in Cesa-Bianchi et al. (2013) to generate the contexts set, using the TF-IDF feature d = 25 and the arm set K = 20 . We plot the per-client normalized rewards of the FedSupLinUCB algorithm with different client numbers M in synchronous and asynchronous cases respectively. Note that the per-client cumulative rewards here are normalized by a random strategy. From Figure 2(a) and Figure 2(b), we can see that in both synchronous and asynchronous experiments, FedSupLinUCB has better rewards than SupLinUCB, and the advantage becomes more significant as the number of users increases.

(a)

FedSupLinUCB

.

(b)

Sync-

FedSupLinUCB

.

2.2

2.2

Normalized reward

2.0

Normalized reward

2.0

1.8

1.8

1.6

SupLinUCB

SupLinUCB

5 clients async

1.6

5clients sync

10 clients async

10 clients sync

1.4

20 clients async

20 clients sync

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Number of arm pulling

Number of arm pulling

9 Conclusion

We studied federated linear bandits with finite adversarial actions, a model that has not been investigated before. We proposed FedSupLinUCB that extends the SupLinUCB and OFUL principles to the federated setting in both asynchronous and synchronous scenarios, and analyzed their regret and communication cost, respectively. The theoretical results proved that FedSupLinUCB is capable of approaching the minimal regret lower bound (up to polylog terms) while only incurring sublinear communication costs, suggesting that it is the finite actions that fundamentally determines the regret behavior in the federated linear bandits setting. Furthermore, we examined the extensions of the algorithm design to the variance-adaptive and adversarial corruption scenarios.

Acknowledgments and Disclosure of Funding

The work of LF and CS was supported in part by the U.S. National Science Foundation (NSF) under grants 2143559, 2029978, and 2132700.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24.

Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422.

Cesa-Bianchi, N., Gentile, C., and Zappella, G. (2013). A gang of bandits. Advances in neural information processing systems , 26.

Chu, W., Li, L., Reyzin, L., and Schapire, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings.

Dani, V., Hayes, T. P., and Kakade, S. M. (2008). Stochastic linear optimization under bandit feedback. 21st Annual Conference on Learning Theory , pages 355-366.

Dubey, A. and Pentland, A. (2020). Differentially-private federated linear bandits. Advances in Neural Information Processing Systems , 33:6003-6014.

Han, Y., Zhou, Z., Zhou, Z., Blanchet, J., Glynn, P. W., and Ye, Y. (2020). Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321 .

Harper, F. M. and Konstan, J. A. (2015). The MovieLens datasets: History and context. ACM Trans. Interact. Intell. Syst. , 5(4):1-19.

He, J., Wang, T., Min, Y., and Gu, Q. (2022a). A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. arXiv preprint arXiv:2207.03106 .

He, J., Zhou, D., Zhang, T., and Gu, Q. (2022b). Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in neural information processing systems .

Huang, R., Wu, W., Yang, J., and Shen, C. (2021). Federated linear contextual bandits. Advances in Neural Information Processing Systems , 34:27057-27068.

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Li, C. and Wang, H. (2022a). Asynchronous upper confidence bound algorithms for federated linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 6529-6553. PMLR.

Li, C. and Wang, H. (2022b). Communication efficient federated learning for generalized linear bandits. arXiv preprint arXiv:2202.01087 .

Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670.

Li, T., Song, L., and Fragouli, C. (2020). Federated recommendation system via differential privacy. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2592-2597. IEEE.

Li, Y., Wang, Y., and Zhou, Y. (2019). Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory , pages 2173-2174. PMLR.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Proc. AISTATS , pages 1273-1282, Fort Lauderdale, FL, USA.

Ruan, Y., Yang, J., and Zhou, Y. (2021). Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87.

Salgia, S. and Zhao, Q. (2023). Distributed linear bandits under communication constraints. In International Conference on Machine Learning , pages 29845-29875. PMLR.

Shi, C., Shen, C., and Yang, J. (2021). Federated multi-armed bandits with personalization. In International Conference on Artificial Intelligence and Statistics , pages 2917-2925. PMLR.

Wang, Y., Hu, J., Chen, X., and Wang, L. (2019). Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations .

Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507 .

Zhou, D., Gu, Q., and Szepesvari, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the linear contextual bandits problem with federated learning of $M$ clients communicating with a central sever. In particular, the paper assumes adversarial finite action, and considers two cases of the communication: asynchronous and synchronous. \n\nFollowing the idea of OFUL, this work extends the previous SupLinUCB in linear contextual bandits to deal with the federated learning scenario, and proposes the algoirthm FedSupLinUCB which achieves $\\widetilde{O}(\\sqrt{dT})$ where $T$ is the total number of pulls (the summation of cumulative pulls over $M$ clients together), and $d$ is the dimension. More importantly, this result is attained with limited communication cost $\\mathcal{O}(\\sqrt{d^3M^3}\\log(d))$ for synchronous case, and $\\mathcal{O}(dM^2\\log(d)\\log(T))$ for asynchronous case. \n\nIn addition, the FedSupLinUCB further extends to the variance-adaptive and adversarial corruption scenarios.\n\nSTRENGTHS:\nThe paper is clearly-written and well-organized. The notations are well-defined and the meanings are explained clearly prior to their usages.  \n\n1. Combining online learning and federated learning is an interesting direction. The proposed algorithm attains the optimal regret bound of such problem up to some $\\log T$ factors, and further ensures limited communication cost, which could be dominating in the distributed systems. Among these results, the most interesting one is that for the synchronous case, which is independent with the horizon $T$. \n\n2. The instance-dependent regret bounds for the variance-adaptive and adversarial corruption scenarios are also very interesting. The worst case regret bound is not meaningful in most real world applications.\n\nWEAKNESSES:\nThis paper does not have any specific weakness.\n\nQUESTIONS:\n1. Any lower bound of the communication cost in the synchronous case or the asynchronous case? \n2. Any idea to improve the regret bounds to the optimal regret bound $\\mathcal{O}(\\sqrt{dT\\log T})$, which can be obtained by Fed-PE?\n\nLIMITATIONS:\nThis work is pure theoretical, and does not have any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies a federated linear bandits model, where M clients communicate with a central server to solve a linear contextual bandits problem with finite adversarial action sets and proposes the FedSupLinUCB algorithm, which extends the SupLinUCB and OFUL principles in linear contextual bandits. Both asynchronous and synchronous cases are considered. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of FedSupLinUCB on both synthetic and real-world datasets.

STRENGTHS:
The method description, theoretical derivation and complexity analysis of the article are very detailed and well organized.

WEAKNESSES:
See Questions part.

QUESTIONS:
1. Do the adversarial corruption actios only exist in the setting of asynchronous case? 
2. Are there differences in intensity for adversarial actions? If so, do different intensities have different effects on the experimental scene? 
3. Is Robust Async-FedSupLinUCB a general robust method or can only be used against a specific adversarial setting？

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors consider the problem of Federated linear bandits with finite adversarial actions. This is the first time investigating a setting where federated clients are faced with a set of actions to choose from that changes over time in an oblivious adversarial manner, and the authors of this paper do so for  both synchronous and asynchronous versions of the problem.\nTo handle this problem, the authors propose two algorithms that are based on SupLinUCB (Chen et al. 2011), which is a standard approach for linear bandits and adapted to federated bandits. Then, they combine this S-LUCB approach with two different protocols to tackle both the asynchronous and the synchronous version of the problem.One of the challenges of federated learning comes with the cost of the communication between the clients and the server. \nIn the asynchronous setting, only one client is active at the time and the the synchronization rate depends on each individual client. For this problem, the authors prove that the proposed algorithm achieves an order optimal (up to log factors) high probability bound of order $O(\\sqrt{dT})$ while ensuring that their total communication cost is logarithmic in T. \nIn the synchronous case, layers of clients work simultaneously which allows for the server to take advantage of the various information provided by the clients. This means that these clients don't just synchronize when the information that they have has changed, but they also do so periodically to ensure that their information is up to date. This extra information provided by the  server to the clients allows to actually lower the total number of communications and obtain a $\\tilde O(\\sqrt{dMT_c})$ regret bound while keeping a time independent bound on the number of communication.\n\nThe authors provide detailed proofs of their results in the appendices and experiments, both on generated and real life datasets to highlight how the regret and the communication costs evolve in terms of arrival patterns and number of clients.\n\nSTRENGTHS:\nThis works studies a new variant of the federated linear bandits problem where the clients are facing finite adversarial arm sets instead of either finite fixed armed sets or infinite arm sets. \nTo do so, they build upon existing algorithms for linear bandits with an adversarial arm set, SupLinUCB (Chu et al. 2011) and for asyncronous federated learning FedLinUCB (He et al. 2022a). The novelty of their approach mainly resides in the improved communication protocol, in particular in the synchronous setting, where they take advantage of a layer structure to limit the communication cost while still synchronizing regularly. \nIn both the asynchronous and the synchronous settings, they recover the regret of  SupLinUCB in the single player setting (Chu et al. 2011), and are within a log T factor of the state of the art results for federated synchronous and asynchronous learning, both in terms of regret and communication costs. \n\nOverall, this is a well presented paper that provides some interesting results for a new variation of the federated learning problem, which might be more relevant in practice, which is reinforced by the fact that they also run experiments on real worlds datasets.\nPresenting results for the corrupted setting is also a nice generalization of their results.\n\nWEAKNESSES:\nIn the experiments section, it would be interesting to see the proposed algorithms compared with other results for federated learning: The results in Appendix H are promising and could benefit from being extended. \nIt  would also be nice if the legends on the plot could be larger so it is a bit easier to read.\n\nQUESTIONS:\nThe problems studied in federated learning are very close to these of multiplayer multiarmed bandits. Have you thought of whether there are results that could bridge the two settings?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the linear bandits problem in the context of federated learning. It proposes a general algorithm called FedSupLinUCB that solves a linear contextual bandit problem with finite adversarial action sets that may vary across clients. The paper also considers practical challenges in federated settings such as synchrony, communication efficiency, and time-evolving scenarios.\n\nSTRENGTHS:\nThis paper explores the linear contextual bandit problem with finite adversarial action sets in the federated learning setting. Additionally, it addresses several practical challenges in federated settings, such as synchrony, communication efficiency, and time-evolving scenarios.\n\nWEAKNESSES:\nBackground and challenge:\n\n- The introduction and preliminaries do not provide a detailed definition of the time-evolving and adversarial arm. To improve comprehension, it would be helpful to explain the impact of these factors on the federated linear bandit problem.\n- Although the paper discusses the main challenges of federated linear bandits, it should emphasize the key differences from traditional linear bandit problems, such as single-player bandits or distributed bandits.\n- The paper lacks strong motivation due to the vague background and challenge.\n\nNovelty:\n\n- The main theoretical contributions of the paper extend previous work on linear bandit problems to the federated setting. For example, FedSumLinUCB is an extension of SupLinUCB and OFUL. Robust Async-FedSupLinUCB incorporates ideas from He et al. (2020b).\n- The theory and methodology employed in the paper appear to be borrowed from the literature.\n\nExperiment setting:\n\n- The reward function utilized in the experiments, where $r_{t, a}=\\theta^{\\top} x_{t, a}+\\epsilon_t$ , does not align with the settings described in the paper. To my knowledge, it is more general to let the reward function different across clients with certain variances according to the environments.\n- The experiments conducted in the paper are limited. Only a classic algorithm proposed in 2011 by Chu et al. is used as a baseline, and not all proposed algorithms are fully evaluated.\n\nWriting:\n\n- It is unusual for a conference paper to have 10 sections.\n\nQUESTIONS:\n- What are the specific challenges in the federated linear bandit problem? Federated learning was originally proposed to address data-sharing issues. Does the federated linear bandit problem face similar challenges related to system efficiency and heterogeneity?\n- What unique solutions or methods does the paper contribute? It appears that some challenges have already been addressed by previous works.\n\nLIMITATIONS:\nNo limitations are discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}