Federated Linear Contextual Bandits

1 Introduction . Federated learning ( FL ) ( McMahan et al. , 2017 ) is an emerging distributed machine learning ( ML ) paradigm where massive number of clients collaboratively learn a shared prediction model while keeping all the training data on local devices . Compared with standard centralized machine learning , FL has the following characteristics ( Kairouz et al. , 2021 ) : • Heterogeneous local datasets . The local datasets , which are often generated at edge devices , are likely drawn from non-independent and identically distributed ( non-IID ) distributions . • Communication efficiency . The communication cost scales with the number of clients , which is one of the primary bottlenecks of FL . It is critical to minimize the communication cost while maintaining the learning accuracy . • Privacy . FL protects local data privacy by only sharing model updates instead of the raw data . While the main focus of the state-of-the-art FL is on the supervised learning setting , recently , a few researchers begin to extend FL to the multi-armed bandits ( MAB ) framework ( Lai and Robbins , 1985 ; Auer et al. , 2002 ; Bubeck and Cesa-Bianchi , 2012 ; Agrawal and Goyal , 2012 , 2013a ) . In the canonical setting of MAB , a player chooses to play one arm from a set of arms at each time slot . An arm , if played , will offer a reward that is drawn from its distribution which is unknown to the player . With all previous observations , the player needs to decide which arm to pull each time in order to maximize the cumulative reward . MAB thus represents an online learning model that naturally captures the intrinsic exploration-exploitation tradeoff in many sequential decision-making problems . Extending FL to the MAB framework is naturally motivated by a corpus of applications , such as recommender systems , clinical trials , and cognitive radio . In those applications , the sequential decision making involves multiple clients and is distributed by nature . While classical MAB models assume immediate access to the sequentially generated data at the learning agent , under the new 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . realm of FL , local datasets can be stored and analyzed at the clients , thus reducing the communication load and potentially protecting the data privacy . Despite the potential benefits of FL , the sequential decision making and bandit feedback bring new challenges to the design of FL algorithms in the MAB setting . Different from the supervised learning setting where static datasets are collected beforehand , under the MAB setting , data is generated sequentially as decisions are made , actions are taken , and observations are collected . In order to maximize the cumulative reward and minimize the corresponding learning regret , it thus requires sophisticated coordination of the actions of the clients . The heterogeneous reward distributions across clients make the coordination process even more convoluted and challenging . Besides , the data privacy and communication efficiency requirements result in significant challenges for efficient information exchange and aggregation between local clients and the central server . In this work , we attempt to address those challenges in a federated linear contextual bandits framework . This particular problem is motivated by the following exemplary applications . • Personalized content recommendation . For content ( arm ) recommendation in web-services , user engagement ( reward ) depends on the profile of a user ( context ) . The central server may deploy a recommender system on each user ’ local device ( client ) in order to personalize recommendations without knowing the personal profile or behavior of the user . • Personalized online education . In order to maximize students performances ( reward ) in online learning , the education platform ( central server ) needs to personalize teaching methods ( arms ) based on the characteristics of individual students ( context ) . With the online learning software installed at local devices ( client ) , it is desirable to personalize the learning experiences without allowing the platform to access students ’ characteristics or scores . In those examples , the reward of pulling the same arm at different clients follows different distributions dependent on the context as in contextual bandits ( Auer , 2003 ; Langford and Zhang , 2008 ) . We note that conventional contextual bandits is defined with respect to a single player , where the time-varying context can be interpreted as different incoming user profiles . In contrast , we consider a multi-client model , where each client is associated with a fixed user profile . The variation of contexts is captured over clients as opposed to over time . Although the set of clients remains fixed through the learning process , the reward of pulling the same arm still varies across clients . Such a model naturally takes data heterogeneity into consideration . Besides , we adopt a linear reward model , which has been widely studied in contextual bandits ( Li et al. , 2010 ; Agrawal and Goyal , 2013b ) . Main contributions . Our main contributions are summarized as follows . First , we propose a new federated linear contextual bandits model that takes the diverse user preferences and data heterogeneity into consideration . Such a model naturally bridges local stochastic bandits with linear contextual bandits , and is well poised to capture the tradeoffs between communication efficiency and learning performances in the federated bandits setting . Second , we design a novel algorithm named Fed-PE and further develop its variants to solve the federated linear contextual bandits problem . Under Fed-PE , clients only upload their local estimates of the global parameters without sharing their local feature vectors or raw observations . It not only keeps the personal information private , but also reduces the upload cost . We explicitly show that Fed-PE and its variants achieve near-optimal regret performances for both disjoint and shared parameter cases with logarithmic communication costs . Third , we generalize the G-optimal design from the single-player setting ( Lattimore and Szepesvári , 2020 ) to the multi-client setting . We develop a block coordinate ascent algorithm to solve the generalized G-optimal design efficiently with convergence guarantees . Such a multi-client G-optimal design plays a vital role in Fed-PE , and may find broad applications in related multi-agent setups . Finally , we introduce a novel concept called collinearly-dependent policy and show that the celebrated LinUCB type of policies ( Li et al. , 2010 ) , Thompson sampling based policies with Gaussian priors ( Agrawal and Goyal , 2013b ) , and least squared estimation based policies , such as Fed-PE , are all in this category . By utilizing the property of collinearly-dependent policies , we are able to characterize a tight minimax regret lower bound in the disjoint parameter setting . We believe that this concept may be of independent interest for the study of bandits with linear rewards . √ Notations . Throughout this paper , we use ‖x‖V to denote √ xᵀV x . The range of a matrixA , denoted by range ( A ) , is the subspace spanned by the column vectors of A . We use A† and Det ( A ) to denote the pseudo-inverse and pseudo-determinant of square matrix A , respectively . The specific definitions can be found in Appendix B of the supplementary material . 2 Related Works . Collaborative and distributed bandits . Our model is closely related to the collaborative and distributed bandits when action collision is not considered . Landgren et al . ( 2016 , 2018 ) and Martínez-Rubio et al . ( 2019 ) study distributed bandits in which multiple agents face the same MAB instance , and the agents collaboratively share their estimates over a fixed communication graph in order to design consensus-based distributed estimation algorithms to estimate the mean of rewards at each arm . Szorenyi et al . ( 2013 ) considers a similar setup where in each round an agent is able to communicate with a few random peers . Korda et al . ( 2016 ) considers the case where clients in different unknown clusters face independent bandit problems , and every agent can communicate with only one other agent per round . The communication and coordination among the clients in those works are fundamentally different from our work . Wang et al . ( 2020 ) investigates communication-efficient distributed linear bandits , where the agents can communicate with a server by sending and receiving packets . It proposes two algorithms , namely , DELB and DisLinUCB , for fixed and time-varying action sets , respectively . The fixed action set setting is similar to our setup , except that it assumes that all agents face the same bandits model , which does not take data heterogeneity into consideration . Federated bandits . A few recent works have touched upon the concept of federated bandits . With heterogeneous reward distributions at local clients , Shi and Shen ( 2021 ) and Shi et al . ( 2021 ) investigate efficient client-server communication and coordination protocols for federated MAB without and with personalization , respectively . Agarwal et al . ( 2020 ) studies regression-based contextual bandits as an example of the federated residual learning framework , where the reward of a client depends on both a global model and a local model . Li et al . ( 2020 ) and Zhu et al . ( 2021 ) focus on differential privacy based local data privacy protection in federated bandits . While the linear contextual bandit model considered in Dubey and Pentland ( 2020 ) is similar to this work , it focuses on federated differential privacy and proposes a LinUCB-based FedUCB algorithm , which incurs a higher regret compared with our result for the shared parameter case . A regret and communication cost comparison between Fed-PE and other baseline algorithms is provided in Table 1 . 3 Problem Formulation . Clients and local bandits model . We consider a federated linear contextual bandits setting where there are M clients pulling the same set of K items ( arms ) denoted as [ K ] : = { 1 , 2 , . . . , K } . At each time t , each client i ∈ [ M ] pulls an arm ai , t ∈ [ K ] based on locally available information . The incurred reward yi , t is given by yi , t = ri , ai , t + ηi , t , where ηi , t is a random noise , and ri , ai , t is 1Results adapted from Dubey and Pentland ( 2020 ) by letting the privacy budget 1/ go to 0 . 2Results adapted from the single-player linear contextual bandits studied in Dimakopoulou et al . ( 2017 ) by assuming instantaneous information exchange and sequential decision-making at the central server . the unknown expected reward by pulling arm ai , t . We note that without additional assumptions or interaction among the clients , each local model is a standard single-player stochastic MAB , where classic algorithms such as UCB ( Auer and Ortner , 2010 ) and Thompson sampling ( Agrawal and Goyal , 2012 ) are known to achieve order-optimal regret . Linear reward structure with global parameters . In order to capture the inherent correlation between rewards of pulling the same arm by different clients , we assume ri , a has a linear structure , i.e. , ri , a = x ᵀ i , aθa , where xi , a ∈ Rd is the feature vector associated with client i and arm a , and θa ∈ Rd is a fixed but unknown parameter vector for each a ∈ [ K ] . Here we use xᵀ to denote the transpose of vector x . The same arm a may have different reward distributions for different clients , due to potentially varying xi , a across clients . Such a linear model naturally captures the heterogeneous data distributions at the clients , yet admits possible collaborations among clients due to the common parameters { θa } a∈ [ K ] . When θa varies for different arm a , it is called the disjoint parameter case ; when θa is known to be a constant across the arms , it is the shared parameter case . We investigate both cases in Sections 4 and 5 , respectively . Communication model . We assume there exists a central server in the system , and similar to FL , the clients can communicate with the server periodically with zero latency . Specifically , the clients can send “ local model updates ” to the central server , which then aggregates and broadcasts the updated “ global model ” to the clients . ( We will specify these components later . ) Note that just as in FL , communication is one of the major bottlenecks and the algorithm has to be conscious about its usage . Similar to Wang et al . ( 2020 ) , we define the communication cost of an algorithm as the number of scalars ( integers or real numbers ) communicated between server and clients . We also make the assumption that clients and server are fully synchronized ( McMahan et al. , 2017 ) . Data privacy concerns . Similar to Dubey and Pentland ( 2020 ) , our contextual bandit problem involves two sets of information that are desirable to be kept private to client i : the feature vectors { xi , a } a∈ [ K ] and the observed rewards { yi , t } t∈ [ T ] . Different from the differential privacy mechanism adopted in Dubey and Pentland ( 2020 ) , in this work , we aim to communicate estimated global model parameters { θa } a between the clients and the server . This is consistent with the FL framework , where only model updates are communicated instead of the raw data . Assumption 1 We make the following assumptions throughout the paper : 1 ) Bounded parameters : For any i ∈ [ M ] , a ∈ [ K ] , we have ‖θa‖2 ≤ s , 0 < ` ≤ ‖xi , a‖2 ≤ L. 2 ) Independent 1-subgaussian noise : ηi , t is a 1-subgaussian noise parameter sampled indepen- dently at each time for each client with E [ ηi , t ] = 0 , E [ exp ( ληi , t ) ] ≤ exp ( λ 2 2 ) for any λ > 0 . Assumption 1.1 is a standard assumption in the bandit literature , which ensures that the maximum regret at any step is bounded . We emphasize that our work does not make any assumption on the knowledge of suboptimality gaps , nor do we assume the existence of a unique optimal arm at each client . Our objective is to minimize the expected cumulative regret among all clients , defined as : E [ R ( T ) ] = E [ M∑ i=1 T∑ t=1 ( xᵀi , a∗i θa∗i − x ᵀ i , ai , t θai , t ) ] , ( 1 ) where a∗i ∈ [ K ] is an optimal arm for client i : ∀b 6= a∗i , x ᵀ i , a∗i θa∗i − x ᵀ i , bθb ≥ 0 .

Summary: This paper discusses the federated multi-armed bandit problem with linear reward functions. The formulation utilizes a linear structure with agent-dependent contexts to model differences between agent reward functions. The authors propose an algorithm based on a multi-agent G-optimal design with competitive theoretical performance under heterogeneity. The authors additionally establish minimax rates under a reasonable collinearity assumption. The algorithms FedPE and Enhanced FedPE additionally provide competitive experimental performance on both simulated and real-world benchmarks. In summary, the paper is a good contribution, however, there are some issues that I would like to see addressed as well.