Federated Linear Contextual Bandits

1 Introduction . Federated learning ( FL ) ( McMahan et al. , 2017 ) is an emerging distributed machine learning ( ML ) paradigm where massive number of clients collaboratively learn a shared prediction model while keeping all the training data on local devices . Compared with standard centralized machine learning , FL has the following characteristics ( Kairouz et al. , 2021 ) : • Heterogeneous local datasets . The local datasets , which are often generated at edge devices , are likely drawn from non-independent and identically distributed ( non-IID ) distributions . • Communication efficiency . The communication cost scales with the number of clients , which is one of the primary bottlenecks of FL . It is critical to minimize the communication cost while maintaining the learning accuracy . • Privacy . FL protects local data privacy by only sharing model updates instead of the raw data . While the main focus of the state-of-the-art FL is on the supervised learning setting , recently , a few researchers begin to extend FL to the multi-armed bandits ( MAB ) framework ( Lai and Robbins , 1985 ; Auer et al. , 2002 ; Bubeck and Cesa-Bianchi , 2012 ; Agrawal and Goyal , 2012 , 2013a ) . In the canonical setting of MAB , a player chooses to play one arm from a set of arms at each time slot . An arm , if played , will offer a reward that is drawn from its distribution which is unknown to the player . With all previous observations , the player needs to decide which arm to pull each time in order to maximize the cumulative reward . MAB thus represents an online learning model that naturally captures the intrinsic exploration-exploitation tradeoff in many sequential decision-making problems . Extending FL to the MAB framework is naturally motivated by a corpus of applications , such as recommender systems , clinical trials , and cognitive radio . In those applications , the sequential decision making involves multiple clients and is distributed by nature . While classical MAB models assume immediate access to the sequentially generated data at the learning agent , under the new 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . realm of FL , local datasets can be stored and analyzed at the clients , thus reducing the communication load and potentially protecting the data privacy . Despite the potential benefits of FL , the sequential decision making and bandit feedback bring new challenges to the design of FL algorithms in the MAB setting . Different from the supervised learning setting where static datasets are collected beforehand , under the MAB setting , data is generated sequentially as decisions are made , actions are taken , and observations are collected . In order to maximize the cumulative reward and minimize the corresponding learning regret , it thus requires sophisticated coordination of the actions of the clients . The heterogeneous reward distributions across clients make the coordination process even more convoluted and challenging . Besides , the data privacy and communication efficiency requirements result in significant challenges for efficient information exchange and aggregation between local clients and the central server . In this work , we attempt to address those challenges in a federated linear contextual bandits framework . This particular problem is motivated by the following exemplary applications . • Personalized content recommendation . For content ( arm ) recommendation in web-services , user engagement ( reward ) depends on the profile of a user ( context ) . The central server may deploy a recommender system on each user ’ local device ( client ) in order to personalize recommendations without knowing the personal profile or behavior of the user . • Personalized online education . In order to maximize students performances ( reward ) in online learning , the education platform ( central server ) needs to personalize teaching methods ( arms ) based on the characteristics of individual students ( context ) . With the online learning software installed at local devices ( client ) , it is desirable to personalize the learning experiences without allowing the platform to access students ’ characteristics or scores . In those examples , the reward of pulling the same arm at different clients follows different distributions dependent on the context as in contextual bandits ( Auer , 2003 ; Langford and Zhang , 2008 ) . We note that conventional contextual bandits is defined with respect to a single player , where the time-varying context can be interpreted as different incoming user profiles . In contrast , we consider a multi-client model , where each client is associated with a fixed user profile . The variation of contexts is captured over clients as opposed to over time . Although the set of clients remains fixed through the learning process , the reward of pulling the same arm still varies across clients . Such a model naturally takes data heterogeneity into consideration . Besides , we adopt a linear reward model , which has been widely studied in contextual bandits ( Li et al. , 2010 ; Agrawal and Goyal , 2013b ) . Main contributions . Our main contributions are summarized as follows . First , we propose a new federated linear contextual bandits model that takes the diverse user preferences and data heterogeneity into consideration . Such a model naturally bridges local stochastic bandits with linear contextual bandits , and is well poised to capture the tradeoffs between communication efficiency and learning performances in the federated bandits setting . Second , we design a novel algorithm named Fed-PE and further develop its variants to solve the federated linear contextual bandits problem . Under Fed-PE , clients only upload their local estimates of the global parameters without sharing their local feature vectors or raw observations . It not only keeps the personal information private , but also reduces the upload cost . We explicitly show that Fed-PE and its variants achieve near-optimal regret performances for both disjoint and shared parameter cases with logarithmic communication costs . Third , we generalize the G-optimal design from the single-player setting ( Lattimore and Szepesvári , 2020 ) to the multi-client setting . We develop a block coordinate ascent algorithm to solve the generalized G-optimal design efficiently with convergence guarantees . Such a multi-client G-optimal design plays a vital role in Fed-PE , and may find broad applications in related multi-agent setups . Finally , we introduce a novel concept called collinearly-dependent policy and show that the celebrated LinUCB type of policies ( Li et al. , 2010 ) , Thompson sampling based policies with Gaussian priors ( Agrawal and Goyal , 2013b ) , and least squared estimation based policies , such as Fed-PE , are all in this category . By utilizing the property of collinearly-dependent policies , we are able to characterize a tight minimax regret lower bound in the disjoint parameter setting . We believe that this concept may be of independent interest for the study of bandits with linear rewards . √ Notations . Throughout this paper , we use ‖x‖V to denote √ xᵀV x . The range of a matrixA , denoted by range ( A ) , is the subspace spanned by the column vectors of A . We use A† and Det ( A ) to denote the pseudo-inverse and pseudo-determinant of square matrix A , respectively . The specific definitions can be found in Appendix B of the supplementary material . 2 Related Works . Collaborative and distributed bandits . Our model is closely related to the collaborative and distributed bandits when action collision is not considered . Landgren et al . ( 2016 , 2018 ) and Martínez-Rubio et al . ( 2019 ) study distributed bandits in which multiple agents face the same MAB instance , and the agents collaboratively share their estimates over a fixed communication graph in order to design consensus-based distributed estimation algorithms to estimate the mean of rewards at each arm . Szorenyi et al . ( 2013 ) considers a similar setup where in each round an agent is able to communicate with a few random peers . Korda et al . ( 2016 ) considers the case where clients in different unknown clusters face independent bandit problems , and every agent can communicate with only one other agent per round . The communication and coordination among the clients in those works are fundamentally different from our work . Wang et al . ( 2020 ) investigates communication-efficient distributed linear bandits , where the agents can communicate with a server by sending and receiving packets . It proposes two algorithms , namely , DELB and DisLinUCB , for fixed and time-varying action sets , respectively . The fixed action set setting is similar to our setup , except that it assumes that all agents face the same bandits model , which does not take data heterogeneity into consideration . Federated bandits . A few recent works have touched upon the concept of federated bandits . With heterogeneous reward distributions at local clients , Shi and Shen ( 2021 ) and Shi et al . ( 2021 ) investigate efficient client-server communication and coordination protocols for federated MAB without and with personalization , respectively . Agarwal et al . ( 2020 ) studies regression-based contextual bandits as an example of the federated residual learning framework , where the reward of a client depends on both a global model and a local model . Li et al . ( 2020 ) and Zhu et al . ( 2021 ) focus on differential privacy based local data privacy protection in federated bandits . While the linear contextual bandit model considered in Dubey and Pentland ( 2020 ) is similar to this work , it focuses on federated differential privacy and proposes a LinUCB-based FedUCB algorithm , which incurs a higher regret compared with our result for the shared parameter case . A regret and communication cost comparison between Fed-PE and other baseline algorithms is provided in Table 1 . 3 Problem Formulation . Clients and local bandits model . We consider a federated linear contextual bandits setting where there are M clients pulling the same set of K items ( arms ) denoted as [ K ] : = { 1 , 2 , . . . , K } . At each time t , each client i ∈ [ M ] pulls an arm ai , t ∈ [ K ] based on locally available information . The incurred reward yi , t is given by yi , t = ri , ai , t + ηi , t , where ηi , t is a random noise , and ri , ai , t is 1Results adapted from Dubey and Pentland ( 2020 ) by letting the privacy budget 1/ go to 0 . 2Results adapted from the single-player linear contextual bandits studied in Dimakopoulou et al . ( 2017 ) by assuming instantaneous information exchange and sequential decision-making at the central server . the unknown expected reward by pulling arm ai , t . We note that without additional assumptions or interaction among the clients , each local model is a standard single-player stochastic MAB , where classic algorithms such as UCB ( Auer and Ortner , 2010 ) and Thompson sampling ( Agrawal and Goyal , 2012 ) are known to achieve order-optimal regret . Linear reward structure with global parameters . In order to capture the inherent correlation between rewards of pulling the same arm by different clients , we assume ri , a has a linear structure , i.e. , ri , a = x ᵀ i , aθa , where xi , a ∈ Rd is the feature vector associated with client i and arm a , and θa ∈ Rd is a fixed but unknown parameter vector for each a ∈ [ K ] . Here we use xᵀ to denote the transpose of vector x . The same arm a may have different reward distributions for different clients , due to potentially varying xi , a across clients . Such a linear model naturally captures the heterogeneous data distributions at the clients , yet admits possible collaborations among clients due to the common parameters { θa } a∈ [ K ] . When θa varies for different arm a , it is called the disjoint parameter case ; when θa is known to be a constant across the arms , it is the shared parameter case . We investigate both cases in Sections 4 and 5 , respectively . Communication model . We assume there exists a central server in the system , and similar to FL , the clients can communicate with the server periodically with zero latency . Specifically , the clients can send “ local model updates ” to the central server , which then aggregates and broadcasts the updated “ global model ” to the clients . ( We will specify these components later . ) Note that just as in FL , communication is one of the major bottlenecks and the algorithm has to be conscious about its usage . Similar to Wang et al . ( 2020 ) , we define the communication cost of an algorithm as the number of scalars ( integers or real numbers ) communicated between server and clients . We also make the assumption that clients and server are fully synchronized ( McMahan et al. , 2017 ) . Data privacy concerns . Similar to Dubey and Pentland ( 2020 ) , our contextual bandit problem involves two sets of information that are desirable to be kept private to client i : the feature vectors { xi , a } a∈ [ K ] and the observed rewards { yi , t } t∈ [ T ] . Different from the differential privacy mechanism adopted in Dubey and Pentland ( 2020 ) , in this work , we aim to communicate estimated global model parameters { θa } a between the clients and the server . This is consistent with the FL framework , where only model updates are communicated instead of the raw data . Assumption 1 We make the following assumptions throughout the paper : 1 ) Bounded parameters : For any i ∈ [ M ] , a ∈ [ K ] , we have ‖θa‖2 ≤ s , 0 < ` ≤ ‖xi , a‖2 ≤ L. 2 ) Independent 1-subgaussian noise : ηi , t is a 1-subgaussian noise parameter sampled indepen- dently at each time for each client with E [ ηi , t ] = 0 , E [ exp ( ληi , t ) ] ≤ exp ( λ 2 2 ) for any λ > 0 . Assumption 1.1 is a standard assumption in the bandit literature , which ensures that the maximum regret at any step is bounded . We emphasize that our work does not make any assumption on the knowledge of suboptimality gaps , nor do we assume the existence of a unique optimal arm at each client . Our objective is to minimize the expected cumulative regret among all clients , defined as : E [ R ( T ) ] = E [ M∑ i=1 T∑ t=1 ( xᵀi , a∗i θa∗i − x ᵀ i , ai , t θai , t ) ] , ( 1 ) where a∗i ∈ [ K ] is an optimal arm for client i : ∀b 6= a∗i , x ᵀ i , a∗i θa∗i − x ᵀ i , bθb ≥ 0 .

Summary: This paper considers a linear contextual bandits problem in a federated setting where different users have different contexts but the rewards are generated through common global parameters. The goal is to minimise the regret for each user while sharing the exploration across users. The paper proposes an algorithm termed Fed-PE with regret bounded as O(\sqrt{dMT}), where d is the dimension of contexts, M is the number of users and T is the time-horizon. A matching lower bound is also presented showing that this regret is tight. The main challenge in this setting is to \emph{balance} the exploration across users as contexts might be very diverse. In order to address this challenge the paper proposes a new G-optimal design which optimises for balanced exploration across users.