Generalized Linear Bandits with Local Differential Privacy 

1 Introduction . Contextual bandit algorithms have received extensive attention for their efficacy for online decision making in many applications such as recommendation system , clinic trials , and online advertisement [ 7 , 35 , 24 ] . Despite their success in many applications , intensive utilization of user-specific information , especially in privacy-sensitive domains such as clinical trials and e-commerce promotions , raises concerns about data privacy protection . Differential privacy , as a provable protection against identification from attackers [ 18 , 19 ] , has been put forth as a competitive candidate for a formal definition of privacy and has received considerable attention from both academic research [ 33 , 17 , 43 , 36 , 8 ] and industry adoption [ 20 , 12 , 37 ] . While increasing attention has been paid to bandit algorithms with joint differential privacy [ 34 , 9 ] , we introduce in this paper a more stringent notion , local differential privacy ( LDP ) , in which users even distrust the server collecting the data , to contextual bandits . * Equal contributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In contextual bandit , at each time round t with individual-specific context Xt , the decision maker can take an action at from a finite set ( arms ) to receive a reward randomly generated from the distribution depending on the context Xt and the chosen arm through its parameter θ ? at which is not unknown to the decision maker . We use the standard notion of expected regret to measure the difference between expected rewards obtained by the action at and the best achievable expected reward in this round . While several papers consider the adversarial setting ( i.e. , Xt can be arbitrary determined in each round ) , this paper considers the stochastic contextual case where Xt is generated i.i.d . from a distribution PX . The goal is to maximize the rewards accumulated over the time horizon . An algorithm achieves LDP guarantee if every user involved in this algorithm is guaranteed that anyone else can only access her context ( and related information such as the arm chosen and the reward ) with limited advantage over a random guess . Recently there is an emerging steam of works combining LDP and bandit . [ 6 , 29 , 10 ] consider the LDP contextual-free bandit and design algorithms to achieve the same regret as in the non-privacy setting . For contextual bandits , [ 44 ] considers the adversarial setting . Despite their pioneering work , their regret bounds O ( T 3/4 ) leave a gap from the corresponding non-privacy results O ( T 1/2 ) , which is conjectured to be inevitable . A natural question arises : can we close this gap for stochastic contextual bandits ? In this paper , we design several algorithms and show that they can achieve the same regret rate in terms of T as in the non-private settings . If we don ’ t assume any structure on the arms ’ parameters , the above formulation is referred to as multi-parameter contextual bandits . If we impose structural assumptions such as all arms share the same parameter ( see Section 2.2 for details ) , then the formulation is referred to as singleparameter contextual bandits . Although multi-parameter and single-parameter settings can be shown to be equivalent , they need independent analysis and design of algorithms because of their distinct properties based on different modeling assumptions ( e.g. , [ 27 ] ) . In this paper , we consider the privacy guarantee in both settings . In fact , multi-parameter setting is more difficult since we need to estimate the parameters for all K arms with sufficient accuracy to make good decisions . However , privacy protection also requires protecting the information about which arm is pulled in each round . Such a requirement hinders the identification of optimal arm and may incur considerable regret in the decision process . A proper balance between privacy protection and estimation accuracy is the key to design algorithms with desired performance guarantee in this setting . Contributions . We organize our results for various settings in Table 1 . Our main contributions can be summarized as follows : 1 . We develop a framework for implementing LDP algorithms by integrating greedy algorithms with a private OLS estimator for linear bandits and a private SGD estimator for generalized linear bandits . We prove that our algorithms achieve regret bound matching the corresponding non-privacy results . 2 . In the multi-parameter setting , to ensure the privacy of the arm pulled in each round , we design a novel LDP strategy by simultaneously updating all the arms with synthetic information instead of releasing the pulled arm . By conducting such synthetic updates for unselected arms , we protect the information of the pulled arm from being identified by the server or other users . This is at the cost of corrupting the estimation of the un-selected arms . To deal with this issue , we design an elimination method that is only based on data collected during a short warm up period . We show that such a mechanism can be combined with the OLS and SGD estimators to achieve the desired performance guarantees . 3 . We introduce the SGD estimator to bandit algorithms to tackle generalized linear reward structure . To the best of our knowledge , few papers have ever considered SGD-based bandit algorithms . Theoretical regret bounds are established in [ 13 ] by combining SGD and Thompson Sampling , while most of the others are limited to empirical studies [ 7 , 32 ] . We establish such theoretical regret bounds for SGD-based bandit algorithms . Our private SGD estimator for bandits is highly computationally efficient , and more importantly , greatly simplifies the data processing mechanism for LDP guarantee . 2 Preliminaries . Notations . We start by fixing some notations that will be used throughout this paper . For a positive integer n , [ n ] denotes the set { 1 , · · · , n } . |A| denotes the cardinality of the set A . ‖·‖2 is Euclidean norm . W ( i , j ) denotes the element in the i-th row and j-th column of matrix W . We write W > 0 if the matrix W is symmetric and positive definite . We denote Id as the d-dimensional identity matrix . Let ⊗ denote the Kronecker product . Let Bdr denote the d-dimensional ball with radius r and Sd−1r denotes the ( d − 1 ) -dimensional sphere for the ball . Given a set A , Unif ( A ) denote the uniform distribution over A . For a tuple ( Zi , j ) i≤N , j≤M and 1 ≤ k1 < k2 ≤ M , we denote Zi , k1 : k2 = ( Zi , k1 , · · · , Zi , k2 ) . We adopt the standard asymptotic notations : for two non-negative sequences { an } and { bn } , { an } = O ( { bn } ) iff lim supn→∞ an/bn < ∞ , an = Ω ( bn ) iff bn = O ( an ) , an = Θ ( bn ) iff an = O ( bn ) and bn = O ( an ) . We also write Õ ( · ) , Ω̃ ( · ) and Θ̃ ( · ) to denote the respective meanings within multiplicative logarithmic factors in n . 2.1 Local Differential Privacy . Definition 2.1 ( Local differential privacy ) . We say a ( randomized ) mechanism M : X → Z is ( ε , δ ) -LDP , if for every x 6= x′ ∈ X and any measurable set C ⊂ Z we have P ( M ( x ) ∈ C ) ≤ eεP ( M ( x′ ) ∈ C ) + δ . When δ = 0 , we simply denote ε-LDP . We now present some tools that will be useful for our analysis . Lemma 2.1 ( Gaussian Mechanism [ 16 , 19 ] ) . For any f : X → Rn , let σε , δ = 1ε supx , x′∈X ‖f ( x ) − f ( x′ ) ‖2 √ 2 ln ( 1.25/δ ) . The Gaussian mechanism , which adds random noise independently drawn from distribution N ( 0 , σ2ε , δIn ) to each output of f , ensures ( ε , δ ) -LDP . Besides the Gaussian mechanism , we also use the following privacy mechanism for bounded vectors . Lemma 2.2 ( Privacy Mechanism for l2-ball [ 14 ] ) . For anyR > 0 , let rε , d = R √ π 2 eε + 1 eε − 1 dΓ ( d+12 ) Γ ( d2 + 1 ) where Γ is the Gamma function . For any x ∈ BdR , consider the mechanism Ψε , R : BdR → Sd−1rε , d of generating Zx as the follows . First , generate a random vector X̃ = ( 2b− 1 ) x where b is a Bernoulli random variable with success probability 12 + ‖x‖2 2R . Next , generate random vector Zx via Zx ∼ { Unif { z ∈ Rd : zT X̃ > 0 , ‖z‖2 = rε , d } with probability eε/ ( 1 + eε ) , Unif { z ∈ Rd : zT X̃ ≤ 0 , ‖z‖2 = rε , d } with probability 1/ ( 1 + eε ) . Then Ψε , R is ε-LDP and E [ Ψε , R ( x ) ] = x. Lemma 2.3 ( Post-Processing property [ 19 ] ) . If M : X → Y is ( ε , δ ) -LDP and f : Y → Z is a fixed map , then f ◦M : X → Z is ( ε , δ ) -LDP . Lemma 2.4 ( Composition property [ 19 ] ) . If M1 : X → Z1 is ( ε1 , δ1 ) -LDP and M2 : X → Z2 is ( ε2 , δ2 ) -LDP , then M = ( M1 , M2 ) : X → Z1 ×Z2 is ( ε1 + ε2 , δ1 + δ2 ) -LDP . 2.2 Local Differential Privacy in Bandit . We consider contextual bandits with LDP guarantee in the context of the user-server communication protocol described in Figure 1 . The user in round t with context Xt ∈ Rd receives ( processed ) historical information St−1 from the server , and chooses an action at ∈ [ K ] to obtain a random reward rt = v ( Xt , at ) + t . Define Ft as the filtration of all historical information up to time t , i.e. , Ft = σ ( X1 , · · · , Xt , 1 , · · · , t−1 ) , and we require t is bounded and E [ t|Ft ] = 0 . Then the user processes the tuple ( Xt , rt ) by some mechanism ψ with LDP guarantee and send the processed information Zt = ψ ( Xt , rt ) to the server . After receiving Zt , the server updates the historical information St to get St+1 . We consider the generalized linear bandits by allowing v ( Xt , at ) = µ ( X T t θ ? at ) , where µ : R→ R is a link function and θ ? i ∈ Rd is the underlying parameter of the i-th arm . For a fix time t , we denote a∗t = arg maxi∈ [ K ] µ ( X T t θ ? i ) . The regret over time horizon T is Reg ( T ) = ∑T t=1 ( µ ( XTt θ ? a∗t ) − µ ( XTt θ ? at ) ) . If we don ’ t assume any structure on { θ ? i } i∈ [ K ] , we refer it as the multi-parameter setting . We also consider d-dimensional single-param setting by assuming θ ? i = ei⊗θ ? for some θ ? ∈ Rd where { ei } i∈ [ K ] is canonical basis of RK . In this case , xt , i ∈ Rd is the i-th segment of Xt ∈ RdK and XTt θ ? i = xTt , iθ ? , so choosing arm i becomes choosing the i-th segment xt , i of the context . In the rest of paper , we always assume that ‖θ ? i ‖2 ≤ 1 , ∀i ∈ [ K ] , the reward is bounded by cr and the Euclidean norm of the context is bounded by CB , our analysis can be easily generalized to the case where t and the context follow sub-gaussian distributions . In fact , when the context and noise are subgaussian , it is guaranteed by the subgaussian concentration that there are at least T −O ( log T ) users have contexts and rewards bounded by C1 , T , C2 , T with overwhelming probability , so we can still use above LDP mechanisms to protect their privacy . For users whose contexts and reward are out of range C1 , T , C2 , T , they can send a private version of ( 0 , 0 ) vector so that there is no privacy leakage . We also impose regularize assumptions on the link function , which are common in previous work [ 44 , 31 , 39 ] and the corresponding family contains a lot of commonly-use model , e.g. , linear model , logistic model . Assumption 1 . The link function µ is continuously differentiable , Lipschitz and there exists some ζ > 0 such that infx∈ [ −CB , CB ] µ ′ ( x ) = ζ > 0 .

Summary: The paper studies (generalized) linear contextual bandits in the context of local differential privacy (DP). At every time step a single user’s sensitive data point arrives, which serves as a context, and a reward is sampled as a function of this context and an unknown (generalized) linear reward-generating function. The difficulty with minimizing regret with local DP is that one has to learn the reward-generating function without compromising users’ privacy. The analysis is split by cases: in the setting of linear rewards, an OLS-based solution is proposed, and in the setting of generalized linear rewards, an SGD-based solution is proposed. In addition, two different bandit models are considered depending on whether the rewards from different arms are related.