Generalized Linear Bandits with Local Differential Privacy 

1 Introduction . Contextual bandit algorithms have received extensive attention for their efficacy for online decision making in many applications such as recommendation system , clinic trials , and online advertisement [ 7 , 35 , 24 ] . Despite their success in many applications , intensive utilization of user-specific information , especially in privacy-sensitive domains such as clinical trials and e-commerce promotions , raises concerns about data privacy protection . Differential privacy , as a provable protection against identification from attackers [ 18 , 19 ] , has been put forth as a competitive candidate for a formal definition of privacy and has received considerable attention from both academic research [ 33 , 17 , 43 , 36 , 8 ] and industry adoption [ 20 , 12 , 37 ] . While increasing attention has been paid to bandit algorithms with joint differential privacy [ 34 , 9 ] , we introduce in this paper a more stringent notion , local differential privacy ( LDP ) , in which users even distrust the server collecting the data , to contextual bandits . * Equal contributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In contextual bandit , at each time round t with individual-specific context Xt , the decision maker can take an action at from a finite set ( arms ) to receive a reward randomly generated from the distribution depending on the context Xt and the chosen arm through its parameter θ ? at which is not unknown to the decision maker . We use the standard notion of expected regret to measure the difference between expected rewards obtained by the action at and the best achievable expected reward in this round . While several papers consider the adversarial setting ( i.e. , Xt can be arbitrary determined in each round ) , this paper considers the stochastic contextual case where Xt is generated i.i.d . from a distribution PX . The goal is to maximize the rewards accumulated over the time horizon . An algorithm achieves LDP guarantee if every user involved in this algorithm is guaranteed that anyone else can only access her context ( and related information such as the arm chosen and the reward ) with limited advantage over a random guess . Recently there is an emerging steam of works combining LDP and bandit . [ 6 , 29 , 10 ] consider the LDP contextual-free bandit and design algorithms to achieve the same regret as in the non-privacy setting . For contextual bandits , [ 44 ] considers the adversarial setting . Despite their pioneering work , their regret bounds O ( T 3/4 ) leave a gap from the corresponding non-privacy results O ( T 1/2 ) , which is conjectured to be inevitable . A natural question arises : can we close this gap for stochastic contextual bandits ? In this paper , we design several algorithms and show that they can achieve the same regret rate in terms of T as in the non-private settings . If we don ’ t assume any structure on the arms ’ parameters , the above formulation is referred to as multi-parameter contextual bandits . If we impose structural assumptions such as all arms share the same parameter ( see Section 2.2 for details ) , then the formulation is referred to as singleparameter contextual bandits . Although multi-parameter and single-parameter settings can be shown to be equivalent , they need independent analysis and design of algorithms because of their distinct properties based on different modeling assumptions ( e.g. , [ 27 ] ) . In this paper , we consider the privacy guarantee in both settings . In fact , multi-parameter setting is more difficult since we need to estimate the parameters for all K arms with sufficient accuracy to make good decisions . However , privacy protection also requires protecting the information about which arm is pulled in each round . Such a requirement hinders the identification of optimal arm and may incur considerable regret in the decision process . A proper balance between privacy protection and estimation accuracy is the key to design algorithms with desired performance guarantee in this setting . Contributions . We organize our results for various settings in Table 1 . Our main contributions can be summarized as follows : 1 . We develop a framework for implementing LDP algorithms by integrating greedy algorithms with a private OLS estimator for linear bandits and a private SGD estimator for generalized linear bandits . We prove that our algorithms achieve regret bound matching the corresponding non-privacy results . 2 . In the multi-parameter setting , to ensure the privacy of the arm pulled in each round , we design a novel LDP strategy by simultaneously updating all the arms with synthetic information instead of releasing the pulled arm . By conducting such synthetic updates for unselected arms , we protect the information of the pulled arm from being identified by the server or other users . This is at the cost of corrupting the estimation of the un-selected arms . To deal with this issue , we design an elimination method that is only based on data collected during a short warm up period . We show that such a mechanism can be combined with the OLS and SGD estimators to achieve the desired performance guarantees . 3 . We introduce the SGD estimator to bandit algorithms to tackle generalized linear reward structure . To the best of our knowledge , few papers have ever considered SGD-based bandit algorithms . Theoretical regret bounds are established in [ 13 ] by combining SGD and Thompson Sampling , while most of the others are limited to empirical studies [ 7 , 32 ] . We establish such theoretical regret bounds for SGD-based bandit algorithms . Our private SGD estimator for bandits is highly computationally efficient , and more importantly , greatly simplifies the data processing mechanism for LDP guarantee . 2 Preliminaries . Notations . We start by fixing some notations that will be used throughout this paper . For a positive integer n , [ n ] denotes the set { 1 , · · · , n } . |A| denotes the cardinality of the set A . ‖·‖2 is Euclidean norm . W ( i , j ) denotes the element in the i-th row and j-th column of matrix W . We write W > 0 if the matrix W is symmetric and positive definite . We denote Id as the d-dimensional identity matrix . Let ⊗ denote the Kronecker product . Let Bdr denote the d-dimensional ball with radius r and Sd−1r denotes the ( d − 1 ) -dimensional sphere for the ball . Given a set A , Unif ( A ) denote the uniform distribution over A . For a tuple ( Zi , j ) i≤N , j≤M and 1 ≤ k1 < k2 ≤ M , we denote Zi , k1 : k2 = ( Zi , k1 , · · · , Zi , k2 ) . We adopt the standard asymptotic notations : for two non-negative sequences { an } and { bn } , { an } = O ( { bn } ) iff lim supn→∞ an/bn < ∞ , an = Ω ( bn ) iff bn = O ( an ) , an = Θ ( bn ) iff an = O ( bn ) and bn = O ( an ) . We also write Õ ( · ) , Ω̃ ( · ) and Θ̃ ( · ) to denote the respective meanings within multiplicative logarithmic factors in n . 2.1 Local Differential Privacy . Definition 2.1 ( Local differential privacy ) . We say a ( randomized ) mechanism M : X → Z is ( ε , δ ) -LDP , if for every x 6= x′ ∈ X and any measurable set C ⊂ Z we have P ( M ( x ) ∈ C ) ≤ eεP ( M ( x′ ) ∈ C ) + δ . When δ = 0 , we simply denote ε-LDP . We now present some tools that will be useful for our analysis . Lemma 2.1 ( Gaussian Mechanism [ 16 , 19 ] ) . For any f : X → Rn , let σε , δ = 1ε supx , x′∈X ‖f ( x ) − f ( x′ ) ‖2 √ 2 ln ( 1.25/δ ) . The Gaussian mechanism , which adds random noise independently drawn from distribution N ( 0 , σ2ε , δIn ) to each output of f , ensures ( ε , δ ) -LDP . Besides the Gaussian mechanism , we also use the following privacy mechanism for bounded vectors . Lemma 2.2 ( Privacy Mechanism for l2-ball [ 14 ] ) . For anyR > 0 , let rε , d = R √ π 2 eε + 1 eε − 1 dΓ ( d+12 ) Γ ( d2 + 1 ) where Γ is the Gamma function . For any x ∈ BdR , consider the mechanism Ψε , R : BdR → Sd−1rε , d of generating Zx as the follows . First , generate a random vector X̃ = ( 2b− 1 ) x where b is a Bernoulli random variable with success probability 12 + ‖x‖2 2R . Next , generate random vector Zx via Zx ∼ { Unif { z ∈ Rd : zT X̃ > 0 , ‖z‖2 = rε , d } with probability eε/ ( 1 + eε ) , Unif { z ∈ Rd : zT X̃ ≤ 0 , ‖z‖2 = rε , d } with probability 1/ ( 1 + eε ) . Then Ψε , R is ε-LDP and E [ Ψε , R ( x ) ] = x. Lemma 2.3 ( Post-Processing property [ 19 ] ) . If M : X → Y is ( ε , δ ) -LDP and f : Y → Z is a fixed map , then f ◦M : X → Z is ( ε , δ ) -LDP . Lemma 2.4 ( Composition property [ 19 ] ) . If M1 : X → Z1 is ( ε1 , δ1 ) -LDP and M2 : X → Z2 is ( ε2 , δ2 ) -LDP , then M = ( M1 , M2 ) : X → Z1 ×Z2 is ( ε1 + ε2 , δ1 + δ2 ) -LDP . 2.2 Local Differential Privacy in Bandit . We consider contextual bandits with LDP guarantee in the context of the user-server communication protocol described in Figure 1 . The user in round t with context Xt ∈ Rd receives ( processed ) historical information St−1 from the server , and chooses an action at ∈ [ K ] to obtain a random reward rt = v ( Xt , at ) + t . Define Ft as the filtration of all historical information up to time t , i.e. , Ft = σ ( X1 , · · · , Xt , 1 , · · · , t−1 ) , and we require t is bounded and E [ t|Ft ] = 0 . Then the user processes the tuple ( Xt , rt ) by some mechanism ψ with LDP guarantee and send the processed information Zt = ψ ( Xt , rt ) to the server . After receiving Zt , the server updates the historical information St to get St+1 . We consider the generalized linear bandits by allowing v ( Xt , at ) = µ ( X T t θ ? at ) , where µ : R→ R is a link function and θ ? i ∈ Rd is the underlying parameter of the i-th arm . For a fix time t , we denote a∗t = arg maxi∈ [ K ] µ ( X T t θ ? i ) . The regret over time horizon T is Reg ( T ) = ∑T t=1 ( µ ( XTt θ ? a∗t ) − µ ( XTt θ ? at ) ) . If we don ’ t assume any structure on { θ ? i } i∈ [ K ] , we refer it as the multi-parameter setting . We also consider d-dimensional single-param setting by assuming θ ? i = ei⊗θ ? for some θ ? ∈ Rd where { ei } i∈ [ K ] is canonical basis of RK . In this case , xt , i ∈ Rd is the i-th segment of Xt ∈ RdK and XTt θ ? i = xTt , iθ ? , so choosing arm i becomes choosing the i-th segment xt , i of the context . In the rest of paper , we always assume that ‖θ ? i ‖2 ≤ 1 , ∀i ∈ [ K ] , the reward is bounded by cr and the Euclidean norm of the context is bounded by CB , our analysis can be easily generalized to the case where t and the context follow sub-gaussian distributions . In fact , when the context and noise are subgaussian , it is guaranteed by the subgaussian concentration that there are at least T −O ( log T ) users have contexts and rewards bounded by C1 , T , C2 , T with overwhelming probability , so we can still use above LDP mechanisms to protect their privacy . For users whose contexts and reward are out of range C1 , T , C2 , T , they can send a private version of ( 0 , 0 ) vector so that there is no privacy leakage . We also impose regularize assumptions on the link function , which are common in previous work [ 44 , 31 , 39 ] and the corresponding family contains a lot of commonly-use model , e.g. , linear model , logistic model . Assumption 1 . The link function µ is continuously differentiable , Lipschitz and there exists some ζ > 0 such that infx∈ [ −CB , CB ] µ ′ ( x ) = ζ > 0 .

Summary: Authors design LDP algorithms for stochastic generalized linear bandits to achieve the same regret bound as in non-privacy settings. The main idea is to develop a stochastic gradient-based estimator and update mechanism to ensure LDP. They also develop an estimator and update mechanism based on Ordinary Least Square (OLS) for linear bandits. Finally, they conduct experiments with both simulation and real-world datasets to demonstrate the consistently superb performance of algorithms to ensure strong privacy protection.