Differentially Private Multi-Armed Bandits in the Shuffle Model

O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k √ log 1δ log T ε ) , and a distribution-independent regret of O ( √ kT log T + k √ log 1δ log T ε ) , where T is the number of rounds , ∆a is the suboptimality gap of the arm a , and k is the total number of arms . Our upper bound almost matches the regret of the best known algorithms for the centralized model , and significantly outperforms the best known algorithm in the local model . 1 Introduction . The multi-armed bandit ( MAB ) problem is a classical sequential decision-making problem in which an agent tries to maximize a cumulative stochastic reward [ 27 , 23 ] under uncertainty . This problem , which is applicable to various areas such as recommender systems , online advertising and clinical trials , embodies the well known exploration-exploitation trade-off between learning the environment and acting optimally based on our current knowledge about the environment . More formally , in the MAB problem at each time t = 1 , . . . , T an agent chooses an arm i from the set [ k ] = { 1 , . . . , k } of k arms , and obtains an iid reward rt drawn from the unknown distribution Ri over { 0 , 1 } with expectation µi = E [ Ri ] . Let a∗ = arg maxa µa be an arm with the largest expected reward , and denote this reward by µ∗ = µa∗ . Let the ( suboptimality ) gap of an arm a to be the gap between its expected reward and that of a∗ , i.e. , ∆a = µ∗ − µa . The agent ’ s goal is to maximize the total expected reward , or rather to minimize the expected regret R ( T ) = T · µ∗ − E [ ∑T i=1 r t ] defined to be the expected gap between the algorithm and the optimal algorithm that knows the distributions Ri . In this work we address the privacy in such a setting . As a motivating example , consider an advertisement system in which the server presents to each user an advertisement a ∈ [ k ] . The user then decides whether to click on the advertisement or not . This click decision depends on different private characteristic of the user . The user then reports to the server whether it clicked on the advertisement ( in which case its reward is r = 1 ) or not ( r = 0 ) . From this example , it is clear that r is private information of the user , and using traditional algorithms for the MAB problem incautiously might leak user-private data . In order to mathematically alleviate privacy concerns , Dwork et al . [ 11 ] defined the notion of differential privacy ( DP ) , which requires that the output of the computation has a limited dependency ∗Google Research . jayten @ google.com . †Blavatnik School of Computer Science , Tel Aviv University and Google Research . haimk @ tau.ac.il . ‡Blavatnik School of Computer Science , Tel Aviv University and Google Research . mansour.yishay @ gmail.com . §Blavatnik School of Computer Science , Tel Aviv University and Google Research . u @ uri.co.il . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . on any single user ’ s data . Formally , a mechanism ( ε , δ ) -DP if for any pair of neighboring inputs ( differing by a single user ’ s data ) , the probability that the mechanism outputs a value in any set B is not different by more than a multiplicative factor of eε and an additive factor of δ . Differential privacy has been extensively studied under many different sub-models of privacy . On one end of the spectrum lies the centralized model of differential privacy , where the users trust the server with their data , and the liability to protect user privacy lies on the server , who must make sure that any data published externally ( e.g. , aggregated statistics ) respects the privacy constraints . On the other end of the spectrum lies the ( strictly stronger ) local model of differentialy privacy ( LDP ) , where the user privatizes its own data prior to sending it to the server.5 Differentially private versions of the MAB problem have been considered in various previous works , where the private information are users ’ rewards ( two neighboring inputs differ by the reward value of a single user ) , and the algorithm ’ s output is the subsequent arm ( s ) it selects . Table 1 summarizes the best known distribution-dependent regret bounds for the various privacy models , together with our new result in the shuffle model which we soon define formally . With real-world algorithms gradually moving away from the centralized model of privacy , the immediate question is “ can we design a private algorithm for MAB with privacy guarantees which are similar to local DP , but with similar regret to centralized DP ( without a multiplicative 1/ε2 factor ) ? ” . To address the inevitable gap between the local and centralized models , which is in fact common in the literature of differential privacy , the alternative shuffle model [ 6 , 9 , 13 ] explores the space in between the local and centralized models by introducing a trusted shuffler that receives user messages and permutes them ( i.e. , disassociates a message from its sender ) before they are delivered to the server . For privacy analysis , we assume that the shuffle is perfectly secure , i.e. , its output contains no information about which user generated each of the messages . This is traditionally achieved by the shuffler stripping implicit metadata from the messages ( e.g. , timestamps , routing information ) , and frequently forwarding this data to remove time and order information . The shuffle model ensures that sufficiently many reports are collected in each round so that any one report can hide in a shuffled batch . In order to apply the shuffle model to the MAB problem in the context of advertisements , we divide the algorithm into batches , where before each batch we decide on the fly its size m , and then present the m next users the same advertisement a , and finally apply a private shuffle model mechanism to their rewards to communicate reward aggregate information to the server.8 A constantly growing body of work presents new and improved mechanisms in the shuffle model for basic statistical tasks [ 16 , 13 , 12 , 2 ] , such as private binary summation , in which the server must privately approximate the sum of a collection of values x1 , ... , xm ∈ { 0 , 1 } held by the m users in 5Formally , in the non-interactive setting , a mechanism is ( ε , δ ) -LDP if for any two user inputs , the probability that the privatizer sends the server a value in any set B is not different by more than a multiplicative factor of eε and an additive factor of δ . 6The corresponding distribution-independent regret bounds usually simply replace the ∑ a∈ [ k ] : ∆a > 0 log T ∆a term with √ kT log T . 7This lower bound can be extended from ( ε , 0 ) -LDP to ( ε , δ ) -LDP using arguments from Bun et al . [ 7 ] and Cheu et al . [ 9 ] since we focus on single-round ( non-interactive ) mechanisms in which the user can only send information to another party once . 8We remark that a given user does not know in advance the size of the batch in which it participates , since this size depends on the algorithm ’ s run . the shuffle . For private binary summation , the optimal achievable errors in the central , local and shuffle model are Θ̃ ( 1/ε ) [ 11 ] , Θ̃ ( √ m/ε ) [ 5 , 8 ] and Θ̃ ( 1/ε ) [ 9 ] respectively , where the Θ̃ ( · ) hides poly-logarithmic terms . Similar errors hold for the private summation problem which approximates the sum of real values in [ 0 , 1 ] . 1.1 Our contributions . To the best of our knowledge , our work is the first to consider the MAB problem under the shuffle model of differential privacy . In order to support the online nature of the MAB problem , we consider a variant of the shuffle model . As opposed to the classical shuffle model , in which the shuffle size is unbounded and the mechanism runs only once , we continuously run shuffle mechanisms for many disjoint batches of users who can only afford a single round of communication.9 We consider the paradigm where the server controls the different users who are cooperative and communicate only with the server . We give a rigorous definition of shuffle differential privacy ( SDP ) for the multi-armed bandit problem ( assuming binary rewards ) , and give and prove the first two such algorithms . Our algorithms Shuffle Differentially Private Arm Elimination ( SDP-AE ) and Variable Batch Shuffle Differentially Private Arm Elimination ( VB-SDP-AE ) are both based on the well-known arm elimination ( AE ) algorithm , using consecutive batches of users and together with an SDP private binary summation mechanism . We show that the simpler but weaker SDP-AE achieves a distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k log 1δ ε2 ) , and a distribution-independent regret of O ( √ kT log T + k log 1δ ε2 ) .10 We then describe VB-SDP-AE , a generalization of SDP-AE to exponentially growing batch sizes , and prove it has a distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k √ log 1δ log T ε ) , and a distribution-independent regret of O ( √ kT log T + k √ log 1δ log T ε ) . Note that , compared to the local model ( Ren et al [ 22 ] ) , the regret of both SDP-AE and VB-SDP-AE is improved , by having the dependency on 1/ε be additive rather than multiplicative . In addition , VB-SDP-AE almost matches the regret of the best known algorithms for the centralized model , that is the distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + kε ) of Tossou and Dimi- trakakis [ 28 ] . 1.2 Related work . The differentially private MAB problem has been considered in many previous works [ 19 , 17 , 21 ] . Shi and Shen [ 26 ] and Dubey and Pentland [ 10 ] studied MAB and linear bandits respectively in the federated setting . Zheng et al . [ 29 ] studied contextual bandits with LDP . Batched MAB with a predetermined number of batches was studied in [ 14 , 15 ] . For the private summation problem , Cheu et al . [ 9 ] gave unbiased ( ε , δ ) -SDP mechanisms over binary inputs and real inputs in [ 0 , 1 ] , with error roughly √ log ( 1/δ ) /ε and log ( 1/δ ) /ε , respectively . In several works of Balle et al . [ 2 , 3 , 1 ] , they gave a biased ( ε , δ ) -SDP mechanism for real inputs in [ 0 , 1 ] with similar error and a constant number of messages , and a single-message ( ε , δ ) -SDP mechanism with optimal error . 9Beimel et al . [ 4 ] showed that every centralized-DP mechanism can be emulated in the shuffle model in two ( communication-intensive ) rounds of communication , however these results are not applicable to our setting since we assume that each online user participates only in one shuffle and then disappears . 10In this paper , we assume that T is known apriori . Otherwise , we can apply standard doubling arguments to get roughly the same results .

Summary: This paper adapts the shuffle model of differential privacy to the multi-armed bandit setting, and aims to design a bandit algorithm which has privacy guarantees similar to that in the local model, while simultaneously having regret guarantees similar to the central DP model. The authors propose two arm-elimination style algorithms, which uses a private binary summation mechanism with optimal error guarantees as a subroutine, in order to obtain improved regret guarantees over the local model. In particular, they show how a simple, constant batch size policy already gets rid of the multiplicative dependence on $1/\epsilon^2$. Then, they show how to adapt this policy to exponentially increasing batch sizes, improving the additive dependence from $\approx k/\epsilon^2$ to $\approx k/\epsilon$.