Differentially Private Multi-Armed Bandits in the Shuffle Model

O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k √ log 1δ log T ε ) , and a distribution-independent regret of O ( √ kT log T + k √ log 1δ log T ε ) , where T is the number of rounds , ∆a is the suboptimality gap of the arm a , and k is the total number of arms . Our upper bound almost matches the regret of the best known algorithms for the centralized model , and significantly outperforms the best known algorithm in the local model . 1 Introduction . The multi-armed bandit ( MAB ) problem is a classical sequential decision-making problem in which an agent tries to maximize a cumulative stochastic reward [ 27 , 23 ] under uncertainty . This problem , which is applicable to various areas such as recommender systems , online advertising and clinical trials , embodies the well known exploration-exploitation trade-off between learning the environment and acting optimally based on our current knowledge about the environment . More formally , in the MAB problem at each time t = 1 , . . . , T an agent chooses an arm i from the set [ k ] = { 1 , . . . , k } of k arms , and obtains an iid reward rt drawn from the unknown distribution Ri over { 0 , 1 } with expectation µi = E [ Ri ] . Let a∗ = arg maxa µa be an arm with the largest expected reward , and denote this reward by µ∗ = µa∗ . Let the ( suboptimality ) gap of an arm a to be the gap between its expected reward and that of a∗ , i.e. , ∆a = µ∗ − µa . The agent ’ s goal is to maximize the total expected reward , or rather to minimize the expected regret R ( T ) = T · µ∗ − E [ ∑T i=1 r t ] defined to be the expected gap between the algorithm and the optimal algorithm that knows the distributions Ri . In this work we address the privacy in such a setting . As a motivating example , consider an advertisement system in which the server presents to each user an advertisement a ∈ [ k ] . The user then decides whether to click on the advertisement or not . This click decision depends on different private characteristic of the user . The user then reports to the server whether it clicked on the advertisement ( in which case its reward is r = 1 ) or not ( r = 0 ) . From this example , it is clear that r is private information of the user , and using traditional algorithms for the MAB problem incautiously might leak user-private data . In order to mathematically alleviate privacy concerns , Dwork et al . [ 11 ] defined the notion of differential privacy ( DP ) , which requires that the output of the computation has a limited dependency ∗Google Research . jayten @ google.com . †Blavatnik School of Computer Science , Tel Aviv University and Google Research . haimk @ tau.ac.il . ‡Blavatnik School of Computer Science , Tel Aviv University and Google Research . mansour.yishay @ gmail.com . §Blavatnik School of Computer Science , Tel Aviv University and Google Research . u @ uri.co.il . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . on any single user ’ s data . Formally , a mechanism ( ε , δ ) -DP if for any pair of neighboring inputs ( differing by a single user ’ s data ) , the probability that the mechanism outputs a value in any set B is not different by more than a multiplicative factor of eε and an additive factor of δ . Differential privacy has been extensively studied under many different sub-models of privacy . On one end of the spectrum lies the centralized model of differential privacy , where the users trust the server with their data , and the liability to protect user privacy lies on the server , who must make sure that any data published externally ( e.g. , aggregated statistics ) respects the privacy constraints . On the other end of the spectrum lies the ( strictly stronger ) local model of differentialy privacy ( LDP ) , where the user privatizes its own data prior to sending it to the server.5 Differentially private versions of the MAB problem have been considered in various previous works , where the private information are users ’ rewards ( two neighboring inputs differ by the reward value of a single user ) , and the algorithm ’ s output is the subsequent arm ( s ) it selects . Table 1 summarizes the best known distribution-dependent regret bounds for the various privacy models , together with our new result in the shuffle model which we soon define formally . With real-world algorithms gradually moving away from the centralized model of privacy , the immediate question is “ can we design a private algorithm for MAB with privacy guarantees which are similar to local DP , but with similar regret to centralized DP ( without a multiplicative 1/ε2 factor ) ? ” . To address the inevitable gap between the local and centralized models , which is in fact common in the literature of differential privacy , the alternative shuffle model [ 6 , 9 , 13 ] explores the space in between the local and centralized models by introducing a trusted shuffler that receives user messages and permutes them ( i.e. , disassociates a message from its sender ) before they are delivered to the server . For privacy analysis , we assume that the shuffle is perfectly secure , i.e. , its output contains no information about which user generated each of the messages . This is traditionally achieved by the shuffler stripping implicit metadata from the messages ( e.g. , timestamps , routing information ) , and frequently forwarding this data to remove time and order information . The shuffle model ensures that sufficiently many reports are collected in each round so that any one report can hide in a shuffled batch . In order to apply the shuffle model to the MAB problem in the context of advertisements , we divide the algorithm into batches , where before each batch we decide on the fly its size m , and then present the m next users the same advertisement a , and finally apply a private shuffle model mechanism to their rewards to communicate reward aggregate information to the server.8 A constantly growing body of work presents new and improved mechanisms in the shuffle model for basic statistical tasks [ 16 , 13 , 12 , 2 ] , such as private binary summation , in which the server must privately approximate the sum of a collection of values x1 , ... , xm ∈ { 0 , 1 } held by the m users in 5Formally , in the non-interactive setting , a mechanism is ( ε , δ ) -LDP if for any two user inputs , the probability that the privatizer sends the server a value in any set B is not different by more than a multiplicative factor of eε and an additive factor of δ . 6The corresponding distribution-independent regret bounds usually simply replace the ∑ a∈ [ k ] : ∆a > 0 log T ∆a term with √ kT log T . 7This lower bound can be extended from ( ε , 0 ) -LDP to ( ε , δ ) -LDP using arguments from Bun et al . [ 7 ] and Cheu et al . [ 9 ] since we focus on single-round ( non-interactive ) mechanisms in which the user can only send information to another party once . 8We remark that a given user does not know in advance the size of the batch in which it participates , since this size depends on the algorithm ’ s run . the shuffle . For private binary summation , the optimal achievable errors in the central , local and shuffle model are Θ̃ ( 1/ε ) [ 11 ] , Θ̃ ( √ m/ε ) [ 5 , 8 ] and Θ̃ ( 1/ε ) [ 9 ] respectively , where the Θ̃ ( · ) hides poly-logarithmic terms . Similar errors hold for the private summation problem which approximates the sum of real values in [ 0 , 1 ] . 1.1 Our contributions . To the best of our knowledge , our work is the first to consider the MAB problem under the shuffle model of differential privacy . In order to support the online nature of the MAB problem , we consider a variant of the shuffle model . As opposed to the classical shuffle model , in which the shuffle size is unbounded and the mechanism runs only once , we continuously run shuffle mechanisms for many disjoint batches of users who can only afford a single round of communication.9 We consider the paradigm where the server controls the different users who are cooperative and communicate only with the server . We give a rigorous definition of shuffle differential privacy ( SDP ) for the multi-armed bandit problem ( assuming binary rewards ) , and give and prove the first two such algorithms . Our algorithms Shuffle Differentially Private Arm Elimination ( SDP-AE ) and Variable Batch Shuffle Differentially Private Arm Elimination ( VB-SDP-AE ) are both based on the well-known arm elimination ( AE ) algorithm , using consecutive batches of users and together with an SDP private binary summation mechanism . We show that the simpler but weaker SDP-AE achieves a distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k log 1δ ε2 ) , and a distribution-independent regret of O ( √ kT log T + k log 1δ ε2 ) .10 We then describe VB-SDP-AE , a generalization of SDP-AE to exponentially growing batch sizes , and prove it has a distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + k √ log 1δ log T ε ) , and a distribution-independent regret of O ( √ kT log T + k √ log 1δ log T ε ) . Note that , compared to the local model ( Ren et al [ 22 ] ) , the regret of both SDP-AE and VB-SDP-AE is improved , by having the dependency on 1/ε be additive rather than multiplicative . In addition , VB-SDP-AE almost matches the regret of the best known algorithms for the centralized model , that is the distribution-dependent regret of O ( ( ∑ a∈ [ k ] : ∆a > 0 log T ∆a ) + kε ) of Tossou and Dimi- trakakis [ 28 ] . 1.2 Related work . The differentially private MAB problem has been considered in many previous works [ 19 , 17 , 21 ] . Shi and Shen [ 26 ] and Dubey and Pentland [ 10 ] studied MAB and linear bandits respectively in the federated setting . Zheng et al . [ 29 ] studied contextual bandits with LDP . Batched MAB with a predetermined number of batches was studied in [ 14 , 15 ] . For the private summation problem , Cheu et al . [ 9 ] gave unbiased ( ε , δ ) -SDP mechanisms over binary inputs and real inputs in [ 0 , 1 ] , with error roughly √ log ( 1/δ ) /ε and log ( 1/δ ) /ε , respectively . In several works of Balle et al . [ 2 , 3 , 1 ] , they gave a biased ( ε , δ ) -SDP mechanism for real inputs in [ 0 , 1 ] with similar error and a constant number of messages , and a single-message ( ε , δ ) -SDP mechanism with optimal error . 9Beimel et al . [ 4 ] showed that every centralized-DP mechanism can be emulated in the shuffle model in two ( communication-intensive ) rounds of communication , however these results are not applicable to our setting since we assume that each online user participates only in one shuffle and then disappears . 10In this paper , we assume that T is known apriori . Otherwise , we can apply standard doubling arguments to get roughly the same results .

Summary: This paper studies the multi-armed bandit problem under the constraint of approximate differential privacy. Previously, multi-armed bandits have been studied in other models of privacy such as the centralized pure DP model, the centralized approximate DP model, and the local pure DP model. However, this is the first work to consider the multi-armed bandit problem under the shuffle model of privacy.  To be clear, the model that the authors consider is as follows. At each round $t$, a batch of $m$ random users are selected. An action (e.g. an ad) is then selected for each user. The reward for each user is then presented to a shuffle DP mechanism.  In terms of results, the authors prove a regret bound which is slightly weaker than the result one can obtain in the central DP model but significantly stronger than the result one can obtain the local DP model. In particular, it removes a multiplicative $1/\epsilon^2$ dependence that appears in the regret bound for the local DP model.  In terms of techniques, the authors combine some ideas from the bandit literature and the differential privacy literature. In particular, their algorithms are based on the non-private arm elimination algorithm and they show how one can use private binary summation from the shuffle DP literature to obtain a shuffle DP algorithm for MAB. The authors also give two differential algorithms. The first algorithm uses static batch sizes in each round. The second algorithm uses dynamic batch sizes and they show that this improves the regret.