Differentially Private Multi-Armed Bandits in the Shuffle Model
Authors: Jay Tenenbaum
Haim Kaplan
Yishay Mansour
Uri Stemmer
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-armed bandits
differential privacy
shuffle model

Abstract: We give an $(\varepsilon,\delta)$-differentially private algorithm for the Multi-Armed Bandit (MAB) problem in the shuffle model with a distribution-dependent regret of $O\left(\left(\sum_{a:\Delta_a>0}\frac{\log T}{\Delta_a}\right)+\frac{k\sqrt{\log\frac{1}{\delta}}\log T}{\varepsilon}\right)$, and a distribution-independent regret of $O\left(\sqrt{kT\log T}+\frac{k\sqrt{\log\frac{1}{\delta}}\log T}{\varepsilon}\right)$, where $T$ is the number of rounds, $\Delta_a$ is the suboptimality gap of the action $a$, and $k$ is the total number of actions. Our upper bound almost matches the regret of the best known algorithms for the centralized model, and significantly outperforms the best known algorithm in the local model.

Differentially Private Multi-Armed Bandits in the Shuffle Model
Jay Tenenbaum"
Haim Kaplan'
Yishay Mansour
Uri Stemmer
Abstract
We give deifferifreylttialll private algorithm for the multi-armed bandit (MAB) problem the shuffle model with a distribution-deeeedden regret of FoR Tog and a distribution -nneeeendnnndeeeeeeee regret
TogT) A.
EaE(H]:A.>0
of O VETlogT Av/og log
where T is the number of rounds, Aa is the
suboptimality gap of the arm a, and l is the total number of arms. Our upper bound almost matches the regret of the best known algorithms for the centralized model, and significantly outperforms the best known algorithm in the local model.
Introduction
The multi-ana aarii--reddddd bandit (MAB) problem is classical sequential decision-making problem in which an agent tries to maximize a cumulative stochastic reward [27 23 under uncertainty. This problem, which is applicable various areas such recommender systems, online advertising and clinical trials, embodies the well known exploration-exploiiation trade-off between learning the environment and acting optimally based our current knowledge about the environment.
More formally, in the MAB problem at each time / = I,..., T an agent chooses an arm i from the set [A] .1..... }} of & arms, and obtains iid reward rf drawn from the unknown distribution R, over (0. 1} with expectation μ4, = E [R.]. Let a arg max,, Hl be an arm with the largest expected reward, and denote this reward by between its expected reward and that a", i.c., A. = μ Ha. The agent's goal is to maximize
Ma Let the (suboptimality) gap of an arm a to be the gap
the total expected reward, or rather minimize the expected regret R(T) T p² E EL defined to be the expected gap between the algorithm and the optimal algorithm that knows the distributions R.
In this work we address the privacy in such a setting. As a motivating example, consider an advertisement system in which the server presents each user an advertisement a [K)] The user then decides whether to click on the advertisement or not. This click decision depends on different private characteristic of the user. The user then reports the server whether it clicked on the advertisement (in which case its reward is r = 1) or not (r = 0). From this example, it is clear that private information of the user, and using traditional algorithms for the MAB problem incautiously might leak user user-private data.
In order mathematically alleviate privacy concerns, Dwork al. [1I] defined the notion of differential privacy (DP), which requires that the output of the computation has a limited dependency
Google Research, jayten google.com
"Blavatnik School of Computer Science, Tel Aviv University and Google Research, haimk @tau.ac.il
'Blavatnik School of Computer Science, Aviv University and Google Research.
man-
sour.yishay @gmmil..com
Blavatnik School Computer Science, Tel Aviv University and Google Research. @@uuri.o.il
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Best-known MAB regret upper and lower bounds for various DP models.
Privacy model
Best-known regret upper and lower bound
Centralized (e, O)-DP
20 EaE(K)
log T Da
+
log T
24
25
Centralized (-D
O
Tog
+
28
Au
Local (s, 0)-DP
θ
EaE[k]-A.>0
log T
122
A.
Shuffle (c, 6)DP (ours)
o
[ (EnE(k-AA20
log T Aa
+
Rv log log
on any single user's data. Formally, a mechanism (e, 8DP if for any pair of neighboring inputs (differing by single user's data), the probability that the mechanism outputs a value in any set B is not different by more than a multiplicative factor of e and an additive factor of 8. Differential privacy has been extensively studied under many different sub-models of privacy. On one end of the spectrum lies the centralized model of differential privacy, where the users trust the server with their data, and the liability protect user privacy lies the server, who must make sure that any data published externaly (e.g.. aggregated statistics) respects the privacy constraints. On the other end of the spectrum lies the (strictly stronger) local model of differentialy privacy (LDP), where the user privatizes its own data prior to sending it to the server
Differentially private versions of the MAB problem have been considered in various previous works, where the private information are users' rewards (two neighboring inputs differ by the reward value of single user), and the algorithm's 's output is the subsequent arm(s) selects. Table I summarizes the best known ietributtio-dependdee new result in the shuffle model which we soon define formally.
regret bounds for the various privacy models, together with
With real- world algorithms gradually moving away from the centralized model of privacy, the immediate question is "can we design private algorithm for MAB with privacy guarantees which are similar to local DP, but with similar regret to centralized DP (without a multiplicative 1/e² factor)?""
To address the inevitable gap between the local and centralized models, which is in fact common in the literature of differential privacy, the alternative shuffle model 66 [3] explores the space in between the local and centralized models by introducing a trusted shuffler that receives user messages and permutes them (i.e., disassociates a message from its sender) before they are delivered to the server. For privacy analysis, we assume that the shuffle is perfectly secure, i.e., its output contains no information about which user generated each of the messages. This is traditionally achieved by the shuffler stripping implicit metadata from the messages (e.g., timestamps, routing information), and frequently forwarding this data to remove time and order information. The shuffle model ensures that sufficiently many reports are collected in each round so that any one report can hide in shuffled batch. In order to apply the shuffle model to the MAB problem in the context of advertisements, we divide the algorithm into batches, where before each batch we decide on the fly its size m, and then present the m next users the same advertisement a, and finally apply a private shuffle model mechanism to their rewards to communicate reward aggregate information to the server {
constantly growing body of work presents new and improved mechanisms in the shuffle model for basic statistical tasks [16 13 12 2, such as private binary summation, in which the server must
privately approximate the sum of collection of values Z .... I'm E {0, 1} held by the m users in
FFormally, in the non- non-interactive setting, mechanism is (e. 8)-LDP if for any two user inputs, the probability that the privatizer sends the server a value in any set B is not different by more than multiplicative factor of and an additive factor of ö.
Thhe corresponding distribution -nneeeeennn-deeededent regret bounds usually simply replace the EmE14)A, 20 A.
T
term with / log
"This lower bound can be extended from (s, 0)-LDP to (e, 8)-LDP using arguments from Bun et al. 7 and Cheu et al. [9] since focus single-round hnoe-interacttve) mechanisms in which the user can only send information to another party once.
We remark that a given user does not know in advance the size of the batch in which it participates, since this size depends on the algorithm's 's run.

---

the shuffle. For private binary summation, the optimal achievable errors in the central, local and shuffle model are (1/=)) [.]. (m/m/)) [ and (11ss) [9] respectively, where the ((-) hides poly-logarithmic terms. Similar errors hold for the private summation problem which approximates the sum of real values in [0, 1].
Our contributions
To the best of our knowledge, our work is the first to consider the MAB problem under the shuffle model differential privacy. order to support the online nature of the MAB problem, we consider variant of the shuffle model. As opposed to the classical shuffle model, in which the shuffle size is unbounded and the mechanism runs only once, we continuously run shuffle mechanisms for many disjoint batches of users who can only afford single round of communication
We consider the paradigm where the server controls the different users who are cooperative and communicate only with the server. We give rigorous definition of shuflle differential privacy (SDP) for the multi-armed bandit problem (assuming binary rewards), and give and prove the first two such algorithms. Our algorithms Shuffle Differentially Private Arm Elimination (SDP-AE) and Variable Batch Shuffle Differentially Private Arm Elimination VV-SDD-AE) are both based on the well-known arm elimination (AE) algorithm, using consecutive batches of users and together with an SDP private binary summation mechanism.
We show that the simpler but weaker SDP-AE achieves a distribution-dependenn re- and
gret of ((EnE)NAAsS0 IogT A.
+
& log
&
distribution-indeeendet regret of
oVETlog &log
10
We then describe VB-SDP-AF, a generalization of SDP-AE to exponentially growing batch sizes,
and prove it has distribution-dee seerbuttion-deeeddentt regret of O (EoE(R)A,
lox A.
+ k log log and
a disttiibttion-indeeeeddd regret of O (VETT log + Avlog log
Note that, compared to the local model (Ren al [22) the regret of both SDP-AE and VB-SDP-AE improved, by having the dependency on 1/s be additive rather than multiplicative. In addition, VB TB-SDP-AE almost matches the regret of the best known algorithms for the centralized model,
that is the distributioon-deeendent regret of O ((Zne))A.>o
log 3.
of Tossou and Dimi- #
trakakis 28
1.2 Related work
The differentially private MAB problem has been considered in many previous works [9
Shi and Shen [26] and Dubey and Pentland [10] studied MAB and linear bandits respectively in federated setting. Zheng et al. (29 studied contextual bandits with LDP. Batched MAB with predetermined number of batches was studied in [14 15].
For the private summation problem, Cheu al. 9 gave unbiased (e, 8DDP mechanisms over binary inputs and real inputs in [0. 1). with error roughly /log(1/8) (1/8)/e and log(1/8) /8) respectively. In several works of Balle et al. 2 I. they gave biased (s, 8D-SDP mechanism for real inputs in [o, 1] with similar error and a constant number of messages, and a Single-messp (e, 8)DDP mechanism with optimal error.
Beimel et al. 4 showed that every centralized DP mechanism can be emulated in the shuffle model in two applicable to setting
ceammunncation-inteiiiee rounds of communication, however these results since we assume that each online user participates only in one shuffle then disappears. I0In this paper, we assume that 7' is known apriori. Otherwise, we can apply standard doubling arguments to get roughly the same results.

---

2 Background and preliminaries
2.1 Shuffle-model privacy
In the well-studied setting of shuffle model privacy, there are users, each with data E X. Each user applies some encoder E : X - Y* to their data and sends the messages (fi,1, Vi.p) (x;)) to a shuffler S Y + Y The shuffler then shuffles all the messages Vi., from all the users, and outputs them in uniformly random order to an analyzer function [(I,...., Fm). Thus, the mechanism M consists of the tuple (E, S, A). We say that such a mechanism M is (e, d.ssfflee differentially private (or (e, 8DPP for short) if the output of (e, 5) -iiifferentially private, or more formally: A mechanism M = (E,S, S, A) is (e, 8)-SDP if for any pair of inputs {x, [x,)==1 and {=/]-= which differ in at most one value, we have for all B C Y":
Y + to estimate some
P E B) S ef P( E +
In the mechanism used in this paper, E outputs the user's reward bit together with set of random bits, and sums all these bits and debiases the result to get an unbiased estimate of the sum of the users' rewards.
2.2 Shuffle-model MAB
Algorithms which are private in the shuffle model typically apply the mechanism once over a set of m users. Here study the MAB problem which is online problem, often deployed in real-world applications and with users which are end-devices such as cellphones with a possibly limited or unreliable internet connection. Hence, to adapt the MAB problem to the shuffle model, we batch sequences of consecutive users, and assume that each user can afford a single round of communication, and is never selected more than once.
Model and objective the T'th player pulls its arm:
The shuffle-mode MAB setting involves repeating the following process until
1. The server selects batch size m, batch of m random fresh new users, and m-user single-round SDP mechanism M II It then pieks an arm a E [M]] that all m users of the batch pull. (For concreteness think that the server picks an ad [k] and sends it to the random batch of m users.)
2. Each user determines binary reward from pulling the arm a (For concreteness think that each user decides whether to click on the ad or not. This defines the reward related to user i, which r; 1 if clicks the ad and r; 0 otherwise.)
Since our m users are random, these rewards are a sample of m independent rewards from the distribution R., associated with arm a.
3. The server computes M ((r, ((r:))eee iebateh)) using the rewards Ti.
The objective is to minimize the (pseudo) regret, which is the expected difference between the sum of the rewards accumulated (over all the users) by the algorithm and the sum of the rewards of the optimal algorithm that apriori knows an arm with the largest expected reward a". Let /4a E [R,] be the expected reward (or simply mean) of the arm a, let μ denote the expected reward of a* let An μr" - Naa be the (suboptimality) gap of the arm a which quantifies the gap between its expected reward and that of a", and let Na be the random variable which counts the total number of times the arm a was pulled during the run of the algorithm.
Formally, the (pseudo) regret of the algorithm for T E N users is defined to be
R(T) = 1
E
Σ r4
E Σ N.A.
DE[K]
Nott that always selecting m reduces this setting the Local model MAB. For simplicity, we assume that the rewards are binary. However, our algorithms and proof's naturally extend to the real jo, 1] reward setting, by replacing our private binary summation mechanism (defined later, see Appendix C) with private summation mechanism for real numbers in [0, 1] with similar guarantees.

---

Privacy Since the private data of each user is its reward, the appropriate adaptation of shuffle model privacy for the multi-an -amedd bandit problem is as follows. An algorithm for the multi-armed bandits problem is (c, 5) ))-shuffle differentially private (SDP) if for any batch of users, the shuffle mechanism that we apply over them in step 3 is (c, 5)-DPP with respect to the rewards of the users (as we recall, the reward of a user whether it clicked on an ad or not depends on its private features). Formally, for every batch we run a shuffle mechanism where the m users are the users of the current batch, and the data 2; of each user is its reward ri, and we require that each such mechanism is (s, d-SDD.
2.3 Concentration bounds
We use the following standard definitions of Sub-Gaussian random variables and Hoeffding's inequal- ity.
Definition 1 SSub-Gaussian random variable). A random variable X with mean p is called sub- Gaussian with variance o², i.e., X ~ -SG(o2) if:
AA M R, E [exp(A(X - -1))] S 2200212.
An equivalent definition shows that if 11 0, max -
- PX- S
exp 20 then X sub-Gaussian with variance a² (up to constant factor). is well known that X; ~ (G are independent random variables for = 1...., n, then Σi= X, ~ SG(E== of) and for any a, b 0. a X1 + b SG(a² of). bounded random variable X E [a is SG - a) a)2/4))
Bub-Gaussian random variables satisfy the following concentration bound, Lemma 2.1 (Hoeffding's inequality (18).. Let {X,) be independent SG(o²) random variables,
then P E X. E E> x.] 2 1
< 2 exp
2nd
3 Differentially private MAB in the shuffle model
In this paper, we use the shuffle model to give a private solution to the multi-armeet bandit problem, attaining similar privacy guarantees to that of the local privacy model (LDP), without sacrificing the regret. That is, our algorithm almost matches the best known regret in the centralized model of differential privacy. Our algorithms rely on the fact that algorithms for the multi-armed bandit problem take decisions based on sums of rewards received from the users. Hence, we rely on particularly efficient and accurate mechanism for private binary summation in the shuffle model as building block in our algorithms. Specifically, for any s, 8 E (0, 1). let M be a private binary summation mechanism, which for any number of users (batch size) m E N. is (s, 0)SD., unbiased, and has error distribution which is independent of the input, and is
with variance 13 Note that our notation of Maum does not include e and ó which will always be clear from context or inherited from the algorithm which runs M The challenge is to combine M with the well studied arm elimination (AE) MAB algorithm to get an SDP algorithm for the MAB problem with almost optimal regret
0² 0
Log 1
3.1 SDP-AE: Shuffle Differentially Private Arm Elimination
We base our algorithm on the (non-private) arm elimination (AE) algorithm for the MAB problem. which informally maintains& a set of viable arms (initially set be [k]). and each phase pulls the set of viable arms sequentially. Once a phase ends, we search for arms which are noticeably suboptimal in comparison to some other arm, and we eliminate them from the set of viable arms.
To adapt AE to the shuffle model, in each phase t each arm is pulled not once, but rather by a whole batch of users. Once all the users in the batch pull the arm and receive their reward, we apply
Note that our methods and algorithms should work similarly if they were built over other common algorithms for the MAB problem which batching makes sense, such as the UCB algorithm. For completeness, in Appendix C we give complete description and proof of such mechanism Maum-
5

---

the private binary summation mechanism M to the batch's rewards, which gives the server an unbiased but noisy estimate of the sum of rewards in the batch. This estimate has sources of error, which we account for when we compute the upper confidence bound - the empirical error due to sampling the reward function of the arm, and the error due to the private binary summation mechanism M sum.
3.1.1 Algorithm outline
In the algorithm below which we call Shufile Differentially Private Arm Elimination (SDP-AE), we update the estimate of the mean reward of each arm after every batch of users who sample a. The algorithm works in phases, and maintains set of viable arms initially set to be [kc] In each phase, for every viable arm have single batch of users sampling it. At the end of phase t. we denote by S the noisy estimate the cumulative sum of the rewards from all previous samples of a in the phases i, 1,...,t, natural estimate for the mean reward of the calculate the upper and lower confidence bounds UCB, and LCB' respectively of each viable arm after each phase using specific bound which takes into account both sources of error. We finally climinate any remaining arm with an upper confidence bound which is strictly smaller than the lower confidence bound of some other arm. Algorithm consolidates the algorithm presentation above.
and denote by Nº the total number of previous samples of arm in the phases .,...,t. .t The
(denoted by it therefore jt', ///Na We then
Algorithm 1: SDP-AE (Shuffle Differentially Private Arm Elimination)
Input: privacy parameters and 6, batch size m and horizon T.
2 Initialize: 00 0, No O and 1 = 0 for every a E [k]:
3 Let o², = 0 log + be the sub-Gaussian variance of the error distribution of M Maami
Let == [k] denote the set of viable arms; for phase <- 1,2.... do for arm a E V do for each
user i +- 1 to do
User i pulls the arm and observes reward ri.. If total arm samples in current algorithm run is T, exit;
10
end
Communication: Perform private binary summation Z + Moum (va
11
Server update: Update S St-1 + Zg No Nt-1 m. and finally Pi x St /
12
end
For each arm a, calculate I 22/oz.. No
Confidence bounds:
2 V2llog T, and
VN!
the upper and lower confidence bounds UCB", +- p/_ 12 and LCB + P I:
15
Elimination: remove all arms from V such that UCB < msx LCB
16
end
3.1.2 Analysis
The privacy trivial, since each batch we use the (c, ()DD) mechanism M Msum- We now focus on regret.
Theorem 3.1 gives a bound on the regret of SDP-AE as a function the batch size m. We follow somewhat standard regret bound analysis for arm elimination, comprising two parts. The first part uses Hoeffding's inequality to derive a high probability bound on j.g | the error between the empirical average reward and the true mean reward of the arm at given phase l. The second part uses this bound to bound the expected number of times we sample each suboptimal arm a, and summing over all suboptimal arms get bound on the regret.

---

Theorem 3.1. The algorithu SDP-AE is (E,8)-SDP, and has distribution -eepbbtddi--deedeeent re- Ing
+ L. logT
+ mda) ) and a Instribution-ineeeedddee regret of mda
gret of Ene[k)
Da
(V(++ Led
KT log'' + mk
Proof sketch. Fix phase t. For any arm we have ni Eiut Meum (fra. Na
where No =
We apply Hoeffding's 's incquality (Lemma 2.1 once the sequence of actual rewards of the users, and once on the sequence of errors that M Msum introduces into the approximated sum of rewards. We conclude that with high probability, Va [k], [T', we have ját
22/io N
Pa S
V N
/2 log T.
Assuming this high probability event, each arm a is necessarily eliminated after the phase to where 1.2 Haa becomes at most roughly A./4. We show that this occurs after at most
log T A2
L. -log pulls of the arm a, and since we must complete the batch of size m, the mA
må. + mda) To get the distribution- dependent regret we sum the regret above over all arms. For the cettribution-independdee regret, we or not, and bound each separately. Namely, bound the contribution of arms with Aa < 3 by BT.
total regret of the arm is most R. O
log llogT A_
split this sum based on the arm a has A. < 3 where 9 = (V (1--0"
Recall that the private binary summation mechanism Moum's 's error distribution is sub-Gaussian with We fix concrete batch size m SDP-AE, and apply Theorem 3.1 to
variance 2 O log
g²
get the following corollary.
Corollary 3.2. SDP-AE with a batch size of m [oc,o| distribution derebnddo regret of
θ
log
is (E,8)-SDP and has
)) =0
0
Log T
A. +
Au log
Σ
log T
+
log
6²
A
62
[EE()]:A.>0
(EE()):A.>0
and a distribution-indeeendde regret of O (VKTlog, log T k
Alog
3.2 VB-SDP-AE: Variable Batch Shuffle Differentially Private Arm Elimination
In this section we modify SDP-AE to give an (s, algorithm for the MAB problem with improved additional regret. Recall that SDP-AE has a distributtion-eeendeen regret of
Tog
klog whereas the best known regret for (c, centralized-DP is
28 and is O ((oeEN)As.> Ing A.
due to Tossou and Dimitrakakis
Since any SDP algo-
rithm can be emulated by a centralized-DP mechanism, we can only wish to match the regret of Tossou and Dimitrakakis (28) Hence, the natural question is: "Can we reduce the dependence e of the additive (second) regret term from to
Note that Naive attempt convert the algorithm of Tossou and Dimitrakakis from the centralized model the shuffle model fails. This since in centralized model they can decrease relative noise as the number of iteration increases by keeping the raw data from the users and reusing it in later iterations. However, in our setting. the noise shuffle batch has to remain constant for privacy, and we do not have access to the raw data of the users.

---

Before presenting our solution, we first indicate two intuitive approaches that fail. The first is to continue using batches constant size (2(1/=²). This idea fails since intuitively in the worst case we can expect a first batch regret of m = (2(1/22) which already surpasses O(1/e). The second to try to directly adapt the algorithm of Tossou and Dimitrakakis (28), which can be interpreted as AE with batches of size [ Unfortunately, this adaptation fails since it requires that the added noise to the empirical mean of each arm decreases with time, whereas in the shuffle model, we must ensure privacy with respect to each batch equally and independently, the total noise cannot decrease with time.
Now for our improved algorithm, observe that using a large batch size increases the regret a lot for arms which are very suboptimal. This is since rather than pulling these arms only a few times until we detect that they are suboptimal, we commit ourselves to pulling them throughout a large batch. On the other hand, fixing the batch size to be small increases the overall estimation error, since every application of the SDP summation algorithm Msun introduces an error of the same magnitude log independently of the batch size. Hence, more executions of Maum translates more noise due privacy. We therefore extend SDP-AE to support variable size batches, which start small and gradually increase. Intuitively, the smaller batches initially, allow us to quickly eliminate very suboptimal arms with only a small number of pulls. We gradually increase the batch size to reduce the per-user error introduced by the private binary summation mechanism. Specifically, we double the batch size after each phase.
(specifically, the error sub-Gaussian with variance o =
3.2.1 Algorithm outline
We consolidate the idea above into Algorithm| 2 below, which we call Variable Batch Shuffle Differ- entially Private Arm Elimination (VB-SDP-AE), which uses a different batch size m' 2' for each phase E
Algorithm 2: VB-SDP-AE (Variable Batch Shuffle Differentially Private Arm Elimination)
Input:
privacy parameters E and 8 and horizon T.
Initialize: 5 0, No 0 and A 0 for every [k:]
Let 02 = 0 log 1 be the sub-Gaussian variance of the error distribution of M Maami
x²
Let V [k) denote the set of viable arms;
5 for phase & + 1,2.... do Let m' < 2 for arm a E V do for each new user +- to m do
User i pulls the arm and observes reward o If total number of arm samples is T, exit;
11
end
Communication: Perform private binary summation Z% +- Moum (trá (fra))i))
12
Server update: Update St. +- Št-1 + Z No + N + m', and finally pt + /
13
14
end
15
Confidence bounds:
For each arm a, calculate r 2vto.. No
22 and
VN!
the upper and lower confidence bounds UCB,, + P1 + In and LCB, + f I.: Elimination: remove all arms a from V such that UCB, max.'es LCB
16
end
3.2.2 Analysis
The privacy is trivial since in each batch we use the (c, 8)-SDP mechanism M We now focus on regret.
To give regret bound on VB-SDP-AE, we follow a similar proof to that of Theorem
8

---

Theorem 3.3. The algorithu VB-SDP-AE is (E,6)-SDP, and has a distribution-dependerr dependent Tog regret of O ( O (VKT log T 20 log
Au + ko,, log ). and a distributioniideeenneent regret of
Proof sketch. A slight modification of the proof of Theorem 3.1 lets recover that with high probabil-
2vTa,
ity, Va [4], V [T', we have that jig Ma. S
<2 T. Assuming this high
probability event, adapting the proof Theorem 3.1 to our exponentially growing batch sizes gives
that each arm is necessarily eliminated after we have already performed O log A
Ing 7 pulls A.
of the arm Unlike the proof of Theorem 3.1 since the batch sizes grow 2 the additional pulls
of the arm a required to complete the last batch before elimination, adds only factor of 2 to the total number of pulls to the arm. Hence, the regret of the arm is at most Rg = 0
log T
+ o,,5 log T).
A.
To get the distribution distributionddependent regret we sum the regret above over all arms. For the distribution-
independent regret, we split this sum based on the arm a has A. < B where =
log
or
not, and bound each sum separately. Namely, bound the regret over arms with A. < B by 3T.
We recall that the private binary summation mechanism Msum 's error distribution is aub-Gaussian with variance o2 O
log 2² i.e., Oz,6
log and apply Theorem 3.3 get the
following corollary.
Corollary 3.4. VB-SDP-AE is (e, 8DD and has distributtinndeppndderr regret of cearrbbutionindepeededd regret of and
Eac(M]ab,>0 1og An
A /vlo & log
/T log + Ay log log T
4 Conclusion and future work
In this paper, we gave and analyzed differentially private algorithms for the MAB problem, closing the inevitable multiplicative 12(1/²²) regret gap between the local model and the centralized model, by considering the (intermediate) shuffle model. Our algorithms are batched variants of AE, which use private binary summation mechanism for the shuffle model as building block. Compared to the non-private AE algorithm's 's regret, our first algorithm SDP-AE has an additive factor of log e²
using constant size batches, and our second algorithm VB-SDP-AE improves the additive
factor to Aviog logT
by using exponentially growing batches, which enable the early detection and elimination of very suboptimal arms.
natural future work is extend our results (i.e., the usage of private binary summation mechanism for the shuffle model) to more general RL settings such as linear/contextual bandits or Markov decision study whether log term in the additional additive
processes. It would also be interesting regret factor can be shaved through a more sophisticated algorithm, or an alternative analysis. Finally, is interesting to study whether there are more refined distribution-dependeen regret bounds that depend on the KL-divergence as in Kaufmann et al. (20
Disclosure of Funding
This work is partially supported by Israel Science Foundation (grants 993/17, 1871/19
German-Israde Foundation (grant 1367/2017), Len Blavatnik and the Blavatnik Family Foundation, the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement 882396), the Yandex Initiative for Machine Learning at Tel Aviv University.

---

References
[1] B. Balle, J. Bell, A. Gascon, and K. Nissim. Differentially private summation with multi- message shufiling. arXiv preprint arXiv:1906.09116, 2019.
[2] B. Balle, J. Bell, A. Gascón, and K. Nissim. The privacy blanket of the shuffle model. In Annual International Cryptology Conference, pages 638-667. Springer, 2019.
[3] B. Balle, J. Bell, A. Gascón, and K. Nissim. Private summation in the multi-message shuffle model. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 657-676, 2020.
[4] A. Beimel, I. Haitner, K. Nissim, and U. Stemmer. On the round complexity of the shuffle model. In Theory of Cryptography Conference, pages 683-712. Springer, 2020.
[5] A. Beimel, K. Nissim, and E. Omri. Distributed private data analysis: Simultancously solving how and what. In Annual International Cryptology Conference, pages 451-468 Springer, 2008.
[6] A. Bittau, U. Erlingsson, P. Maniatís, I. Mironov, A. Raghunathan, D. Lie, M. Rudominer, U. Kode, Tinnes, and B. Seefeld. Prochlo: Strong privacy for analyties in the crowd. In Proceedings of the 26th Symposium Operating Systems Principles, pages 441-459 2017.
[7] M. Bun, J. Nelson, and U. Stemmer. Heavy hitters and the structure of local privacy. ACM Transactions on Algorithms (TALG), 15(4):1-40, 2019.
[8] T. H. Chan, E. Shi, and D. Song. Optimal lower bound for differentially private multi-party aggregation. In European Symposium Algorithms, pages 277-288. Springer, 2012.
[9] A. Cheu, A. Smith, J. Ullman, D. Zeber, and M. Zhilyaev. Distributed differential privacy via shuffling. In Anmual International Conference on the Theory and Applications of Cryptographic Techniques, pages 375-403. Springer, 2019.
[10] A. Dubey and A. Pentland. Differentiaaly-priine federated linear bandits. arXiv preprint arXiv:2010.1 1425,, 2020.
[11] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[12] U. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, S. Song, K. Talwar, and A. Thakurta. Encode, shuffle, analyze privacy revisited: Formalizations and empirical evaluation. arXiv preprint arXiv:2001. 036618, 2020.
[13] Ó. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, and A. Thakurta. Amplifi- cation by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Anmual ACM-SIAM Symposium on Discrete Algorithms, pages 2019.
privvayy
SIAM,
[14] H. Esfandiari, A. Karbasi, Mehrabian, and V. Mirrokni. Regret bounds for batched bandits. arXiv preprint arXiv: 190.0499,, 2019.
[15] 7. Gao, Y. Han, 7. Ren, and 7.. Zhou. Batched multi-armed bandits problem. arXiv preprint arXiv: 1904. 01 7633, 2019.
[16] B. Ghazi, R. Pagh, and A. Velingker. Scalable and differentially private distributed aggregation in the shuffled model. arXiv preprint arXiv: 1906. 08320, 2019.
[17] A. Guha Thakurta and A. Smith. (nearly) optimal algorithms for private online learning in full-information and bandit settings. Advances in Neural Information Processing Systems, 26:2733-2741, 2013.
[18] W. Hoeffding. Probability inequalities for sums of bounded random variables. In The Collected Works of Wassily Hoeffding, pages 409-426. Springer, 1994.
[19] Jain, Kothari, and A. Thakurta. Differentially private online learning. In Conference on Learning Theory, pages 24-1. JMLR Workshop and Conference Proceedings, 2012.
10

---

[20] E. Kaufmann, O. Cappé, and A. Garivier. On the complexity of best-arm identification in multi-armes bandit models. The Journal of Machine Learning Research, 17(1):1-42, 2016.
[21] N. Mishra and A. Thakurta. (nearly) optimal differentially private stochastic multi-arm bandits. In Proceedings of the Thirty-Firsl Conference Uncertainty in Artificial Intelligence, pages 592-601, 2015.
[22] W. Ren, X. Zhou, J. Liu, and N. B. Shroff. Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.03/2), 2020.
[23] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
[24] T. Sajed and O. Sheffet. An optimal private stochastic- mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning, pages 5579-5588. PMLR, 2019.
[25] R. Shariff and O. Sheffet. Differentially private contextual linear bandits. arXiv preprint arXiv:1810.00068, 2018.
[26] C. Shi and C. Shen. Federated multi-armed bandits. arXiv preprint arXiv:1101. 2021.
[27] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4))255-994, 1933.
[28] A. Tossou and C. Dimitrakakis. Algorithmss for differentially private multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[29] K. Zheng, T. Cai, W. Huang, Z. Li, and L. Wang. Locally differentially private (contextual) bandits learning, arXiv preprint arXiv:2006.0077 2020.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We discuss the importance of preserving using privacy the introduction. The main contribution of this work is to help preserve user privacy, which clearly has positive societal impact. cannot foresee any negative societal impact related to our theoretical work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All the assumptions appear either in the Sections T and Section 2
(b) Did you include complete proofs of all theoretical results? [Yes] All the proofs which don't I appear in the body of the paper, do appear in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "fnNS3CKIgv", "reviewer": "Reviewer_iPAH", "summary": "This paper adapts the shuffle model of differential privacy to the multi-armed bandit setting, and aims to design a bandit algorithm which has privacy guarantees similar to that in the local model, while simultaneously having regret guarantees similar to the central DP model. The authors propose two arm-elimination style algorithms, which uses a private binary summation mechanism with optimal error guarantees as a subroutine, in order to obtain improved regret guarantees over the local model. In particular, they show how a simple, constant batch size policy already gets rid of the multiplicative dependence on $1/\\epsilon^2$. Then, they show how to adapt this policy to exponentially increasing batch sizes, improving the additive dependence from $\\approx k/\\epsilon^2$ to $\\approx k/\\epsilon$.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the ideas and techniques considered in this paper are fairly standard, the results of this work seem nice. I found it surprising that almost the same regret guarantees are achievable in the shuffle model as in the central model, so I quite enjoyed reading this work.\n\nOverall, I found the paper to be very well written. The exposition of the models considered in the paper, as well as the theoretical results and proof ideas, were very easy to follow. I especially enjoyed the section on the variable batch size which discussed two intuitive approaches which fail.\n\nI am wondering what dependence on the parameter \\delta is to be expected in the regret analysis. In Table 1, the Centralized $(\\epsilon,\\delta)$-DP result displays no (explicit) dependence on $\\delta$, although I suspect this dependence is hidden perhaps in $\\epsilon$? Do the authors believe that the $\\sqrt{\\log{\\frac{1}{\\delta}}}$ dependence on the additive term in their analysis is tight? Is this dependence the same as in the central model, or is this the main separation?\n\nAdditionally, could the authors elaborate on the $\\log(T)$ term that appears in the additive $1/\\epsilon$ term in their analysis? In particular, why does this term not appear in the central $(\\epsilon,\\delta)$-DP analysis? Could you provide any intuition why you would expect this term is not necessary?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, adequately addressed.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "M06FqkyblL", "reviewer": "Reviewer_Kux9", "summary": "This paper studies the multi-armed bandit problem under the constraint of approximate differential privacy. Previously, multi-armed bandits have been studied in other models of privacy such as the centralized pure DP model, the centralized approximate DP model, and the local pure DP model. However, this is the first work to consider the multi-armed bandit problem under the shuffle model of privacy.\n\nTo be clear, the model that the authors consider is as follows. At each round $t$, a batch of $m$ random users are selected. An action (e.g. an ad) is then selected for each user. The reward for each user is then presented to a shuffle DP mechanism.\n\nIn terms of results, the authors prove a regret bound which is slightly weaker than the result one can obtain in the central DP model but significantly stronger than the result one can obtain the local DP model. In particular, it removes a multiplicative $1/\\epsilon^2$ dependence that appears in the regret bound for the local DP model.\n\nIn terms of techniques, the authors combine some ideas from the bandit literature and the differential privacy literature. In particular, their algorithms are based on the non-private arm elimination algorithm and they show how one can use private binary summation from the shuffle DP literature to obtain a shuffle DP algorithm for MAB. The authors also give two differential algorithms. The first algorithm uses static batch sizes in each round. The second algorithm uses dynamic batch sizes and they show that this improves the regret.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper studies the MAB problem under the shuffle approximate DP setting. This has not been studied before.\n- The paper provides novel results.\n- The results are obtained using a nice combination of techniques from the DP and MAB literature including private binary summation from the DP literature and the arm elimination algorithm from the MAB literature.\n- The algorithms are fairly easy to understand.\n- Overall the paper is written well and the ideas are clearly explained.\n\n**Weaknesses**\n- My main concern is that the paper does not have that much new technical ideas. Algorithm 1 mainly combined known ideas from the DP and MAB literature (in particular, private binary summation and arm elimination). Algorithm 2 tacks on the additional idea of using increasing batch sizes.\n\n\n**Target Audience**\n- The results will be of interest to researchers working on online learning and theoretical aspects of differential privacy.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ChpLiFm2Vwx", "reviewer": "Reviewer_HQch", "summary": "An interesting paper that utilizes the shuffle model to achieve low regret and differential privacy simultaneously. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies MAB under local differential privacy up to a trusted shuffler (the shuffle model). This privacy guarantee is a weaker notion than LDP and it requires the user to not repeat through the entire horizon, but it could relax the 1/eps^2 dependency on the additive regret to 1/eps. The algorithm is mostly the same as MAB under LDP where each reward is with a Bernoulli noise, but m rewards are shuffled and aggregated until they are observed by the algorithm, which makes the noise level much smaller than LDP. Setting m=2^t further improves the regret. The adjustment of the noise level borrows the lemma from the shuffle model. This manuscript is very well-written and the presentation is very clear. The intuitive description of the algorithm and the proof are also sufficient that they could be easily understood. The manuscript also properly credits the referred techniques and the related work, which helps the readers to understand a larger scale of the area. I would like to see this manuscript accepted, though I would not be too surprised if otherwise as the manuscript could still be improved in several ways.\n\n1. I think Bernoulli bandits is quite a limit and it seems to be possible to give regret bounds to general sub-Gaussian arms. The concentration needed can again be borrowed from the shuffle model. This can improve the overall technical contribution as well (which is currently lacking). It might also be interesting to consider other relaxations such as allowing the users to repeat for some limited amount of times (which is critical for the shuffle model to make sense).\n\n2. I believe that it would be helpful to at least implement the algorithm and compare it with other LDP/DP methods to see the effect of batch learning and reduced noise. I am also curious if m=2^t will be too large in practice and if so what could be the practical choice of the batch size.\n\nMinor comments:\n\nline 6 and other places: almost -> nearly (almost could mean w.p. 1)  \nline 138: move Section 2.3 to Appendix? I don't see them useful.  \nline 160: Is this a \"challenge\"? If so, elaborate on it.  \nline 220: delta dependence missing  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BegENCuAqh4", "reviewer": "Reviewer_dj8Y", "summary": "The paper studies the classic multi-armed bandit problem in the shuffled model of differential privacy. Paper's first contribution is the definition of shuffled differential privacy (SDP) in the multi-armed bandit setting.  The paper's main contribution is an SDP algorithm with the same regret guarantees as the best-known multi-armed bandit algorithm in the central privacy model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\na) The definition of shuffled differential privacy for sequential decision-making problems is interesting and novel and would inspire future privacy research for these problems.\n\nb) The algorithms proposed by the authors combine known MAB algorithms and SDP mechanisms with minimal tweaking. \n\nc) However, one key innovation is the variable batch size selection in the VB-SDP-AE algorithm, which allows the algorithm to enjoy the same regret as that enjoyed by algorithms in the central privacy model.\n\nPresentation and Correctness of claims: The paper is well written and easy to follow. The analysis, too, is easy to follow and correct.\n\nTo summarize, I like this paper and recommend it be accepted.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes new algorithms for differentially private multi-armed bandits in the shuffle model. Their algorithms nearly match the results in the central model. The paper is well-written and the reviewers were positive.
I would suggest the authors incorporate the feedback from the reviewers in the final version. In particular, I found the discussion on the Naive attempt to convert the algorithm of Tossou and Dimitrakakis from the centralized model to the shuffle model to be useful, and adding some of this to the paper would make it better.
I am happy to recommend acceptance for this paper.
Decision: Accept (Poster)