Differentially Private Multi-Armed Bandits in the Shuffle Model
Authors: Jay Tenenbaum
Haim Kaplan
Yishay Mansour
Uri Stemmer
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-armed bandits
differential privacy
shuffle model

Abstract: We give an $(\varepsilon,\delta)$-differentially private algorithm for the Multi-Armed Bandit (MAB) problem in the shuffle model with a distribution-dependent regret of $O\left(\left(\sum_{a:\Delta_a>0}\frac{\log T}{\Delta_a}\right)+\frac{k\sqrt{\log\frac{1}{\delta}}\log T}{\varepsilon}\right)$, and a distribution-independent regret of $O\left(\sqrt{kT\log T}+\frac{k\sqrt{\log\frac{1}{\delta}}\log T}{\varepsilon}\right)$, where $T$ is the number of rounds, $\Delta_a$ is the suboptimality gap of the action $a$, and $k$ is the total number of actions. Our upper bound almost matches the regret of the best known algorithms for the centralized model, and significantly outperforms the best known algorithm in the local model.

Differentially Private Multi-Armed Bandits in the Shuffle Model
Jay Tenenbaum"
Haim Kaplan'
Yishay Mansour
Uri Stemmer
Abstract
We give deifferifreylttialll private algorithm for the multi-armed bandit (MAB) problem the shuffle model with a distribution-deeeedden regret of FoR Tog and a distribution -nneeeendnnndeeeeeeee regret
TogT) A.
EaE(H]:A.>0
of O VETlogT Av/og log
where T is the number of rounds, Aa is the
suboptimality gap of the arm a, and l is the total number of arms. Our upper bound almost matches the regret of the best known algorithms for the centralized model, and significantly outperforms the best known algorithm in the local model.
Introduction
The multi-ana aarii--reddddd bandit (MAB) problem is classical sequential decision-making problem in which an agent tries to maximize a cumulative stochastic reward [27 23 under uncertainty. This problem, which is applicable various areas such recommender systems, online advertising and clinical trials, embodies the well known exploration-exploiiation trade-off between learning the environment and acting optimally based our current knowledge about the environment.
More formally, in the MAB problem at each time / = I,..., T an agent chooses an arm i from the set [A] .1..... }} of & arms, and obtains iid reward rf drawn from the unknown distribution R, over (0. 1} with expectation μ4, = E [R.]. Let a arg max,, Hl be an arm with the largest expected reward, and denote this reward by between its expected reward and that a", i.c., A. = μ Ha. The agent's goal is to maximize
Ma Let the (suboptimality) gap of an arm a to be the gap
the total expected reward, or rather minimize the expected regret R(T) T p² E EL defined to be the expected gap between the algorithm and the optimal algorithm that knows the distributions R.
In this work we address the privacy in such a setting. As a motivating example, consider an advertisement system in which the server presents each user an advertisement a [K)] The user then decides whether to click on the advertisement or not. This click decision depends on different private characteristic of the user. The user then reports the server whether it clicked on the advertisement (in which case its reward is r = 1) or not (r = 0). From this example, it is clear that private information of the user, and using traditional algorithms for the MAB problem incautiously might leak user user-private data.
In order mathematically alleviate privacy concerns, Dwork al. [1I] defined the notion of differential privacy (DP), which requires that the output of the computation has a limited dependency
Google Research, jayten google.com
"Blavatnik School of Computer Science, Tel Aviv University and Google Research, haimk @tau.ac.il
'Blavatnik School of Computer Science, Aviv University and Google Research.
man-
sour.yishay @gmmil..com
Blavatnik School Computer Science, Tel Aviv University and Google Research. @@uuri.o.il
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Best-known MAB regret upper and lower bounds for various DP models.
Privacy model
Best-known regret upper and lower bound
Centralized (e, O)-DP
20 EaE(K)
log T Da
+
log T
24
25
Centralized (-D
O
Tog
+
28
Au
Local (s, 0)-DP
θ
EaE[k]-A.>0
log T
122
A.
Shuffle (c, 6)DP (ours)
o
[ (EnE(k-AA20
log T Aa
+
Rv log log
on any single user's data. Formally, a mechanism (e, 8DP if for any pair of neighboring inputs (differing by single user's data), the probability that the mechanism outputs a value in any set B is not different by more than a multiplicative factor of e and an additive factor of 8. Differential privacy has been extensively studied under many different sub-models of privacy. On one end of the spectrum lies the centralized model of differential privacy, where the users trust the server with their data, and the liability protect user privacy lies the server, who must make sure that any data published externaly (e.g.. aggregated statistics) respects the privacy constraints. On the other end of the spectrum lies the (strictly stronger) local model of differentialy privacy (LDP), where the user privatizes its own data prior to sending it to the server
Differentially private versions of the MAB problem have been considered in various previous works, where the private information are users' rewards (two neighboring inputs differ by the reward value of single user), and the algorithm's 's output is the subsequent arm(s) selects. Table I summarizes the best known ietributtio-dependdee new result in the shuffle model which we soon define formally.
regret bounds for the various privacy models, together with
With real- world algorithms gradually moving away from the centralized model of privacy, the immediate question is "can we design private algorithm for MAB with privacy guarantees which are similar to local DP, but with similar regret to centralized DP (without a multiplicative 1/e² factor)?""
To address the inevitable gap between the local and centralized models, which is in fact common in the literature of differential privacy, the alternative shuffle model 66 [3] explores the space in between the local and centralized models by introducing a trusted shuffler that receives user messages and permutes them (i.e., disassociates a message from its sender) before they are delivered to the server. For privacy analysis, we assume that the shuffle is perfectly secure, i.e., its output contains no information about which user generated each of the messages. This is traditionally achieved by the shuffler stripping implicit metadata from the messages (e.g., timestamps, routing information), and frequently forwarding this data to remove time and order information. The shuffle model ensures that sufficiently many reports are collected in each round so that any one report can hide in shuffled batch. In order to apply the shuffle model to the MAB problem in the context of advertisements, we divide the algorithm into batches, where before each batch we decide on the fly its size m, and then present the m next users the same advertisement a, and finally apply a private shuffle model mechanism to their rewards to communicate reward aggregate information to the server {
constantly growing body of work presents new and improved mechanisms in the shuffle model for basic statistical tasks [16 13 12 2, such as private binary summation, in which the server must
privately approximate the sum of collection of values Z .... I'm E {0, 1} held by the m users in
FFormally, in the non- non-interactive setting, mechanism is (e. 8)-LDP if for any two user inputs, the probability that the privatizer sends the server a value in any set B is not different by more than multiplicative factor of and an additive factor of ö.
Thhe corresponding distribution -nneeeeennn-deeededent regret bounds usually simply replace the EmE14)A, 20 A.
T
term with / log
"This lower bound can be extended from (s, 0)-LDP to (e, 8)-LDP using arguments from Bun et al. 7 and Cheu et al. [9] since focus single-round hnoe-interacttve) mechanisms in which the user can only send information to another party once.
We remark that a given user does not know in advance the size of the batch in which it participates, since this size depends on the algorithm's 's run.

---

the shuffle. For private binary summation, the optimal achievable errors in the central, local and shuffle model are (1/=)) [.]. (m/m/)) [ and (11ss) [9] respectively, where the ((-) hides poly-logarithmic terms. Similar errors hold for the private summation problem which approximates the sum of real values in [0, 1].
Our contributions
To the best of our knowledge, our work is the first to consider the MAB problem under the shuffle model differential privacy. order to support the online nature of the MAB problem, we consider variant of the shuffle model. As opposed to the classical shuffle model, in which the shuffle size is unbounded and the mechanism runs only once, we continuously run shuffle mechanisms for many disjoint batches of users who can only afford single round of communication
We consider the paradigm where the server controls the different users who are cooperative and communicate only with the server. We give rigorous definition of shuflle differential privacy (SDP) for the multi-armed bandit problem (assuming binary rewards), and give and prove the first two such algorithms. Our algorithms Shuffle Differentially Private Arm Elimination (SDP-AE) and Variable Batch Shuffle Differentially Private Arm Elimination VV-SDD-AE) are both based on the well-known arm elimination (AE) algorithm, using consecutive batches of users and together with an SDP private binary summation mechanism.
We show that the simpler but weaker SDP-AE achieves a distribution-dependenn re- and
gret of ((EnE)NAAsS0 IogT A.
+
& log
&
distribution-indeeendet regret of
oVETlog &log
10
We then describe VB-SDP-AF, a generalization of SDP-AE to exponentially growing batch sizes,
and prove it has distribution-dee seerbuttion-deeeddentt regret of O (EoE(R)A,
lox A.
+ k log log and
a disttiibttion-indeeeeddd regret of O (VETT log + Avlog log
Note that, compared to the local model (Ren al [22) the regret of both SDP-AE and VB-SDP-AE improved, by having the dependency on 1/s be additive rather than multiplicative. In addition, VB TB-SDP-AE almost matches the regret of the best known algorithms for the centralized model,
that is the distributioon-deeendent regret of O ((Zne))A.>o
log 3.
of Tossou and Dimi- #
trakakis 28
1.2 Related work
The differentially private MAB problem has been considered in many previous works [9
Shi and Shen [26] and Dubey and Pentland [10] studied MAB and linear bandits respectively in federated setting. Zheng et al. (29 studied contextual bandits with LDP. Batched MAB with predetermined number of batches was studied in [14 15].
For the private summation problem, Cheu al. 9 gave unbiased (e, 8DDP mechanisms over binary inputs and real inputs in [0. 1). with error roughly /log(1/8) (1/8)/e and log(1/8) /8) respectively. In several works of Balle et al. 2 I. they gave biased (s, 8D-SDP mechanism for real inputs in [o, 1] with similar error and a constant number of messages, and a Single-messp (e, 8)DDP mechanism with optimal error.
Beimel et al. 4 showed that every centralized DP mechanism can be emulated in the shuffle model in two applicable to setting
ceammunncation-inteiiiee rounds of communication, however these results since we assume that each online user participates only in one shuffle then disappears. I0In this paper, we assume that 7' is known apriori. Otherwise, we can apply standard doubling arguments to get roughly the same results.

---

2 Background and preliminaries
2.1 Shuffle-model privacy
In the well-studied setting of shuffle model privacy, there are users, each with data E X. Each user applies some encoder E : X - Y* to their data and sends the messages (fi,1, Vi.p) (x;)) to a shuffler S Y + Y The shuffler then shuffles all the messages Vi., from all the users, and outputs them in uniformly random order to an analyzer function [(I,...., Fm). Thus, the mechanism M consists of the tuple (E, S, A). We say that such a mechanism M is (e, d.ssfflee differentially private (or (e, 8DPP for short) if the output of (e, 5) -iiifferentially private, or more formally: A mechanism M = (E,S, S, A) is (e, 8)-SDP if for any pair of inputs {x, [x,)==1 and {=/]-= which differ in at most one value, we have for all B C Y":
Y + to estimate some
P E B) S ef P( E +
In the mechanism used in this paper, E outputs the user's reward bit together with set of random bits, and sums all these bits and debiases the result to get an unbiased estimate of the sum of the users' rewards.
2.2 Shuffle-model MAB
Algorithms which are private in the shuffle model typically apply the mechanism once over a set of m users. Here study the MAB problem which is online problem, often deployed in real-world applications and with users which are end-devices such as cellphones with a possibly limited or unreliable internet connection. Hence, to adapt the MAB problem to the shuffle model, we batch sequences of consecutive users, and assume that each user can afford a single round of communication, and is never selected more than once.
Model and objective the T'th player pulls its arm:
The shuffle-mode MAB setting involves repeating the following process until
1. The server selects batch size m, batch of m random fresh new users, and m-user single-round SDP mechanism M II It then pieks an arm a E [M]] that all m users of the batch pull. (For concreteness think that the server picks an ad [k] and sends it to the random batch of m users.)
2. Each user determines binary reward from pulling the arm a (For concreteness think that each user decides whether to click on the ad or not. This defines the reward related to user i, which r; 1 if clicks the ad and r; 0 otherwise.)
Since our m users are random, these rewards are a sample of m independent rewards from the distribution R., associated with arm a.
3. The server computes M ((r, ((r:))eee iebateh)) using the rewards Ti.
The objective is to minimize the (pseudo) regret, which is the expected difference between the sum of the rewards accumulated (over all the users) by the algorithm and the sum of the rewards of the optimal algorithm that apriori knows an arm with the largest expected reward a". Let /4a E [R,] be the expected reward (or simply mean) of the arm a, let μ denote the expected reward of a* let An μr" - Naa be the (suboptimality) gap of the arm a which quantifies the gap between its expected reward and that of a", and let Na be the random variable which counts the total number of times the arm a was pulled during the run of the algorithm.
Formally, the (pseudo) regret of the algorithm for T E N users is defined to be
R(T) = 1
E
Σ r4
E Σ N.A.
DE[K]
Nott that always selecting m reduces this setting the Local model MAB. For simplicity, we assume that the rewards are binary. However, our algorithms and proof's naturally extend to the real jo, 1] reward setting, by replacing our private binary summation mechanism (defined later, see Appendix C) with private summation mechanism for real numbers in [0, 1] with similar guarantees.

---

Privacy Since the private data of each user is its reward, the appropriate adaptation of shuffle model privacy for the multi-an -amedd bandit problem is as follows. An algorithm for the multi-armed bandits problem is (c, 5) ))-shuffle differentially private (SDP) if for any batch of users, the shuffle mechanism that we apply over them in step 3 is (c, 5)-DPP with respect to the rewards of the users (as we recall, the reward of a user whether it clicked on an ad or not depends on its private features). Formally, for every batch we run a shuffle mechanism where the m users are the users of the current batch, and the data 2; of each user is its reward ri, and we require that each such mechanism is (s, d-SDD.
2.3 Concentration bounds
We use the following standard definitions of Sub-Gaussian random variables and Hoeffding's inequal- ity.
Definition 1 SSub-Gaussian random variable). A random variable X with mean p is called sub- Gaussian with variance o², i.e., X ~ -SG(o2) if:
AA M R, E [exp(A(X - -1))] S 2200212.
An equivalent definition shows that if 11 0, max -
- PX- S
exp 20 then X sub-Gaussian with variance a² (up to constant factor). is well known that X; ~ (G are independent random variables for = 1...., n, then Σi= X, ~ SG(E== of) and for any a, b 0. a X1 + b SG(a² of). bounded random variable X E [a is SG - a) a)2/4))
Bub-Gaussian random variables satisfy the following concentration bound, Lemma 2.1 (Hoeffding's inequality (18).. Let {X,) be independent SG(o²) random variables,
then P E X. E E> x.] 2 1
< 2 exp
2nd
3 Differentially private MAB in the shuffle model
In this paper, we use the shuffle model to give a private solution to the multi-armeet bandit problem, attaining similar privacy guarantees to that of the local privacy model (LDP), without sacrificing the regret. That is, our algorithm almost matches the best known regret in the centralized model of differential privacy. Our algorithms rely on the fact that algorithms for the multi-armed bandit problem take decisions based on sums of rewards received from the users. Hence, we rely on particularly efficient and accurate mechanism for private binary summation in the shuffle model as building block in our algorithms. Specifically, for any s, 8 E (0, 1). let M be a private binary summation mechanism, which for any number of users (batch size) m E N. is (s, 0)SD., unbiased, and has error distribution which is independent of the input, and is
with variance 13 Note that our notation of Maum does not include e and ó which will always be clear from context or inherited from the algorithm which runs M The challenge is to combine M with the well studied arm elimination (AE) MAB algorithm to get an SDP algorithm for the MAB problem with almost optimal regret
0² 0
Log 1
3.1 SDP-AE: Shuffle Differentially Private Arm Elimination
We base our algorithm on the (non-private) arm elimination (AE) algorithm for the MAB problem. which informally maintains& a set of viable arms (initially set be [k]). and each phase pulls the set of viable arms sequentially. Once a phase ends, we search for arms which are noticeably suboptimal in comparison to some other arm, and we eliminate them from the set of viable arms.
To adapt AE to the shuffle model, in each phase t each arm is pulled not once, but rather by a whole batch of users. Once all the users in the batch pull the arm and receive their reward, we apply
Note that our methods and algorithms should work similarly if they were built over other common algorithms for the MAB problem which batching makes sense, such as the UCB algorithm. For completeness, in Appendix C we give complete description and proof of such mechanism Maum-
5

---

the private binary summation mechanism M to the batch's rewards, which gives the server an unbiased but noisy estimate of the sum of rewards in the batch. This estimate has sources of error, which we account for when we compute the upper confidence bound - the empirical error due to sampling the reward function of the arm, and the error due to the private binary summation mechanism M sum.
3.1.1 Algorithm outline
In the algorithm below which we call Shufile Differentially Private Arm Elimination (SDP-AE), we update the estimate of the mean reward of each arm after every batch of users who sample a. The algorithm works in phases, and maintains set of viable arms initially set to be [kc] In each phase, for every viable arm have single batch of users sampling it. At the end of phase t. we denote by S the noisy estimate the cumulative sum of the rewards from all previous samples of a in the phases i, 1,...,t, natural estimate for the mean reward of the calculate the upper and lower confidence bounds UCB, and LCB' respectively of each viable arm after each phase using specific bound which takes into account both sources of error. We finally climinate any remaining arm with an upper confidence bound which is strictly smaller than the lower confidence bound of some other arm. Algorithm consolidates the algorithm presentation above.
and denote by Nº the total number of previous samples of arm in the phases .,...,t. .t The
(denoted by it therefore jt', ///Na We then
Algorithm 1: SDP-AE (Shuffle Differentially Private Arm Elimination)
Input: privacy parameters and 6, batch size m and horizon T.
2 Initialize: 00 0, No O and 1 = 0 for every a E [k]:
3 Let o², = 0 log + be the sub-Gaussian variance of the error distribution of M Maami
Let == [k] denote the set of viable arms; for phase <- 1,2.... do for arm a E V do for each
user i +- 1 to do
User i pulls the arm and observes reward ri.. If total arm samples in current algorithm run is T, exit;
10
end
Communication: Perform private binary summation Z + Moum (va
11
Server update: Update S St-1 + Zg No Nt-1 m. and finally Pi x St /
12
end
For each arm a, calculate I 22/oz.. No
Confidence bounds:
2 V2llog T, and
VN!
the upper and lower confidence bounds UCB", +- p/_ 12 and LCB + P I:
15
Elimination: remove all arms from V such that UCB < msx LCB
16
end
3.1.2 Analysis
The privacy trivial, since each batch we use the (c, ()DD) mechanism M Msum- We now focus on regret.
Theorem 3.1 gives a bound on the regret of SDP-AE as a function the batch size m. We follow somewhat standard regret bound analysis for arm elimination, comprising two parts. The first part uses Hoeffding's inequality to derive a high probability bound on j.g | the error between the empirical average reward and the true mean reward of the arm at given phase l. The second part uses this bound to bound the expected number of times we sample each suboptimal arm a, and summing over all suboptimal arms get bound on the regret.

---

Theorem 3.1. The algorithu SDP-AE is (E,8)-SDP, and has distribution -eepbbtddi--deedeeent re- Ing
+ L. logT
+ mda) ) and a Instribution-ineeeedddee regret of mda
gret of Ene[k)
Da
(V(++ Led
KT log'' + mk
Proof sketch. Fix phase t. For any arm we have ni Eiut Meum (fra. Na
where No =
We apply Hoeffding's 's incquality (Lemma 2.1 once the sequence of actual rewards of the users, and once on the sequence of errors that M Msum introduces into the approximated sum of rewards. We conclude that with high probability, Va [k], [T', we have ját
22/io N
Pa S
V N
/2 log T.
Assuming this high probability event, each arm a is necessarily eliminated after the phase to where 1.2 Haa becomes at most roughly A./4. We show that this occurs after at most
log T A2
L. -log pulls of the arm a, and since we must complete the batch of size m, the mA
må. + mda) To get the distribution- dependent regret we sum the regret above over all arms. For the cettribution-independdee regret, we or not, and bound each separately. Namely, bound the contribution of arms with Aa < 3 by BT.
total regret of the arm is most R. O
log llogT A_
split this sum based on the arm a has A. < 3 where 9 = (V (1--0"
Recall that the private binary summation mechanism Moum's 's error distribution is sub-Gaussian with We fix concrete batch size m SDP-AE, and apply Theorem 3.1 to
variance 2 O log
g²
get the following corollary.
Corollary 3.2. SDP-AE with a batch size of m [oc,o| distribution derebnddo regret of
θ
log
is (E,8)-SDP and has
)) =0
0
Log T
A. +
Au log
Σ
log T
+
log
6²
A
62
[EE()]:A.>0
(EE()):A.>0
and a distribution-indeeendde regret of O (VKTlog, log T k
Alog
3.2 VB-SDP-AE: Variable Batch Shuffle Differentially Private Arm Elimination
In this section we modify SDP-AE to give an (s, algorithm for the MAB problem with improved additional regret. Recall that SDP-AE has a distributtion-eeendeen regret of
Tog
klog whereas the best known regret for (c, centralized-DP is
28 and is O ((oeEN)As.> Ing A.
due to Tossou and Dimitrakakis
Since any SDP algo-
rithm can be emulated by a centralized-DP mechanism, we can only wish to match the regret of Tossou and Dimitrakakis (28) Hence, the natural question is: "Can we reduce the dependence e of the additive (second) regret term from to
Note that Naive attempt convert the algorithm of Tossou and Dimitrakakis from the centralized model the shuffle model fails. This since in centralized model they can decrease relative noise as the number of iteration increases by keeping the raw data from the users and reusing it in later iterations. However, in our setting. the noise shuffle batch has to remain constant for privacy, and we do not have access to the raw data of the users.

---

Before presenting our solution, we first indicate two intuitive approaches that fail. The first is to continue using batches constant size (2(1/=²). This idea fails since intuitively in the worst case we can expect a first batch regret of m = (2(1/22) which already surpasses O(1/e). The second to try to directly adapt the algorithm of Tossou and Dimitrakakis (28), which can be interpreted as AE with batches of size [ Unfortunately, this adaptation fails since it requires that the added noise to the empirical mean of each arm decreases with time, whereas in the shuffle model, we must ensure privacy with respect to each batch equally and independently, the total noise cannot decrease with time.
Now for our improved algorithm, observe that using a large batch size increases the regret a lot for arms which are very suboptimal. This is since rather than pulling these arms only a few times until we detect that they are suboptimal, we commit ourselves to pulling them throughout a large batch. On the other hand, fixing the batch size to be small increases the overall estimation error, since every application of the SDP summation algorithm Msun introduces an error of the same magnitude log independently of the batch size. Hence, more executions of Maum translates more noise due privacy. We therefore extend SDP-AE to support variable size batches, which start small and gradually increase. Intuitively, the smaller batches initially, allow us to quickly eliminate very suboptimal arms with only a small number of pulls. We gradually increase the batch size to reduce the per-user error introduced by the private binary summation mechanism. Specifically, we double the batch size after each phase.
(specifically, the error sub-Gaussian with variance o =
3.2.1 Algorithm outline
We consolidate the idea above into Algorithm| 2 below, which we call Variable Batch Shuffle Differ- entially Private Arm Elimination (VB-SDP-AE), which uses a different batch size m' 2' for each phase E
Algorithm 2: VB-SDP-AE (Variable Batch Shuffle Differentially Private Arm Elimination)
Input:
privacy parameters E and 8 and horizon T.
Initialize: 5 0, No 0 and A 0 for every [k:]
Let 02 = 0 log 1 be the sub-Gaussian variance of the error distribution of M Maami
x²
Let V [k) denote the set of viable arms;
5 for phase & + 1,2.... do Let m' < 2 for arm a E V do for each new user +- to m do
User i pulls the arm and observes reward o If total number of arm samples is T, exit;
11
end
Communication: Perform private binary summation Z% +- Moum (trá (fra))i))
12
Server update: Update St. +- Št-1 + Z No + N + m', and finally pt + /
13
14
end
15
Confidence bounds:
For each arm a, calculate r 2vto.. No
22 and
VN!
the upper and lower confidence bounds UCB,, + P1 + In and LCB, + f I.: Elimination: remove all arms a from V such that UCB, max.'es LCB
16
end
3.2.2 Analysis
The privacy is trivial since in each batch we use the (c, 8)-SDP mechanism M We now focus on regret.
To give regret bound on VB-SDP-AE, we follow a similar proof to that of Theorem
8

---

Theorem 3.3. The algorithu VB-SDP-AE is (E,6)-SDP, and has a distribution-dependerr dependent Tog regret of O ( O (VKT log T 20 log
Au + ko,, log ). and a distributioniideeenneent regret of
Proof sketch. A slight modification of the proof of Theorem 3.1 lets recover that with high probabil-
2vTa,
ity, Va [4], V [T', we have that jig Ma. S
<2 T. Assuming this high
probability event, adapting the proof Theorem 3.1 to our exponentially growing batch sizes gives
that each arm is necessarily eliminated after we have already performed O log A
Ing 7 pulls A.
of the arm Unlike the proof of Theorem 3.1 since the batch sizes grow 2 the additional pulls
of the arm a required to complete the last batch before elimination, adds only factor of 2 to the total number of pulls to the arm. Hence, the regret of the arm is at most Rg = 0
log T
+ o,,5 log T).
A.
To get the distribution distributionddependent regret we sum the regret above over all arms. For the distribution-
independent regret, we split this sum based on the arm a has A. < B where =
log
or
not, and bound each sum separately. Namely, bound the regret over arms with A. < B by 3T.
We recall that the private binary summation mechanism Msum 's error distribution is aub-Gaussian with variance o2 O
log 2² i.e., Oz,6
log and apply Theorem 3.3 get the
following corollary.
Corollary 3.4. VB-SDP-AE is (e, 8DD and has distributtinndeppndderr regret of cearrbbutionindepeededd regret of and
Eac(M]ab,>0 1og An
A /vlo & log
/T log + Ay log log T
4 Conclusion and future work
In this paper, we gave and analyzed differentially private algorithms for the MAB problem, closing the inevitable multiplicative 12(1/²²) regret gap between the local model and the centralized model, by considering the (intermediate) shuffle model. Our algorithms are batched variants of AE, which use private binary summation mechanism for the shuffle model as building block. Compared to the non-private AE algorithm's 's regret, our first algorithm SDP-AE has an additive factor of log e²
using constant size batches, and our second algorithm VB-SDP-AE improves the additive
factor to Aviog logT
by using exponentially growing batches, which enable the early detection and elimination of very suboptimal arms.
natural future work is extend our results (i.e., the usage of private binary summation mechanism for the shuffle model) to more general RL settings such as linear/contextual bandits or Markov decision study whether log term in the additional additive
processes. It would also be interesting regret factor can be shaved through a more sophisticated algorithm, or an alternative analysis. Finally, is interesting to study whether there are more refined distribution-dependeen regret bounds that depend on the KL-divergence as in Kaufmann et al. (20
Disclosure of Funding
This work is partially supported by Israel Science Foundation (grants 993/17, 1871/19
German-Israde Foundation (grant 1367/2017), Len Blavatnik and the Blavatnik Family Foundation, the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement 882396), the Yandex Initiative for Machine Learning at Tel Aviv University.

---

References
[1] B. Balle, J. Bell, A. Gascon, and K. Nissim. Differentially private summation with multi- message shufiling. arXiv preprint arXiv:1906.09116, 2019.
[2] B. Balle, J. Bell, A. Gascón, and K. Nissim. The privacy blanket of the shuffle model. In Annual International Cryptology Conference, pages 638-667. Springer, 2019.
[3] B. Balle, J. Bell, A. Gascón, and K. Nissim. Private summation in the multi-message shuffle model. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 657-676, 2020.
[4] A. Beimel, I. Haitner, K. Nissim, and U. Stemmer. On the round complexity of the shuffle model. In Theory of Cryptography Conference, pages 683-712. Springer, 2020.
[5] A. Beimel, K. Nissim, and E. Omri. Distributed private data analysis: Simultancously solving how and what. In Annual International Cryptology Conference, pages 451-468 Springer, 2008.
[6] A. Bittau, U. Erlingsson, P. Maniatís, I. Mironov, A. Raghunathan, D. Lie, M. Rudominer, U. Kode, Tinnes, and B. Seefeld. Prochlo: Strong privacy for analyties in the crowd. In Proceedings of the 26th Symposium Operating Systems Principles, pages 441-459 2017.
[7] M. Bun, J. Nelson, and U. Stemmer. Heavy hitters and the structure of local privacy. ACM Transactions on Algorithms (TALG), 15(4):1-40, 2019.
[8] T. H. Chan, E. Shi, and D. Song. Optimal lower bound for differentially private multi-party aggregation. In European Symposium Algorithms, pages 277-288. Springer, 2012.
[9] A. Cheu, A. Smith, J. Ullman, D. Zeber, and M. Zhilyaev. Distributed differential privacy via shuffling. In Anmual International Conference on the Theory and Applications of Cryptographic Techniques, pages 375-403. Springer, 2019.
[10] A. Dubey and A. Pentland. Differentiaaly-priine federated linear bandits. arXiv preprint arXiv:2010.1 1425,, 2020.
[11] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[12] U. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, S. Song, K. Talwar, and A. Thakurta. Encode, shuffle, analyze privacy revisited: Formalizations and empirical evaluation. arXiv preprint arXiv:2001. 036618, 2020.
[13] Ó. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, and A. Thakurta. Amplifi- cation by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Anmual ACM-SIAM Symposium on Discrete Algorithms, pages 2019.
privvayy
SIAM,
[14] H. Esfandiari, A. Karbasi, Mehrabian, and V. Mirrokni. Regret bounds for batched bandits. arXiv preprint arXiv: 190.0499,, 2019.
[15] 7. Gao, Y. Han, 7. Ren, and 7.. Zhou. Batched multi-armed bandits problem. arXiv preprint arXiv: 1904. 01 7633, 2019.
[16] B. Ghazi, R. Pagh, and A. Velingker. Scalable and differentially private distributed aggregation in the shuffled model. arXiv preprint arXiv: 1906. 08320, 2019.
[17] A. Guha Thakurta and A. Smith. (nearly) optimal algorithms for private online learning in full-information and bandit settings. Advances in Neural Information Processing Systems, 26:2733-2741, 2013.
[18] W. Hoeffding. Probability inequalities for sums of bounded random variables. In The Collected Works of Wassily Hoeffding, pages 409-426. Springer, 1994.
[19] Jain, Kothari, and A. Thakurta. Differentially private online learning. In Conference on Learning Theory, pages 24-1. JMLR Workshop and Conference Proceedings, 2012.
10

---

[20] E. Kaufmann, O. Cappé, and A. Garivier. On the complexity of best-arm identification in multi-armes bandit models. The Journal of Machine Learning Research, 17(1):1-42, 2016.
[21] N. Mishra and A. Thakurta. (nearly) optimal differentially private stochastic multi-arm bandits. In Proceedings of the Thirty-Firsl Conference Uncertainty in Artificial Intelligence, pages 592-601, 2015.
[22] W. Ren, X. Zhou, J. Liu, and N. B. Shroff. Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.03/2), 2020.
[23] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
[24] T. Sajed and O. Sheffet. An optimal private stochastic- mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning, pages 5579-5588. PMLR, 2019.
[25] R. Shariff and O. Sheffet. Differentially private contextual linear bandits. arXiv preprint arXiv:1810.00068, 2018.
[26] C. Shi and C. Shen. Federated multi-armed bandits. arXiv preprint arXiv:1101. 2021.
[27] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4))255-994, 1933.
[28] A. Tossou and C. Dimitrakakis. Algorithmss for differentially private multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[29] K. Zheng, T. Cai, W. Huang, Z. Li, and L. Wang. Locally differentially private (contextual) bandits learning, arXiv preprint arXiv:2006.0077 2020.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We discuss the importance of preserving using privacy the introduction. The main contribution of this work is to help preserve user privacy, which clearly has positive societal impact. cannot foresee any negative societal impact related to our theoretical work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All the assumptions appear either in the Sections T and Section 2
(b) Did you include complete proofs of all theoretical results? [Yes] All the proofs which don't I appear in the body of the paper, do appear in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "xBdK39EiskQ", "reviewer": "Reviewer_gRZS", "summary": "This paper studies the problem of multi-armed bandits under the shuffled model of DP. The authors provide two algorithms, one of which almost matches the regret of the best known algorithms for the centralized model. To do so, they give a variant of the arm elimination algorithm that leverages the private summation algorithm by Cheu et al. [9] and uses exponentially growing batches. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written. It is easy to follow and read through, and it addresses a timely and important topic in our community. It's interesting to see that one can immediately get an optimal (up to log(T) factor) algorithm under the shuffled model of DP. \n\nThat said, the paper does not present any fundamentally new algorithms or analyses and does not provide any experimentation or analyses for things other than regret (e.g. communication efficiency: number and length of messages to be shuffled). The shuffled model of DP has been (almost fully) solved for several canonical problems: summation, histograms, convex optimization, etc. And it seems that one can simply just apply the shuffled algorithms in those canonical settings as a black-box to get a \"good enough\" algorithm for other problem domains (like the one studied in this paper). From that perspective, it is unclear how novel or deep these attempts are. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The main limitation of this work stems from the lack of algorithmic novelty and lack of depth in the provided proofs. The solution seems to be largely a mix and match of known results (applying the private summation protocol as a black-box to the arm elimination algorithm with very little changes). I think shaving the log(T) term would have been interesting since it does not seem as straightforward. \n\n2. Since the algorithms are not as novel, I would have expected/hoped that the authors give other types of analyses (e.g. analyzing communication costs or implementation issues related to shuffling and conducting extensive experiments). ", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OWT7ze4g2Bi", "reviewer": "Reviewer_3BED", "summary": "This paper proposes multi-armed bandit algorithms that are differentially private in the shuffle model.\nFor the proposed algorithms,\na gap-dependent regret bound and a distribution independent regret bound,\nwhich are better than known regret bounds achieved by exsiting algorithms that are differentially private in the local model.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThis is the first work that consider bandit problems under the shuffle model of differential privacy.\nThe proposed algorithms heavily rely on a well-known arm elimination approach and \na private binary summation mechanism in the shuffle model.\nGiven these two existing framework,\nthe proposed algorithms appear to be natural ones to come up with.\n\n\nQuality and clarity: \n\nThe paper is clearly written and well organized.\n\nSignificance:\n\nGiven a private binary summation mechanism with $\\tilde{\\Theta}(1 / \\varepsilon)$-error in the shuffle model,\nthe results of the proposed multi-armed bandit algorithms and theoretical analysis are not very surprising.\nOn the other hand,\nthe work steadily advances the research area of optimal trade-offs between privacy and decision-making efficiency.\nIt would be better if there was a detailed explanation of why the shuffle model is important in the context multi-armed bandit problems.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think limitations and societal impact have been well discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OUvIAwm3vIh", "reviewer": "Reviewer_321T", "summary": "The authors give a differentially private algorithm for the Multi-Armed Bandit (MAB) problem in the shuffle model. The algorithm attains similar privacy guarantees to the local model, but has almost matched regret bounds compared to the best known centralized model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well written, I can quickly get the contributions of this work. Although I didn't follow the state-of-the-art in private MAB well, based on what the authors summarize I think this is a good work to the community. This also makes me find the shuffle model interesting. In the private MAB setting, this model somehow achieves the best-of-both-world: having similar privacy as local models while having similar regret as centralized models. The model seems very promising and is worthy of further investigation in the future.  \n- My main question/confusion is about the batches. Specifically, the regret definition above Line 130 is confusing. First, the definition of B is not very clear to me. It doesn't appear in the algorithm (but the authors say it is used by the algorithm during its run) or elsewhere, so it is a little bit hard for me to relate. A formal definition and more explanation to it would be helpful. Additionally, I assume $\\mathbb{E}[\\sum_{b\\in B}\\sum_{i\\in b}r_i]$ corresponds to the accumulated reward of the learner but why is there no dependence on T? And why does the last equality hold? I think there are more explanations required there.   \n- Related to the last comment, I think the role of batches should be better explained. It seems that in the standard online learning problems, we have m=1. But in this paper the authors set something specific in the first algorithm and even make it time-varying in the second algorithm. Is this a parameter that the algorithm can set arbitrarily? What does it mean in the real applications? I think these are something the authors may clarify. \n- In footnote 6, the authors claim that their method also works for UCB. Then how about the regret bounds? Why do authors end up choosing arm elimination based algorithms instead of UCB based?\n- In the caption of Table 1, where is LB?\n\n=========\n\nI thank the reviewer for the detailed responses. Now I have a better understanding of the shuffle model. After reading all the review and the author responses, I share a similar concern with other reviewers regarding the algorithmic novelty. It seems like the main algorithmic contribution is basically introducing the exponentially growing batch size (non-trivial) to improve the regret. Although this idea and the analysis are non-trivial, the overall contributions are slightly weaker than I thought. I decrease my score accordingly, but I still tend to recommend  the acceptance. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Lsx6i9vmPSe", "reviewer": "Reviewer_qxTz", "summary": "The paper proposes a shuffle model of DP for multi-armed bandits. The paper proposes two algorithms, SDP-AE and VB-SDP-AE, for bandits with shuffle DP and Bernoulli rewards. Distribution-dependent and independent regret bounds are derived for both the algorithms. VB-SDP-AE achieves additive regret due to privacy which has same order of dependence on $\\epsilon$ but logarithmically worse dependence on $T$ than centralised DP.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThis paper is the first to propose shuffled DP for multi-armed bandits with Bernoulli rewards. It incrementally proposes two algorithms for SDP model and also provides theoretical analysis to support their efficiencies and shortcomings. This allows us to find an alternative model of DP which can have future applications.\n\nQuality:\n+ The submission is technically sound. The definitions and the setup are clear. The bounds and proves are also understandable enough.\n+ The intermediate nature of SDP wrt LDP and centralised DP is clear from Table 1 and following discussions.\n- But lack of lower bound makes it questionable whether it is an artefact of the algorithm or fundamentally unavoidable. A discussion on that or a lower bound will really help.\n- The paper lacks experimental evaluation of the SDP algorithm against LDP and centralised DP algorithms. DP is a practical problem and the SDP model proposed is an interesting alternative. Though this is a generic problem of this category of papers, an experimental analysis (even in a simple synthetic setup) does not only firms the claims but also increases its applicability.\n+ The algorithms proposed in this paper are novel and add new techniques merging private bandits and SDP.\n+ In general, the authors compare it well with existing problem-dependent bounds for private bandits. They also discuss the strength and weakness against them.\n- Though the authors have problem-independent bounds, there's no discussion along this line. There are multiple existing upper and lower minimax bounds in literature for both local and centralised DP in bandits. A discussion is needed to clarify how much SDP gets in both worst-case and problem-dependent domains.\n- Another issue is that often the problem-dependent lower bounds in private bandits assume $O(1/\\Delta_a)$ as the optimal problem dependent factor. But in bandit literature, it is long known that this dependence is suboptimal and the problem dependent factor should be $O(1/KL(R_a||R^*))$. This does not lessen the algorithm design or contribution but stating it clearly and using such existing bounds in private bandit literature rigorously clarifies the gaps in literature.\n\n\nClarity:\n+ The contributions of the papers and the problem setup are clear. \n+ In general the write-up is smooth and easy to follow.\n- Line 189-190: It is not really trivial for a reader from bandit community or even no experience with shuffled DP. It should be explained a bit more explicitly and rigorously even if it requires stating existing results.\n- It would be very helpful for a reader if a running example is used to explain SDP.  Authors can probably extend the accessible example in introduction (line 22-28).\n\n\nSignificance:\nThis introduces a new way of ensuing DP in bandits. Though it is presently limited to Bernoulli rewards,  extensions to practical settings with experimental evaluations can become useful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The paper in general talks about the weakness and strength of their bounds against that of centralised and local DP respectively. But it should also remind the reader that this is limited to Bernoulli reward only. In lines 63-64, it is loosely mentioned that it can be extended to continuous case but it should be discussed a bit more in details.\n2. The paper does not have a lower bound for shuffled DP model. Thus, it is worth mentioning that suboptimality of SDP here is unknown though it is compared with global and local DPs.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of differentially private multi-armed bandits which has been well-studied both in the local as well as global DP model. The paper proposes to study the recently popular shuffled model of privacy. The paper analyses two settings keeping a fixed batch and an exponentially increasing batch size showing the precise regret dependence with privacy loss. The latter algorithm with the increasing batch size represents the primary algorithmic contribution of the paper - the reviewers (after some discussion and a mixed verdict) agreed that the idea/analysis is non-trivial and somewhat novel as well as the fact that the same results as global DP can be achieved is a strength of the approach. 

I would like to point out that the paper's technical contribution is primarily in the space of multi-armed bandits and the privacy aspect is merely the part that actually defines the entire setup. Overall the paper contains some novel ideas and the algorithm/results are useful in general again in the space of multi-armed bandits. The authors should discuss the shuffle model better though and discuss the practical privacy impacts of their algorithm - in particular the exponentially increasing batch size, if they wish to make this paper more useful to their targeted application which is privacy. 

Overall the reviewers agree on the paper's contributions being (close to) but above the borderline for acceptance and that forms the rationale behind my recommendation. 
Decision: Accept (Poster)