Title: On the Complexity of Differentially Private Best-Arm Identification with Fixed Confidence
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

Multi-armed Bandits

Best Arm Identification

Fixed Confidence
Abstract: Best Arm Identification (BAI) problems are progressively used for data-sensitive applications, such as designing adaptive clinical trials, tuning hyper-parameters, and conducting user studies to name a few. Motivated by the data privacy concerns invoked by these applications, we study the problem of BAI with fixed confidence under $\epsilon$-global Differential Privacy (DP). First, to quantify the cost of privacy, we derive a lower bound on the sample complexity of any $\delta$-correct BAI algorithm satisfying $\epsilon$-global DP. Our lower bound suggests the existence of two privacy regimes depending on the privacy budget $\epsilon$. In the high-privacy regime (small $\epsilon$), the hardness depends on a coupled effect of privacy and a novel information-theoretic quantity, called the Total Variation Characteristic Time. In the low-privacy regime (large $\epsilon$), the sample complexity lower bound reduces to the classical non-private lower bound. Second, we propose AdaP-TT, an $\epsilon$-global DP variant of the Top Two algorithm. AdaP-TT runs in *arm-dependent adaptive episodes* and adds *Laplace noise* to ensure a good privacy-utility trade-off. We derive an asymptotic upper bound on the sample complexity of AdaP-TT that matches with the lower bound up to multiplicative constants in the high-privacy regime. Finally, we provide an experimental analysis of AdaP-TT that validates our theoretical results.

On the Complexity of Differentially Private Best-Arm Identification with Fixed Confidence

Abstract

Best Arm Identification (BAI) problems are progressively used for data-sensitive applications, such as designing adaptive clinical trials, tuning hyper-parameters, and conducting user studies to name a few. Motivated by the data privacy concerns invoked by these applications, we study the problem of BAI with fixed confidence under ϵ -global Differential Privacy (DP) . First, to quantify the cost of privacy, we derive a lower bound on the sample complexity of any δ -correct BAI algorithm satisfying ϵ -global DP. Our lower bound suggests the existence of two privacy regimes depending on the privacy budget ϵ . In the high-privacy regime (small ϵ ), the hardness depends on a coupled effect of privacy and a novel informationtheoretic quantity, called the Total Variation Characteristic Time . In the low-privacy regime (large ϵ ), the sample complexity lower bound reduces to the classical nonprivate lower bound. Second, we propose AdaP -TT , an ϵ -global DP variant of the Top Two algorithm. AdaP -TT runs in arm-dependent adaptive episodes and adds Laplace noise to ensure a good privacy-utility trade-off. We derive an asymptotic upper bound on the sample complexity of AdaP -TT that matches with the lower bound up to multiplicative constants in the high-privacy regime. Finally, we provide an experimental analysis of AdaP -TT that validates our theoretical results.

1 Introduction

Westudy the stochastic multi-armed bandit problem [LS20], which allows us to reflect on fundamental information-utility trade-offs involved in interactive sequential learning. Specifically, in a bandit problem, a learning agent is exposed to interact with K unknown probability distributions { ν 1 , . . . , ν K } with bounded expectations, referred as the reward distributions (or arms ). ν ≜ { ν 1 , . . . , ν K } is called a bandit instance . At every step t > 0 , the agent chooses to interact with one of the reward distributions ν A t for an A t ∈ [ K ] , and obtains a sample (or reward ) r t from it. The goal of the agent can be of two types: (a) maximise the reward accumulated over time, or equivalently to minimise the regret, and (b) to find the reward distribution (or arm) with highest expected reward. The first problem is called the regret-minimisation problem [ACBF02], while the second one is called the Best Arm

Identification (BAI) problem [KCG16]. In this paper, we focus on the BAI problem, i.e. to compute

With its advent in 1950s [Bec54, Bec58] and resurgence in last two decades [MT04, GGL12, JMNB14, KCG16, DKM19], BAI has been extensively studied with different structural assumptions (Fixed-confidence: [JN14]; Fixed-budget: [CL16]; Non-stochastic: [JT16]; Best-of-bothworlds: [AYBG + 18]; Linear: [SLM14]). In this paper, we specifically investigate the Fixed Confidence BAI problem , in brief FC-BAI, that yields a δ -correct recommendation ˆ a ∈ [ K ] satisfying Pr(ˆ a = a ⋆ ) ≤ δ . FC-BAI is increasingly deployed for different applications, such as clinical trials [AKR21], hyper-parameter tuning [LJD + 17], communication networks [LPJ22], online advertisement [CLK + 14], crowd-sourcing [ZCL14], user studies [LEHT22], and pandemic mitigation [LVR + 19] to name a few. All of these applications often involve the sensitive and personal data of users, which raises serious data privacy concerns [TBD + 16], as illustrated in Example 1.

̸

Example 1 (Adaptive dose finding trial) . In a dose-finding trial, one physician decides K possible dose levels of a medicine based on preliminary studies ( Typically, K ∈ { 3 , . . . , 10 } in practice [AKR21] ) . At each step t , a patient is chosen from a local pool of volunteers and a dose level a t ∈ [ K ] is applied to the patient. Following that, the effectiveness of the dose on the patient, i.e. r t ∈ R is observed. The goal of the physician is to recommend after the trial, which dose level is most effective on average, i.e. the dose level a ∗ that maximises the expected reward. Here, every application of a dose level and the patient's reaction to it exposes information regarding the medical conditions of the patient . Additionally, at each step t of an adaptive sequential trial, the physician can use an FC-BAI algorithm that observes the previous history of dose levels { a s } s<t and their effectiveness { r s } s<t to decide on the next dose level a t to test. When releasing the experimental findings of the trial to health authorities, the physician should thoroughly detail the experimental protocol. This includes the dose allocated to each patient { a s } s ≤ t and the final recommended dose level a ⋆ . Thus, even if the sequence of reactions to doses { r s } s ≤ t is kept secret, publishing the sequence of chosen dose levels { a s } s ≤ t and the final recommended dose level a ⋆ computed using the history can leak information regarding patients involved in the trial .

This example demonstrates the need for privacy in best-arm identification. In this paper, we investigate privacy-utility trade-offs for a privacy-preserving algorithm in FC-BAI . Specifically, we use the celebrated Differential Privacy (DP) [DR14] as the framework to preserve data privacy. DP ensures that an algorithm's output is unaffected by changes in input by a single data point. By limiting the amount of sensitive information that an adversary can deduce from the output, DP renders an individual corresponding to a data point ' indistinguishable '. A popular way to achieve DP is to inject a calibrated amount of noise, from a Laplace [DR14] or Gaussian distribution [DRS22], into the algorithm. The scale of the noise is set to be proportional to the algorithm's sensitivity and inversely proportional to the privacy budget ϵ . Specifically, we study ϵ -global DP , where users trust the centralised decision-maker with access to the raw sensitive rewards. For example, in an adaptive dose-finding trial, the patients trust the physician conducting the trial. Thus, at any time t , she has access to all the true history { a s , r s } s<t , and it is her duty to design an algorithm such that publishing { a s } s ≤ t and the recommended optimal dose a ⋆ obeys ϵ -DP given the sensitive input , i.e. the effectiveness of the dose levels on the patients { r s } s ≤ t . We define the notion of ϵ -global DP for BAI rigorously in Section 2.

For different settings of bandits, the cost of ϵ -global DP and optimal algorithm design techniques are widely studied in the regret-minimization problem [MT15, TD16, SS19, SS18, NR18, BDT19, AB22]. Recently, a problem-dependent lower bound on regret of stochastic multi-armed bandits with ϵ -global DP and an algorithm matching the regret lower bound is proposed by [AB22]. In contrast, DP is meagerly studied in the FC-BAI problem of bandits [SS19, KNSS21]. Though efficient algorithm design in FC-BAI literature is traditionally propelled by deriving tight lower bounds, we do not have any explicit sample complexity lower bound for FC-BAI satisfying ϵ -global DP. Here, by an 'efficient' algorithm, we refer to the FC-BAI algorithms that aim to minimise the expected number of samples required (alternatively called, expected stopping time ) to find a δ -correct recommendation. Presently, we know neither the minimal cost in terms of sample complexity for ensuring DP in FC-BAI, nor the feasibility of efficient algorithm design to achieve the minimal cost.

Motivated by this gap in the literature, we aim to address two questions:

What is the fundamental hardness of ensuring Differential Privacy in the Best-Arm Identification with fixed confidence problem ( ϵ -DP-FC-BAI) in terms of a lower bound on the sample complexity?

How to design an efficient ϵ -DP-FC-BAI algorithm that achieves the lower bound order-optimally?

Our contributions. These questions have led to the following contributions:

1. Hardness as lower bounds: We commence our study by deriving the lower bound on sample complexity (or expected stopping time) of any FC-BAI algorithm ensuring ϵ -global DP, in brief ϵ -DP-FC-BAI (Section 3). In Theorem 2, we prove that the sample complexity of ϵ -DP-FC-BAI depends on the minimum of two information-theoretic quantities one that depends on privacy, and the other that originates from the classical sample complexity of non-private FC-BAI [KCG16]. The term dependent on privacy depends on the privacy budget ϵ , and a novel information-theoretic quantity, referred to as the Total Variation Characteristic Time T ∗ TV . T ∗ TV depends on the Total Variation (TV) distance between the reward distributions in the bandit problem and corresponding most confusing instance. The lower bound also indicates that, in a similar spirit as the regret minimisation with ϵ -global DP [AB22], there are two regimes of hardness for ϵ -DP-FC-BAI. For lower level of privacy (i.e. higher ϵ ), the sample complexity of ϵ -DP-FC-BAI is identical to that of the non-private FC-BAI. But for higher level of privacy (i.e. lower ϵ ), the sample complexity depends on ϵ and T ∗ TV .

2. Algorithm design: Following the lower bounds, we aim to design an efficient ϵ -DP-FC-BAI algorithm that simultaneously achieves the lower bound order-optimally and is computationally efficient (Section 4). Due to the superior empirical performance and computational efficiency of Top Two algorithms, we design an ϵ -DP variant of Top Two algorithms, named AdaP -TT . Specifically, we show two simple design techniques, i.e. adaptive episodes for each arm and Laplacian mechanism, if properly used, lead to AdaP -TT from the non-private TTUCB [JD22]. We further derive an asymptotic (as δ → 0 ) upper bound on the sample complexity of AdaP -TT (Theorem 5). In the high-privacy regime, the sample complexity upper bound of AdaP -TT coincides with the lower bound up to multiplicative constants. Thus, it is an order-optimal ϵ -DP-FC-BAI algorithm in this regime, whereas the looseness in the low-privacy regime is as same as the looseness of the non-private Top Two algorithm. In Section 5, we experimentally show that AdaP -TT is more sample efficient than the existing ϵ -DP-FC-BAI algorithm, i.e. DP-SE [SS19]. We also show that its sample complexity is independent of the privacy budget in the low-privacy regime, as already indicated by the lower bound.

3. Technical tools: (a) To derive the lower bound, we provide an ϵ -global DP version of the transportation lemma [KK13] (Lemma 1), which we prove using a sequential coupling argument. We also define and study the TV characteristic time ( T ⋆ TV ) quantifying the hardness of FC-BAI in high privacy regimes (Proposition 1). (b) To design the algorithm, we propose a generic wrapper, which adapts the existing FC-BAI algorithm to tackle DP-FC-BAI . It builds on two components: (i) Adaptive episodes with per-arm doubling and forgetting, (ii) A private GLR stopping rule obtained by plugging in private empirical means in the non-private GLR stopping rule used of FC-BAI with Gaussian distributions. To use this proposed wrapper, one can choose among the numerous existing sampling rules to tackle FC-BAI. In this work, we consider the Top Two algorithms since they have good theoretical guarantees and empirical performance. To provide the sample complexity upper bound, we study step-by-step the effects of doubling, forgetting, and adding noise on the performance of the algorithm. Building on [JDB + 22], we provide a generic analysis of the class of Top Two algorithms when combined with our wrapper.

1.1 Related works

Lower bound. Efficient algorithm design in BAI literature is propelled by the derivation of lower bounds on sample complexity. [KCG16] derive the first lower bounds for classical fixed-confidence BAI setting without privacy, which is further improved in [GK16] by introducing KL characteristic time T ⋆ KL (Corollary 1). Motivated by this, we prove the first-known lower bound on sample complexity for FC-BAI with ϵ -global DP (Theorem 2). The proof employs a similar sequential coupling argument as in the regret lower bound for bandits with ϵ -global DP [SS18, AB22]. This similarity is also reflected in the existence of two privacy regimes depending on the privacy budget ϵ . And for both lower bounds, the Total Variation (TV) appears to be the information-theoretic measure that captures the hardness in the high privacy regime. Specifically, the TV characteristic time ( T ⋆ TV , Corollary 1) serves as the BAI counterpart to the TV-distinguishability gap ( t inf ) in the problem-dependent regret lower bound for bandits with ϵ -global DP as in [AB22, Theorem 3].

Algorithms for BAI with fixed confidence (FC-BAI). The optimal sample complexity for the non-private FC-BAI problem (i.e., T ⋆ KL ) is well-understood [GK16], and algorithms are proposed with the aim to achieve this lower bound. Early approaches involved Successive Elimination (SE) based algorithms [EDMMM06] with uniform sampling to find the optimal arm. Inspired by the success of Upper Confidence Bound (UCB) algorithms in the regret setting, the Lower Upper Confidence Bound (LUCB) algorithm was proposed [GGL12]. However, neither SE nor LUCB algorithms achieve asymptotic optimality. The Track-and-Stop (TnS) algorithm introduced in [GK16] was the first to asymptotically achieve the exact optimal sample complexity T ⋆ KL . TnS attains asymptotic optimality by solving a plug-in estimate of the lower bound optimization problem at each step. The game-based approach presented in [DKM19] relaxes this requirement by casting the optimization problem as an unknown game and proposing sampling rules based on iterative strategies to estimate and converge to its saddle point. Finally, the Top Two algorithms arose as an identification strategy based on the praised Thompson Sampling algorithm for regret minimization [Rus16]. In recent years, numerous variants have been analyzed and shown to be asymptotically near optimal [JDB + 22]. At every step, a Top Two sampling rule selects the next arm to sample from among two candidate arms, a leader and a challenger. In addition to their great empirical performance, and easy implementation compared to TnS and Game-based algorithms, the Top Two algorithms achieve near asymptotic optimality. In Sec. 4, we derive an ϵ -global DP version of a Top Two algorithm: AdaP -TT .

ϵ -DP BAI algorithms. DP-SE [SS19] is an ϵ -global DP version of the Successive Elimination algorithm. Although the algorithm was proposed and analysed for the regret minimisation setting in [SS19], it is possible to derive a sample complexity from the analysis in [SS19]. We compare in-depth DP-SE and AdaP -TT , both theoretically (Section 4) and experimentally (Section 5). In both aspects, our proposed algorithm AdaP -TT outperforms DP-SE . Another adaptation of DP-SE, namely DP-SEQ, is proposed in [KNSS21] for the problem of privately finding the arm with the highest quantile at a fixed level. But this is a different setting of interest than the present paper. [RBCS23] also studies privacy for BAI under fixed confidence but with multiple agents. They propose and analyse the sample complexity of DP-MASE, a multi-agent version of DP-SE. They show that multi-agent collaboration leads to better sample complexity than independent agents, even under privacy constraints. While the multi-agent setting with federated learning allows tackling large-scale clinical trials taking place at several locations simultaneously, we study the single-agent setting, which is relevant for many small-scale clinical trials (see Example 1).

2 Differential privacy and best arm identification

Background: Differential Privacy (DP). DP ensures protection of an individual's sensitive information when her data is used for analysis. A randomised algorithm satisfies DP if the output of the algorithm stays almost the same, regardless of whether any single individual's data is included in or excluded from the input. This is achieved by adding controlled noise to the algorithm's output.

Definition 1 ( ( ϵ, δ ) -DP [DR14]) . A randomised algorithm A satisfies ( ϵ, δ ) -Differential Privacy (DP) if for any two neighbouring datasets D and D ′ that differ only in one entry, i.e. d Ham ( D , D ′ ) = 1 , and for all sets of output O ⊆ Range( A ) ,

where the probability space is over the coin flips of the mechanism A , and for some ( ϵ, δ ) ∈ R ≥ 0 × R ≥ 0 . If δ = 0 , we say that A satisfies ϵ -DP . A lower privacy budget ϵ implies higher privacy.

The Laplace mechanism [DNPR10, DR14] ensures ϵ -DP by injecting controlled random noise into the output of the algorithm, which is sampled from a calibrated Laplace distribution (as specified in Theorem 1). We use Lap ( b ) to denote the Laplace distribution with mean 0 and variance 2 b 2 .

Theorem 1 ( ϵ -DP of Laplace mechanism (Theorem 3.6, [DR14])) . Let us consider an algorithm f : X → R d with sensitivity s ( f ) ≜ max D , D ′ s.t |D-D ′ | Hamming =1 | f ( D ) -f ( D ′ ) | 1 . Here, ∥·∥ 1 is the L 1

norm on R d . If d noise samples { N i } d i =1 are generated independently from Lap ( s ( f ) ϵ ) , then the output injected with the noise, i.e. f ( D ) + [ N 1 , . . . , N d ] , satisfies ϵ -DP.

Background: BAI with fixed confidence. Now, we describe the canonical best-arm identification problem with fixed confidence ( FC-BAI ). BAI is a variant of pure exploration, where the goal is to

Algorithm 1 Sequential interaction between a BAI strategy and users

and a recommen-

identify the optimal arm. In FC-BAI, the learner is provided with a confidence level 1 -δ ∈ (0 , 1) 1 . Learner aims to recommend an arm that is optimal with probability at least 1 -δ , while using as few samples as possible. To achieve this, the learner defines a FC-BAI strategy to interact with the bandit instance ν = { ν a : a ∈ [ K ] } . We denote the action played at step t by a t , and the corresponding observed reward by r t ∼ ν a t . H t = ( a 1 , r 1 , . . . , a t , r t ) is the history of actions played and rewards collected until time t . We augment the action set by a stopping action ⊤ , and write a t = ⊤ to denote that the algorithm has stopped before step t . A FC-BAI strategy π is composed of i. A pair of sampling and stopping rules (S t : H t -1 →P ([ | 1 , K | ] ∪ {⊤} )) t ≥ 1 . For an action a ∈ [ K ] , S t ( a |H t -1 ) denotes the probability of playing action a given history H t -1 . On the other hand, S t ( ⊤ | H t -1 ) is the probability of the algorithm halting given H t -1 . For any history H t -1 , a consistent sampling and stopping rule S t satisfies S t ( ⊤ | H t -1 ) = 1 if ⊤ has been played before t .

ii. A recommendation rule (Rec t : H t -1 →P ([ | 1 , K | ])) t> 1 . A recommendation rule dictates Rec t ( a |H t -1 ) , i.e. the probability of returning action a as a guess for the best action given H t -1 .

We denote by τ the stopping time of the algorithm, i.e. the first step t demonstrating a t = ⊤ . A BAI strategy π is called δ -correct for a class of bandit instances M , if for every instance ν ∈ M , π recommends the optimal action a ⋆ ( ν ) = arg max a ∈ [ K ] µ a with probability at least 1 -δ , i.e. P ν ,π ( τ < ∞ , ̂ a = a ⋆ ( ν )) ≥ 1 -δ .

FC-BAI with ϵ -global DP ( ϵ -DP-FC-BAI). Now, we formally define ϵ -global DP for FC-BAI, where the BAI strategy (a.k.a. the centralised decision maker) is trusted with all the intermediate rewards. We represent each user u t by the vector x t ≜ ( x t, 1 , . . . , x t,K ) ∈ R K , where x t,a represents the potential reward observed, if action a was recommended to user u t . Due to the bandit feedback, only r t = x t,a t ∼ ν a t is observed at step t . We use an underline to denote any sequence. Thus, we denote the sequence of sampled actions until T as a T = ( a 1 , . . . , a T ) . We further represent a set of users { u t } T t =1 until T by the table of potential rewards d T ≜ { x 1 , . . . , x T } ∈ ( R K ) T . First, we observe that d T is the sensitive input generated through interaction with the users, and ( a T , ̂ a, T ) is the output of the BAI strategy. Hence, we define the probability that the BAI strategy π samples the action sequence a T , recommends the action ̂ a , and halts at time T , as

where T users under interaction are represented by the table of potential rewards d T

Thus, a BAI strategy satisfies ϵ -global DP if the probability defined in Eq. (2) is similar when the BAI strategy interacts with two neighbouring tables of rewards differing by a user (i.e. a row in d T ).

Definition 2 ( ϵ -global DP for BAI) . A BAI strategy satisfies ϵ -global DP , if for all T ≥ 1 , all neighbouring table of rewards d T and d ′ T , i.e. d Ham ( d T , d ′ T ) = 1 , all sequences of sampled actions a T ∈ [ K ] T and recommended actions ̂ a ∈ [ K ] we have that

Definition 2 can be seen as a BAI counterpart of the ϵ -global DP definition proposed in [AB22] for regret minimization. We demonstrate the BAI strategy-Users interaction in Algorithm 1.

Remark 1. It is possible to consider that the output of a BAI strategy is only the final recommended action ˆ a , i.e. not publishing the intermediate actions a T . This gives a weaker definition of privacy compared to Definition 2, since the latter defends against adversaries that may look inside the execution of the BAI strategy, i.e. pan-privacy [DNP + 10]. In addition, Definition 2 is needed in practice. For example, in the case of dose-finding (Example 1), the experimental protocol, i.e. the intermediate actions, needs to be published too.

The goal in ϵ -DP-FC-BAI is to design a δ -correct ϵ -global DP algorithm, with τ as small as possible.

3 Lower bound on sample complexity for FC-BAI with ϵ -global DP

The central question that we address in this section is

How many additional samples a BAI strategy must select for ensuring ϵ -global DP?

In response, we prove a lower bound on the sample complexity of any δ -correct ϵ -DP BAI strategy. Our lower bound features problem-dependent characteristic times reminiscent of the FC-BAI setting.

̸

Let ν ≜ { ν a : a ∈ [ K ] } be a bandit instance, consisting of K arms with finite means { µ a } a ∈ [ K ] . Now, we define the set of alternative instances as Alt( ν ) ≜ { λ : a ⋆ ( λ ) = a ⋆ ( ν ) } , i.e. the bandit instances with a different optimal arm than ν . For two probability distributions P , Q on the same measurable space (Ω , F ) , the Total Variation (TV) distance is defined as TV( P ∥ Q ) ≜ sup A ∈F { P ( A ) -Q ( A ) } , while the KL divergence (or relative entropy) is KL( P ∥ Q ) ≜ ∫ log ( d P d Q ( ω ) ) d P ( ω ) , when P ≪ Q , and ∞ otherwise. We denote the probability simplex by Σ K ≜ { ω ∈ [0 , 1] K : ∑ K a =1 ω a = 1 } .

First, we derive an ϵ -global DP variant of the 'transportation' lemma, i.e. Lemma 1 in [KCG16].

Lemma 1 (Transportation lemma under ϵ -global DP) . Let δ ∈ (0 , 1) and ϵ > 0 . Let ν be a bandit instance and λ ∈ Alt( ν ) . For any δ -correct ϵ -global DP BAI strategy, we have that

Proof sketch. We use Sequential Karwa-Vadhan Lemma [AB22, Lemma 2] with a data-processing inequality in the BAI canonical model. Extra care is needed to deal with the stopping times in the coupling, compared to a fixed horizon T in regret minimization . The proof is deferred to Appendix B.

Leveraging Lemma 1, we derive a sample complexity lower bound for any ϵ -DP-FC-BAI strategy.

Theorem 2 (Sample complexity lower bound for ϵ -DP-FC-BAI) . Let δ ∈ (0 , 1) and ϵ > 0 . For any δ -correct ϵ -global DP BAI strategy, we have that

Comments on the lower bound. Similar to the lower bound for the non-private BAI [GK16], the lower bound of Theorem 2 is the value of a two-player zero-sum game between a MIN player and MAX player. MIN plays an alternative instance λ close to ν in order to confuse MAX. The latter plays an allocation ω ∈ Σ K to explore the different arms, with the purpose of maximising the divergence between ν and the confusing instance λ that MIN played. On top of the KL divergence present in the non-private lower bound, our bound features the TV distance that appears naturally when incorporating the ϵ -global DP constraint. The proof is deferred to Appendix B. In order to compare the lower bound of an ϵ -global BAI strategy with the non-private lower bound of [GK16], we relax Theorem 2 to further derive a simpler bound, as in Corollary 1.

Corollary 1. For any δ -correct ϵ -global DP BAI strategy, we have that

where ( T ⋆ d ( ν )) -1 ≜ sup ω ∈ Σ K inf λ ∈ Alt( ν ) ∑ K a =1 ω a d ( ν a , λ a ) , and d is either KL or TV .

Proof. The proof is direct by observing that T ⋆ ( ν ; ϵ ) ≥ T ⋆ KL ( ν ) and T ⋆ ( ν ; ϵ ) ≥ 1 6 ϵ T ⋆ TV ( ν ) .

Comparison with the non-private lower bound. T ⋆ KL is the characteristic time in the non-private lower bound [GK16], and we refer to Section 2.2 of [GK16] for a detailed discussion on its properties. The sample complexity lower bound suggests the existence of two hardness regimes depending on ϵ , T ⋆ KL and T ⋆ TV . (1) Low-privacy regime : When ϵ > T ⋆ TV( ν ) / (6 T ⋆ KL ( ν )) , the lower bound retrieves the non-private lower bound, i.e. T ⋆ KL ( ν ) , and thus, privacy can be achieved for free . (2) High-privacy regime: When ϵ < T ⋆ TV ( ν ) / (6 T ⋆ KL ( ν )) , the lower bound becomes T ⋆ TV / (6 ϵ ) and ϵ -global DP δ -BAI requires more samples than non-private ones.

In the following proposition, we characterise T ⋆ TV for Bernoulli instances.

Proposition 1 (TV characteristic time for Bernoulli instances) . Let ν be a bandit instance, i.e. such that ν a = Bernoulli ( µ a ) and µ 1 > µ 2 ≥ · · · ≥ µ K . Let ∆ a ≜ µ 1 -µ a and ∆ min ≜ min a =1 ∆ a . We have that

̸

Proof sketch. The proof is direct by solving the optimisation problem defining T ⋆ TV and using that TV( Bernoulli ( p ) ∥ Bernoulli ( q )) = | p -q | . We refer to Appendix B for details.

Comment. The aforementioned bound on TV characteristic time for Bernoulli instances is ϵ -global DP parallel of the KL-characteristic time bound T ⋆ KL ( ν ) ≤ ∑ K a =1 ∆ -2 a [GK16]. Using Pinsker's inequality, one can connect the TV and KL characteristic times by T ⋆ TV ( ν ) ≥ √ 2 T ⋆ KL ( ν ) .

4 Algorithm design: Private Top Two with adaptive episodes ( AdaP -TT )

In this section, we propose AdaP -TT , an ϵ -global DP version of the TTUCB algorithm [JD22]. We show that AdaP -TT satisfies ϵ -global DP, is δ -correct, and has an asymptotic sample complexity that matches the high privacy lower bounds up to multiplicative constants.

TTUCB belongs to the family of Top Two algorithms [Rus16, SHM + 20, JDB + 22], which selects at each time two arms called leader and challenger, and sample among them. After initialisation, TTUCB uses a UCB-based leader and a Transportation Cost (TC) challenger, expressed by

̸

Here, (ˆ µ n , N n ) are the empirical means and counts on the whole history. The theoretical motivation behind the TC challenger comes from the theoretical lower bound in FC-BAI, which involves the KL-characteristic time T ⋆ KL ( µ ) = min β ∈ (0 , 1) T ⋆ KL ,β ( µ ) . For Gaussian distributions, we have

The maximiser of the above equation is denoted by ω ⋆ KL ,β ( µ ) , and is further referred to as the β -optimal allocation as it is unique. Let N a n,b denote the number of times arm b was pulled when a was the leader, and L n,a denotes the number of times arm a was the leader. In order to select the next arm to sample I n , TTUCB relies on K tracking procedures, i.e. set I n = B n if N B n n,B n ≤ βL n +1 ,B n , else I n = C n . This ensures that max a ∈ [ K ] ,n>K | N a n,a -βL n,a | ≤ 1 . Standing on this premise, we now describe how we design an ϵ -global DP extension of TTUCB.

Private algorithm design. As illustrated in Algorithm 2, AdaP -TT relies on three ingredients: adaptive episodes with doubling , forgetting , and adding calibrated Laplacian noise . (1) AdaP -TT maintains K episodes, i.e. one per arm. The private empirical estimate of the mean of an arm is only updated at the end of an episode, that means when the number of times that a particular arm was played doubles (Line 5). (2) For each arm a , AdaP -TT forgets rewards from previous phases of arm a , i.e. the private empirical estimate of arm a is only computed using the rewards collected in the last phase of arm a (Line 8). This assures that the means of each arm are estimated using a non-overlapping sequence of rewards. (3) Thanks to this doubling and forgetting , AdaP -TT is ϵ -global DP as soon as each empirical mean (Line 9) is made ϵ -DP, and thus, avoiding any use of privacy composition. This is achieved by adding Laplace noise. We formalise this intuition in Lemma 2 of Appendix C.

Remark 2. The aforementioned generic wrapper can be used to construct a near-optimal differentially private version of any existing FC-BAI algorithm that deploys a sampling rule with the empirical means of rewards. In this work, we consider and rigorously analyse the Top Two algorithms since they demonstrate both good theoretical guarantees and empirical performance.

Theorem 3 (Privacy analysis) . For rewards in [0 , 1] , AdaP -TT satisfies ϵ -global DP.

Proof sketch. A change in one user only affects the empirical mean calculated at one episode of an arm, which is made private using the Laplace Mechanism and Lemma 2. Since the sampled actions, recommended action, and stopping time are computed only using the private empirical means, AdaP -TT satisfies ϵ -global DP thanks to post-processing lemma. We refer to Appendix C for details.

Private GLR stopping rule. We consider the private GLR stopping rule based on the private means and on the pulling counts from the last phase (Line 12), and recommend the arm with the highest private mean (Line 11). Lemma 4 yields a threshold function ensuring that any sampling rule is δ -correct, when using the private GLR stopping rule.

Theorem 4 ( δ -correctness) . Let δ ∈ (0 , 1) , ϵ > 0 . Let s > 1 and ζ be the Riemann ζ function. Let c k ( n, m, δ ) = 2 C G (log(( K -1) ζ ( s ) k s /δ ) / 2)+2log(4+log n )+2log(4+log m ) be the threshold without privacy. Given any sampling rule, the following threshold

with the GLR stopping rule yields a δ -correct algorithm for sub-Gaussian distributions. The function C G is defined in (15) . It satisfies C G ( x ) ≈ x +ln( x ) .

Remark. We observe that approximately c ϵ,k 1 ,k 2 ( n, m, δ ) ≈ 2 log(1 /δ )+(1 /n +1 /m ) log(1 /δ ) 2 /ϵ 2 .

̸

Proof sketch. Proving δ -correctness of a GLR stopping rule is done by leveraging concentration results. Specifically, we start by decomposing the failure probability P µ ( τ δ < + ∞ , ˆ a = a ⋆ ) into a non-private and a private part using the basic property of P ( X + Y ≥ a + b ) ≤ P ( X ≥ a )+ P ( Y ≥ b ) . The two-factor in front of c k 1 k 2 originates from the looseness of this decomposition. To remove it, we would need a tighter stopping threshold that jointly controls both the non-private and the private parts. We conclude using concentration results from sub-Gaussian random variables for the non-private part, and Laplace random variables for the private part.

̸

Theorem 5 (Asymptotic upper bound on expected sample complexity) . Let ( δ, β ) ∈ (0 , 1) 2 and ϵ > 0 . Combined with the private GLR stopping rule using threshold as in (4) , AdaP -TT is δ -correct and satisfies that, for all µ ∈ R K such that min a = b | µ a -µ b | > 0 ,

Proof sketch. We adapt the asymptotic proof of the TTUCB algorithm, which is based on the unified analysis of Top Two algorithms from [JDB + 22]. Below, we present high-level ideas of the proof and specify the effect of different elements of AdaP -TT on the expected sample complexity.

Consequences of Theorem 5. (1) The non-private TTUCB algorithm [JD22] achieves a sample complexity of T ⋆ KL ,β ( µ ) for sub-Gaussian random variables. The proof relies on showing that the empirical pulling counts are converging towards the β -optimal allocation ω ⋆ KL ,β ( µ ) . (2) The effect of doubling and forgetting is a multiplicative four-factor, i.e. 4 T ⋆ KL ,β ( µ ) . The first two-factor is due to forgetting since we throw away half of the samples. The second two-factor is due to doubling since we have to wait for the end of an episode to evaluate the stopping condition. (3) The Laplace noise only affects the empirical estimate of the mean. Since the Laplace noise has no bias and a sub-exponential tail, the private means will still converge towards their true values. Therefore, the empirical counts of AdaP -TT will also converge to ω ⋆ KL ,β ( µ ) asymptotically. (4) While the Laplace noise has little effect on the sampling rule itself, it changes drastically the dependency in log(1 /δ ) of the threshold used in the GLR stopping rule. The private threshold c ϵ,k 1 ,k 2 has an extra factor O (log 2 (1 /δ )) compared to the non-private one c k . Using the convergence towards ω ⋆ KL ,β ( µ ) , the stopping condition is met as soon as n T ⋆ KL ,β ( µ ) ≲ 2 log(1 /δ ) + ∆ 2 max 2 ϵ 2 T ⋆ KL ,β ( µ ) n log 2 (1 /δ ) . Solving the inequality for n concludes the proof while adding a multiplicative four-factor.

Algorithm 2 AdaP -TT . Private statistics are in red. Changes due to privacy are in blue.

̸

̸

̸

Discussion. For β = 1 / 2 , it is well known that T ⋆ KL , 1 / 2 ( µ ) ≤ 2 T ⋆ KL ( µ ) ≤ 8 ∑ a = a ⋆ ∆ -2 a . We consider Bernoulli instances ( 0 < ∆ min ≤ ∆ max < 1 ), where the gaps have the same order of magnitude, i.e. Condition 1 : there exists a constant C ≥ 1 such that ∆ max / ∆ min ≤ C . For such instances, there exists a universal constant c , such that

̸

Without privacy, i.e. ϵ → + ∞ , AdaP -TT yields a multiplicative eight-factor. On top of the four-factor due to doubling and forgetting, another multiplicative two comes from 2 c k 1 k 2 in Equation (4).

Comparison to the lower bound. For Bernoulli bandits verifying Condition 1 , the upper bound of Theorem 5 matches the T ⋆ TV ( µ ) /ϵ lower bound of Corollary 1 up to constants in the high-privacy regime, i.e. when ϵ ⪯ T ⋆ TV ( µ ) /T ⋆ KL ( µ ) . In the low-privacy regime, the upper bound reduces to T ⋆ KL , 1 / 2 ( µ ) . In Appendix E.7, we discuss in-depth why this difference is necessary for private BAI algorithms based on the GLR stopping rule, which poses an interesting open problem.

̸

Comparison to DP-SE. DP-SE is a private version of the successive-elimination algorithm studied in [SS19] for the regret minimisation setting. The algorithm samples active arms uniformly during phases of geometrically increasing length. Based on the private confidence bounds, DP-SE eliminates provably sub-optimal arms at the end of each phase. Due to its phased-elimination structure, DP-SE can be easily converted into an ϵ -DP-FC-BAI algorithm, where we stop once there is only one active arm left. In particular, the proof of Theorem 4.3 of [SS19] shows that with high probability any suboptimal arm a = a ⋆ is sampled no more than O (∆ 2 a +( ϵ ∆ a ) -1 ) . From this result, it is straightforward to extract a sample complexity upper bound for DP-SE, i.e. O ( ∑ a = a ⋆ ∆ -2 a + ∑ a = a ⋆ ( ϵ ∆ a ) -1 ) . This shows that DP-SE too achieves (ignoring constants) the high-privacy lower bound T ⋆ TV ( µ ) /ϵ for Bernoulli instances. However, due to its uniform sampling within the phases, DP-SE is less adaptive than AdaP -TT . Inside a phase, DP-SE continues to sample arms that might already be known to be bad, while AdaP -TT adapts its sampling rule based on the transportation costs that reflect the amount of evidence collected in favour of the hypothesis that the leader is the best arm. Finally, AdaP -TT has the advantage of being anytime, i.e. its sampling strategy does not depend on the risk δ .

̸

̸

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 5

10 6

10 7

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 5

10 6

10 7

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

5 Experimental analysis

We perform experiments to show that: (i) AdaP -TT has better empirical performance compared to DP -SE , and (ii) the transition between high and low-privacy regimes is reflected empirically.

Experimental setup. We compare the performances of AdaP -UCB and DP-SE for FC-BAI in different Bernoulli instances as in [SS19]. The first instance has means µ 1 = (0 . 95 , 0 . 9 , 0 . 9 , 0 . 9 , 0 . 5) and the second instance has means µ 2 = (0 . 75 , 0 . 7 , 0 . 7 , 0 . 7 , 0 . 7) . As a benchmark, we also compare to the non-private TTUCB. We set the risk δ = 10 -2 . We implement all the algorithms in Python (version 3 . 8 ) and on an 8-core 64-bits Intel i5@1.6 GHz CPU. We run each algorithm 100 times, and plot corresponding average and standard deviations of stopping times in Figure 1. We also test the algorithms on other Bernoulli instances and report the results in Appendix F.

Result analysis. a. Efficiency in performance. AdaP -TT requires less samples than DP-SE to provide a δ -correct answer. In the high privacy regime, i.e. small ϵ , AdaP -TT outperforms DP-SE in all the instances tested. In the low privacy regimes, i.e. large ϵ , both algorithms have similar performance that in the worst case is four times the samples required of TTUCB, as shown theoretically. b. Impact of privacy regimes. As indicated by the theoretical sample complexity lower bounds and upper bounds, the experimental performance of AdaP -TT demonstrates two regimes: a high-privacy regime (for ϵ < 0 . 2 ), where the stopping time of AdaP -TT depends on the privacy budget ϵ , and a low privacy regime (for ϵ > 0 . 2 ), where the performance of AdaP -TT does not depend on ϵ .

6 Conclusion and future works

We study FC-BAI with ϵ -global DP. We derive a sample complexity lower bound that quantifies the additional samples needed by a δ -correct BAI strategy in order to ensure ϵ -global DP. The lower bound further suggests the existence of two privacy regimes. In the low-privacy regime , no additional samples are needed, and privacy can be achieved for free . For the high-privacy regime , the lower bound reduces to Ω( ϵ -1 T ⋆ TV ) , and more samples are required . We also propose AdaP -TT , an ϵ -global DP variant of the Top Two algorithms, that runs in adaptive phases and adds Laplace noise. AdaP -TT achieves the high privacy regime lower bound up to multiplicative constants.

The upper bound matches the lower bound by a multiplicative constant in the high privacy regime, and is also loose in some instances in the low privacy regime, due to the mismatch between the KL divergence of Bernoulli distributions and that of Gaussian. It would be an interesting technical challenge to merge this gap. One possible direction to solve this issue is to use transportation costs tailored to Bernoulli for both the Top Two Sampling and the stopping. Another interesting direction would be to extend the proposed technique to other variants of pure DP, namely ( ϵ, δ ) -DP and RényiDP [Mir17], or other trust models, namely local DP [DJW13] and shuffle DP [Che21, GDD + 21].

Acknowledgments and Disclosure of Funding

This work has been partially supported by the THIA ANR program 'AI_PhD@Lille". A. Al-Marjani acknowledges the support of the Chaire SeqALO (ANR-20-CHIA-0020). D. Basu acknowledges the Inria-Kyoto University Associate Team 'RELIANT' for supporting the project, and the ANR JCJC for the REPUBLIC project (ANR-22-CE23-0003-01). We thank Emilie Kaufmann and Aurélien Garivier for the interesting conversations. We also thank Philippe Preux for his support.

References

[AB22] Achraf Azize and Debabrota Basu. When privacy meets partial information: A refined analysis of differentially private bandits. Advances in Neural Information Processing Systems , 35:32199-32210, 2022.

[ACBF02] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3):235-256, 2002.

[AKR21] Maryam Aziz, Emilie Kaufmann, and Marie-Karelle Riviere. On multi-armed bandit designs for dose-finding clinical trials. The Journal of Machine Learning Research , 22(1):686-723, 2021.

[ASZ21] Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private assouad, fano, and le cam. In Algorithmic Learning Theory , pages 48-78. PMLR, 2021.

[AYBG + 18] Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of both worlds: Stochastic & adversarial best-arm identification. In Conference on Learning Theory , pages 918-949. PMLR, 2018.

[BDT19] Debabrota Basu, Christos Dimitrakakis, and Aristide Tossou. Differential privacy for multi-armed bandits: What is it and what is its cost? arXiv preprint arXiv:1905.12298 , 2019.

[Bec54] Robert E Bechhofer. A single-sample multiple decision procedure for ranking means of normal populations with known variances. The Annals of Mathematical Statistics , pages 16-39, 1954.

[Bec58] Robert E Bechhofer. A sequential multiple-decision procedure for selecting the best one of several normal populations with a common unknown variance, and its use with various experimental designs. Biometrics , 14(3):408-429, 1958.

[Che21] Albert Cheu. Differential privacy in the shuffle model: A survey of separations. arXiv preprint arXiv:2107.11839 , 2021.

[CL16] Alexandra Carpentier and Andrea Locatelli. Tight (lower) bounds for the fixed budget best arm identification bandit problem. In Conference on Learning Theory , pages 590-604. PMLR, 2016.

[CLK + 14] Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. Advances in neural information processing systems , 27, 2014.

[CWZ21] T Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics , 49(5):2825-2850, 2021.

[DJW13] John C Duchi, Michael I Jordan, and Martin J Wainwright. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438. IEEE, 2013.

[DKM19] Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. Advances in Neural Information Processing Systems , 32, 2019.

[DNP + 10] Cynthia Dwork, Moni Naor, Toniann Pitassi, Guy N Rothblum, and Sergey Yekhanin. Pan-private streaming algorithms. In ICS , pages 66-80, 2010.

[DNPR10] Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N Rothblum. Differential privacy under continual observation. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 715-724. ACM, 2010.

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

[DRS22] Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):3-37, 2022.

[EDMMM06] Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

[GDD + 21] Antonious M Girgis, Deepesh Data, Suhas Diggavi, Ananda Theertha Suresh, and Peter Kairouz. On the renyi differential privacy of the shuffle model. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages 2321-2341, 2021.

[GGL12] Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. Advances in Neural Information Processing Systems , 25, 2012.

[GK16] Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR, 2016.

[JD22] Marc Jourdan and Rémy Degenne. Non-asymptotic analysis of a ucb-based top two algorithm. arXiv preprint arXiv:2210.05431 , 2022.

[JDB + 22] Marc Jourdan, Rémy Degenne, Dorian Baudry, Rianne de Heide, and Emilie Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems , 35:26791-26803, 2022.

[JDK23] Marc Jourdan, Rémy Degenne, and Emilie Kaufmann. Dealing with unknown variances in best-arm identification. International Conference on Algorithmic Learning Theory , 2023.

[JMNB14] Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory , pages 423-439. PMLR, 2014.

[JN14] Kevin Jamieson and Robert Nowak. Best-arm identification algorithms for multiarmed bandits in the fixed confidence setting. In 2014 48th Annual Conference on Information Sciences and Systems (CISS) , pages 1-6. IEEE, 2014.

[JT16] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics , pages 240-248. PMLR, 2016.

[KCG16] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17:1-42, 2016.

[KK13] Emilie Kaufmann and Shivaram Kalyanakrishnan. Information complexity in bandit subset selection. In Conference on Learning Theory , pages 228-251. PMLR, 2013.

[KK21] Emilie Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research , 22(246):1-44, 2021.

[KNSS21] Dionysios S Kalogerias, Kontantinos E Nikolakakis, Anand D Sarwate, and Or Sheffet. Quantile multi-armed bandits: Optimal best-arm identification and a differentially private scheme. IEEE Journal on Selected Areas in Information Theory , 2(2):534-548, 2021.

[KV17] Vishesh Karwa and Salil Vadhan. Finite sample differentially private confidence intervals, 2017.

[LEHT22] David E Losada, David Elsweiler, Morgan Harvey, and Christoph Trattner. A day at the races: using best arm identification algorithms to reduce the cost of information retrieval user studies. Applied Intelligence , 52(5):5617-5632, 2022.

[LJD + 17] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research , 18(1):6765-6816, 2017.

[LPJ22] Simon Lindståhl, Alexandre Proutiere, and Andreas Johnsson. Measurement-based admission control in sliced networks: A best arm identification approach. In GLOBECOM2022-2022 IEEE Global Communications Conference , pages 1484-1490. IEEE, 2022.

[LS20] Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

[LVR + 19] Pieter JK Libin, Timothy Verstraeten, Diederik M Roijers, Jelena Grujic, Kristof Theys, Philippe Lemey, and Ann Nowé. Bayesian best-arm identification for selecting influenza mitigation strategies. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018 , 2019.

[Mir17] Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pages 263-275. IEEE, 2017.

[MT04] Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research , 5(Jun):623-648, 2004.

[MT15] Nikita Mishra and Abhradeep Thakurta. (Nearly) optimal differentially private stochastic multi-arm bandits. In UAI , 2015.

[NKS19] Konstantinos E Nikolakakis, Dionysios S Kalogerias, and Anand D Sarwate. Optimal rates for learning hidden tree structures. arXiv preprint arXiv:1909.09596 , 2019.

[NR18] Seth Neel and Aaron Roth. Mitigating bias in adaptive data gathering via differential privacy. In International Conference on Machine Learning , pages 3720-3729. PMLR, 2018.

[RBCS23] Alexandre Rio, Merwan Barlier, Igor Colin, and Marta Soare. Multi-agent best arm identification with private communications. In Proceedings of the 40th International Conference on Machine Learning , 2023.

[Rus16] Daniel Russo. Simple bayesian algorithms for best arm identification. In Conference on Learning Theory , pages 1417-1418. PMLR, 2016.

[SHM + 20] Xuedong Shang, Rianne Heide, Pierre Menard, Emilie Kaufmann, and Michal Valko. Fixed-confidence guarantees for bayesian best-arm identification. In International Conference on Artificial Intelligence and Statistics , pages 1823-1832. PMLR, 2020.

[SLM14] Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

[SS18] Roshan Shariff and Or Sheffet. Differentially private contextual linear bandits. In Advances in Neural Information Processing Systems , pages 4296-4306, 2018.

[SS19] Touqir Sajed and Or Sheffet. An optimal private stochastic-mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning , pages 5579-5588. PMLR, 2019.

[TBD + 16] Katherine Tucker, Janice Branson, Maria Dilleen, Sally Hollis, Paul Loughlin, Mark J Nixon, and Zoë Williams. Protecting patient privacy when sharing patient-level data from clinical trials. BMC medical research methodology , 16(1):5-14, 2016.

[TD16] Aristide CY Tossou and Christos Dimitrakakis. Algorithms for differentially private multi-armed bandits. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.

[ZCL14] Yuan Zhou, Xi Chen, and Jian Li. Optimal pac multiple arm identification with applications to crowdsourcing. In International Conference on Machine Learning , pages 217-225. PMLR, 2014.

Appendix

Table of Contents

A Outline

The appendices are organized as follows:

The proof of our lower bound is detailed in Appendix B (Theorem 2).

The proof of the privacy of AdaP -TT is done in Appendix C (Theorem 3).

The AdaP -TT algorithm is presented in more details in Appendix D, and we show the δ -correctness of the non-private GLR stopping rule (Theorem 4).

The asymptotic upper bound on the expected sample complexity of AdaP -TT is proven in Appendix E (Theorem 5).

Extended experiments are presented in Appendix F.

B Lower bounds on sample complexity

In this section, we provide the proofs for the sample complexity lower bounds. First, we present the canonical model for BAI to introduce the relevant quantities. Then, we prove an ϵ -global version of the transportation lemma, i.e. Lemma 1. Using this lemma, we prove Theorem 2. Finally, we prove the formula expressing the TV characteristic time for Bernoulli instances.

B.1 Canonical model for BAI

Let ν ≜ { ν a : a ∈ [ K ] } be a bandit instance, consisting of K arms with finite means { µ a } a ∈ [ K ] . Now, we recall the interaction between a BAI strategy π and the bandit instance ν in the Protocol 1. The BAI strategy π halts at τ , samples a sequence of actions A τ , and recommends the action ˆ A . Let P ν ,π be the probability distribution over the triplets ( τ, A τ , ˆ A ) , when the BAI strategy π interacts with the bandit instance ν .

For a fixed T > 1 , a sequence of actions a T = ( a 1 , . . . , a T ) ∈ [ K ] T and a recommendation ˆ a ∈ [ K ] , we define the event E = { τ = T, A τ = a T , ˆ A = ˆ a } . We have that

where

Remark on the bandit feedback. Let π be an ϵ -DP BAI strategy. Let T ≥ 1 , a T ∈ [ K ] T a sequence sampled actions and ̂ a ∈ [ K ] a recommended actions. This time, let r T = { r 1 , . . . , r T } ∈ R T and r ′ T ∈ R T two neighbouring sequence of rewards, i.e. d Ham ( r T , r ′ T ) ≜ ∑ T t =1 1 { r t = r ′ t } = 1 . Consider the table of rewards d T consisting of concatenating r T colon-wise K times, i.e. d T t,i = r T t for all i ∈ [ K ] and all t ∈ [ T ] . Define d ′ T similarly with respect to r ′ T .

In this case, by definition of π , d T and d ′ T , it is direct that

Which means that

In other words, if π is ϵ -pure DP for neighbouring table of rewards d T , then π is also ϵ -pure DP for neighbouring sequence of observed rewards r T .

̸

B.2 Transportation lemma under ϵ -global DP: Proof of Lemma 1

Lemma 1 (Transportation lemma under ϵ -global DP) . Let δ ∈ (0 , 1) and ϵ > 0 . Let ν be a bandit instance and λ ∈ Alt( ν ) . For any δ -correct ϵ -global DP BAI strategy, we have that

where kl( x, y ) ≜ x log x y +(1 -x ) log 1 -x 1 -y for x, y ∈ (0 , 1) .

Proof. Step 1: Distinguishability due to δ -correctness. Let π be a δ -correct ϵ -global DP BAI strategy. Let ν be a bandit instance and λ ∈ Alt( ν ) .

Let P ν ,π denote the probability distribution of ( A, ̂ A,τ ) when the BAI strategy π interacts with ν . For any alternative instance λ ∈ Alt( ν ) , the data-processing inequality gives that

where the second inequality is because π is δ -correct i.e. P ν ,π ( ̂ A = a ⋆ ( ν ) ) ≥ 1 -δ and P λ ,π ( ̂ A = a ⋆ ( ν ) ) ≤ δ , and the monotonicity of the kl .

Step 2: Connecting KL and TV under ϵ -global DP. On the other hand, by the definition of the KL, we have that

where

Since π is ϵ -global DP, using the sequential Karwa-Vadhan lemma [AB22, Lemma 2], we get that

Which gives that

Combining Inequalities 5 and 6 concludes the proof.

B.3 Lower bound on sample complexity: Proof of Theorem 2

Theorem 2 (Sample complexity lower bound for ϵ -DP-FC-BAI) . Let δ ∈ (0 , 1) and ϵ > 0 . For any δ -correct ϵ -global DP BAI strategy, we have that

Proof. Let π be a δ -correct ϵ -global DP BAI strategy. Let ν be a bandit instance and λ ∈ Alt( ν ) . Let E denote the expectation under P ν ,π , ie E ≜ E ν ,π .

By Lemma 1, we have that 6 ϵ ∑ K a =1 E [ N a ( τ )] TV ( ν a ∥ λ a ) ≥ kl(1 -δ, δ ) .

Lemma 1 from [KCG16] gives that ∑ K a =1 E [ N a ( τ )] KL ( ν a ∥ λ a ) ≥ kl(1 -δ, δ ) .

Since these two inequalities hold for all λ ∈ Alt( ν ) , we get

(a) is due to the fact that E [ τ ] does not depend on λ . (b) is obtained by noting that the vector ( ω a ) a ∈ [ K ] ≜ ( E ν,π [ N a ( τ )] E ν,π [ τ ] ) a ∈ [ K ] belongs to the simplex Σ K .

The theorem follows by noting that for δ ∈ (0 , 1) , kl(1 -δ, δ ) ≥ log(1 / 3 δ ) .

B.4 TV characteristic time for Bernoulli instances: Proof of Proposition 1

̸

Proposition 1 (TV characteristic time for Bernoulli instances) . Let ν be a bandit instance, i.e. such that ν a = Bernoulli ( µ a ) and µ 1 > µ 2 ≥ · · · ≥ µ K . Let ∆ a ≜ µ 1 -µ a and ∆ min ≜ min a =1 ∆ a . We have that

Proof. Step 1: Let ν be a bandit instance, i.e. such that ν a ≜ Bernoulli ( µ a ) and µ 1 > µ 2 ≥ · · · ≥ µ K .

For the alternative bandit instance λ , we refer to the mean of arm a as ρ a , i.e. λ a ≜ Bernoulli ( ρ a ) . By the definition of T ⋆ TV , we have that

̸

where g a ( x a ) ≜ min(1 , x a )∆ a .

Equality (a) is obtained due to the fact that Alt( ν ) = ⋃ a =1 { λ : ρ a > ρ 1 } , and for Bernoullis, TV( ν a ∥ λ a ) = | µ a -ρ a | .

̸

Equality (b) is true, since inf λ : ρ a >ρ 1 ω 1 | µ 1 -ρ 1 | + ω a | µ a -ρ a | = min( ω 1 , ω a )∆ a .

̸

̸

̸

This leads to the inequalities

̸

Thus,

̸

This means that for every ( x 2 , . . . , x K ) ∈ ( R + ) K -1 ,

̸

Here, the upper bound is achievable for x ⋆ a = ∆ min , since g a ( x ⋆ a ) = ∆ min for all a = 1 .

∆ a This concludes that

̸

̸

Step 3: The lower and upper bounds on ( T ⋆ TV ( ν )) follow from the fact that 1 ∆ a ≥ 0 for all a , and 1 ∆ a ≤ 1 ∆ min for all a = 1 .

Hence, we conclude the proof.

B.5 On the total variation distance and the hardness of privacy

Our lower bound suggests that the hardness of the DP-FC-BAI problem is characterized by T ⋆ TV , which is a total variation counterpart of the classic KL-based characteristic time T ⋆ KL in FCBAI [GK16]. The total variation distance appears to be the natural measure to quantify the hardness of privacy in other settings such as regret minimization [AB22], Karwa-Vadhan lemma [KV17] and Differentially Private Assouad, Fano, and Le Cam [ASZ21]. The high-level intuition is that: Pure DP can be seen as a multiplicative stability constraint of e ϵ when one data point changes. With group privacy, if two datasets differ in d ham points, then one incurs a factor e d ham ϵ . Now, by sampling n i.i.d points from a distribution P and n i.i.d points from a distribution Q , the Karwa-Vadhan lemma states that the incurred factor is e ( nTV ( P,Q )) ϵ . This is proved by building a maximal coupling, which is the coupling that minimizes the Hamming distance in expectation. In brief, the total variation naturally appears in lower bounds since it is the quantity that characterises the hardness of the optimal transport problem minimizing the hamming distance , i.e TV ( P, Q ) = inf ( X,Y ) ∼ ( P,Q ) E (1 X = Y ) . However, it is possible that the problem can be characterized by other f-divergences. Finally, one can always go from TV to KL using Pinsker's inequality, though that would always be less tight than the TV-based lower bound.

̸

On the relation between T ⋆ TV and T ⋆ KL A direct application of Pinsker's inequality gives that T ⋆ TV ( ν ) ≥ √ 2 T ⋆ KL ( ν ) . For completeness, we present here the exact calculations:

̸

Equality (c) holds true, since ω 1 = 1 (if ω 1 = 0 , the value of the objective is 0).

Equality (d) is obtained by the change of variable x a ≜ ω a ω 1

Step 2: Let ( x 2 , . . . , x K ) ∈ ( R + ) K -1 . By the definition of g a , we have that

̸

̸

̸

For every alternative mean parameter λ and every arm a , using Pinkser's inequality, we have that d TV ( µ a , λ a ) ≤ √ 1 2 d KL ( µ a , λ a ) . Therefore, for every allocation over arms ω , we have

Taking the supremum over the simplex and the infimum over the set of alternative mean parameters yields ( T ⋆ TV ( ν )) -1 ≤ √ 1 2 ( T ⋆ KL ( ν )) -1 . This concludes the proof.

C Privacy analysis

In this section, we prove that AdaP -TT satisfies ϵ -DP. We first provide the privacy lemma that justifies using doubling and forgetting in AdaP -TT . Using the privacy lemma and the post-processing property of DP, we conclude the privacy analysis of AdaP -TT .

C.1 Privacy lemma for non-overlapping sequences

Lemma 2 (Privacy of non-overlapping sequence of empirical means) . Let M be a mechanism that takes a set as input and outputs the private empirical mean, i.e.

Let ℓ < T and t 1 , . . . t ℓ , t ℓ +1 be in [1 , T ] such that 1 = t 1 < · · · < t ℓ < t ℓ +1 -1 = T . Let's define the following mechanism

In other words, G is the mechanism we get by applying M to the non-overlapping partition of the sequence { r 1 , . . . , r T } according to t 1 < · · · < t ℓ < t ℓ +1 , i.e.

where µ i ∼ M { r t i ,...,r t i +1 -1 } .

For r t ∈ [0 , 1] , the mechanism G is ϵ -DP.

Proof. Let r T ≜ ( r 1 , . . . , r T ) and r ′ T ≜ ( r ′ 1 , . . . , r ′ T ) be two neighbouring reward sequences in [0,1]. This implies that ∃ j ∈ [1 , T ] such that r j = r ′ j and ∀ t = j , r t = r ′ t .

̸

̸

Let ℓ ′ be such that t ℓ ′ ≤ j ≤ t ℓ ′ +1 -1 , and follows the convention that t 0 = 1 and t ℓ +1 = T +1 .

Let µ ≜ ( µ 1 , . . . , µ ℓ ) a fixed sequence of outcomes. Then,

where the last inequality holds true because M satisfies ϵ -DP following Theorem 1.

C.2 Privacy analysis of AdaP -TT : Proof of Theorem 3

Theorem 3 (Privacy analysis) . For rewards in [0 , 1] , AdaP -TT satisfies ϵ -global DP.

̸

Proof. Let T ≥ 1 . Let d T = { x 1 , . . . , x T } and d ′ T = { d ′ 1 , . . . , d ′ T } two neighbouring reward tables in ( R K ) T . Let j ∈ [1 , T ] such that, for all t = j , d t = d ′ t .

We also fix a sequence of sampled actions a T = { a 1 , . . . , a T } ∈ [ K ] T and a recommended action ˆ a ∈ K .

We refer to AdaP -TT BAI strategy by π .

The main idea is that the change of reward in the j -th reward only affects the empirical mean computed in one episode, which is made private using the Laplace Mechanism and Lemma 2.

Step 1. Sequential decomposition of the output probability

We observe that due to the sequential nature of the interaction, the output probability can be decomposed to a part that depends on d j -1 ≜ { x 1 , . . . , x j -1 } , which is identical for both d T and d ′ T and a second conditional part on the history.

Specifically, we have that

where

Similarly

Which means that

Step 2. The adaptive episodes are the same, before step j

Let ℓ such that t ℓ ≤ j < t ℓ +1 when π interacts with d T . Let us call it ψ π d T ( j ) ≜ ℓ .

T

Similarly, let ℓ ′ such that t ℓ ′ ≤ j < t ℓ ′ +1 when π interacts with d ′ . Let us call it ψ π d ′ T ( j ) ≜ ℓ ′ .

Since ψ π d T ( j ) only depends on d j -1 , which is identical for d T and d ′ T , we have that ψ π d T ( j ) = ψ π d ′ T ( j ) with probability 1 .

We call ξ j the last time-step of the episode ψ π d T ( j ) , i.e ξ j ≜ t ψ π d T ( j )+1 -1 .

Step 3. Private sufficient statistics

Let r t ≜ d T t,a t , be the reward corresponding to the action a t in the table d T . Similarly, r ′ t ≜ d ′ T t,a t for d ′ T .

Let us define L j ≜ G { r 1 ,...,r ξ j } and L ′ j ≜ G { r ′ 1 ,...,r ′ ξ j } , where G is defined as in Eq. 8, using the same episodes for d and d ′ . In other words, L j is the list of private empirical means computed on a non-overlapping sequence of rewards before step ξ j .

Using the forgetting structure of AdaP -TT , there exists a randomised mapping f d >ξ j such that

In other words, the interaction of π with d and d ′ from step ξ j + 1 until T only depends on the sufficient statistics L j , which summarises what happened before ξ j , and the new inputs d >ξ j , which are the same for d and d ′ .

Step 4. Concluding with Lemma 2 and the post-processing lemma

Since rewards are in [0 , 1] , using Lemma 2, we have that G is ϵ -DP.

Since P π d ( . | a j ) is just a post-processing of the output of G , we have that

and Eq. (9) concludes the proof.

D AdaP -TT : A private Top Two algorithm with adaptive episodes

We propose a generic wrapper to adapt existing BAI algorithms to tackle private BAI (Appendix D.1). Then, we show how to instantiate our wrapper with an instance of Top Two algorithm (Appendix D.2), namely TTUCB [JD22].

D.1 Generic wrapper on existing BAI algorithms

We propose a generic wrapper to adapt existing BAI algorithms to tackle the ϵ -DP-FC-BAI problem.

It uses adaptive episodes with doubling and forgetting (Appendix D.1.1). This builds on the idea used to make the UCB algorithm private for regret minimisation [AB22].

It relies on a private GLR (generalised likelihood ratio) stopping rule (Appendix D.1.2). The GLR stopping rule [GK16] is widely used in the BAI literature, since it ensures δ -correctness of the stopping rule regardless of the sampling rule.

For each arm a ∈ [ K ] , we maintain a phase at time n whose index will be denoted by k n,a ∈ N . We switch phase as soon as the number of times that the arm was played is doubled. We only evaluate the stopping condition when we switch phase for an arm.

D.1.1 Adaptive episodes with doubling and forgetting

As initialisation, we start by pulling each arm once, and set k n,a = 1 , T 1 ( a ) = K +1 and N n,a = 1 for all a ∈ [ K ] . In the following, we will consider n > K and we denote the global pulling count of arm a before time n by N n,a ≜ ∑ t ∈ [ n -1] 1 ( I t = a ) . For each arm a ∈ [ K ] , the random stopping time denoting the end of k a -1 and the beginning of phase k a > 1 is denoted by

At the beginning of phase k a for an arm a ∈ [ K ] , we update the empirical mean based on the observations on arm a ∈ [ K ] collected during this last phase, i.e.

where the local pulling count pf arm a is denoted by ˜ N k a ,a ≜ N T ka ( a ) ,a -N T ka -1 ( a ) ,a , meaning it is the number of collected samples during the phase k a -1 . To ensure privacy, we add a Laplace

noise to define the private empirical mean, i.e.

We emphasise that only the private version the estimator ˆ µ k a ,a , i.e. ˜ µ k a ,a , is used by the algorithm until the end of phase k a for arm a . Since ( k n,a ) a ∈ [ K ] denotes the current phases at time n , our algorithm relies on (ˆ µ k n,a ,a , ˜ µ k a ,a , ˜ N k n,a ,a , N n,a ) a ∈ [ K ] .

Due to the doubling, the growth of the global and local pulling counts is exponential (Lemma 3).

Lemma 3. For all a ∈ [ K ] and all k ∈ N such that E ν [ T k ( a )] < + ∞ , we have

Proof. Let a ∈ [ K ] . After initialisation, we have k = 1 , T 1 ( a ) = K +1 and N T 1 ( a ) ,a = 1 . Using the definition of the adaptive phase switch (Equation (10)), it is direct to see that N T 2 ( a ) ,a = 2 and ˜ N 2 ,a = 1 when E ν [ T 2 ( a )] < + ∞ .

Now, we proceed by recurrence. Suppose that N T k ( a ) ,a = 2 k -1 and ˜ N k,a = 2 k -2 when E ν [ T k ( a )] < + ∞ . If E ν [ T k +1 ( a )] < + ∞ , then it means that the phase k ends for arm a almost surely. Since we sample only one arm at each round, at the beginning of phase k + 1 for arm a , we have N T k +1 ( a ) ,a = 2 N T k ( a ) ,a = 2 k by using the definition of the adaptive phase switch (10). Then, we have directly that ˜ N k +1 ,a = N T k +1 ( a ) ,a -N T k ( a ) ,a = 2 k -2 k -1 = 2 k -1 .

D.1.2 GLR stopping rule

Non-private GLR stopping rule with phases. Given a set of non-private threshold functions ( c k ) k ∈ N , such that c k : N × N × (0 , 1) → R + for all k ∈ N , the non-private GLR stopping rule can be evaluated at the beginning of each phase for each arm, namely

̸

where ˆ a NP n = arg max a ∈ [ K ] ˆ µ k n,a ,a is the non-private candidate answer until we switch phase again (for any arm). We emphasise that this stopping condition is only evaluated at the beginning of each phase for each arm since it involves quantities that are fixed until we switch phase again .

Lemma 4 gives non-private threshold functions ensuring that the non-private GLR stopping rule is δ -correct for all δ ∈ (0 , 1) , independently of the sampling rule.

Lemma 4. Let δ ∈ (0 , 1) . Let s > 1 and ζ be the Riemann ζ function. Given any sampling rule, the non-private GLR stopping rule (Equation (13) ) with non-private threshold functions

ensures δ -correctness for 1 -sub-Gaussian distributions. The function C G is defined in Equation (15) . It satisfies C G ≈ x +log x .

Proof. The non-private GLR stopping rule matches the one used for Gaussian bandits. Proving δ -correctness of a GLR stopping rule is done by leveraging concentration results. In particular, we build upon Theorem 9 of [KK21], which is restated below.

Lemma 5. Consider sub-Gaussian bandits with means µ ∈ R K . Let S ⊆ [ K ] and x > 0 .

where C G is defined in [KK21] as

Here, ζ is the Riemann ζ function and C G ( x ) ≈ x +log( x ) .

̸

Using the non-private GLR stopping rule (13) with non-private threshold functions ( c k ) k ∈ N and the above manipulations, we obtain

̸

̸

̸

The inequality (a) is obtained with ( b, x, y ) = ( a ⋆ , µ a , µ a ⋆ ) . The inequality (b) drops the condition a = arg max b ∈ [ K ] ˆ µ k b ,b , hence we can restrict to ( k a , k a ⋆ ) ∈ N 2 since it doesn't depend on other phase indices. The inequality (c) relies on a direct union bound. For all a = a ⋆ and all ( k a , k a ⋆ ) ∈ N 2 , the estimators ˆ µ k a ,a (resp. ˆ µ k a ⋆ ,a ⋆ ) are based solely on the observations collected for arm a (resp. arm a ⋆ ) between times n ∈ { T k a -1 ( a ) , · · · , T k a ( a ) -1 } (resp. n ∈ { T k a ⋆ -1 ( a ⋆ ) , · · · , T k a ⋆ ( a ⋆ ) -1 } ) with local counts ˜ N k a ,a (resp. ˜ N k a ⋆ ,a ⋆ ), i.e. dropping past observations. Using Lemma 5 for all a = a ⋆ and all ( k a , k a ⋆ ) ∈ N 2 , we obtain

̸

̸

Private GLR stopping rule with phases. Since we want to ensure privacy, the non-private GLR stopping rule cannot be used since it relies on the empirical means ˆ µ k n,a ,a , which are not private. To alleviate this problem, we propose the private GLR stopping rule, which is based on the private empirical means ˜ µ k n,a ,a .

Given a set of private threshold functions ( c ϵ,k 1 ,k 2 ) ( ϵ,k 1 ,k 2 ) ∈ R ⋆ + × N 2 , such that c ϵ,k 1 ,k 2 : N × N × (0 , 1) → R + for all ( ϵ, k 1 , k 2 ) ∈ R ⋆ + × N 2 , the non-private GLR stopping rule is evaluated at the beginning of each phase for each arm, namely

̸

̸

̸

̸

̸

where ˆ a n = arg max a ∈ [ K ] ˜ µ k n,a ,a is the private candidate answer until we switch phase again (for any arm). We emphasise that this stopping condition is only evaluated at the beginning of each phase for each arm since it involves quantities that are fixed until we switch phase again.

Theorem 4 gives private threshold functions ensuring that the private GLR stopping rule is δ -correct for all δ ∈ (0 , 1) and all ϵ ∈ R ⋆ + , independently of the sampling rule.

Theorem 4 ( δ -correctness of the private GLR stopping rule) . Let δ ∈ (0 , 1) and ϵ ∈ R ⋆ + . Let s > 1 and ζ be the Riemann ζ function and non-private threshold function ( c k ) k ∈ N as in (14) . Given any sampling rule, the private GLR stopping rule (Equation (16) ) with private threshold functions

ensures δ -correctness for 1 -sub-Gaussian distributions.

Proof. Let ϵ ∈ R ⋆ + . Since Y k n,a ,a ∼ Lap ( ( ϵ ˜ N k n,a ,a ) -1 ) , we have that ˜ N k n,a ,a | Y k n,a ,a | ∼ E ( ϵ ) for all a ∈ [ K ] and all n ∈ N , where E ( · ) denotes the exponential distribution. Using concentration results for exponential distribution, a direct union bound yields that

Let us denote ˜ c ϵ,k 1 ,k 2 ( n, m, δ ) the threshold associated to the Laplace noise, i.e.

Using the private GLR stopping rule (Equation (16)) with private threshold functions ( c ϵ,k 1 ,k 2 ) ( ϵ,k 1 ,k 2 ) ∈ R ⋆ + × N 2 , similar manipulations as above yields

̸

̸

The inequality (a) uses that P ( X + Y ≥ a + b ) ≤ P ( X ≥ a ) + P ( Y ≥ b ) and ( x -y ) 2 ≤ 2 x 2 +2 y 2 . The inequality (c) leverages Lemma 4 and a direct inclusion of event. The inequality (c) deploys Equation (18) to conclude.

D.2 AdaP -TT : Instantiating our wrapper with a Top Two sampling rule

A blueprint of Top Two algorithm design. At time n > K , a Top Two sampling rule defines a leader B n and a challenger C n . Then, it selects among them the next arm to sample I n . Given a proportion β ∈ (0 , 1) fixed beforehand, the choice of I n ∈ { B n , C n } should ensure that the leader is sampled close to β of the times where it was chosen as leader. In early works on Top Two algorithms, this choice is randomised. Following [JD22], we use K independent tracking procedures.

̸

̸

We denote by N a n,b ≜ ∑ t ∈ [ n -1] 1 ( B t = a, I t = C t = b ) the number of times the arm b was pulled while the arm a was the leader, and by L n,a ≜ ∑ t ∈ [ n -1] 1 ( B t = a ) the number of times arm a was the leader. At time n > K , the next arm to be pulled I n is defined as

In other words, we sample the leader if we have not yet sampled it a fraction β of the times it was leader. Those K independent tracking procedure satisfy the desired property (Lemma 6).

Lemma 6 (Lemma 2.2 in [JD22]) . For all n > K and all a ∈ [ K ] , we have

To finish specifying a Top Two algorithm, we simply need to specify the choice of the leader/challenger pair . Intuitively, a good choice of the leader/challenger pair should ensure (1) sufficient exploration , (2) convergence of the leader towards the best arm a ⋆ , and (3) convergence of the global pulling proportions to the β -optimal allocation ω KL ,β ( µ ) , which is defined for Gaussian distributions as

̸

While we consider TTUCB algorithm [JD22] in AdaP -TT , we emphasise that other Top Two algorithms could be used with the same type of guarantees. TTUCB is a Top Two algorithm which combines a UCB-based leader and a Transportation Cost (TC) challenger. Its key novelty lies in the use of K tracking procedures. Since it is deterministic, the analysis is less cumbersome.

Non-private leader/challenger pair. The non-private leader/challenger pair is inspired by the TTUCB algorithm [JD22]. At time n > K , the non-private UCB leader is defined as

where √ k n,a / ˜ N k n,a ,a is a bonus to cope for the uncertainty which depends on the current phase k n,a . Later, we show that log( n ) /k n,a = Θ(1) for all a ∈ [ K ] . Hence, it has the same scaling as the bonus used in standard UCB. Since it depends solely on the local counts ( ˜ N k n,a ,a ) a ∈ [ K ] and the empirical means (ˆ µ k n,a ,a ) a ∈ [ K ] , the non-private UCB leader is fixed until we switch phase again.

At time n > K , the non-private TC challenger is defined as

̸

While it depends on the empirical means (ˆ µ k n,a ,a ) a ∈ [ K ] that are fixed till we switch phase again, it also depends on the global counts ( N n,a ) a ∈ [ K ] . Therefore, the non-private TC challenger is chosen in an adaptive manner. This is the key to obtain guarantees on the expected sample complexity of the non-private algorithm.

We derive upper bounds on the expected sample complexity of the non-private AdaP -TT algorithm in the asymptotic regime of δ → 0 (Theorem 6). In particular, it shows that the cost of doubling and forgetting is multiplicative four-factor compared to the TTUCB algorithm, which achieves T ⋆ KL ,β ( µ ) (see Theorem 2.3 in [JD22]). We defer its proof to Appendix E.8.

̸

Theorem 6 (Asymptotic upper bound on expected sample complexity of non-private AdaP -TT ) . Let ( δ, β ) ∈ (0 , 1) 2 . Combined with the non-private GLR stopping rule (Equation (13) ) using non-private threshold functions as in Equation (14) , the non-private AdaP -TT algorithm is δ -correct and satisfies that, for all 1 -sub-Gaussian distributions ν with means µ ∈ R K such that min a = b | µ a -µ b | > 0 ,

where T ⋆ KL ,β ( µ ) is the β -characteristic time for Gaussian distributions, such that

Private leader/challenger pair. Since we want to ensure privacy, the non-private leader/challenger pair cannot be used since it relies on the empirical means ˆ µ k n,a ,a , which are not private. To alleviate this problem, we propose a private leader/challenger pair which is based on the private empirical means ˜ µ k n,a ,a .

At time n > K , the private UCB leader is defined as

where k n,a / ( ϵ ˜ N k n,a ,a ) is a bonus to cope for the uncertainty due to the Laplace noise. It also depends on the current phase k n,a and has the same scaling as the private UCB indices. Likewise, the private UCB leader is fixed until we switch phase again.

At time n > K , the private Transportation Cost (TC) challenger is defined as

̸

Likewise, the private TC challenger is chosen in an adaptive manner, which is key to obtain guarantees on the expected sample complexity of the private algorithm.

We derive upper bounds on the expected sample complexity of the private AdaP -TT algorithm in the asymptotic regime of δ → 0 (Theorem 5). We defer its proof to Appendix E.

̸

Theorem 5 (Asymptotic upper bound on expected sample complexity of AdaP -TT ) . Let ( δ, β ) ∈ (0 , 1) 2 . Combined with the non-private GLR stopping rule (Equation (13) ) using non-private threshold functions as in Equation (14) , the non-private AdaP -TT algorithm is δ -correct and satisfies that, for all bandit instances ν with 1 -sub-Gaussian distributions and means µ ∈ R K such that min a = b | µ a -µ b | > 0 ,

where T ⋆ KL ,β ( µ ) is the β -characteristic time for Gaussian distributions.

E Analysis of AdaP -TT : Proof of Theorem 5

̸

Let β ∈ (0 , 1) , ϵ ∈ R ⋆ + , and ν be a bandit instance consisting of K , 1 -sub-Gaussian distributions with distinct means µ ∈ R K , i.e. min a = b | µ a -µ b | > 0 . For conciseness, we denote ∆ a ≜ µ a ⋆ -µ a , ∆ min ≜ min a = a ⋆ ∆ a , and ∆ max ≜ max a = a ⋆ ∆ a .

̸

̸

For Gaussian distributions, the unique β -optimal allocation ω ⋆ KL ,β ( µ ) = { ω ⋆ β,a } is defined as

̸

At equilibrium, we have equality of the transportation costs (see [JD22] for example), namely

̸

Our proof follows the unified sample complexity analysis of Top Two algorithms from [JDB + 22].

Let γ > 0 . We denote by T µ ,γ the convergence time towards ω ⋆ β , which is a random variable quantifies the number of samples required for the global empirical allocations N n / ( n -1) to be γ -close to ω ⋆ β for any subsequent time, namely

The rest of Appendix E is organised as follows. First, we prove that AdaP -TT ensures sufficient exploration (Appendix E.2) Second, we prove that there is convergence towards the β -optimal

allocation (Appendix E.3) in finite time. Finally, we conclude the proof of Theorem 5 (Appendix E.5). In Appendix E.6, we compare our asymptotic upper bound with our asymptotic lower bound on the expected sample complexity (Theorem 2). In Appendix E.7, we discuss the limitation of our result and pose an open problem. Appendix E.8 will detail the slight modification that needs to be made in order to obtain Theorem 6.

E.1 Technical results

Before delving into the proofs, we first recall some useful technical results extracted from the literature.

Concentration results. In order to control the randomness of (˜ µ k a ,a ) a ∈ [ K ] , we use a standard concentration result on the empirical mean of sub-Gaussian random variables and on sub-exponential observations (Lemma 7). Since Bernoulli distributions are 1 / 2 -sub-Gaussian and the absolute value of a Laplace is an exponential distribution, Lemma 7 applies to our setting.

Lemma 7. There exists a sub-Gaussian random variable W µ such that, almost surely,

There exists a sub-exponential random variable W ϵ such that, almost surely,

In particular, any random variable which is polynomial in ( W ϵ , W µ ) has a finite expectation.

Proof. The first part is a known result, e.g. Appendix E.2 in [JDB + 22]. Let us define

By definition, we have that, almost surely,

Since ˜ N k,i | Y k,i | ∼ E ( ϵ ) , Lemma 72 in [JDB + 22] yields that W ϵ is a sub-exponential random variable. Since W µ is sub-Gaussian and W ϵ is a sub-exponential, any random variable which is polynomial in ( W ϵ , W µ ) has a finite expectation.

Inversion results. Lemma 8 gathers properties on the function W -1 , which is used in the literature to obtain concentration results.

Lemma 8 ([JDK23]) . Let W -1 ( x ) ≜ -W -1 ( -e -x ) for all x ≥ 1 , where W -1 is the negative branch of the Lambert W function. The function W -1 is increasing on (1 , + ∞ ) and strictly concave on (1 , + ∞ ) . In particular, W ′ -1 ( x ) = ( 1 -1 W -1 ( x ) ) -1 for all x > 1 . Then, for all y ≥ 1 and x ≥ 1 ,

Moreover, for all x > 1 ,

Lemma 9 is an inversion result to upper bound a time, which is implicitly defined. It is a direct consequence of Lemma 8.

Lemma 9. Let W defined in Lemma 8. Let A > 0 , B > 0 such that B/A +log A > 1 and

Then, C ( A,B ) < h 1 ( A,B ) with h 1 ( z, y ) = zW -1 ( y/z +log z ) .

Proof. Since B/A +log A > 1 , we have C ( A,B ) ≥ A , hence

Using Lemma 8 yields that

E.2 Sufficient exploration

The first step of in the generic analysis of Top Two algorithms [JDB + 22] is to show that AdaP -TT ensures sufficient exploration. The main idea is to show that, if there are still undersampled arms, either the leader or the challenger will be among them. Therefore, after a long enough time, no arm can still be undersampled. We emphasise that there are multiple ways to select the leader/challenger pair in order to ensure sufficient exploration. Therefore, while we conduct the proof for AdaP -TT , other choices of leader/challenger pair would yield similar results.

Given an arbitrary phase p ∈ N , we define the sampled enough set, i.e. the arms having reached phase p , and the arm with highest mean in this set (when not empty) as

̸

Since min a = b | µ a -µ b | > 0 , a ⋆ n is unique.

Let p ∈ N such that ( p -1) / 4 ∈ N . We define the highly and the mildly under-sampled sets as

They correspond to the arms having not reached phase ( p -1) / 2 and phase 3( p -1) / 4 , respectively.

Lemma 10 shows that, when the leader is sampled enough, it is the arm with highest true mean among the sampled enough arms.

̸

Lemma 10. Let S p n and a ⋆ n as in (27). There exists p 0 with E ν [exp( αp 0 )] < + ∞ for all α > 0 such that if p ≥ p 0 , for all n such that S p n = ∅ , B n ∈ S p n implies that B n = a ⋆ n = arg max a ∈ S p n ˜ µ k n,a ,a .

̸

Proof. Let p 0 to be specified later. Let p ≥ p 0 . Let n ∈ N such that S p n = ∅ , where S p n and a ⋆ n as in Equation (27). Let ( k n,a ) a ∈ [ K ] be the phases indices for all arms. Since N n,a ≥ 2 p -1 for all a ∈ S p n , we have k n,a ≥ p and ˜ N k n,a ,a ≥ 2 p -2 by using Lemma 3. Using Lemma 7, we obtain that

Here, we use that x → log( e + x ) /x is decreasing.

̸

Let ∆ min = min a = b | µ a -µ b | . By assumption on the considered instances, we know that ∆ min > 0 . Let p 1 = ⌈ log 2 ( X 1 -e ) ⌉ +2 and p 2 = ⌈ log 2 (( X 2 -e -2) log 2 + 1) ⌉ +2 with

̸

where we used Lemma 9, and h 1 defined therein. Then, for all p ∈ N such that p ≥ max { p 1 , p 2 } +1 and all n ∈ N such that S p n = ∅ , we have ˜ µ k n,a ⋆ n ,a ⋆ n ≥ µ a ⋆ n -∆ min / 4 and ˜ µ k n,a ,a ≤ µ a +∆ min / 4 for all a ∈ S p n \ { a ⋆ n } , hence a ⋆ n = arg max a ∈ [ K ] ˜ µ k n,a ,a .

We have, for all α ∈ R + ,

where we used Lemma 7 and h 1 ( x, e ) ∼ x → + ∞ x log x to obtain that exp( αp 1 ) is at most polynomial in W µ . Likewise, we obtain that E ν [exp( αp 2 )] < + ∞ for all α ∈ R + .

Let us define the UCB indices by I k n,a ,a = ˜ µ k n,a ,a + √ k n,a / ˜ N k n,a ,a + k n,a / ( ϵ ˜ N k n,a ,a ) . Using the above, we have

where we used Lemma 3 and the fact that x → log( e + x ) /x and x → x 2 2 -x are decreasing function for x ≥ 2 . Let p 3 = ⌈ log 2 X 3 ⌉ +2 and p 4 = ⌈ log 2 X 4 ⌉ +2 with

̸

where we used Lemma 9, and h 1 defined therein. We highlight that ( p 3 , p 4 ) are deterministic values, hence their expectation is finite. Then, for all p ∈ N such that p ≥ p 0 = max { p 1 , p 2 , p 3 , p 4 } +1 and all n ∈ N such that S p n = ∅ , we have I k n,a ⋆ n ,a ⋆ n ≥ µ a ⋆ n -∆ min / 4 and I k n,a ,a ≤ µ a +∆ min / 2 for all a ∈ S p n \ { a ⋆ n } , hence a ⋆ n = B n since we have B n = arg max a ∈ [ K ] I k n,a ,a .

Since we have E ν [exp( αp 0 )] < + ∞ for all α ∈ R + , this concludes the proof.

Lemma 11 shows that the transportation costs between the sampled enough arms with largest true means and the other sampled enough arms are increasing fast enough.

̸

Lemma 11. Let S p n and a ⋆ n are as in Equation (27) . There exists p 1 with E ν [exp( αp 1 )] < + ∞ for all α > 0 such that if p ≥ p 1 , for all n such that S p n = ∅ , for all b ∈ S p n \ { a ⋆ n } , we have

where C µ > 0 is a problem dependent constant.

Proof. Let p 1 to be specified later. Let p ≥ p 1 . Let n ∈ N such that S p n = ∅ , where S p n and a ⋆ n as in Equation (27). Let ( k n,a ) a ∈ [ K ] be the phases indices for all arms. Since N n,a ≥ 2 p -1 for all a ∈ S p n , we have k n,a ≥ p and ˜ N k n,a ,a ≥ 2 p -2 by using Lemma 3. Let ∆ min = min a = b | µ a -µ b | , which satisfies ∆ min > 0 by assumption on the instance considered.

Using Lemma 7, for all b ∈ S p n \ { a ⋆ n } , we obtain

̸

where we used Lemma 9, and h 1 defined therein. Then, for all p ∈ N such that p ≥ p 1 = max { p 3 , p 2 } +1 and all n ∈ N such that S p n = ∅ , we have, for all b ∈ S p n \ { a ⋆ n } ,

As in the proof of Lemma 10, we obtain that E ν [exp( αp 1 )] < + ∞ for all α ∈ R + .

Then, for all b ∈ S p n \ { a ⋆ n } , we have

where we used that min { ˜ N k n,a ⋆ n , ˜ N k n,b ,b } ≥ 2 p -2 . Setting C µ = ∆ min / 2 5 / 2 yields the result.

̸

̸

Lemma12showsthat the transportation costs between sampled enough arms and undersampled arms are not increasing too fast.

̸

Lemma 12. Let S p n be as in Equation (27) . For all p ≥ 1 and all n such that S p n = ∅

where D µ > 0 is a problem dependent constant and ( W µ , W ϵ ) are the random variables defined in Lemma 7.

̸

Proof. Let p ≥ 1 . Let n ∈ N such that S p n = ∅ , where S p n as in Equation (27). Let ( k n,a ) a ∈ [ K ] be the phases indices for all arms. Since N n,a ≥ 2 p -1 for all a ∈ S p n , we have k n,a ≥ p and ˜ N k n,a ,a ≥ 2 p -2 by using Lemma 3. Likewise, N n,a < 2 p -1 for all a / ∈ S p n , we have k n,a < p and ˜ N k n,a ,a < 2 p -2 . Let ∆ max = min a = b | µ a -µ b | , which satisfies ∆ max > 0 by assumption on the instance considered. Using Lemma 7, for all a ∈ S p n and b / ∈ S p n , we obtain

̸

where we used that ˜ N k n,b ,b ≥ 1 , k n,b < p , ˜ N k n,b ,b < 2 p -2 ≤ ˜ N k n,a ,a and x → log( e + x ) /x is decreasing. Taking D µ = ∆ max / 2 yields the result.

Lemma 13 shows that the challenger is mildly undersampled if the leader is not mildly undersampled.

̸

Lemma 13. Let V p n be as in Equation (28) . There exists p 2 with E ν [exp( αp 2 )] < + ∞ for all α > 0 such that if p ≥ p 2 , for all n such that U p n = ∅ , B n / ∈ V p n implies C n ∈ V p n .

̸

Proof. Let p 2 to be specified later. Let p ≥ p 2 . Let n ∈ N such that U p n = ∅ and V p n = [ K ] , where U p n ⊆ V p n are defined in Equation (28). In the following, we suppose that B n / ∈ V p n .

̸

Let ( k n,a ) a ∈ [ K ] be the phases indices for all arms. Let p 0 as in Lemma 10. Let b ⋆ n = arg max b/ ∈ V p n µ b . Then, for all p ≥ 4 p 0 / 3 -1 / 3 and all n such that B n / ∈ V p n , Lemma 10 yields that B n = b ⋆ n = arg max a/ ∈ V p n ˜ µ k n,a ,a .

Let p 1 and C µ as in Lemma 11, and D µ as in Lemma 12. Then, for all p ≥ 4 max { p 0 , p 1 } / 3 -1 / 3 and all n such that B n / ∈ V p n , we have B n = b ⋆ n and

where we used Lemmas 11 and 12. Let p 3 = 16 ⌈ log 2 (2 D µ /C µ ) ⌉ +1 , then we have 2 ( p -1) / 16 > D µ C µ for all p ≥ p 3 . Let p 4 = 16 9 ⌈ log 2 X 4 ⌉ +25 and p 5 = 32 9 ⌈ log 2 X 5 ⌉ +7 where

As in the proof of Lemma 10, using Lemma 7 yields that E ν [exp( αp 4 )] < + ∞ and E ν [exp( αp 5 )] < + ∞ for all α ∈ R + . Let p 2 = max { p 3 , p 4 , p 5 , 4 max { p 0 , p 1 } / 3 -1 / 3 } +1 . Then, we have shown that for all p ≥ p 2 , for all n such that B n / ∈ V p n , we have B n = b ⋆ n and

̸

Therefore, by definition of the TC challenger C n = arg min b = b ⋆ n ˜ µ k n,b ⋆ n ,b ⋆ n -˜ µ k n,b ,b √ 1 / ˜ N k n,b ⋆ n ,b ⋆ n +1 / ˜ N k n,b ,b , we obtain

̸

that C n ∈ V p n . Otherwise, there would be a contradiction given that we assumed that U p n = ∅ . Given all the condition exhibited above, it is direct to see that E ν [exp( αp 2 )] < + ∞ for all α > 0 . This concludes the proof.

Lemma 14 shows that all the arms are sufficient explored for large enough n .

Lemma 14. There exists N 0 with E ν [ N 0 ] < + ∞ such that for all n ≥ N 0 and all a ∈ [ K ] ,

̸

Proof. Let p 0 and p 2 as in Lemmas 10 and 13. Combining Lemmas 10 and 13 yields that, for all p ≥ p 3 = max { p 2 , 4 p 0 / 3 -1 / 3 } and all n such that U p n = ∅ , we have B n ∈ V p n or C n ∈ V p n . We have E ν [2 p 2 ] < + ∞ . We have 2 p -1 ≥ K 2 3( p -1) / 4 for all p ≥ p 4 = 4 ⌈ log 2 K ⌉ + 1 . Let p ≥ max { p 3 , p 4 } .

Suppose towards contradiction that U p K 2 p -1 is not empty. Then, for any 1 ≤ t ≤ K 2 p -1 , U p t and V p t are non empty as well. Using the pigeonhole principle, there exists some a ∈ [ K ] such that N 2 p -1 ,a ≥ 2 3( p -1) / 4 . Thus, we have ∣ ∣ V p 2 p -1 ∣ ∣ ≤ K -1 . Our goal is to show that | V p 2 p | ≤ K -2 . A sufficient condition is that one arm in V p 2 p -1 is pulled at least 2 3( p -1) / 4 times between 2 p -1 and 2 p -1 .

Case 1. Suppose there exists a ∈ V p 2 p -1 such that L 2 p ,a -L 2 p -1 ,a ≥ 2 3( p -1) / 4 β +3 / (2 β ) . Using Lemma 6, we obtain

hence a is sampled 2 3( p -1) / 4 times between 2 p -1 and 2 p -1 .

Case 2. Suppose that for all a ∈ V p 2 p -1 , we have L 2 p ,a -L 2 p -1 ,a < 2 3( p -1) / 4 /β +3 / (2 β ) . Then,

Using Lemma 6, we obtain

Combining all the above, we obtain

where the last inequality is obtained for p ≥ p 5 with

The LHS summation is exactly the number of times where an arm a / ∈ V p 2 p -1 was leader but wasn't sampled, hence

For any 2 p -1 ≤ t ≤ 2 p -1 , U p t is non-empty, hence we have B t / ∈ V p 2 p -1 (hence B t / ∈ V p t ) implies C t ∈ V p t ⊆ V p 2 p -1 . Therefore, we have shown that

Therefore, there is at least one arm in V p 2 p -1 that is sampled 2 3( p -1) / 4 times between 2 p -1 and 2 p -1 .

In summary, we have shown | V p 2 p | ≤ K -2 for all p ≥ p 6 = max { p 3 , p 4 , p 5 } . By induction, for any 1 ≤ k ≤ K , we have ∣ ∣ V p k 2 p -1 ∣ ∣ ≤ K -k , and finally U p K 2 p -1 = ∅ for all p ≥ p 6 . Defining N 0 = K 2 p 6 -1 , we have E ν [ N 0 ] < + ∞ by using Lemmas 10 and 13 for p 3 = max { p 2 , 4 p 0 / 3 -1 / 3 } and p 4 and p 5 are deterministic. For all n ≥ N 0 , we let 2 p -1 = n K . Then, by applying the above, we have U p K 2 p -1 = U log 2 ( n/K )+1 n is empty, which shows that N n,a ≥ √ n/K for all a ∈ [ K ] . Using Lemma 3, we obtain that k n,a ≥ log( n/K ) 2 log 2 +1 for all a ∈ [ K ] . This concludes the proof.

E.3 Convergence towards β -optimal allocation

The second step of in the generic analysis of Top Two algorithms [JDB + 22] is to show that AdaP -TT ensures convergence of its empirical proportions towards the β -optimal allocation. First, we show that the leader coincides with the best arm. Hence, the tracking procedure will ensure that the empirical proportion of time we sample it is exactly β . Second, we show that a sub-optimal arm whose empirical proportion overshoots its β -optimal allocation will not be sampled next as challenger. Therefore, this 'overshoots implies not sampled' mechanism will ensure the convergence towards the β -optimal allocation. We emphasise that there are multiple ways to select the leader/challenger pair in order to ensure convergence towards the β -optimal allocation. Therefore, while we conduct the proof for AdaP -TT , other choices of leader/challenger pair would yield similar results. Note that our results heavily rely on having obtained sufficient exploration first.

Convergence for the best arm. Lemma 15 exhibits a random phase which ensures that the leader and the candidate answer are equal to the best arm for large enough n .

Lemma 15. Let N 0 be as in Lemma 14. There exists N 1 ≥ N 0 with E ν [ N 1 ] < + ∞ such that, for all n ≥ N 1 , we have ˆ a n = B n = a ⋆ .

Proof. Let k ≥ 1 and ( T k ( a )) a ∈ [ K ] as in Equation (10). Suppose that E ν [max a ∈ [ K ] T k ( a )] < + ∞ . Then, Lemma 3 yields that N T k ( a ) ,a = 2 k -1 and ˜ N k,a = 2 k -2 . Using Lemma 7, we obtain that

̸

̸

where we used Lemma 9, and h 1 defined therein. Then, for all k ∈ N K such that min a ∈ [ K ] k a > p 0 = max { p 1 , p 2 } such that E ν [max a ∈ [ K ] T k a ( a )] < + ∞ , we have ˜ µ k,a ⋆ ≥ µ a ⋆ -∆ min / 4 and ˜ µ k,a ≤ µ a +∆ min / 4 for all a = a ⋆ , hence a ⋆ = arg max a ∈ [ K ] ˜ µ k,a . We have, for all α ∈ R + ,

where we used Lemma 7 and h 1 ( x, e ) ∼ x → + ∞ x log x to obtain that exp( αp 1 ) is at most polynomial in W µ . Likewise, we obtain that E ν [exp( αp 2 )] < + ∞ for all α ∈ R + . Therefore, we have E ν [exp( αp 0 )] < + ∞ for all α ∈ R + .

Let us define the UCB indices by I k,a = ˜ µ k,a + √ k/ ˜ N k,a + k/ ( ϵ ˜ N k,a ) . Using the above, we have

̸

Therefore, we have a ⋆ = arg max a ∈ [ K ] I k,a for all k ∈ N K such that min a ∈ [ K ] k a > max { p 1 , p 2 } such that E ν [max a ∈ [ K ] T k a ( a )] < + ∞ .

Let N 0 as in Lemma 14. Using Lemma 14, we obtain that, for all n ≥ N 0 and all a ∈ [ K ] , k n,a ≥ log 2 ( n/K ) / 2 + 1 . Therefore, we obtain min a ∈ [ K ] k n,a > max { p 1 , p 2 } is implied by n ≥ N 1 = max { K 4 max { p 1 ,p 2 } , N 0 } . Using the above, we conclude that E ν [ N 1 ] < + ∞ and ˆ a n = B n = a ⋆ for all n ≥ N 1 .

Lemma 16 shows that that the pulling proportion of the best arm converges towards β , provided the phase defined in Lemma 15 is reached in finite time for all arms.

Lemma 16. Let γ > 0 , and N 1 be as in Lemma 15. There exists a deterministic constant C 0 ≥ 1 such that, for all n ≥ C 0 N 1 ,

̸

Proof. Let γ > 0 . Let N 1 as in Lemma 15. Let M ≥ N 1 . Using Lemma 15, we obtain B n = a ⋆ for all n ≥ M . Therefore, we obtain L n,a ⋆ ≥ n -M and ∑ a = a ⋆ N a n,a ⋆ ≤ M for all n ≥ M . Using Lemma 6 yields that

̸

where the last inequality is obtained by taking n ≥ max { M, (1 / 2 + 2 β ( M -1)) /γ +1 } .

Convergence for the sub-optimal arms Lemma 17 exhibits a random phase which ensures that if a sub-optimal arm overshoots its β -optimal allocation then it cannot be selected as challenger for large enough n .

Lemma 17. Let γ > 0 . Let N 1 and C 0 be as in Lemma 15 and 16. There exists N 2 ≥ C 0 N 1 with E ν [ N 2 ] < + ∞ such that, for all n ≥ N 2 ,

̸

̸

Proof. Let γ > 0 and ˜ γ > 0 . Let N 1 as in Lemma 15 and C 0 as in Lemma 16 for ˜ γ . Let n ≥ C 0 N 1 .

̸

Let a = a ⋆ such that N n,a n -1 ≥ ω ⋆ β,a + γ . Suppose towards contradiction that N n,b n -1 > ω ⋆ β,a for all b / ∈ { a ⋆ , a } . Then, for all n ≥ C 0 N 1 , we have

̸

̸

which yields a contradiction for ˜ γ ≤ γ . Therefore, for all n ≥ C 0 N 1 , we have

̸

Then, we have

̸

̸

In the following, we use Lemma 7 and similar manipulations as in the proof of Lemma 15. Therefore, we obtain that, for all c = a ⋆ ,

where we used Lemma 9, and h 1 defined therein. We have, for all α ∈ R + ,

where we used Lemma 7 and h 1 ( x, e ) ∼ x → + ∞ x log x to obtain that exp( αp 3 ) is at most polynomial in W µ . Likewise, we obtain that E ν [exp( αp 2 )] < + ∞ for all α ∈ R + .

̸

Using Lemma 14 (with C 0 N 1 ≥ N 1 ≥ N 0 ), we obtain that, for all n ≥ C 0 N 1 and all a ∈ [ K ] , k n,a ≥ log 2 ( n/K ) / 2 + 1 . Therefore, we obtain min a ∈ [ K ] k n,a > max { p 2 , p 3 } is implied by n ≥ N 2 = max { K 4 max { p 3 ,p 2 } , C 0 N 1 } . Using the above, we conclude that E ν [ N 2 ] < + ∞ and max c = a ⋆ | ˜ µ k n,a ⋆ ,a ⋆ -˜ µ k n,c ,c -∆ c | ≤ η for all n ≥ N 2 .

Then, for all n ≥ N 2 , we have B n = a ⋆ and

where the last inequality is obtained by taking η and ˜ γ sufficiently small and by using Equation (25), i.e.

Therefore, we have shown that B n = a ⋆ and

This concludes the proof.

Lemma 18 shows that that the pulling proportion of the best arm converges towards β for large enough n .

Lemma 18. Let γ > 0 and T µ ,γ be as in Equation (26) . Then, we have E ν [ T µ ,γ ] < + ∞ .

Proof. Let γ > 0 and ˜ γ > 0 . Let N 2 as in Lemma 17 for ˜ γ . Let M ≥ N 2 . Using Lemmas 15, 16 and 17 for all n ≥ M , we obtain that B n = a ⋆ , ∣ ∣ ∣ N n,a ⋆ n -1 -β ∣ ∣ ∣ ≤ ˜ γ and

̸

̸

For all a = a ⋆ , let us define t n,a (˜ γ ) = max { t | M ≤ t ≤ n, N t,a / ( n -1) < ω ⋆ β,a + ˜ γ } . Since N t,a / ( n -1) ≤ N t,a / ( t -1) for t ≤ n , we have

̸

The second inequality uses Lemma 17, and the two last inequalities use the definition of t n,a (˜ γ ) . Using that ∑ a ∈ [ K ] N n,a n -1 = ∑ a ∈ [ K ] ω ⋆ β,a = 1 , we obtain

̸

̸

Taking ˜ γ ≤ γ/ (2( K -1)) and n ≥ max { M, 2( K -1)( M -1) /γ +1 } yields that

Let T µ ,γ as in 26. Then, we showed that T µ ,γ ≤ max { M, 2( K -1)( M -1) /γ +1 } . Therefore, we have

which concludes the proof.

E.4 Cost of doubling and forgetting

Compared to the generic analysis of Top Two algorithms [JDB + 22], for AdaP -TT , we need to control the sample complexity cost of doubling and forgetting. Due to this reason, we have to pay a multiplicative four-factor: one two-factor due to doubling, and another two-factor due to forgetting. It is possible to show that this cost exists when adapting any 'reasonable' BAI algorithm, meaning for any BAI algorithm in which the empirical proportions are converging towards an allocation ω such that min a ω a > 0 . Those BAI algorithms are 'reasonable' because the asymptotic lower bound stipulates that all arms have to be sampled linearly in order to be near optimal.

Let ω ∈ Σ K be any allocation over arms such that min a ω a > 0 . Let γ > 0 . We denote by T µ ,γ ( ω ) the convergence time towards ω , which is a random variable quantifying the number of samples required for the global empirical allocations N n / ( n -1) to be γ -close to ω for any subsequent time, namely

Lemma 19 shows that the phase switches of the arms happen in a round-robin fashion, which means that an arm switches phase for a second time after all other arms first switch their own phases.

Lemma 19. Let ω ∈ Σ K such that min a ω a > 0 . Assume that there exists γ µ > 0 such that for E ν [ T µ ,γ ( ω )] < + ∞ for all γ ∈ (0 , γ µ ) , where T µ ,γ ( ω ) is defined in Equation (29) . Let η > 0 . There exists ˜ γ µ ∈ (0 , γ µ ) such that, for all γ ∈ (0 , ˜ γ µ ) , there exists N 3 ≥ T µ ,γ ( ω ) with E ν [ N 3 ] < + ∞ which satisfies

Proof. Let η > 0 . Let ˜ γ µ ∈ (0 , γ µ ) such that 2 max a ∈ [ K ] ( ω a + γ ) / ( ω a -γ ) ≤ 2 + η , which is possible since min a ω a > 0 . Let γ ∈ (0 , ˜ γ µ ) . By assumption, we have E ν [ T µ ,γ ( ω )] < + ∞ . Then, for all n ≥ T µ ,γ ( ω ) ,

Let M ≥ T µ ,γ ( ω ) . Let use denote by k M = ( k M,a ) a ∈ [ K ] the current phases for all arms a ∈ [ K ] at time M . Then, for all n ≥ M and all a ∈ [ K ] , we have N n,a ≥ ( n -1)( ω a -γ ) . Therefore, taking

n ≥ max a ∈ [ K ] 2 k M,a ( ω a -γ ) -1 +1 , we obtain that N n,a ≥ 2 k M,a for all a ∈ [ K ] , hence we have max a ∈ [ K ] T k M,a +1 ( a ) ≤ n . Since min a ∈ [ K ] T k M,a +1 ( a ) ≥ M , we have

Likewise, taking n ≥ max a ∈ [ K ] 2 k M,a +1 ( ω a -γ ) -1 + 1 , we obtain that N n,a ≥ 2 k M,a +1 for all a ∈ [ K ] , hence we have max a ∈ [ K ] T k M,a +2 ( a ) ≤ n . Let a 1 = arg min a ∈ [ K ] T k M,a +2 ( a ) . By definition and using Lemma 3, we have

̸

Let a 2 = arg max a ∈ [ K ] T k M,a +2 ( a ) . By definition and using Lemma 3, we have

Therefore, combining the above yields

where the last inequality uses that γ ∈ (0 , ˜ γ µ ) and ˜ γ µ ∈ (0 , γ µ ) is such that 2 max a ∈ [ K ] ( ω a + γ ) / ( ω a -γ ) ≤ 2 + η . We take n ≥ N 3 = max a ∈ [ K ] T k M,a +2 ( a ) , hence we have k n,a ≥ k M,a +2 for all a ∈ [ K ] . Since E ν [ T µ ,γ ( ω )] < + ∞ (i.e. arms are sampled linearly), it is direct to see that E ν [max a ∈ [ K ] T k M,a +2 ( a )] < + ∞ . This concludes the proof.

Note that T µ ,γ defined in 26 is such that T µ ,γ = T µ ,γ ( ω ⋆ KL ,β ) where T µ ,γ ( ω ) as in Equation (29). Lemma 18 showed that E ν [ T µ ,γ ] < + ∞ for all γ > 0 . Therefore, the condition of Lemma 19 are fulfilled by AdaP -TT .

E.5 Asymptotic expected sample complexity

The final step of the generic analysis of Top Two algorithms [JDB + 22] is to invert the private GLR stopping rule by leveraging the convergence of the empirical proportions towards the β -optimal allocation. Compared to the non-private GLR stopping rule, the private threshold in the private GLR stopping rule involves an additive term in O (log(1 /δ ) 2 ) . This difference is the largest price that we pay to obtain a private BAI algorithm. We defer the reader to Appendix E.7 for a more detailed discussion on it.

Asymptotically β -optimal ϵ -DP-FC-BAI algorithm. The inversion of the GLR stopping rule by leveraging the convergence of the empirical proportions towards the ( β -)optimal allocation is a generic method used in the BAI literature. Provided this convergence is shown, it only depends on the threshold that ensures δ -correctness. More precisely, it only depends on its asymptotic dependence in log(1 /δ ) . In addition to the multiplicative four-factor, the price of privacy for asymptotically β -optimal BAI algorithms when combined with the non-private GLR stopping rule is a problem dependent multiplicative factor 1 + √ 1 + ∆ 2 max / (2 ϵ 2 ) (Lemma 20).

Lemma 20. Let ( δ, β ) ∈ (0 , 1) 2 . Assume that there exists γ µ > 0 such that for E ν [ T µ ,γ ] < + ∞ for all γ ∈ (0 , γ µ ) , where T µ ,γ is defined in Equation (26) . Combining the private GLR stopping rule (Equation (16) ) with private threshold (Equation (4) ) yields a δ -correct algorithm which satisfies that, for all ν with mean µ such that | a ⋆ ( µ ) | = 1 ,

Proof. Theorem 4 yields the δ -correctness.

Let a ⋆ be the unique best arm, i.e. a ⋆ ( µ ) = { a ⋆ } . Let ζ > 0 . Using Equation (25) and the continuity of

̸

yields that there exists γ ζ > 0 such that ∥ ∥ ∥ N n n -1 -ω ⋆ β ∥ ∥ ∥ ∞ ≤ γ ζ and max a ∈ [ K ] | ˜ µ k n,a +1 ,a -µ a | ≤ γ ζ implies that

̸

We choose such a γ ζ . Let γ µ > 0 be such that for E ν [ T µ ,γ ] < + ∞ for all γ ∈ (0 , γ µ ) , where T µ ,γ is defined in Equation (26). Let η > 0 . Let ˜ γ µ ∈ (0 , γ µ ) as in Lemma 19 for this η . In the following, let us consider γ ∈ (0 , min { ˜ γ µ , γ ζ , β/ 4 , ∆ min / 4 } ) .

Let N 3 ≥ T µ ,γ with E ν [ N 3 ] < + ∞ as Lemma 19 for those ( γ, η ) . Then, we have E ν [ T µ ,γ ] < + ∞ and

Since arms are sampled linearly, it is direct to construct N 4 ≥ N 3 with E ν [ N 4 ] < + ∞ such that, for all n ≥ N 4 , we have max a ∈ [ K ] max k ∈{ k n,a ,k n,a +1 } | ˜ µ k,a -µ a | ≤ γ , Therefore, we have ˆ a n = a ⋆ .

Let κ ∈ (0 , 1) . Let n ≥ N 4 /κ and ( k n,a ) a ∈ [ K ] be the current phases at time n . Combining the above, we have ˆ a n = a ⋆ and

̸

Let a 1 = arg min a ∈ [ K ] T k n,a ( a ) and a 2 = arg max a ∈ [ K ] T k n,a ( a ) . Therefore, we obtain, for all a = a ⋆ ,

̸

Similarly, we can show that, for all a = a ⋆ ,

̸

Let ( c k ) k ∈ N as in Equation (14). Using Lemma 3, we obtain, for all a = a ⋆ ,

Likewise, we obtain, for all a ∈ [ K ] ,

Let us denote by T + k n +1 = max b ∈ [ K ] T k n,b +1 ( b ) , T + k n +2 = max b ∈ [ K ] T k n,b +2 ( b ) , T -k n +1 = min b ∈ [ K ] T k n,b +1 ( b ) , T -k n = min b ∈ [ K ] T k n,b ( b ) . Let T be a time such that T ≥ T + k n +1 ≥ κT . Using Lemmas 3 and 19, we have

Using the private GLR stopping rule (Equation (16)), we have

̸

Let T ζ ( δ ) defined as the largest deterministic time such that the above condition is satisfied when replacing T -k n by (1 -κ ) T . Let k δ be the largest random vector of phases such that that T + k δ +1 ≤ T ζ ( δ ) almost surely, hence T + k δ +2 > T ζ ( δ ) almost surely. Then, using the above yields that τ δ ≤ T + k δ +2 almost surely, hence

where the second inequality uses Lemma 19 twice, i.e. T + k δ +2 ≤ (2 + η ) T -k δ +2 ≤ (2 + η ) 2 T + k δ +1 , and the last one used the definition of k δ and that T ζ ( δ ) is deterministic.

Since we are only interested in upper bounding lim sup δ → 0 T ζ ( δ ) log(1 /δ ) , we can safely drop the second orders terms in T and log(1 /δ ) . This allows us to remove the terms in O (log log T ) and in O (log log(1 /δ )) . Using that C G ( x ) = x + O (log x ) , tedious manipulations yields that

where

The last inequality uses that x 2 -2 bx -c < 0 for all x ∈ [0 , b (1 + √ 1 + c/b 2 )) . Therefore, we have shown that

Letting κ , η and ζ goes to zero yields that

̸

Concluding the proof of Theorem 5. Combining Lemmas 14, 18, 19 and 20 concludes the proof of Theorem 5. We restrict the result to instances such that min a = b | µ a -µ b | > 0 in order for Lemma 14 to hold. Note that this is an artifact of the asymptotic proof which could be alleviated with more careful considerations.

Asymptotically optimal ϵ -DP-FC-BAI algorithm. While Lemma 20 is derived for BAI algorithms converging towards the β -optimal allocation ω ⋆ KL ,β ( µ ) , it is direct to see that a similar inversion results can be obtained for BAI algorithms that converge towards the unique optimal allocation ω ⋆ KL ( µ ) = { w ⋆ } defined as

̸

At equilibrium, we have equality of the transportation costs (see [JD22] for example), namely

̸

In addition to the multiplicative four-factor, the price of privacy for asymptotically optimal BAI algorithms when combined with the non-private GLR stopping rule is a problem dependent multiplicative factor 1 + √ 1 + ∆ 2 max / (2 ϵ 2 ) (Lemma 21). We omit the proof since it is the same as the one of Lemma 20.

Lemma 21. Let δ ∈ (0 , 1) Assume that there exists γ µ > 0 such that for E ν [ T µ ,γ ( ω ⋆ )] < + ∞ for all γ ∈ (0 , γ µ ) , where T µ ,γ ( w ) is defined in Equation (29) and ω ⋆ is defined in Equation (30) . Combining the private GLR stopping rule (Equation (16) ) with private threshold (Equation (4) ) yields a δ -correct algorithm which satisfies that, for all ν with mean µ such that | a ⋆ ( µ ) | = 1 ,

E.6 Connection to the lower bound

In this section, we compare the sample complexity lower bound of Corollary 1 with the sample complexity upper bound of Theorem 5.

Simplification of the upper bound. The asymptotic expected sample complexity of AdaP -TT (Theorem 5) is upper bounded by

where T ⋆ KL ,β ( µ ) is the β -characteristic time for Gaussian bandits, and (a) is due to the sub-additivity of the square root.

On the other hand, [GK16] showed that H ( µ ) ≤ T ⋆ KL ( µ ) ≤ 2 H ( µ ) , where H ( µ ) ≜ ∑ a ∈ [ K ] 2∆ -2 a with ∆ a ⋆ = ∆ min .

.

Plugging these two inequalities in the upper bound of Theorem 5 with β = 1 / 2 gives that

Since we consider Bernoulli distributions, we know that 0 < ∆ min ≤ ∆ max < 1 . If we restrict ourselves to instances such that all the gaps have the same order of magnitude (Condition 1): there exists a constant C ≥ 1 such that ∆ max ≤ C ∆ min .

For such instances, we obtain

where the last inequality is due to H ( µ )∆ min ≤ 2 ∆ min + ∑ K a =2 2 ∆ a .

Finally using that a + b ≤ 2 max( a, b ) , we get that

for the universal constant c = 45 . 26 .

The lower bound for Bernoulli instances. For Bernoulli instances, Corollary 1 gives that the lower bound of the expected sample complexity of any δ -correct ϵ -global DP BAI strategy is

where we use Proposition 1 to replace T ⋆ TV ( ν ) and T ⋆ KL ( ν ) is the characteristic time for Bernoulli bandits .

Upper-lower bound discussion for the two privacy regimes. In the low privacy regime, our upper bound retrieves T ⋆ KL , 1 / 2 ( µ ) for Gaussian distributions. Since the rewards in the analysis are supposed Bernoulli, the mismatch from exact optimality is coming from the mismatch between the KL divergence of Bernoulli distributions and that of Gaussian, which is generally controllable in most instances where the means are far from the borders, i.e. 0 , and 1 . This is in essence, similar to the mismatch between UCB and KL-UCB in the regret-minimization literature (Chapter 10 in [LS20]). To overcome this mismatch, it is necessary to adapt the transportation costs to the family of distributions considered. In our setting, this can be done by using the Bernoulli KL rather than Gaussian KL in lines 12 and 16 of the algorithm. While the Top Two algorithms for Bernoulli distributions have been studied in [JDB + 22], the analysis is more involved. Therefore, it would obfuscate where and how privacy is impacting the expected sample complexity.

In the high privacy regime, and for instances verifying Condition 1, our upper bound matches the lower bound ϵ -1 T ⋆ TV ( ν ) up to a constant. Having matching upper and lower bounds only for high privacy regimes is an interesting phenomenon that appears in different settings of differential privacy literature, such as regret minimization [AB22], parameter estimation [CWZ21] and hidden probabilistic graphical models [NKS19]. We speculate two facets of this phenomenon:

1. Explicit bounds in high and low privacy regimes: We have matching bounds only in the high or low privacy regimes because the lower bounds are generally harder to explicit and understand in transitional phases. Thus, it is harder to claim optimality in those phases.

2. Information-theoretic roots: There might be a more profound information-theoretic reason in relation to [NKS19]. Indeed, there seems to be a link between the privacy budget ϵ and the information thresholds introduced in [NKS19]. Specifically, if the randomized mapping F in [3] satisfies DP, then the noisy information threshold and noiseless information threshold can be similarly written as a function of ϵ and the total variation. Finding a rigorous link between these two quantities is an interesting question to explore.

E.7 Limitation and open problem

In the previous section (Appendix E.6), we argue that the upper and lower bounds match up to multiplicative constants in both privacy regimes, provided we restrict ourselves to instances satisfying Condition 1 (i.e. there exists C ≥ 1 such that ∆ max ≤ C ∆ min ).

While this holds for numerous instances, it does not account for instances in which the gaps have different orders of magnitude. One example would be the regime where ∆ min → 0 while ∆ max is fixed, hence yielding T ⋆ KL ( µ ) → + ∞ .

In Appendix E.5 (see Lemma 20 and 21), we show that combining the private GLR stopping rule (Equation (16)) with any BAI algorithms whose empirical proportions converge towards the β -optimal allocation ω ⋆ KL ,β ( µ ) for Gaussian bandits will incur a problem dependent multiplicative cost

in addition to the multiplicative four-factor due to doubling and forgetting.

In order to have matching upper and lower bounds, we would need to have ∆ min instead of ∆ max in Equation (32). Unfortunately, our results show that it is not possible when using the private GLR stopping rule (Equation (16)) and a sampling rule that is tailored to asymptotic ( β -)optimality for Gaussian bandits. Therefore, this impossibility result holds for a large class of BAI sampling rules when adapting them to tackle private BAI by using the private GLR stopping rule (Equation (16)).

Origin of this limitation. The term ∆ max appears due to the private stopping threshold that ensures δ -correctness of the private GLR stopping rule (Equation (16)).

For asymptotically ( β -)optimal algorithms, the additive term due to privacy in the threshold is of the order

̸

Therefore, the private GLR stopping rule (Equation (16)) will stop when, for all a = a ⋆ ,

which yields the problem-dependent multiplicative cost 1 + √ 1 + ∆ 2 max 2 ϵ 2 .

Open problem. This impossibility result is specific to the way we derive an ϵ -DP version of the GLR stopping rule. Therefore, a natural question is whether it is possible to derive a better private GLR stopping rule to match the lower bound for all Bernoulli instances.

A two-phase algorithm. AdaP -TT tracks the non-private lower bound (i.e. T ⋆ KL ) as a non-private algorithm would do, and the additional cost tracking that privately is shown to be T ⋆ TV /ϵ (up to constants for bandits verifying Condition 1), where the additional cost comes from the added Laplace noise. Another approach would be to first perform a test to determine in which privacy regime the policy resides, and then track (privately) the corresponding characteristic time. Such an algorithm would empirically estimate both T ⋆ TV and T ⋆ KL in the first phase. If the privacy budget ϵ is bigger than the empirical estimate of T ⋆ TV /T ⋆ KL , then this means that we are in the low privacy regime and the algorithm tracks (privately) the KL characteristic time in the second phase (as AdaP -TT does). However, if the privacy budget ϵ is smaller than the empirical estimate of T ⋆ TV /T ⋆ KL , then it is the high privacy regime and the algorithm tracks (privately) the TV characteristic time in the second phase. For such a two-phase algorithm to achieve optimality, properly tuning the amount of time spent in its first phase is a crucial step. Whether it is possible to analyze such an algorithm and quantify the proper tuning (if it even exists) is an interesting direction for future work.

Non-asymptotic sample complexity of AdaP -TT . In the non-private FC-BAI literature (i.e. ϵ = + ∞ ), there is no tight lower bound in the non-asymptotic regime (i.e. for any value of δ ) . This is the main open problem in FC-BAI, and hence in DP-FC-BAI too. In the class of asymptotically ( β -)optimal algorithms, TTUCB [JD22] is one of the few to have non-asymptotic guarantees. Adapting

the non-asymptotic analysis of [JD22] to the private AdaP-TT is an interesting direction for research. We conjecture that such an adaptation is possible (up to technicalities) by adding concentration terms linked to Laplace distribution and losing at least a multiplicative four-factor compared to TTUCB due to doubling and forgetting.

E.8 Analysis of non-private AdaP -TT : Proof of Theorem 6

The proofs detailed in Appendices E.2 and E.3 can easily by adapted to provide guarantees on the non-private AdaP -TT , which relies on the non-private leader/challenger defined in Equation (20) and Equation (21).

The key difference between the non-private AdaP -TT and the private AdaP -TT lies in the definition of the stopping threshold. While the private GLR stopping rule (Equation (16)) has an additive terms in O (log(1 /δ ) 2 /ϵ ) to cope for the uncertainty due to the Laplace noise, the non-private GLR stopping rule (Equation (13)) scales simply as log(1 /δ ) . This has drastic consequences in terms of asymptotic upper bound on the expected sample complexity.

It is direct to see that most manipulations from Appendix E.5 still holds for the non-private AdaP -TT . Therefore, we use the notations and conditions defined therein. We show that the non-private GLR stopping rule (Equation (13)) yields

̸

Let T ζ ( δ ) defined as the largest deterministic time such that the above condition is satisfied when replacing T -k n by (1 -κ ) T . Let k δ be the largest random vector of phases such that that T + k δ +1 ≤ T ζ ( δ ) almost surely, hence T + k δ +2 > T ζ ( δ ) almost surely. Then, using the above yields that τ δ ≤ T + k δ +2 almost surely, hence

First-order terms. Since we are only interested in upper bounding lim sup δ → 0 T ζ ( δ ) log(1 /δ ) , we can safely drop the second orders terms in T and log(1 /δ ) . This allows us to remove the terms in O (log log T ) and in O (log log(1 /δ )) . Using that C G ( x ) = x + O (log x ) , tedious manipulations yields that

Letting κ , η and ζ goes to zero yields that

F Extended experimental analysis

In this section, we perform additional experiments to compare AdaP -TT and DP-SE for FC-BAI. We test the two algorithms in six bandit environments with Bernoulli distributions, as defined by [SS19], namely

For each Bernoulli instance, we implement the algorithms with

and a risk level δ = 0 . 01 . We verify empirically that the algorithms are δ -correct by running each algorithm 100 times. In Figure 2, we plot the evolution of the average stopping time and standard deviation with respect to the privacy budget ϵ . All the algorithms are implemented in Python (version 3 . 8 ) and are tested with an 8-core 64-bits Intel i5@1.6 GHz CPU.

All the experiments validate the same conclusions as the ones reached in Section 5, i.e.

AdaP -TT requires fewer samples to provide a δ -correct answer,

there exists two privacy regimes, and in the low-privacy regime, the sample complexity is independent of the privacy budget.

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 5

10 6

10 7

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(a)

µ

1

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 5

10 6

10 7

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(b)

µ

2

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 3

10 4

10 5

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(c)

µ

3

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 4

10 5

10 6

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(d)

µ

4

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 4

10 5

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(e)

µ

5

10

3

10

2

10

1

10 0

10 1

Privacy Budget

10 5

10 6

10 7

Stopping Time

High Privacy Regime

Low Privacy Regime

TTUCB

AdaP-TT

DP-SE

(f)

µ

6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the best arm identification (BAI) problem under differential privacy (DP) constraints, where the goal is to privately identify the arm with the highest expected reward with probability at least $1 - \\delta$. The paper proposes a DP variant of the top two algorithm, which achieves near-optimal performance in the high privacy regime. The paper also proves an information-theoretic instance-dependent lower bound for the DP-BAI problem. The paper also conducts experiments on synthetic data to demonstrate the efficiency of the proposed algorithm.\n\nSTRENGTHS:\nThe proposed upper and lower bounds match each other in the high-privacy regime when the true means are away from 0 or 1. This provides an up-to-constant characterization of the asymptotic sample complexity for DP-BAI.\n\nThe proposed algorithm empirically outperforms a natural DP variant of the successive elimination algorithm. \n\nThe paper is also nicely written and easy to read.\n\nWEAKNESSES:\nWhile DP-BAI is an interesting problem to study, the techniques used in the paper are primarily natural variants of previous work. In particular, nonprivate-BAI and the closely related DP regret minimization in the bandit setting have been studied. Hence the technical novelty of the paper is restricted. \n\nIt would be nice if the authors could comment on the non-asymptotic optimality of the proposed algorithm.\n\nMinor comments:\n1. Not sure if colored texts are supported for NeurIPS submissions.\n2. Definition 1: The letter d is a bit overloaded. Maybe it is better to use $D$ or $\\mathcal{D}$ to refer to datasets, which is used later in the paper.\n3. Line 207: Might be good to have parentheses after sup.\n4. Can you comment on how $T^*_{TV}(\\nu)$ and  $T^*_{KL}(\\nu)$ relate to each other? \n5. Algorithm 2: $L_n$'s are not initialized.\n\nQUESTIONS:\nSee comments above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This work provides some insight into using the Laplace mechanism on the algorithms concerning the best arm identification problem bandits with a constraint of fixed confidence. The analysis carried out in this work enumerates general properties of differentially private FC-BAI and then proposes an algorithm based on those bounds.

STRENGTHS:
Analysis carried out in this work is general to all algorithms that are designed to solve the BAI problem where there are some guarantees presented on the sample complexity of such class of algorithms.

WEAKNESSES:
No weaknesses found in this paper, as per the knowledge of the reviewer.

QUESTIONS:
-

LIMITATIONS:
No additional limitations can be seen in this work. All the limitations, to the reviewer's knowledge, have been addressed properly in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of Best Arm Identification (BAI) for bandit problems under the constraints of differential privacy. There are some $K$ distributions and the solver interacts with the arms over a sequence of rounds. In each round they choose a distribution and receive a reward according to that distribution. The goal is to identify the arm with the highest expected reward with certainty $1-\\delta$ where $\\delta$ is a Fixed Confidence (FC) parameter given to the solver. \n\nMotivated by real-world problems like clinical trials for testing doses/efficacy of drugs, the authors suggest studying this problem under the constraint of differential privacy - this is relevant when the rewards received might be private information, such as the medical data of a patient's response to the drug/dose they were administered.\n\nThe contributions of this paper are as follows:\n1. The authors derive a lower bound on the sample complexity for DP-FC-BAI problem - their results suggest that the problem can be characterized in the low-privacy and high privacy regimes. In the former it seems that the sample complexity of the problem is unaffected by the DP constraint.\n2. They present an algorithm, AdaP-TT, which is a differentially private version of the Top Two Upper Confidence Bound (TTUCB) algorithm from prior work. The sample complexity of this algorithm matches the lower bound asymptotically in the high-privacy regime.\n3. The conduct some experiments showing that their algorithm has lower sample complexity compared to another differentially private approach (DP Successive-Elimination).\n\nSTRENGTHS:\n1. The problem being studied is very well-motivated - in particular the example of dose finding and the necessity of privacy preserving BAI is compelling.\n2. The authors provide a lower bound and characterize the hardness of the problem.\n3. The algorithm they provide (privatizing the optimal sample complexity achieving method in the non-private setting) seems to work well in practice based on their experiments. At a cursory look the performance of the algorithms seem to corroborate the constant factors in the sample complexity terms that the authors derived.\n\nWEAKNESSES:\n1. I found this paper very hard to read - the questions being considered by the authors make sense but the expressions derived are not easy to make sense of. The presentation is hard to follow - some terms are introduced without much context (such as ``Top Two algorithms\" on the second page) and this paper seems hard for non-experts to follow.\n2. On page 2 the authors say \"A calibrated amount of noise, from a Laplace [DR14] or Gaussian distribution [DRS22], is injected into an algorithm to ensure DP. The noise scale is set to be proportional to the algorithm’s sensitivity and inversely proportional to the privacy budget $\\varepsilon$.\" I think this statement is somewhat misleading - DP can be achieved in many ways and obfuscating noise is routinely drawn from many distributions (including the Gumbel distribuion, Poisson distributions, or other ad-hoc methods depending on the setting). Indeed, as the authors themselves find in their lower bound, sometimes privacy can be achieved for free. Similarly, the way the scale of the noise is chosen can be very involved - it is not always possible to derive a uniform bound on the sensitivity of an algorithm or such a method could give a very sub-optimal result. It is also important to note that not all problems admit DP solutions - there is no meaningful DP way to compute the XOR of $n$ private bits, for instance.\n3. The expressions in the lower bound are hard to interpret - ideally one might want to characterize them only in terms of the problem parameters i.e. the number of arms $K$, the confidence parameter $\\delta$ and the privacy parameter $\\varepsilon$ - the given bound seems to be more general which is great (as it describes the hardness in terms of the expected rewards), but is it possible to instead substitute a hard instance for the $\\mu_i$ and $\\Delta_a$ values and derive a more readable corollary?\n4. The approach does not seem to be very novel as it reduces to essentially just applying the Laplace mechanism at one point of the TTUCB algorithm from prior work. The sensitivity admits a straightforward bound as well.\n\nQUESTIONS:\n1. You define a low and high privacy regime based on $\\varepsilon$, $T_{KL}^*$ and $T_{TV}^*$, but the expressions for these depend upon $\\mu_i$ which the solver does not have access to. Operationally, is there any way for the solver to know whether they are in the low privacy or high privacy regime?\n2. The claim that there is no additional cost of privacy in terms of sample complexity for the low-privacy regime seems to need some more justification - there is a matching upper bound in the high-privacy regime but since there is no matching upper bound for the low-privacy regime, couldn't there just be a stronger lower bound which holds that we are not yet aware of?\n\nLIMITATIONS:\nPotential negative societal impacts do not need to be discussed here as this is primarily a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides novel theoretical guarantees to quantify the cost of privacy in DP-FC-BAI (and $\\epsilon$ global DP). Specifically, the authors show that the complexity depends on an information theoretic total variation metric. Although, this holds in the high privacy regime, this is reasonable and possibly connected with results on strong data processing inequalities in prior works. Through their lower bound ,the authors show consistency and recover the non-private regime as well. Additionally, the authors propose a variant of existing algorithms with asymptotically matching upper bound.\n\nSTRENGTHS:\nThe originality of the paper is clear. The presentation is clear and the contributions are significant. The total variation characteristic time is a novel quantity that it is introduced in this work. The proposed algorithm (AdaP-TT) is also novel and achieves matching guarantees with the proposed lower bound. The authors manage to characterize a phase transition effect: in the low privacy regime the privacy can be achieved for free. This is correct and has been observed in prior works as well (but for Thomson sampling). The results of the work are the first lower bounds for DP BAI, and they successfully characterize the hardness of the problem.\n\nWEAKNESSES:\nThe proof techniques appear to be mostly applications of known proof techniques. Sketches of proofs for the lower and upper bounds can potentially improve further the quality of the paper. The sketches should provide details on the novelty of the proof technique and several steps of the proof (similarly to proof of Lemma 1 in the Appendix).   \n\n\nA few references are missing, for instance \"On the Renyi differential privacy of the shuffle model\" by Girgis et al., \"Secure best arm identification in multi-armed bandits\"  by Ciucanu at al.\n\nFinally, the format of some references in the bibliography can be improved, some references point to older version of prior works on arxiv, while the papers have been published, for instance see [KNSS20].\n\nQUESTIONS:\nWhich parts of the proofs technique are novel? Could you please include proof sketches that emphasize on the proof technique?\n\nDoes the total variation uniquely characterize the hardness of the problem? Can the lower bounds be expressed in terms of the KL divergence instead?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of BAI with fixed confidence under ϵ-global Differential Privacy (DP). The authors derive a lower bound on the sample complexity of any δ-correct BAI algorithm satisfying ϵ-global DP. In addition, the authors also design an ϵ-global DP variant of the Top Two algorithm, named AdaP-TT. AdaP-TT runs in arm-dependent adaptive episodes and adds Laplace noise to ensure a good privacy-utility trade-off. The authors provide an asymptotic upper bound on the sample complexity of AdaP-TT, which matches the lower bound up to multiplicative constants in the high-privacy regime. Finally, the authors also conduct experiments to validate their theoretical results.\n\nSTRENGTHS:\n1.\tThe studied problem, differential privacy in best arm identification, is interesting and finds applications such as clinical trials and hyper-parameter tuning.\n2.\tThe provided lower bound result is very interesting. The authors give a deep discussion on the problem hardness in two regimes. In the high-privacy regime, the hardness depends on a coupled effect of privacy and a novel information-theoretic quantity, i.e., total variation characteristic time. In the low-privacy regime, the sample complexity lower bound is the same as that in classic (non-private) best arm identification.\n\nWEAKNESSES:\n1.\tThe writing of this paper should be improved. This paper is very dense and difficult to follow.\n2.\tThe authors only provide asymptotic sample complexity for their algorithm AdaP-TT. Can the authors provide a non-asymptotic sample complexity, or give some ideas?\n3.\tCan the authors discuss the technical novelty compared to existing works on DP with regret minimization? What is the unique challenge brought by the best arm identification setting?\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nPlease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of best arm identification with fixed confidence under ϵ-global Differential Privacy. First, they provide a lower bound of the BAI-DP problem showing that two different regimes depend on the privacy budget of $\\epsilon$. They propose an ϵ-global DP variant of the Top Two algorithm, called AdaP-TT, and provide its asymptotic sample complexity. They also conduct experiments that confirm the theoretical validity.\n\nSTRENGTHS:\n(originality) \n\n-They first derive the hardness result giving a lower bound for the problem of best arm identification with fixed confidence under ϵ-global Differential Privacy.\nFor the high-private regime, the lower bound depends on the Total Variation Characteristic Time, which shares a similar spirit with the regret minimization with ϵ-global DP [AB22]. For low-private regimes, the lower bound reduces to the classical no-private lower bound.\n\n-As an algorithmic contribution, they devise variants of Top Two algorithms, called AdaP-TT, based on non-private TTUCB[JD22]. They propose a sample complexity in an asymptotic regime, which is the order-optimal up to some constants.\n\n\n(quality) The paper is well-structured and easy to follow.\n\nWEAKNESSES:\n-As discussed in the paper, in BAI setting the only Information is leaked when publishing all the sequence of actions $\\{a_s\\}_{s<t}$ and the final recommendation a^*, which might not be frequently common situations, since BAI setting, the decision of final output is more interested. The presented problem is more of an artificial problem, although theoretical results are valid and new insights. \n\n-Related above, the privacy issues should be more important for the case where multiple players may exist rather than only a single player is involved, as in federated learning. With regard to multi-agent bandit problems, recently the following paper studied the best arm identification with DP setting and propose the DP successive elimination based on [SS19].\n\n\n [Rio et al 2023] Multi-Agent Best Arm Identification with Private Communications.\nAlexandre Rio, Merwan Barlier, Igor Colin, Marta Soare. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:29082-29102, 2023.\n\n\nI believe that a comparison with this paper should be required before the publication, even though [Rio et al 2023] might have been public after the NeurIPS submission date (So lacking this comparison in the current is not the reason for rejection). \n [Rio et al 2023]  consider a lower bound but only for the extreme non-private setting, where agents share all their raw reward samples with the central coordinator. Therefore, the submitted paper’s analysis for lower bound might be still novel. However, DP successive elimination proposed in [Rio et al 2023] could cover the single agent BAI-DP. Therefore the authors could do a comparison with this paper.\n======After rebuttal======\nMy concerns are fully addressed. Hence I changed my score from 5 to 7.\n\nQUESTIONS:\nCould you provide comparison with [Rio et al 2023]?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}