SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Achraf Azize
Équipe Scool, Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRISTAL F-59000 Lille, France achraf.azize@inria.fr
Équipe Scool, Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRISTAL F-59000 Lille, France marc.jourdan@inria.fr
Aymen Al Marjani
UMPA, ENS Lyon Lyon, France aymen.al_marjani@ens-lyon.fr
Équipe Scool, Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRISTAL F-59000 Lille, France debabrota.basu@inria.fr
Best Arm Identification (BAI) problems are progressively used for data-sensitive applications, such as designing adaptive clinical trials, tuning hyper-parameters, and conducting user studies to name a few. Motivated by the data privacy concerns invoked by these applications, we study the problem of BAI with fixed confidence under  $\epsilon$ -global Differential Privacy (DP). First, to quantify the cost of privacy, we derive a lower bound on the sample complexity of any  $\delta$ -correct BAI algorithm satisfying  $\epsilon$ -global DP. Our lower bound suggests the existence of two privacy regimes depending on the privacy budget  $\epsilon$ . In the high-privacy regime (small  $\epsilon$ ), the hardness depends on a coupled effect of privacy and a novel information-theoretic quantity, called the Total Variation Characteristic Time. In the low-privacy regime (large  $\epsilon$ ), the sample complexity lower bound reduces to the classical non-private lower bound. Second, we propose AdaP-TT, an  $\epsilon$ -global DP variant of the Top Two algorithm. AdaP-TT runs in arm-dependent adaptive episodes and adds Laplace noise to ensure a good privacy-utility trade-off. We derive an asymptotic upper bound on the sample complexity of AdaP-TT that matches with the lower bound up to multiplicative constants in the high-privacy regime. Finally, we provide an experimental analysis of AdaP-TT that validates our theoretical results.
We study the stochastic multi-armed bandit problem [LS20], which allows us to reflect on fundamental information-utility trade-offs involved in interactive sequential learning. Specifically, in a bandit problem, a learning agent is exposed to interact with  $K$  unknown probability distributions  $\{\nu_{1},\dots,\nu_{K}\}$  with bounded expectations, referred as the reward distributions (or arms).  $\pmb{\nu} \triangleq \{\nu_{1},\dots,\nu_{K}\}$  is called a bandit instance. At every step  $t > 0$ , the agent chooses to interact with one of the reward distributions  $\nu_{A_t}$  for an  $A_{t} \in [K]$ , and obtains a sample (or reward)  $r_t$  from it. The goal of the agent can be of two types: (a) maximise the reward accumulated over time, or equivalently to minimise the regret, and (b) to find the reward distribution (or arm) with highest expected reward. The first problem is called the regret-minimisation problem [ACBF02], while the second one is called the Best Arm
Identification (BAI) problem [KCG16]. In this paper, we focus on the BAI problem, i.e. to compute
$$
a ^ {\star} \triangleq \underset {a \in [ K ]} {\arg \max } \underset {r \sim \nu_ {a}} {\mathbb {E}} [ r ] \triangleq \underset {a \in [ K ]} {\arg \max } \mu_ {a}. \tag {BAI}
$$
With its advent in 1950s [Bec54, Bec58] and resurgence in last two decades [MT04, GGL12, JMNB14, KCG16, DKM19], BAI has been extensively studied with different structural assumptions (Fixed-confidence: [JN14]; Fixed-budget: [CL16]; Non-stochastic: [JT16]; Best-of-both-worlds:  $\mathrm{[AYBG^{+}18]}$ ; Linear: [SLM14]). In this paper, we specifically investigate the Fixed Confidence BAI problem, in brief FC-BAI, that yields a  $\delta$ -correct recommendation  $\hat{a} \in [K]$  satisfying  $\operatorname*{Pr}(\hat{a} \neq a^{\star}) \leq \delta$ . FC-BAI is increasingly deployed for different applications, such as clinical trials [AKR21], hyper-parameter tuning  $\mathrm{[LJD^{+}17]}$ , communication networks [LPJ22], online advertisement  $\mathrm{[CLK^{+}14]}$ , crowd-sourcing [ZCL14], user studies [LEHT22], and pandemic mitigation  $\mathrm{[LVR^{+}19]}$  to name a few. All of these applications often involve the sensitive and personal data of users, which raises serious data privacy concerns  $\mathrm{[TBD^{+}16]}$ , as illustrated in Example 1.
Example 1 (Adaptive dose finding trial). In a dose-finding trial, one physician decides  $K$  possible dose levels of a medicine based on preliminary studies (Typically,  $K \in \{3, \dots, 10\}$  in practice [AKR21]). At each step  $t$ , a patient is chosen from a local pool of volunteers and a dose level  $a_t \in [K]$  is applied to the patient. Following that, the effectiveness of the dose on the patient, i.e.  $r_t \in \mathbb{R}$  is observed. The goal of the physician is to recommend after the trial, which dose level is most effective on average, i.e. the dose level  $a^*$  that maximises the expected reward. Here, every application of a dose level and the patient's reaction to it exposes information regarding the medical conditions of the patient. Additionally, at each step  $t$  of an adaptive sequential trial, the physician can use an FC-BAI algorithm that observes the previous history of dose levels  $\{a_s\}_{s < t}$  and their effectiveness  $\{r_s\}_{s < t}$  to decide on the next dose level  $a_t$  to test. When releasing the experimental findings of the trial to health authorities, the physician should thoroughly detail the experimental protocol. This includes the dose allocated to each patient  $\{a_s\}_{s < t}$  and the final recommended dose level  $a^*$ . Thus, even if the sequence of reactions to doses  $\{r_s\}_{s \leq t}$  is kept secret, publishing the sequence of chosen dose levels  $\{a_s\}_{s \leq t}$  and the final recommended dose level  $a^*$  computed using the history can leak information regarding patients involved in the trial.
This example demonstrates the need for privacy in best-arm identification. In this paper, we investigate privacy-utility trade-offs for a privacy-preserving algorithm in FC-BAI. Specifically, we use the celebrated Differential Privacy (DP) [DR14] as the framework to preserve data privacy. DP ensures that an algorithm's output is unaffected by changes in input by a single data point. By limiting the amount of sensitive information that an adversary can deduce from the output, DP renders an individual corresponding to a data point 'indistinguishable'. A popular way to achieve DP is to inject a calibrated amount of noise, from a Laplace [DR14] or Gaussian distribution [DRS22], into the algorithm. The scale of the noise is set to be proportional to the algorithm's sensitivity and inversely proportional to the privacy budget  $\epsilon$ . Specifically, we study  $\epsilon$ -global  $DP$ , where users trust the centralised decision-maker with access to the raw sensitive rewards. For example, in an adaptive dose-finding trial, the patients trust the physician conducting the trial. Thus, at any time  $t$ , she has access to all the true history  $\{a_s, r_s\}_{s < t}$ , and it is her duty to design an algorithm such that publishing  $\{a_s\}_{s \leq t}$  and the recommended optimal dose  $a^*$  obeys  $\epsilon$ -DP given the sensitive input, i.e., the effectiveness of the dose levels on the patients  $\{r_s\}_{s \leq t}$ . We define the notion of  $\epsilon$ -global  $DP$  for BAI rigorously in Section 2.
For different settings of bandits, the cost of  $\epsilon$ -global DP and optimal algorithm design techniques are widely studied in the regret-minimization problem [MT15, TD16, SS19, SS18, NR18, BDT19, AB22]. Recently, a problem-dependent lower bound on regret of stochastic multi-armed bandits with  $\epsilon$ -global DP and an algorithm matching the regret lower bound is proposed by [AB22]. In contrast, DP is meagerly studied in the FC-BAI problem of bandits [SS19, KNSS21]. Though efficient algorithm design in FC-BAI literature is traditionally propelled by deriving tight lower bounds, we do not have any explicit sample complexity lower bound for FC-BAI satisfying  $\epsilon$ -global DP. Here, by an 'efficient' algorithm, we refer to the FC-BAI algorithms that aim to minimise the expected number of samples required (alternatively called, expected stopping time) to find a  $\delta$ -correct recommendation. Presently, we know neither the minimal cost in terms of sample complexity for ensuring DP in FC-BAI, nor the feasibility of efficient algorithm design to achieve the minimal cost.
Motivated by this gap in the literature, we aim to address two questions:
1. What is the fundamental hardness of ensuring Differential Privacy in the Best-Arm Identification with fixed confidence problem ( $\epsilon$ -DP-FC-BAI) in terms of a lower bound on the sample complexity?
2. How to design an efficient  $\epsilon$ -DP-FC-BAI algorithm that achieves the lower bound order-optimally?
Our contributions. These questions have led to the following contributions:
Lower bound. Efficient algorithm design in BAI literature is propelled by the derivation of lower bounds on sample complexity. [KCG16] derive the first lower bounds for classical fixed-confidence BAI setting without privacy, which is further improved in [GK16] by introducing KL characteristic time  $T_{\mathrm{KL}}^{\star}$  (Corollary 1). Motivated by this, we prove the first-known lower bound on sample complexity for FC-BAI with  $\epsilon$ -global DP (Theorem 2). The proof employs a similar sequential coupling argument as in the regret lower bound for bandits with  $\epsilon$ -global DP [SS18, AB22]. This similarity is also reflected in the existence of two privacy regimes depending on the privacy budget  $\epsilon$ . And for both lower bounds, the Total Variation (TV) appears to be the information-theoretic measure that captures the hardness in the high privacy regime. Specifically, the TV characteristic time  $(T_{\mathrm{TV}}^{\star}, \text{Corollary 1})$  serves as the BAI counterpart to the TV-distinguishability gap  $(t_{\mathrm{inf}})$  in the problem-dependent regret lower bound for bandits with  $\epsilon$ -global DP as in [AB22, Theorem 3].
Algorithms for BAI with fixed confidence (FC-BAI). The optimal sample complexity for the non-private FC-BAI problem (i.e.,  $T_{\mathrm{KL}}^{\star}$ ) is well-understood [GK16], and algorithms are proposed with the aim to achieve this lower bound. Early approaches involved Successive Elimination (SE) based algorithms [EDMMM06] with uniform sampling to find the optimal arm. Inspired by the success of Upper Confidence Bound (UCB) algorithms in the regret setting, the Lower Upper Confidence Bound (LUCB) algorithm was proposed [GGL12]. However, neither SE nor LUCB algorithms achieve asymptotic optimality. The Track-and-Stop (TnS) algorithm introduced in [GK16] was the first to asymptotically achieve the exact optimal sample complexity  $T_{\mathrm{KL}}^{\star}$ . TnS attains asymptotic optimality by solving a plug-in estimate of the lower bound optimization problem at each step. The game-based approach presented in [DKM19] relaxes this requirement by casting the optimization problem as an unknown game and proposing sampling rules based on iterative strategies to estimate and converge to its saddle point. Finally, the Top Two algorithms arose as an identification strategy based on the praised Thompson Sampling algorithm for regret minimization [Rus16]. In recent years, numerous variants have been analyzed and shown to be asymptotically near optimal [JDB+22]. At every step, a Top Two sampling rule selects the next arm to sample from among two candidate arms, a leader and a challenger. In addition to their great empirical performance, and easy implementation compared to TnS and Game-based algorithms, the Top Two algorithms achieve near asymptotic optimality. In Sec. 4, we derive an  $\epsilon$ -global DP version of a Top Two algorithm: AdaP-TT.
$\epsilon$ -DP BAI algorithms. DP-SE [SS19] is an  $\epsilon$ -global DP version of the Successive Elimination algorithm. Although the algorithm was proposed and analysed for the regret minimisation setting in [SS19], it is possible to derive a sample complexity from the analysis in [SS19]. We compare in-depth DP-SE and AdaP-TT, both theoretically (Section 4) and experimentally (Section 5). In both aspects, our proposed algorithm AdaP-TT outperforms DP-SE. Another adaptation of DP-SE, namely DP-SEQ, is proposed in [KNSS21] for the problem of privately finding the arm with the highest quantile at a fixed level. But this is a different setting of interest than the present paper. [RBCS23] also studies privacy for BAI under fixed confidence but with multiple agents. They propose and analyse the sample complexity of DP-MASE, a multi-agent version of DP-SE. They show that multi-agent collaboration leads to better sample complexity than independent agents, even under privacy constraints. While the multi-agent setting with federated learning allows tackling large-scale clinical trials taking place at several locations simultaneously, we study the single-agent setting, which is relevant for many small-scale clinical trials (see Example 1).
2 Differential privacy and best arm identification
Background: Differential Privacy (DP). DP ensures protection of an individual's sensitive information when her data is used for analysis. A randomised algorithm satisfies DP if the output of the algorithm stays almost the same, regardless of whether any single individual's data is included in or excluded from the input. This is achieved by adding controlled noise to the algorithm's output.
Definition 1  $((\epsilon, \delta)$ -DP [DR14]). A randomised algorithm  $\mathcal{A}$  satisfies  $(\epsilon, \delta)$ -Differential Privacy (DP) if for any two neighbouring datasets  $\mathcal{D}$  and  $\mathcal{D}'$  that differ only in one entry, i.e.  $d_{Ham}(\mathcal{D}, \mathcal{D}') = 1$ , and for all sets of output  $\mathcal{O} \subseteq \mathrm{Range}(\mathcal{A})$ ,
$$
\Pr [ \mathcal {A} (\mathcal {D}) \in \mathcal {O} ] \leq e ^ {\epsilon} \Pr [ \mathcal {A} \left(\mathcal {D} ^ {\prime}\right) \in \mathcal {O} ] + \delta , \tag {1}
$$
where the probability space is over the coin flips of the mechanism  $\mathcal{A}$ , and for some  $(\epsilon, \delta) \in \mathbb{R}^{\geq 0} \times \mathbb{R}^{\geq 0}$ . If  $\delta = 0$ , we say that  $\mathcal{A}$  satisfies  $\epsilon$ -DP. A lower privacy budget  $\epsilon$  implies higher privacy.
The Laplace mechanism [DNPR10, DR14] ensures  $\epsilon$ -DP by injecting controlled random noise into the output of the algorithm, which is sampled from a calibrated Laplace distribution (as specified in Theorem 1). We use  $Lap(b)$  to denote the Laplace distribution with mean 0 and variance  $2b^{2}$ .
Theorem 1 (ε-DP of Laplace mechanism (Theorem 3.6, [DR14])). Let us consider an algorithm  $f: \mathcal{X} \to \mathbb{R}^d$  with sensitivity  $s(f) \triangleq \max_{\mathcal{D}, \mathcal{D}' \text{ s.t } |\mathcal{D} - \mathcal{D}'|_{\mathrm{Hamming}} = 1} |f(\mathcal{D}) - f(\mathcal{D}')|1$ . Here,  $\| \cdot \|_1$  is the  $L_1$  norm on  $\mathbb{R}^d$ . If  $d$  noise samples  $\{N_i\}_{i=1}^d$  are generated independently from  $\text{Lap}\left(\frac{s(f)}{\epsilon}\right)$ , then the output injected with the noise, i.e.  $f(\mathcal{D}) + [N_1, \ldots, N_d]$ , satisfies  $\epsilon$ -DP.
Background: BAI with fixed confidence. Now, we describe the canonical best-arm identification problem with fixed confidence (FC-BAI). BAI is a variant of pure exploration, where the goal is to
Algorithm 1 Sequential interaction between a BAI strategy and users
1: Input: A BAI strategy  $\pi = (S_t, \mathrm{Rec}_t)_{t \geq 1}$  and Users  $\{u_t\}_{t \geq 1}$  represented by the table  $\underline{\mathbf{d}}$   
2: Output: A stopping time  $\tau$ , a sequence of samples actions  $\underline{a}^{\tau} = (a_1, \dots, a_\tau)$  and a recommendation  $\hat{a}$  satisfying  $\epsilon$ -global DP  
3: for  $t = 1, \dots, \mathbf{do}$   
4:  $\pi$  recommends action  $a_t \sim S_t(\cdot | a_1, r_1, \dots, a_{t-1}, r_{t-1})$   
5: if  $a_t = \top$  then  
6: Halt. Return  $\tau = t$  and  $\hat{a} \sim \mathrm{Rec}_t(\cdot | a_1, r_1, \dots, a_{t-1}, r_{t-1})$   
7: else  
8:  $u_t$  sends the sensitive reward  $r_t \triangleq \underline{\mathbf{d}}_{t, a_t}$  to  $\pi$   
9: end if  
10: end for
identify the optimal arm. In FC-BAI, the learner is provided with a confidence level  $1 - \delta \in (0,1)^{1}$ . Learner aims to recommend an arm that is optimal with probability at least  $1 - \delta$ , while using as few samples as possible. To achieve this, the learner defines a FC-BAI strategy to interact with the bandit instance  $\pmb{\nu} = \{\nu_{a}:a\in [K]\}$ . We denote the action played at step  $t$  by  $a_{t}$ , and the corresponding observed reward by  $r_t\sim \nu_{a_t}$ .  $\mathcal{H}_t = (a_1,r_1,\dots ,a_t,r_t)$  is the history of actions played and rewards collected until time  $t$ . We augment the action set by a stopping action  $\top$ , and write  $a_{t} = \top$  to denote that the algorithm has stopped before step  $t$ . A FC-BAI strategy  $\pi$  is composed of
i. A pair of sampling and stopping rules  $(\mathrm{S}_t:\mathcal{H}_{t - 1}\to \mathcal{P}([|1,K|]\cup \{\top \}))_{t\geq 1}$ . For an action  $a\in [K]$ ,  $\mathrm{S}_t(a\mid \mathcal{H}_{t - 1})$  denotes the probability of playing action  $a$  given history  $\mathcal{H}_{t - 1}$ . On the other hand,  $\mathrm{S}_t(\top \mid \mathcal{H}_{t - 1})$  is the probability of the algorithm halting given  $\mathcal{H}_{t - 1}$ . For any history  $\mathcal{H}_{t - 1}$ , a consistent sampling and stopping rule  $\mathrm{S}_t$  satisfies  $\mathrm{S}_t(\top \mid \mathcal{H}_{t - 1}) = 1$  if  $\top$  has been played before  $t$ .
ii. A recommendation rule  $(\operatorname{Rec}_t: \mathcal{H}_{t-1} \to \mathcal{P}([|1, K|]))_{t>1}$ . A recommendation rule dictates  $\operatorname{Rec}_t(a \mid \mathcal{H}_{t-1})$ , i.e. the probability of returning action  $a$  as a guess for the best action given  $\mathcal{H}_{t-1}$ .
We denote by  $\tau$  the stopping time of the algorithm, i.e. the first step  $t$  demonstrating  $a_{t} = \top$ . A BAI strategy  $\pi$  is called  $\delta$ -correct for a class of bandit instances  $\mathcal{M}$ , if for every instance  $\pmb{\nu} \in \mathcal{M}$ ,  $\pi$  recommends the optimal action  $a^{\star}(\pmb{\nu}) = \arg \max_{a \in [K]} \mu_{a}$  with probability at least  $1 - \delta$ , i.e.  $\mathbb{P}_{\pmb{\nu},\pi}(\tau < \infty, \widehat{a} = a^{\star}(\pmb{\nu})) \geq 1 - \delta$ .
FC-BAI with  $\epsilon$ -global DP ( $\epsilon$ -DP-FC-BAI). Now, we formally define  $\epsilon$ -global DP for FC-BAI, where the BAI strategy (a.k.a. the centralised decision maker) is trusted with all the intermediate rewards. We represent each user  $u_{t}$  by the vector  $\mathbf{x}_{t} \triangleq (x_{t,1}, \ldots, x_{t,K}) \in \mathbb{R}^{K}$ , where  $x_{t,a}$  represents the potential reward observed, if action  $a$  was recommended to user  $u_{t}$ . Due to the bandit feedback, only  $r_{t} = x_{t,a_{t}} \sim \nu_{a_{t}}$  is observed at step  $t$ . We use an underline to denote any sequence. Thus, we denote the sequence of sampled actions until  $T$  as  $\underline{a}^T = (a_1, \ldots, a_T)$ . We further represent a set of users  $\{u_{t}\}_{t=1}^{T}$  until  $T$  by the table of potential rewards  $\underline{\mathbf{d}}^{\mathbf{T}} \triangleq \{\mathbf{x}_1, \ldots, \mathbf{x}_T\} \in (\mathbb{R}^K)^T$ . First, we observe that  $\underline{\mathbf{d}}^T$  is the sensitive input generated through interaction with the users, and  $(\underline{a}^T, \hat{a}, T)$  is the output of the BAI strategy. Hence, we define the probability that the BAI strategy  $\pi$  samples the action sequence  $\underline{a}^T$ , recommends the action  $\hat{a}$ , and halts at time  $T$ , as
$$
\pi \left(\underline {{a}} ^ {T}, \hat {a}, T \mid \underline {{\mathbf {d}}} ^ {T}\right) \triangleq \operatorname {R e c} _ {T + 1} (\hat {a} \mid \mathcal {H} _ {T}) \mathrm {S} _ {T + 1} (\top \mid \mathcal {H} _ {T}) \prod_ {t = 1} ^ {T} \mathrm {S} _ {t} \left(a _ {t} \mid \mathcal {H} _ {t - 1}\right) \tag {2}
$$
where  $T$  users under interaction are represented by the table of potential rewards  $\underline{\mathbf{d}}^T$
Thus, a BAI strategy satisfies  $\epsilon$ -global DP if the probability defined in Eq. (2) is similar when the BAI strategy interacts with two neighbouring tables of rewards differing by a user (i.e., a row in  $\underline{\mathbf{d}}^T$ ).
Definition 2 (ε-global DP for BAI). A BAI strategy satisfies ε-global DP, if for all  $T \geq 1$ , all neighbouring table of rewards  $\underline{\boldsymbol{d}}^T$  and  $\underline{\boldsymbol{d}}^T$ , i.e.  $d_{Ham}(\underline{\boldsymbol{d}}^T, \underline{\boldsymbol{d}}^T) = 1$ , all sequences of sampled actions  $\underline{\boldsymbol{a}}^T \in [K]^T$  and recommended actions  $\widehat{\underline{a}} \in [K]$  we have that
$$
\pi (\underline {{a}} ^ {T}, \widehat {a}, T \mid \underline {{d}} ^ {T}) \leq e ^ {\epsilon} \pi (\underline {{a}} ^ {T}, \widehat {a}, T \mid \underline {{d}} ^ {T}).
$$
<sup>1</sup>We remind not to confuse risk level  $\delta$  with the  $\delta$  of  $(\epsilon, \delta)$ -DP. Hereafter, we consider  $\epsilon$ -global DP as the privacy definition, and  $\delta$  always represents the risk (or probability of mistake) of the BAI strategy.
Definition 2 can be seen as a BAI counterpart of the  $\epsilon$ -global DP definition proposed in [AB22] for regret minimization. We demonstrate the BAI strategy-Users interaction in Algorithm 1.
Remark 1. It is possible to consider that the output of a BAI strategy is only the final recommended action  $\hat{a}$ , i.e. not publishing the intermediate actions  $\underline{a}^T$ . This gives a weaker definition of privacy compared to Definition 2, since the latter defends against adversaries that may look inside the execution of the BAI strategy, i.e. pan-privacy  $[DNP^{+}10]$ . In addition, Definition 2 is needed in practice. For example, in the case of dose-finding (Example 1), the experimental protocol, i.e. the intermediate actions, needs to be published too.
The goal in  $\epsilon$ -DP-FC-BAI is to design a  $\delta$ -correct  $\epsilon$ -global DP algorithm, with  $\tau$  as small as possible.
3 Lower bound on sample complexity for FC-BAI with  $\epsilon$ -global DP
The central question that we address in this section is
How many additional samples a BAI strategy must select for ensuring  $\epsilon$ -global DP?
In response, we prove a lower bound on the sample complexity of any  $\delta$ -correct  $\epsilon$ -DP BAI strategy. Our lower bound features problem-dependent characteristic times reminiscent of the FC-BAI setting.
Let  $\pmb{\nu} \triangleq \{\nu_{a} : a \in [K]\}$  be a bandit instance, consisting of  $K$  arms with finite means  $\{\mu_{a}\}_{a \in [K]}$ . Now, we define the set of alternative instances as  $\mathrm{Alt}(\pmb{\nu}) \triangleq \{\pmb{\lambda} : a^{\star}(\pmb{\lambda}) \neq a^{\star}(\pmb{\nu})\}$ , i.e. the bandit instances with a different optimal arm than  $\pmb{\nu}$ . For two probability distributions  $\mathbb{P}, \mathbb{Q}$  on the same measurable space  $(\Omega, \mathcal{F})$ , the Total Variation (TV) distance is defined as TV  $(\mathbb{P} \parallel \mathbb{Q}) \triangleq \sup_{A \in \mathcal{F}} \{\mathbb{P}(A) - \mathbb{Q}(A)\}$ , while the KL divergence (or relative entropy) is KL  $(\mathbb{P} \parallel \mathbb{Q}) \triangleq \int \log \left(\frac{\mathrm{d}\mathbb{P}}{\mathrm{d}\mathbb{Q}}(\omega)\right) \mathrm{d}\mathbb{P}(\omega)$ , when  $\mathbb{P} \ll \mathbb{Q}$ , and  $\infty$  otherwise. We denote the probability simplex by  $\Sigma_K \triangleq \{\omega \in [0,1]^K : \sum_{a=1}^{K} \omega_a = 1\}$ .
First, we derive an  $\epsilon$ -global DP variant of the 'transportation' lemma, i.e. Lemma 1 in [KCG16].
Lemma 1 (Transportation lemma under  $\epsilon$ -global DP). Let  $\delta \in (0,1)$  and  $\epsilon > 0$ . Let  $\nu$  be a bandit instance and  $\lambda \in \mathrm{Alt}(\nu)$ . For any  $\delta$ -correct  $\epsilon$ -global DP BAI strategy, we have that
$$
6 \epsilon \sum_ {a = 1} ^ {K} \mathbb {E} _ {\boldsymbol {\nu}, \pi} \left[ N _ {a} (\tau) \right] \operatorname {T V} \left(\nu_ {a} \| \lambda_ {a}\right) \geq \mathrm {k l} (1 - \delta , \delta),
$$