SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C Confidence Intervals for Exchangeable Bernoulli Means 18
C.1 Non-Asymptotic Confidence Intervals 19
C.2 Asymptotic Confidence Intervals 23
C.3 Scaling of Higher-Order Bernstein Bounds (Proposition 4) 25
D Canary Design for Lifted DP: Details 25
D.1 Data Poisoning 26
D.2 Random Gradients 27
E Simulations with the Gaussian Mechanism: Details and More Results 27
E.1 Experiment Setup 27
E.2 Additional Experimental Results 30
F Experiments: Details and More Results 32
F.1 Training Details: Datasets, Models 32
F.2 DP and Auditing Setup 33
F.3 Miscellaneous Details 33
F.4 Additional Experimental Results 33
Prior to [17], privacy auditing required some access to the description of the mechanism. [5-7, 24, 43, 50, 51, 62] provide platforms with a specific set of functions to be used to implement the mechanism, where the end-to-end privacy of the source code can be automatically verified. Closest to our setting is the work of [18], where statistical testing was first proposed for privacy auditing, given access to an oracle that returns the exact probability measure of the mechanism. However, the guarantee is only for a relaxed notion of DP from [8] and the run-time depends super-linearly on the size of the output domain.
The pioneering work of [17] was the first to propose practical methods to audit privacy claims given a black-box access to a mechanism. The focus was on simple queries that do not involve any training of models. This is motivated by [16, 40], where even simple mechanisms falsely reported privacy guarantees. For such mechanisms, sampling a large number, say 500,000 in [17], of outputs is computationally easy, and no effort was made in [17] to improve the statistical trade-off. However, for private learning algorithms, training such a large number of models is computationally prohibitive. The main focus of our framework is to improve this statistical trade-off for auditing privacy. We first survey recent breakthroughs in designing stronger canaries, which is orthogonal to our main focus.
A.1 Auditing Private Machine Learning with Strong Canaries
Recent breakthroughs in auditing are accelerated by advances in privacy attacks, in particular membership inference. An attacker performing membership inference would like to determine if a particular data sample was part of the training set. Early work on membership inference [11, 23, 30, 52] considered algorithms for statistical methods, and more recent work demonstrates black-box attacks on ML algorithms [14, 33, 53, 67, 68]. [33] compares different notions of privacy by measuring privacy empirically for the composition of differential privacy [35], concentrated differential privacy [10, 21], and Rényi differential privacy [44]. This is motivated by [49], which compares different DP mechanisms by measuring the success rates of membership inference attacks. [31] attempts to measure the intrinsic privacy of stochastic gradient descent (without additional noise as in DP-SGD) using canary designs from membership inference attacks. Membership inference attacks have been shown to have higher success when the adversary can poison the training dataset [59].
A more devastating privacy attack is the extraction or reconstruction of the training data, which is particularly relevant for generative models such as large language models (LLMs). Several papers showed that LLMs tend to memorize their training data [13, 15], allowing an adversary to prompt the generative models and extract samples of the training set. The connection between the ability of an attacker to perform training data reconstruction and DP guarantees has been shown in recent work [26, 29].
When performing privacy auditing, a stronger canary design increases the success of the adversary in the distinguishing test and improves the empirical privacy bounds. The resulting hypothesis test can tolerate larger confidence intervals and requires less number of samples. Recent advances in privacy auditing have focused on designing such stronger canaries. [32] designs data poisoning canaries, in the direction of the lowest variance of the training data. This makes the canary out of distribution, making it easier to detect. [45] proposes attack surfaces of varying capabilities. For example, a gradient attack canary returns a gradient of choice when accessed by DP-SGD. It is shown that, with more powerful attacks, the canaries become stronger and the lower bounds become higher. [60] proposes using an example from the baseline training dataset, after changing the label, and introduces a search procedure to find a strong canary. More recently, [46] proposes a significantly improved auditing scheme for DP-SGD under a white-box access model where  $(i)$  the auditor knows that the underlying mechanism is DP-SGD with a spherical Gaussian noise with unknown variance, and  $(ii)$  all intermediate models are revealed. Since each coordinate of the model update provides an independent sample from the same Gaussian distribution, sample complexity is dramatically improved. [41] proposes CANIFE, a novel canary design method that finds a strong data poisoning canary adaptively under the federated learning scenario.
Prior work in this space shows that privacy auditing can be performed via privacy attacks, such as membership inference or reconstruction, and strong canary design results in better empirical privacy bounds. We emphasize that our aim is not to innovate on optimizing canary design for performing
privacy auditing. Instead, our framework can seamlessly adopt recently designed canaries and inherit their strengths as demonstrated in §5 and §6.
A.2 Improving Statistical Trade-offs in Auditing
Eq. (2) points to two orthogonal directions that can potentially improve the sample dependence: designing stronger canaries and improving the sample dependence of the confidence intervals. The former was addressed in the previous section. There is relatively less work in improving the statistical dependence, which our framework focuses on.
Given a pair of neighboring datasets,  $(D_0, D_1)$ , and for a query with discrete output in a finite space, the statistical trade-off of estimating privacy parameters was studied in [25] where a plug-in estimator is shown to achieve an error of  $O(\sqrt{de^{2\varepsilon} / n})$ , where  $d$  is the size of the discrete output space. [38] proposes a sub-linear sample complexity algorithm that achieves an error scaling as  $\sqrt{de^{\varepsilon} / n\log n}$  based on polynomial approximation of a carefully chosen degree to optimally trade-off bias and variance motivated by [27, 64]. Similarly, [37] provides a lower bound for auditing Rényi differential privacy. [9] trains a classifier for the binary hypothesis test and uses the classifier to design rejection sets. [3] proposes local search to find the rejection set efficiently. More recently, [69] proposes numerical integration over a larger space of false positive rate and true positive rate to achieve better sample complexity of the confidence region in the two-dimensional space. Our framework can be potentially applied to this confidence region scenario, which is an important future research direction.
A.3 Connections to Other Notions of Differential Privacy
Similar to Lifted DP, a line of prior works [36, 55, 61, 66] generalizes DP to include a distributional assumption over the dataset. However, unlike Lifted DP, they are motivated by the observation that DP is not sufficient for preserving privacy when the samples are highly correlated. For example, upon releasing the number of people infected with a highly contagious flu within a tight-knit community, the usual Laplace mechanism (with sensitivity one) is not sufficient to conceal the likelihood of one member getting infected when the private count is significantly high. One could apply group differential privacy to hide the contagion of the entire population, but this will suffer from excessive noise. Ideally, we want to add a noise proportional to the expected size of the infection. Pufferfish privacy, introduced in [36], achieves this with a generalization of DP that takes into account prior knowledge of a class of potential distributions over the dataset. A special case of  $\varepsilon$ -Pufferfish with a specific choice of parameters recovers a special case of our Lifted DP in Eq. (3) with  $\delta = 0$  [36, Section 3.2], where a special case of Theorem 3 has been proven for pure DP [36, Theorem 3.1]. However, we want to emphasize that our Lifted DP is motivated by a completely different problem of auditing differential privacy and is critical to breaking the barriers in the sample complexity.
B Properties of Lifted DP and Further Details
B.1 Equivalence Between DP and LiDP
In contrast to the usual  $(\varepsilon, \delta)$ -DP in Eq. (1), the probability in LiDP is over both the internal randomness of the algorithm  $\mathcal{A}$  and the distribution  $\mathcal{P}$  over the triplet  $(D_0, D_1, R)$ . Since we require that the definition holds only for lifted distributions  $\mathcal{P}$  that are independent of the algorithm  $\mathcal{A}$ , it is easy to show its equivalence to the usual notion of  $(\varepsilon, \delta)$ -DP.
Proof. Suppose  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -LiDP. Fix a pair of neighboring datasets  $D_0, D_1$  and an outcome  $R \subset \mathcal{R}$ . Define  $\mathcal{P}_{D_0, D_1, R}$  as the point mass on  $(D_0, D_1, R)$ , i.e.,
$$
\mathrm {d} \mathcal {P} _ {D _ {0}, D _ {1}, R} \left(D _ {0} ^ {\prime}, D _ {1} ^ {\prime}, R ^ {\prime}\right) = \mathbb {1} \left(D _ {0} ^ {\prime} = D _ {0}, D _ {1} ^ {\prime} = D _ {1}, R ^ {\prime} = R\right),
$$
so that  $\mathbb{P}_{\mathcal{A},\mathcal{P}_{D_0,D_1,R}}(\mathcal{A}(D_0)\in R) = \mathbb{P}(\mathcal{A}(D_0)\in R)$  and similarly for  $D_{1}$ . Then, applying the definition of  $(\varepsilon ,\delta)$ -LiDP w.r.t. the distribution  $\mathcal{P}_{D_0,D_1,R}$  gives
$$
\mathbb {P} _ {\mathcal {A}} \left(\mathcal {A} \left(D _ {1}\right) \in R\right) \leq e ^ {\varepsilon} \mathbb {P} _ {\mathcal {A}} \left(\mathcal {A} \left(D _ {0}\right) \in R\right) + \delta .
$$
Since this holds for any neighboring datasets  $D_0, D_1$  and outcome set  $R$ , we get that  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -DP.
Conversely, suppose that  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -DP. For any distribution  $\mathcal{P}$  over pairs of neighboring datasets  $D_0, D_1$  and outcome set  $R$ , we have by integrating the DP definition
$$
\int \mathbb {P} (\mathcal {A} (D _ {1}) \in R) \mathrm {d} \mathcal {P} (D _ {0}, D _ {1}, R) \leq e ^ {\varepsilon} \int \mathbb {P} (\mathcal {A} (D _ {0}) \in R) \mathrm {d} \mathcal {P} (D _ {0}, D _ {1}, R) + \delta . \tag {11}
$$
Next, we use the law of iterated expectation to get
$$
\begin{array}{l} \mathbb {P} _ {\mathcal {A}, \mathcal {P}} \left(\mathcal {A} \left(\boldsymbol {D} _ {0}\right) \in \boldsymbol {R}\right)) = \mathbb {E} _ {\mathcal {A}, \mathcal {P}} \left[ \mathbb {I} \left(\mathcal {A} \left(\boldsymbol {D} _ {0}\right) \in \boldsymbol {R}\right) \right] \\ = \mathbb {E} _ {\left(\boldsymbol {D} _ {0}, \boldsymbol {D} _ {1}, \boldsymbol {R}\right) \sim \mathcal {P}} \left[ \mathbb {E} _ {\mathcal {A}} \left[ \mathbb {I} \left(\mathcal {A} \left(\boldsymbol {D} _ {0}\right) \in \boldsymbol {R}\right) \mid \boldsymbol {D} _ {0}, \boldsymbol {R} \right] \right] \\ = \int \mathbb {E} _ {\mathcal {A}} \left[ \mathbb {I} \left(\mathcal {A} (\boldsymbol {D} _ {0}) \in \boldsymbol {R}\right) \mid \boldsymbol {D} _ {0} = D _ {0}, \boldsymbol {R} = R \right] \mathrm {d} \mathcal {P} (D _ {0}, D _ {1}, R) \\ \stackrel {(*)} {=} \int \mathbb {E} _ {\mathcal {A}} \left[ \mathbb {I} \big (\mathcal {A} (D _ {0}) \in R) \right] \mathrm {d} \mathcal {P} (D _ {0}, D _ {1}, R) \\ = \int \mathbb {P} _ {\mathcal {A}} \left(\mathcal {A} \left(D _ {0}\right) \in R\right) \mathrm {d} \mathcal {P} \left(D _ {0}, D _ {1}, R\right), \\ \end{array}
$$
where  $(\ast)$  followed from the independence of  $\mathcal{A}$  and  $\mathcal{P}$ . Plugging this and the analogous expression for  $\mathbb{P}(\mathcal{A}(D_1) \in \mathbf{R})$  into (11) gives us that  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -LiDP.
B.2 Auditing LiDP with Different Notions of Neighborhood
We describe how to modify the recipe of §3 for other notions of neighborhoods of datasets. The notion of the neighborhood in Definition 1 is also known as the "add-or-remove" neighborhood.
Replace-one Neighborhood. Two datasets  $D, D' \in \mathcal{Z}^*$  are considered neighboring if  $|D| = |D'|$  and  $|D \setminus D'| = |D' \setminus D| = 1$ . Roughly speaking, this leads to privacy guarantees that are roughly twice as strong as the add-or-remove notion of the neighborhood in Definition 1, as the worst-case sensitivity of the operation is doubled. We refer to [48, 63] for more details. Just like Definition 1, Definition 2 can also be adapted to this notion of neighborhood.
Auditing LiDP with Replace-one Neighborhood. The recipe of §3.2 can be slightly modified for this notion of neighborhood. The main difference is that the null hypothesis must now use  $K$  canaries as well, with one fresh canary.
The alternative hypothesis is the same — we train a model on a randomized training dataset  $D_{1} = D \cup \{c_{1}, \ldots, c_{K}\}$  augmented with  $K$  random canaries drawn i.i.d. from  $\mathcal{P}_{\mathrm{canary}}$ . Under the  $j^{\mathrm{th}}$  null hypothesis for each  $j \in [K]$ , we construct a coupled dataset  $D_{0,j} = D \cup \{c_{1}, \ldots, c_{j-1}, c_{j}', c_{j+1}, \ldots, c_{K}\}$ , where  $c_{j}'$  is a fresh canary drawn i.i.d. from  $\mathcal{P}_{\mathrm{canary}}$ . This coupling ensures that  $(D_{0,j}, D_{1})$  are neighboring with probability one. We restrict the rejection region  $R_{j}$  to now depend only on  $c_{j}$  and not in the index  $j$ , e.g., we test for the present of  $c_{j}$ .<sup>2</sup>
Note the symmetry of the setup. Testing for the presence of  $c_{j}$  in  $D_{0,j}$  is exactly identical to testing for the presence of  $c_{j}'$  in  $D_{1}$ . Thus, we can again rewrite the LiDP condition as
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {P} \left(\mathcal {A} \left(\boldsymbol {D} _ {1}\right) \in \boldsymbol {R} _ {k}\right) \leq \frac {e ^ {\varepsilon}}{m} \sum_ {j = 1} ^ {m} \mathbb {P} \left(\mathcal {A} \left(\boldsymbol {D} _ {1}\right) \in \boldsymbol {R} _ {j} ^ {\prime}\right) + \delta . \tag {12}
$$
Note the subtle difference between (4) and (12): both sides depend only on  $\mathcal{A}(D_1)$  and we have completely eliminated the need to train models on  $K - 1$  canaries.
From here on, the rest of the recipe is identical to §3 and Algorithm 1; we construct XBern confidence intervals for both sides of (12) and get a lower bound  $\hat{\varepsilon}$ .
C Confidence Intervals for Exchangeable Bernoulli Means
We give a rigorous definition of the multivariate Exchangeable Bernoulli (XBern) distributions and derive their confidence intervals. We also give proof of the correctness of the confidence intervals.
2Note that the test could have depended on  $c_{j}^{\prime}$  as well, but we do not need it here.
Definition 5 (XBern Distributions). A random vector  $(\pmb{x}_1, \dots, \pmb{x}_K) \in \{0, 1\}^K$  is said to be distributed as  $\mathrm{XBern}_K(\mu_1, \dots, \mu_K)$  if:
-  $\pmb{x}_1, \dots, \pmb{x}_K$  is exchangeable, i.e., the vector  $(\pmb{x}_1, \dots, \pmb{x}_K)$  is identical in distribution to  $(\pmb{x}_{\pi(1)}, \dots, \pmb{x}_{\pi(K)})$  for any permutation  $\pi : [K] \to [K]$ , and,
for each  $\ell = 1,\dots ,K$  , we have  $\mathbb{E}[m_{\ell}] = \mu_{\ell}$  , where
$$
\boldsymbol {m} _ {\ell} := \frac {1}{\binom {K} {\ell}} \sum_ {j _ {1} <   \dots <   j _ {\ell} \in [ K ]} \boldsymbol {x} _ {j _ {1}} \dots \boldsymbol {x} _ {j _ {\ell}}. \tag {13}
$$
We note that the XBern $_K$  distribution is fully determined by its  $K$  moments  $\mu_1, \ldots, \mu_K$ . For  $K = 1$ ,  $\mathrm{XBern}_1(\mu_1) = \mathrm{Bernoulli}(\mu_1)$  is just the Bernoulli distribution.
The moments  $m_{\ell}$  satisfy a computationally efficient recurrence.
Proposition 6. Let  $\pmb{x} \sim X\text{Bern}_K(\mu_1, \dots, \mu_K)$ . We have the following for all  $1 \leq \ell \leq K\pmb{m}_1$ :
$$
\boldsymbol {m} _ {\ell} = \frac {\binom {K m _ {1}} {\ell}}{\binom {K} {\ell}} \quad \text {a n d} \quad \boldsymbol {m} _ {\ell + 1} = \boldsymbol {m} _ {\ell} \left(\frac {K \boldsymbol {m} _ {1} - \ell}{K - \ell}\right). \tag {14}
$$
Computing the first  $\ell$  moments can thus be done in  $O(K + \ell)$  time rather than  $O(K^{\ell})$  by naively computing the sum in (13).
Proof of Proposition 6. We show that this holds for any fixed vector  $(x_{1},\ldots ,x_{K})\in \{0,1\}^{K}$  and their corresponding moments  $m_{1},\ldots ,m_{K}$  as defined in (13). Define the sums  $s_1\coloneqq \sum_{j = 1}^{K}x_{j} = Km_{1}$  and
$$
s _ {\ell} := \binom {K} {\ell} \sum_ {j _ {1} <   \dots <   j _ {\ell}} x _ {j _ {1}} \dots x _ {j _ {\ell}}. \tag {15}
$$
We can compute  $s_{\ell}$  by a counting argument: each term in the sum is non-zero only if all  $\ell$  of the  $x_{j_1}, \ldots, x_{j_\ell}$  are non-zero. The number of non-zero terms is equal to number of ways of selecting  $\ell$  items from a total of  $\sum_{j=1}^{K} x_j = s_1$  items, i.e.,
$$
s _ {\ell} = \binom {s _ {1}} {\ell} = \binom {K m _ {1}} {\ell}. \tag {16}
$$
Combining Equations (15) and (16) completes the proof.
Notation. In this section, we are interested in giving confidence intervals on the mean  $\mu_{1}$  of a XBern $_{K}(\mu_1,\ldots ,\mu_K)$  random variable from  $n$  i.i.d. observations:
$$
\boldsymbol {x} ^ {(1)}, \ldots , \boldsymbol {x} ^ {(n)} \stackrel {{\text {i . i . d .}}} {{\sim}} \operatorname {X B e r n} _ {K} (\mu_ {1}, \ldots , \mu_ {K}).
$$
We will define the confidence intervals using the empirical mean
$$
\hat {\boldsymbol {\mu}} _ {1} = \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {m} _ {1} ^ {(i)} \quad \text {w h e r e} \quad \boldsymbol {m} _ {1} ^ {(i)} = \frac {1}{K} \sum_ {j = 1} ^ {K} \boldsymbol {x} _ {j} ^ {(i)},
$$
as well as the higher-order moments for  $\ell \in [K]$
$$
\hat {\boldsymbol {\mu}} _ {\ell} = \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {m} _ {\ell} ^ {(i)} \quad \text {w h e r e} \quad \boldsymbol {m} _ {\ell} ^ {(i)} = \frac {1}{\binom {K} {\ell}} \sum_ {j _ {1} <   \dots <   j _ {\ell} \in [ K ]} \boldsymbol {x} _ {j _ {1}} ^ {(i)} \dots \boldsymbol {x} _ {j _ {\ell}} ^ {(i)}.
$$
C.1 Non-Asymptotic Confidence Intervals
We start by giving non-asymptotic confidence intervals for the XBern distributions based on the Bernstein bound, recalled below [e.g. 20, Thm. 1.3.2].
Algorithm 2 First-Order Bernstein Intervals
Input: Random vectors  $\pmb{x}^{(1)},\dots,\pmb{x}^{(n)}\sim \mathrm{XBern}_K(\mu_1,\dots,\mu_K)$  with unknown parameters, failure probability  $\beta \in (0,1)$ .
Output: Confidence intervals  $[\underline{\mu}_1, \overline{\mu}_1]$  such that  $\mathbb{P}(\mu_1 < \underline{\mu}_1) \leq \beta$  and  $\mathbb{P}(\mu_1 > \overline{\mu}_1) \leq \beta$ .
1: Set  $\underline{\pmb{\mu}}_1$  as the unique solution of  $x\in [0,\hat{\pmb{\mu}}_1]$  such that
$$
\hat {\boldsymbol {\mu}} _ {1} - x - \sqrt {\left(\frac {2}{n} \log \frac {1}{\beta}\right) x (1 - x)} = \frac {2}{3 n} \log \frac {1}{\beta}
$$
if it exists, else set  $\pmb{\mu}_{1} = 0$
2: Set  $\overline{\mu}_1$  as the unique solution of  $x\in [\hat{\mu}_1,1]$  such that
$$
x - \hat {\mu} _ {1} - \sqrt {\left(\frac {2}{n} \log \frac {1}{\beta}\right) x (1 - x)} = \frac {2}{3 n} \log \frac {1}{\beta}
$$
if it exists, else set  $\overline{\mu}_1 = 1$ .
3: return  $\underline{\mu}_1, \overline{\mu}_1$ .
Lemma 7 (Bernstein inequality). Let  $\pmb{x}_1, \dots, \pmb{x}_n$  be independent random variables with  $\mathbb{E}[\pmb{x}_i] = 0$ ,  $\operatorname{Var}(\pmb{x}_i) \leq \sigma^2$ , and  $|\pmb{x}_i| \leq b$  almost surely. Let  $\pmb{\mu}_1 \coloneqq (1/n)\sum_{i=1}^{n}\pmb{x}_i$ . For any  $t > 0$ , we have
$$
\mathbb {P} \left(\boldsymbol {\mu} _ {1} > t\right) \leq \exp \left(- \frac {n t ^ {2}}{2 \sigma^ {2} + 2 b t / 3}\right).
$$
Equivalently, we have with probability at least  $1 - \beta$  that
$$
\boldsymbol {\mu} _ {1} \leq \sqrt {\frac {2 \sigma^ {2}}{n} \log \frac {1}{\beta} + \frac {b ^ {2}}{9 n ^ {2}} \log^ {2} \frac {1}{\beta}} + \frac {b}{3 n} \log \frac {1}{\beta} \leq \sqrt {\frac {2 \sigma^ {2}}{n} \log \frac {1}{\beta}} + \frac {2 b}{3 n} \log \frac {1}{\beta}.
$$
The last simplification is obtained by using  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  for scalars  $a, b \geq 0$ .
C.1.1 First-Order Bernstein Intervals
The first-order Bernstein interval only depends on the empirical mean  $\mu_{1}$  and is given in Algorithm 2.
Proposition 8. Consider Algorithm 2 with inputs  $n$  i.i.d. samples  $\pmb{x}^{(1)},\dots,\pmb{x}^{(n)} \stackrel{i.i.d.}{\sim} XBern_{K}(\mu_{1},\dots,\mu_{K})$  for some  $K \geq 1$ . Then, its outputs  $\underline{\mu}_1,\overline{\mu}_1$  satisfy  $\mathbb{P}(\mu_1\leq \underline{\mu}_1)\geq 1 - \beta$  and  $\mathbb{P}(\mu_1\geq \overline{\mu}_1)\geq 1 - \beta$ .
Proof. Applying Bernstein's inequality to  $m_1 \coloneqq (1 / K) \sum_{j=1}^{K} x_j$ , we have with probability  $1 - \beta$  that
$$
\mu_ {1} - \hat {\mu} _ {1} \leq \sqrt {\frac {2 \operatorname {V a r} (\boldsymbol {m} _ {1})}{n} \log \frac {1}{\beta}} + \frac {2}{3 n} \log \frac {1}{\beta} \leq \sqrt {\frac {2 \mu_ {1} (1 - \mu_ {1})}{n} \log \frac {1}{\beta}} + \frac {2}{3 n} \log \frac {1}{\beta},
$$
where we used that  $\operatorname{Var}(m_1) \leq \mu_1(1 - \mu_1)$  since  $m_1 \in [0, 1]$  a.s. We see from Figure 2 that  $\overline{\mu}_1$  is that largest value of  $\mu_1$  that satisfies the above inequality, showing that it is an upper confidence bound. Similarly, we get that  $\underline{\mu}_1$  is a valid lower confidence bound.
Algorithm 3 Second-Order Bernstein Intervals
1: For each  $i \in [n]$ , set  $\pmb{m}_1^{(i)} = (1 / K)\sum_{j=1}^{K}\pmb{x}_j^{(i)}$  and  $\pmb{m}_2^{(i)} = \pmb{m}_1^{(i)}\left(\frac{K\pmb{m}_1^{(i)} - 1}{K - 1}\right)$ .
2: Set  $\hat{\pmb{\mu}}_{\ell} = (1 / n)\sum_{i = 1}^{n}\pmb{m}_{\ell}^{(i)}$  for  $\ell = 1,2$
3: Set  $\overline{\pmb{\mu}}_2$  as the unique solution of  $x\in [\hat{\pmb{\mu}}_2,1]$  such that
$$
x - \hat {\mu} _ {2} - \sqrt {\left(\frac {2}{n} \log \frac {2}{\beta}\right) x (1 - x)} = \frac {2}{3 n} \log \frac {2}{\beta}.
$$
if it exists, else set  $\overline{\mu}_2 = 1$ .
4: Set  $\underline{\mu}_1$  as the unique solution of  $x\in [0,\hat{\mu}_1]$  such that
$$
\hat {\pmb {\mu}} _ {1} - x - \sqrt {\left(\frac {2}{n} \log \frac {2}{\beta}\right) \left(\frac {x}{K} - x ^ {2} + \frac {K - 1}{K} \overline {{\pmb {\mu}}} _ {2}\right)} = \frac {2}{3 n} \log \frac {2}{\beta}
$$
if it exists, else set  $\underline{\mu}_1 = 0$ .
5: Set  $\overline{\pmb{\mu}}_1$  as the unique solution of  $x\in [\hat{\pmb{\mu}}_1,1]$  such that
$$
x - \hat {\mu} _ {1} - \sqrt {\left(\frac {2}{n} \log \frac {2}{\beta}\right) \left(\frac {x}{K} - x ^ {2} + \frac {K - 1}{K} \bar {\mu} _ {2}\right)} = \frac {2}{3 n} \log \frac {2}{\beta}
$$
6: return  $\underline{\mu}_1, \overline{\mu}_1$ .
C.1.2 Second-Order Bernstein Intervals
The second-order Bernstein interval only depends on the first two empirical moments  $\mu_{1}$  and  $\mu_{2}$ . It is given in Algorithm 3. The algorithm is based on the calculation
$$
\begin{array}{l} \operatorname {V a r} \left(\boldsymbol {m} _ {1}\right) = \mathbb {E} \left[ \boldsymbol {m} _ {1} ^ {2} \right] - \mu_ {1} ^ {2} \\ = \mathbb {E} \left[ \frac {1}{K ^ {2}} \sum_ {j = 1} ^ {K} \boldsymbol {x} _ {j} ^ {2} + \frac {2}{K ^ {2}} \sum_ {j _ {1} <   j _ {2} \in [ K ]} \boldsymbol {x} _ {j _ {1}} \boldsymbol {x} _ {j _ {2}} \right] - \mu_ {1} ^ {2} \tag {17} \\ = \frac {\mu_ {1}}{K} - \mu_ {1} ^ {2} + \frac {K - 1}{K} \mu_ {2}, \\ \end{array}
$$
where we used  $\pmb{x}_j^2 = \pmb{x}_j$  since it is an indicator.