SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unified Enhancement of Privacy Bounds for Mixture Mechanisms via  $f$ -Differential Privacy
Chendi Wang*
Philadelphia, PA, USA, 19104 &
Shenzhen Research Institute of Big data
Shenzhen, Guangdong, China, 518000
chendi@wharton.upenn.edu
Buxin Su*
Philadelphia, PA, USA, 19104
subuxin@sas.upenn.edu
Jiayuan Ye
jiayuan@comp.nus.edu.sg
Reza Shokri
reza@comp.nus.edu.sg
Wharton Statistics and Data Science Department
Differentially private (DP) machine learning algorithms incur many sources of randomness, such as random initialization, random batch subsampling, and shuffling. However, such randomness is difficult to take into account when proving differential privacy bounds because it induces mixture distributions for the algorithm's output that are difficult to analyze. This paper focuses on improving privacy bounds for shuffling models and one-iteration differentially private gradient descent (DP-GD) with random initializations using  $f$ -DP. We derive a closed-form expression of the trade-off function for shuffling models that outperforms the most up-to-date results based on  $(\epsilon, \delta)$ -DP. Moreover, we investigate the effects of random initialization on the privacy of one-iteration DP-GD. Our numerical computations of the trade-off function indicate that random initialization can enhance the privacy of DP-GD. Our analysis of  $f$ -DP guarantees for these mixture mechanisms relies on an inequality for trade-off functions introduced in this paper. This inequality implies the joint convexity of  $F$ -divergences. Finally, we study an  $f$ -DP analog of the advanced joint convexity of the hockey-stick divergence related to  $(\epsilon, \delta)$ -DP and apply it to analyze the privacy of mixture mechanisms.
Differential privacy (DP, [16, 17]) is a rigorous mathematical framework for ensuring data privacy and has become a cornerstone of privacy-preserving data analysis over the past two decades. DP has found widespread applications in various data science fields, such as machine learning [12, 6, 44],
query answering [18, 15], and synthetic data generation [37, 48, 29, 28]. A randomized mechanism is considered differentially private if the outputs of two neighboring datasets that differ in at most one element are indistinguishable from each other. The closeness of these outputs can be measured in various ways, resulting in the definition of  $(\epsilon, \delta)$ -DP in [16] and its various relaxations.
The distinguishability between the outputs can be measured by statistical divergences. For example,  $(\epsilon, \delta)$ -DP is associated with the so-called hockey-stick divergence [35]. Another divergence relevant to differential privacy is the Rényi divergence [20, 38] which leads to Rényi DP [30, 10] and concentrated DP [11]. In addition to divergence-based DP, a hypothesis testing perspective on differential privacy was proposed in [43]. More recently, [14] established  $f$ -DP for differential privacy where the privacy is measured by the trade-off function of type I and type II errors.
In real-world applications of differential privacy, including differentially private machine learning, it is common to analyze the privacy budget of mechanisms that involve mixture distributions, where the mixture is introduced by stochastic components in the algorithm. Examples of such mechanisms include sub-sampled mechanisms [4, 52, 40, 31], shuffled mechanisms [13, 22, 23], and variants of the differentially private stochastic gradient descent (DP-SGD) algorithm [19, 26, 3, 45] that involves random initialization and multiple rounds of mini-batch sampling. Recently, privacy amplification by iteration [24] has drawn much attention as it can be used to analyze the privacy bounds for DP-SGD [45, 2]) which leads to tighter privacy bounds compared to classical analysis based on the composition theorem [33, 39, 50].
While mixture mechanisms are essential in differentially private machine learning, the absence of an  $f$ -DP guarantee for their analysis remains a significant challenge. Moreover, existing divergence-based DP bounds for most of these mechanisms are not tight. This is primarily because the complex distribution resulting from the mixture makes it challenging to accurately quantify privacy guarantees. In order to illustrate this perspective, we consider the examples of shuffling models and DP gradient descent (DP-GD) with random initialization, as follows.
- In shuffling models, each user's data record is locally privatized using a local DP algorithm [21]. Subsequently, a curator shuffles the dataset containing all users' data. The shuffling procedure introduces additional mixtures of binomial noise [22], thereby potentially amplifying the privacy provided by the local randomizer. Shuffling is commonly employed in machine learning algorithms for batch generation [45, 47]. To deal with this mixture, Hoeffding's inequality was used in previous literature [22, 23] that leads to the loss of information. Using  $f$ -DP in this paper, we derive an exact analytical trade-off function for the mixture of binomial distributions which is sharp.
- In deep learning, random initialization is usually adopted in the stochastic gradient descent to enhance the performance of deep neural networks [36]. Intuitively, the inherent randomness introduced by initialization should contribute to the privacy amplification of DP-GD. However, Rényi differential privacy (DP) falls short in quantitatively measuring this randomness, even when applied to the simplest linear model. In this paper, we demonstrate how  $f$ -DP can effectively evaluate and quantify this inherent randomness from initialization.
Our contributions. This paper makes a two-fold contribution. Firstly, we propose a unified theory to analyze the privacy of mixture mechanisms within the framework of  $f$ -DP. Precisely, we derive an  $f$ -DP inequality for mixture distributions which implies the joint convexity of  $F$ -divergences for any convex  $F$ . We name this result the "joint concavity of trade-off functions", as it is a lower bound for trade-off functions. The tightness of the joint concavity is also investigated. Moreover, we propose the "advanced joint concavity of trade-off functions" which is an  $f$ -DP analog of the advanced joint convexity of the hockey-stick divergence and results in sharper bounds in certain cases.
Building on our inequality, we have refined the privacy analysis of both shuffling models and DP-GD with random initialization using  $f$ -DP. Specifically, for shuffling models, we obtain trade-off functions in a closed-form representation, leading to tighter bounds compared to existing state-of-the-art results based on  $(\epsilon, \delta)$ -DP. As for DP-GD, given the challenges in the trajectory analysis of multi-step iterations, we have chosen to explore a more straightforward one-iteration DP-GD. We demonstrate that using random initialization significantly enhances the privacy of the output from a single iteration.
2 Preliminaries on differential privacy
Let  $\mathcal{D} = \{z_i\}_{i=1}^n \subset \mathcal{Z}$  be a fixed dataset of size  $n$ . Consider a randomized algorithm  $\mathcal{A}: \mathcal{Z}^n \to S$  that maps a dataset  $\mathcal{D}$  to  $\mathcal{A}(\mathcal{D})$  in some probability space  $S$ . Differential privacy requires that the change of one element in a dataset has a restricted impact on the output of  $\mathcal{A}$ . Mathematically, we say  $\mathcal{A}$  satisfies  $(\epsilon, \delta)$ -DP for some  $\epsilon \geq 0$  and  $0 \leq \delta \leq 1$  if
$$
\mathbb {P} [ \mathcal {A} (\mathcal {D} _ {0}) \in S ] \leq e ^ {\epsilon} \mathbb {P} [ \mathcal {A} (\mathcal {D} _ {1}) \in S ] + \delta ,
$$
for any event  $S \in S$  and any neighboring datasets  $\mathcal{D}_0$  and  $\mathcal{D}_1$ . When  $\delta = 0$ , we simply call  $(\epsilon, 0)$ -DP as  $\epsilon$ -DP. Based on the definition, we see that for small values of  $\epsilon$  and  $\delta$ , it is challenging to distinguish between  $\mathcal{D}_0$  and  $\mathcal{D}_1$  based on the outputs of  $\mathcal{A}(\mathcal{D}_0)$  and  $\mathcal{A}(\mathcal{D}_1)$ , as the distribution of  $\mathcal{A}(\mathcal{D}_0)$  closely resembles that of  $\mathcal{A}(\mathcal{D}_1)$ .
The definition of  $(\epsilon, \delta)$ -DP corresponds to the hockey-stick divergence. Let  $P$  and  $Q$  be two distributions with probability density functions (pdfs)  $p$  and  $q$ , respectively. The hockey-stick divergence between  $P$  and  $Q$  is defined by  $H_{\gamma}(P \| Q) = \int (p(x) - \gamma q(x))_+ dx$  for  $\gamma \geq 1$  with  $(\cdot)_+ = \max \{0, \cdot\}$ . With a little bit abuse of notations, in this paper, we define the divergence (or the trade-off function) between two random variables as the divergence (or the trade-off function) between their distributions. Then, a mechanism  $\mathcal{A}$  is  $(\epsilon, \delta)$ -DP if and only if  $H_{e^{\epsilon}}(\mathcal{A}(\mathcal{D}_0) \| \mathcal{A}(\mathcal{D}_1)) \leq \delta$  for any neighboring datasets  $\mathcal{D}_0$  and  $\mathcal{D}_1$ , which also implies  $H_{e^{\epsilon}}(\mathcal{A}(\mathcal{D}_1) \| \mathcal{A}(\mathcal{D}_0)) \leq \delta$ .
The Rényi-DP (RDP) is defined based on the Rényi divergence. The Rényi divergence of order  $\widetilde{\alpha} > 1$  between  $P$  and  $Q$  is given by
$$
R _ {\widetilde {\alpha}} (P \| Q) = \frac {1}{\widetilde {\alpha} - 1} \log \int \left(\frac {p (x)}{q (x)}\right) ^ {\widetilde {\alpha}} q (x) d x.
$$
For  $\widetilde{\alpha} = 1$  or  $+\infty$ ,  $R_{1}$  or  $R_{\infty}$  is the limit of  $R_{\widetilde{\alpha}}$  as  $\widetilde{\alpha}$  tends to 1 or  $+\infty$ . A mechanism  $\mathcal{A}$  is said to satisfy  $(\widetilde{\alpha}, \epsilon)$ -RDP if  $R_{\widetilde{\alpha}}(\mathcal{A}(\mathcal{D}_0) \| \mathcal{A}(\mathcal{D}_1)) \leq \epsilon$  for any neighboring  $\mathcal{D}_0$  and  $\mathcal{D}_1$ .
The distinguishability between  $\mathcal{A}(\mathcal{D}_0)$  and  $\mathcal{A}(\mathcal{D}_1)$  can be quantified using hypothesis testing, which aligns with the concept of  $f$ -DP. Consider a hypothesis testing problem  $H_0: P$  v.s.  $H_1: Q$  and a rejection rule  $\phi \in [0, 1]$ . We define the type I error as  $\alpha_{\phi} = \mathbb{E}_P[\phi]$ , which is the probability that we reject the null hypothesis  $H_0$  by mistake. The type II error  $\beta_{\phi} = 1 - \mathbb{E}_Q[\phi]$  is the probability that we accept the alternative  $H_1$  wrongly.
The trade-off function  $T(P, Q)$  is the minimal type II error at level  $\alpha$  of the type I error, that is,
$$
T (P, Q) (\alpha) = \inf  _ {\phi} \left\{\beta_ {\phi}: \alpha_ {\phi} \leq \alpha \right\}.
$$
We say a mechanism  $\mathcal{A}$  satisfies  $f$ -DP if  $T(\mathcal{A}(\mathcal{D}_0), \mathcal{A}(\mathcal{D}_1)) \geq f$  for any neighboring datasets  $\mathcal{D}_0$  and  $\mathcal{D}_1$ . In particular,  $\mathcal{A}$  is said to satisfy  $\mu$ -GDP if it is  $G_{\mu}$ -DP, where  $G_{\mu}(x) = \Phi (\Phi^{-1}(1 - x) - \mu)$ , for  $\mu \geq 0$ , is the Gaussian trade-off function with  $\Phi$  being the cumulative distribution function (cdf) of  $\mathcal{N}(0,1)$ .  $\mathcal{A}$  is considered to be more private if the corresponding trade-off function takes larger values. When  $\mathcal{A}$  achieves perfect privacy and  $\mathcal{A}(\mathcal{D}_0)$  and  $\mathcal{A}(\mathcal{D}_1)$  become completely indistinguishable, the trade-off function is  $\operatorname{Id}(x) = 1 - x$ . Consequently, for any trade-off function  $f$ , we have  $f \leq \operatorname{Id}$ .
We say a trade-off function is symmetric if  $T(P, Q) = T(Q, P)$ . Note that a trade-off function  $f$  may not necessarily be symmetric. But one can symmetrize it as shown in [14]. The symmetrization of a trade-off function will be used when we analyze the shuffled mechanisms.
3 Joint concavity of trade-off functions
Let  $\{P_i\}_{i=1}^m$  and  $\{Q_i\}_{i=1}^m$  be two sequences of probability distributions. Denote the probability density functions (pdfs) of  $P_i$  and  $Q_i$  as  $p_i$  and  $q_i$ , respectively. Consider the mixture distributions  $P_{\mathbf{w}}$  and  $Q_{\mathbf{w}}$  with pdfs  $p_{\mathbf{w}} = \sum_{i=1}^{m} w_i p_i$  and  $q_{\mathbf{w}} = \sum_{i=1}^{m} w_i q_i$ , where the weight  $\mathbf{w} = (w_1, \dots, w_m)$  is such that  $w_i \geq 0$  and  $\sum_{i=1}^{m} w_i = 1$ . The following lemma is to bound the trade-off function  $T(P_{\mathbf{w}}, Q_{\mathbf{w}})$ . Upon finalizing this paper, we noted that Lemma 3.1 and Proposition 3.2 appeared independently in another paper [42].
Lemma 3.1 (Joint concavity of trade-off functions). For two mixture distributions  $P_{\mathbf{w}}$  and  $Q_{\mathbf{w}}$ , it holds
$$
T \left(P _ {\mathbf {w}}, Q _ {\mathbf {w}}\right) (\alpha (t, c)) \geq \sum_ {i = 1} ^ {m} w _ {i} T \left(P _ {i}, Q _ {i}\right) \left(\alpha_ {i} (t, c)\right),
$$
where  $\alpha_{i}(t,c) = \mathbb{P}_{X\sim P_{i}}\left[\frac{q_{i}}{p_{i}}(X) > t\right] + c\mathbb{P}_{X\sim P_{i}}\left[\frac{q_{i}}{p_{i}}(X) = t\right]$  is the type I error for testing  $P_{i}$  v.s.  $Q_{i}$  using the likelihood ratio test and  $\alpha(t,c) = \sum_{i=1}^{m} w_{i} \alpha_{i}(t,c)$ .
The main idea of the proof is to make the mixture distributions more distinguishable by releasing the indices. Precisely, for  $X \sim P_{\mathbf{w}}$  and  $Y \sim Q_{\mathbf{w}}$ , let  $X|I$  be a random variable such that  $X|I = i \sim P_i$  with  $I$  being the indices, i.e.,  $\mathbb{P}[I = i] = w_i$ . Let  $(X|I, I)$  be a random variable where we observe both  $X|I$  and the indices  $I$ . Then, the right hand side of Lemma 3.1 is the trade-off function  $T((X|I, I), (Y|I, I))$  between two joint distributions. This is a lower bound for the trade-off function between mixture distributions because  $(X|I, I) \to X$  is a data-independent post-processing procedure that only removes the observation of indices  $I$ , and DP is immune to post-processing [14, 19].
Under the setting of  $f$ -DP, we usually require that the trade-off function is symmetric. The symmetry of the trade-off function in Lemma 3.1 is guaranteed by the following proposition.
Proposition 3.2. Suppose that for each  $i$ ,  $T(P_i, Q_i)$  is a symmetric trade-off function. Then the trade-off function  $T((X|I, I), (Y|I, I))$  is symmetric.
The joint convexity of  $F$ -divergences plays an important role in the analysis of divergence-based DP for mixture mechanisms [4, 22]. We now show that Lemma 3.1 is an extension of the joint convexity of  $F$ -divergences, including the scaled exponentiation of the Rényi divergence and the hockey-stick divergence, to trade-off functions. A trade-off function is always convex and is thus differentiable almost everywhere. Thus, without loss of generality, we consider  $f_{i}$  that is differentiable, symmetric, with  $f_{i}(0) = 1$ .
Proposition 3.3 (An application of Lemma 3.1 to the  $F$ -divergences). Let  $D_F(P \| Q) = \int F(p(x) / q(x))q(x)dx$  be an  $F$ -divergence between any two distributions  $P$  and  $Q$  with some convex  $F$ . Then, for  $f_i = T(P_i, Q_i)$ , we have
$$
D _ {F} \left(P _ {\mathbf {w}} \| Q _ {\mathbf {w}}\right) \leq \sum_ {i = 1} ^ {m} \int_ {0} ^ {1} F \left(\frac {1}{| f _ {i} ^ {\prime} (x) |}\right) | f _ {i} ^ {\prime} (x) | d x = \sum_ {i = 1} ^ {m} w _ {i} D _ {F} \left(P _ {i} \| Q _ {i}\right).
$$
Conversion from a trade-off function to  $F$ -divergences is straightforward using Section B in [14]. However, conversion from an  $F$ -divergence to a trade-off function is highly non-trivial. In fact,  $F$ -divergence is an integral of a functional of the trade-off function over the whole space while Lemma 3.1 holds pointwisely, which is a local property. This explains why the divergence-based DP is not as informative as  $f$ -DP since some information is lost due to the integration.
4 Privacy analysis of the shuffled mechanisms
In this section, we explore the  $f$ -DP analysis of shuffled mechanisms. Drawing upon [22, 23], the shuffling procedure incorporates a mixture of binomial noise. This noise can be tightly bounded using our  $f$ -DP inequality for mixture distributions.
4.1 Theoretical privacy guarantee
In shuffling models, the record of each user is privatized by some local randomizer (such as a randomized response mechanism [41]) and all records are then shuffled by a curator. Mathematically, consider a dataset  $\mathcal{D} = \{z_i\}_{i=1}^n \subseteq \mathcal{Z}$  of size  $n$  and each data point  $z_i$  is privatized by an local randomizer  $\mathcal{A}_0: \mathcal{Z} \mapsto \widetilde{\mathcal{Z}}$  that satisfies  $\epsilon_0$ -DP. Then, the mechanism  $\mathcal{A}: \mathcal{Z}^n \to \widetilde{\mathcal{Z}}^n$  that maps  $\mathcal{D}$  to  $\widetilde{\mathcal{D}} = \{\mathcal{A}_0(z_i)\}_{i=1}^n$  is  $\epsilon_0$ -DP. A shuffler  $\mathcal{A}_{\mathrm{Shuffle}}$  takes the privatized dataset  $\widetilde{\mathcal{D}}$  as input and applies a uniformly random permutation to  $\widetilde{\mathcal{D}}$ , which introduces the mixture of binomial noise to  $\mathcal{A}$  and results in privacy amplification.
As noted in [23], the shuffling procedure introduces mixtures of binomial distributions. More specifically, the outputs generated by the shuffled mechanism for two neighboring datasets result from post-processing random variables  $X \sim P$  and  $Y \sim Q$  with  $P = (1 - w)P_{0} + wQ_{0}$  and  $Q = (1 - w)Q_{0} + wP_{0}$ , where the weight  $w = \frac{1}{e^{\epsilon_{0}} + 1}$ , and the distributions  $P_{0}$  and  $Q_{0}$  are defined as  $(A + 1, C - A) \sim P_{0}$ , and  $(A, C - A + 1) \sim Q_{0}$  with  $A \sim \operatorname{Binom}(C, 1/2)$  and  $C \sim \operatorname{Binom}(n - 1, 2/(e^{\epsilon_{0}} + 1))$ . It is easy to see that  $P_{0}$  is the mixture of  $\{(A_{i} + 1, i - A_{i})\}_{i=0}^{n-1}$  with weights  $w_{i}^{0} := \mathbb{P}[C = i]$  and  $Q_{0}$  is the mixture of  $\{(A_{i}, i - A_{i} + 1)\}_{i=0}^{n-1}$  with the same weights. In this context,  $\operatorname{Binom}(k, p)$  is a binomial distribution with parameters  $k \in \mathbb{N}$  and  $p \in [0, 1]$  and each  $A_{i}$  is distributed as  $\operatorname{Binom}(i, 1/2)$ . Advancing our analysis, we adopt the joint concavity, as outlined in Lemma [3.1] to establish a lossless bound for the trade-off function  $T(P_{0}, Q_{0})$ .
Proposition 4.1. Let  $F_{i}$  be the distribution function of  $\mathrm{Binom}(i,1 / 2)$  and let  $w_{i}^{0} = \mathbb{P}[C = i]$  for  $C\sim \mathrm{Binom}(n - 1,2 / (e^{\epsilon_0} + 1))$ . Then, we have  $T(P_0,Q_0)$  is a piecewise linear function with
$$
T (P _ {0}, Q _ {0}) (\alpha (t)) = \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} \left\{1 - F _ {i} \left[ F _ {i} ^ {- 1} (\alpha_ {i} (t)) + 1 \right] \right\},
$$
for each knot  $\alpha(t) = \sum_{i=0}^{n-1} w_i^0 \alpha_i(t) \coloneqq \sum_{i=0}^{n-1} w_i^0 F_i\left(i - \frac{i+1}{t+1}\right)$ .
Remark. Proposition 4.1 holds with equality and the bound for  $T(P_0, Q_0)$  is sharp.
Before stating our results for  $T(P, Q)$ , we define some notations related to  $f$ -DP. For a function  $g: \mathbb{R} \to \mathbb{R}$ , let  $g^{*}(y) := \max_{x} \{xy - g(x)\}$  be its convex conjugate. For a trade-off function  $f$ , let  $\mathcal{C}(f) = \min \{f, f^{-1}\}^{**}$  be its symmetrization, where  $f^{-1}$  is the left inverse function of  $f$ , i.e.,  $f^{-1} \circ f(x) = x$ .
Theorem 4.2. The shuffled mechanism  $\mathcal{A}_{\mathrm{Shuffle}} \circ \mathcal{A}$  is  $\mathcal{C}(f_{\mathrm{Shuffle}}) - DP$ . Here  $f_{\mathrm{Shuffle}}(\alpha(t))$  is a piecewise linear function where each knot  $\alpha(t)$  has the form
$$
\alpha (t) = \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} \alpha_ {i} (t) := \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} F _ {i} \left(i - \frac {i + 1}{t + 1}\right) \in [ 0, 1 ], \qquad f o r a l l t \geq 0,
$$
with  $F_{i}$  being the distribution function of  $\mathrm{Binom}(i,1 / 2)$  and  $w_{i}^{0} = \mathbb{P}[C = i]$  for  $C\sim \mathrm{Binom}(n - 1,2 / (e^{\epsilon_0} + 1))$ , and the value of  $f_{\mathrm{Shuffle}}$  at a knot  $\alpha (t)$  is
$$
f _ {\mathrm {S h u f f l e}} (\alpha (t)) = 2 w \cdot \operatorname {I d} (\alpha (t)) + (1 - 2 w) \cdot \left[ \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} \left\{1 - F _ {i} \left[ F _ {i} ^ {- 1} (\alpha_ {i} (t)) + 1 \right] \right\} \right],
$$
with  $w = \frac{1}{1 + e^{e_0}}$  and  $\operatorname{Id}(x) = 1 - x$  being the identity trade-off function.
Remark. The bound in Theorem 4.2 is near-optimal. In fact, the proof of Theorem 4.2 is based on a post-processing procedure in [23], joint concavity (Proposition 4.1), and advanced joint concavity (Proposition 6.4). The post-processing procedure is sharp for specific mechanisms, such as the randomized response mechanism, as shown by Theorem 5.2 and Theorem 5.3 in [23]. Proposition 4.1 holds with equality and is optimal. The advanced joint concavity, which is an  $f$ -DP analog of the advanced joint convexity in [4], is optimal for specific distributions. Compared to existing analysis of shuffled mechanisms (e.g., [23]), the main advantage of using  $f$ -DP is that we avoid the use of Hoeffding's inequality and the Chernoff bound to bound the distance between  $P_0$  and  $Q_0$  in Proposition 4.1, which is adopted in [22, 23] and leads to loose bounds, to bound the mixture of binomial distributions. Moreover, Theorem 3.2 in [23] holds with an assumption  $\epsilon_0 \leq \log \left(\frac{n}{8\log(2 / \delta)} - 1\right)$ , which is removed by using  $f$ -DP in our paper.
To convert  $f$ -DP to  $(\epsilon, \delta)$ -DP, we use the primal-dual perspective in [14] and obtain the following Corollary.
Corollary 4.3. Let  $l(t) \coloneqq -\frac{\sum_{i=0}^{n-1} w_i^0 p_i \left( \left| i+1-\frac{i+1}{t+1} \right| \right)}{\sum_{i=0}^{n-1} w_i^0 p_i \left( \left| i-\frac{i+1}{t+1} \right| \right)}$  with  $p_i$  being the probability mass function of Binom(i,1/2). Then, we have  $\mathcal{A}_{\text{Shuffle}} \circ \mathcal{A}$  is  $(\epsilon, \delta_{f-DP}(\epsilon))$ -DP for any  $\epsilon > 0$  with
$$
\delta_ {f \cdot D P} (\epsilon) = (- e ^ {\epsilon} + 2 w) \left[ \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} F _ {i} \left(i - \frac {i + 1}{t _ {\epsilon} + 1}\right) \right] + (1 - 2 w) \left[ \sum_ {i = 0} ^ {n - 1} w _ {i} ^ {0} F _ {i} \left(i + 1 - \frac {i + 1}{t _ {\epsilon} + 1}\right) \right],
$$
where  $t_\epsilon = \inf \{t: -2w + (1 - 2w)l(t) \geq -e^\epsilon\}$  and  $w = \frac{1}{e^{\epsilon_0} + 1}$ .
(a)  $\epsilon_0 = 5.444 > 4.444$
(b)  $\epsilon_0 = 4.444$
(c)  $\epsilon_0 = 3.444$
Figure 1: A comparison between the trade-off function given by Theorem [4.2] and  $(\epsilon, \delta)$ -DP with  $n = 10000$  and  $\delta = n^{-1.5}$  given by [23]. [23] requires that  $\epsilon_0 \leq \log \left(\frac{n}{8 \log(2/\delta)} - 1\right) \approx 4.444$ . Thus, there is no result for  $(\epsilon, \delta)$ -DP when  $\epsilon_0 > 4.444$ .
4.2 Numerical results and comparisons
To the best of our understanding, the leading privacy analysis for shuffled mechanisms is given in [23]. In this section, we compare the privacy bounds from our Theorem 4.2 and Corollary 4.3 with those found in Theorem 3.2 of [23]. Additionally, we assess the tightness of our bound against the empirical lower bounds obtained through binary search.
Specifically, Figure 1 presents a comparison of the trade-off function derived from our Theorem 4.2 to that of [23]. This comparison clearly illustrates that  $f$ -DP offers tighter privacy bounds, given that its trade-off function aligns closer to the identity trade-off function.
In our Table  $\boxed{1}$ , we compare the values of  $\delta_{f\text{-DP}}(\epsilon)$ , as derived from Corollary  $\boxed{4.3}$  with  $\delta(\epsilon)$  in  $\boxed{23}$ . The results indicate that  $\delta_{f\text{-DP}}(\epsilon)$  is significantly smaller than  $\delta(\epsilon)$ .
In Table 2, we present  $\epsilon_{f\text{-DP}}$  alongside the numerical upper bound of  $\epsilon$  from [23] and the numerical lower bound determined by binary search. Given its closeness to the lower bound, our Theorem 4.2 can be considered near-optimal.
Table 1: Comparisons with [23]
ε	0.5	0.6	0.7	0.8	0.9	1.0
δ in [23]	0.9494	0.3764	0.1038	0.0181	0.0018	8 × 10-5
δf-DP (ours)	3 × 10-6	10-7	4 × 10-9	9 × 10-11	2 × 10-12	2 × 10-14