SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With a slight modification (i.e., mapping the output space from  $\{0,1\}$  to  $\{-1,1\}$ ),  $sto-sign(x,A)$  can be understood as a special case of the binomial mechanism with  $M = 1$  and  $p_i(x_i) = \frac{A + x_i}{2A}$ . In this case, we have  $p_{max} = \frac{A + c}{2A}$  and  $p_{min} = \frac{A - c}{2A}$ . Applying the results in Theorem 2 yields
$$
f ^ {s t o - s i g n} (\alpha) = \beta_ {\phi , \inf } ^ {+} (\alpha) = \beta_ {\phi , \inf } ^ {-} (\alpha) = \left\{ \begin{array}{l l} 1 - \frac {A + c}{A - c} \alpha , & f o r \alpha \in [ 0, \frac {A + c}{2 A} ], \\ \frac {A - c}{A + c} - \frac {A - c}{A + c} \alpha , & f o r \alpha \in [ \frac {A + c}{2 A}, 1 ]. \end{array} \right. \tag {9}
$$
Combining (9) with (3) suggests that the sto-sign compressor ensures  $(\ln (\frac{A + c}{A - c}),0)$ -DP.
Example 2. The second sign-based compressor that we examine is  $CLDP_{\infty}(\cdot)$  [23].
Definition 5 ( $CLDP_{\infty}(\cdot)$  [23]). For any given  $x \in [-c, c]$ , the compressor  $CLDP_{\infty}(\cdot)$  outputs  $CLDP_{\infty}(\epsilon)$ , which is given by
$$
C L D P _ {\infty} (\epsilon) = \left\{ \begin{array}{l} + 1, \text {w i t h p r o b a b i l i t y} \frac {1}{2} + \frac {x}{2 c} \frac {e ^ {\epsilon} - 1}{e ^ {\epsilon} + 1}, \\ - 1, \text {w i t h p r o b a b i l i t y} \frac {1}{2} - \frac {x}{2 c} \frac {e ^ {\epsilon} - 1}{e ^ {\epsilon} + 1}. \end{array} \right. \tag {10}
$$
$CLDP_{\infty}(\epsilon)$  can be understood as a special case of sto-sign  $(x,A)$  with  $A = \frac{c(e^{\epsilon} + 1)}{e^{\epsilon} - 1}$ . In this case, according to (9), we have
$$
f ^ {C L D P _ {\infty}} (\alpha) = \left\{ \begin{array}{l} 1 - e ^ {\epsilon} \alpha , \quad f o r \alpha \in [ 0, \frac {A + c}{2 A} ], \\ e ^ {- \epsilon} (1 - \alpha), \quad f o r \alpha \in [ \frac {A + c}{2 A}, 1 ]. \end{array} \right. \tag {11}
$$
Combining the above result with (3) suggests that  $CLDP_{\infty}(\epsilon)$  ensures  $(\epsilon, 0)$ -DP, which recovers the result in [23]. It is worth mentioning that  $CLDP_{\infty}(\epsilon)$  can be understood as the composition of sto-sign with  $A = c$  followed by the randomized response mechanism [22], and is equivalent to the one-dimensional case of the compressor in [13]. Moreover, the one-dimensional case of the schemes in [10, 11] can also be understood as special cases of sto-sign.
5 The Proposed Ternary Compressor
The output of the binomial mechanism with  $M = 1$  lies in the set  $\{0,1\}$ , which coincides with the sign-based compressor. In this section, we extend the analysis to the ternary case, which can be understood as a combination of sign-based quantization and sparsification (when the output takes value 0, no transmission is needed since it does not contain any information) and leads to improved communication efficiency. More specifically, we propose the following ternary compressor.
Definition 6 (Ternary Stochastic Compressor). For any given  $x \in [-c, c]$ , the compressor ternary outputs ternary $(x, A, B)$ , which is given by
$$
\operatorname {t e r n a r y} (x, A, B) = \left\{ \begin{array}{l} 1, \text {w i t h p r o b a b i l i t y} \frac {A + x}{2 B}, \\ 0, \text {w i t h p r o b a b i l i t y} 1 - \frac {A}{B}, \\ - 1, \text {w i t h p r o b a b i l i t y} \frac {A - x}{2 B}, \end{array} \right. \tag {12}
$$
where  $B > A > c$  are the design parameters that control the level of sparsity.
For the ternary stochastic compressor in Definition 6, we establish its privacy guarantee as follows.
Theorem 3. The ternary stochastic compressor is  $f^{\text{ternary}}(\alpha)$ -differentially private with
$$
f ^ {\text {t e r n a r y}} (\alpha) = \left\{ \begin{array}{c} 1 - \frac {A + c}{A - c} \alpha , \quad \text {f o r} \alpha \in [ 0, \frac {A - c}{2 B} ], \\ 1 - \frac {c}{B} - \alpha , \text {f o r} \alpha \in [ \frac {A - c}{2 B}, 1 - \frac {A + c}{2 B} ], \\ \frac {A - c}{A + c} - \frac {A - c}{A + c} \alpha , \text {f o r} \alpha \in [ 1 - \frac {A + c}{2 B}, 1 ]. \end{array} \right. \tag {13}
$$
Remark 3 (Privacy amplification by sparsification). It can be observed from (9) and (13) that  $f^{\text{ternary}}(\alpha) > f^{\text{sto-sign}}$  when  $\alpha \in [\frac{A - c}{2B}, 1 - \frac{A + c}{2B}]$ , and  $f^{\text{ternary}}(\alpha) = f^{\text{sto-sign}}$ , otherwise. Fig. 3 shows  $f^{\text{ternary}}(\alpha)$  and  $f^{\text{sto-sign}}$  for  $c = 0.1, A = 0.25, B = 0.5$ ,
and the shaded gray area corresponds to the improvement in privacy. It can be observed that communication efficiency and privacy are improved simultaneously. It is worth mentioning that, if we convert the privacy guarantees to  $(\epsilon, 0)$ -DP, we have  $\epsilon = \ln \left(\frac{7}{3}\right)$  for both compressors. However, the ternary compressor ensures  $(\ln (2), 0.05)$ -DP (i.e.,  $f^{ternary}(\alpha) \geq \max \{0, 0.95 - 2\alpha, 0.5(0.95 - \alpha)\}$ ) while the sto-sign compressor does not. We note that for the same  $A$ , as  $B$  increases (i.e., communication cost decreases),  $f^{ternary}(\alpha)$  approaches  $f(\alpha) = 1 - \alpha$  (which corresponds to perfect privacy).
In the following, we present a special case of the proposed ternary stochastic compressor.
Example 3. The ternary-based compressor proposed in [37] is formally defined as follows.
Definition 7 (ternarize  $(\cdot)$  [37]). For any given  $x \in [-c, c]$ , the compressor ternarize  $(\cdot)$  outputs ternarize  $(x, B) = \text{sign}(x)$  with probability  $|x| / B$  and ternarize  $(x, B) = 0$  otherwise, in which  $B > c$  is the design parameter.
ternarize  $(x, B)$  can be understood as a special case of ternary  $(x, A, B)$  with  $A = |x|$ . According to Theorem 3,  $f^{\text{ternary}}(\alpha) = 1 - \frac{c}{B} - \alpha$  for  $\alpha \in [0, 1 - \frac{c}{B}]$  and  $f^{\text{ternary}}(\alpha) = 0$  for  $\alpha \in [1 - \frac{c}{B}, 1]$ . Combining the above result with (3), we have  $\delta = \frac{c}{B}$  and  $\epsilon = 0$ , i.e., ternarize  $(\cdot)$  provides perfect privacy protection ( $\epsilon = 0$ ) with a violation probability of  $\delta = \frac{c}{B}$ . Specifically, the attacker cannot distinguish  $x_i$  from  $x_i'$  if the output of ternarize  $(\cdot) = 0$  (perfect privacy protection), while no differential privacy is provided if the output of ternarize  $(\cdot) \neq 0$  (violation of the privacy guarantee).
Remark 4. It is worth mentioning that, in [37], the users transmit a scaled version of ternarize(·) and the scaling factor reveals the magnitude information of  $x_{i}$ . Therefore, the compressor in [37] is not differentially private.
Figure 3: Sparsification improves privacy.
6 Breaking the Communication-Privacy-Accuracy Tradeoff
In this section, we extend the results in Section 5 to the vector case in two different approaches, followed by discussions on the three-way tradeoff between communication, privacy, and accuracy. The results in Section 4 can be extended similarly. Specifically, in the first approach, we derive the  $\mu$ -GDP in closed form, while introducing some loss in privacy guarantees. In the second approach, a tight approximation is presented. Given the results in Section 5, we can readily convert  $f$ -DP in the scalar case to Gaussian differential privacy in the vector case as follows.
Theorem 4. Given a vector  $x_{i} = [x_{i,1}, x_{i,2}, \dots, x_{i,d}]$  with  $|x_{i,j}| \leq c, \forall j$ . Applying the ternary compressor to the  $j$ -th coordinate of  $x_{i}$  independently yields  $\mu$ -GDP with  $\mu = -2\Phi^{-1}\left(\frac{1}{1 + \left(\frac{A + c}{A - c}\right)^d}\right)$ .
Remark 5. Note that  $||x_{i}||_{2}\leq c$  is a sufficient condition for  $|x_{i,j}|\leq c,\forall j$ . In the proof of Theorem 4, we first convert  $f^{\text{ternary}}(\alpha)$ -DP to  $(\epsilon ,0)$ -DP for the scalar case, and then obtain  $(d\epsilon ,0)$ -DP
for the  $d$ -dimensional case, followed by the conversion to GDP. One may notice that some loss in privacy guarantee is introduced since the extreme case  $|x_{i,j}| = c, \forall j$  actually violates the condition  $||x_i||_2 \leq c$ . To address this issue, following a similar method in [13, 38, 9], one may introduce Kashin's representation to transform the  $l_2$  geometry of the data into the  $l_\infty$  geometry. More specifically, [39] shows that for  $D > d$ , there exists a tight frame  $U$  such that for any  $x \in \mathbb{R}^d$ , one can always represent each  $x_i$  with  $y_i \in [-\gamma_0 / \sqrt{d}, -\gamma_0 / \sqrt{d}]^D$  for some  $\gamma_0$  and  $x_i = U y_i$ .
In Theorem 4, some loss in privacy guarantees is introduced when we convert  $f$ -DP to  $\mu$ -GDP. In fact, since each coordinate of the vector is processed independently, the extension from the scalar case to the  $d$ -dimensional case may be understood as the  $d$ -fold composition of the mechanism in the scalar case. The composed result can be well approximated or numerically obtained via the central limit theorem for  $f$ -DP in [15] or the Edgeworth expansion in [33]. In the following, we present the result for the ternary compressor by utilizing the central limit theorem for  $f$ -DP.
Theorem 5. For a vector  $x_{i} = [x_{i,1}, x_{i,2}, \dots, x_{i,d}]$  with  $|x_{i,j}| \leq c, \forall j$ , the ternary compressor with  $B \geq A > c$  is  $f^{\text{ternary}}(\alpha)$ -DP with
$$
G _ {\mu} (\alpha + \gamma) - \gamma \leq f ^ {t e r n a r y} (\alpha) \leq G _ {\mu} (\alpha - \gamma) + \gamma , \tag {14}
$$
$$
\mu = \frac {2 \sqrt {d c}}{\sqrt {A B - c ^ {2}}}, \gamma = \frac {0 . 5 6 \left[ \frac {A - c}{2 B} \left| 1 + \frac {c}{B} \right| ^ {3} + \frac {A + c}{2 B} \left| 1 - \frac {c}{B} \right| ^ {3} + \left(1 - \frac {A}{B}\right) \left| \frac {c}{B} \right| ^ {3} \right]}{\left(\frac {A}{B} - \frac {c ^ {2}}{B ^ {2}}\right) ^ {3 / 2} d ^ {1 / 2}}. \tag {15}
$$
Given the above results, we investigate the communication-privacy-accuracy tradeoff and compare the proposed ternary stochastic compressor with the state-of-the-art method SQKR in [13] and the classic Gaussian mechanism. According to the discussion in Remark 5, given the  $l_{2}$  norm constraint, Kashin's representation can be applied to transform it into the  $l_{\infty}$  geometry. Therefore, for ease of discussion, we consider the setting in which each user  $i$  stores a vector  $x_{i} = [x_{i,1}, x_{i,2}, \dots, x_{i,d}]$  with  $|x_{i,j}| \leq c = \frac{C}{\sqrt{d}}, \forall j$ , and  $||x_{i}||_{2} \leq C$ .
Ternary Stochastic Compressor: Let  $Z_{i,j} = \text{ternary}(x_{i,j}, A, B)$ , then  $\mathbb{E}[BZ_{i,j}] = x_{i,j}$  and  $Var(BZ_{i,j}) = AB - x_{i,j}^2$ . In this sense, applying the ternary stochastic compressor to each coordinate of  $x_i$  independently yields an unbiased estimator with a variance of  $ABd - ||x_i||_2^2$ . The privacy guarantee is given by Theorem 5, and the communication overhead is  $(\log_2(d) + 1)\frac{A}{B} d$  bits in expectation.
SQKR: In SQKR, each user first quantizes each coordinate of  $x_{i}$  to  $\{-c,c\}$  with 1-bit stochastic quantization. Then, it samples  $k$  coordinates (with replacement) and privatizes the  $k$  bit message via the  $2^{k}$  Random response mechanism with  $\epsilon$ -LDP [14]. The SQKR mechanism yields an unbiased estimator with a variance of  $\frac{d}{k}\left(\frac{e^{\epsilon} + 2^{k} - 1}{e^{\epsilon} - 1}\right)^{2}C^{2} - ||x_{i}||_{2}^{2}$ . The privacy guarantee is  $\epsilon$ -LDP, and the corresponding communication overhead is  $(\log_2(d) + 1)k$  bits.
Gaussian Mechanism: We apply the Gaussian mechanism (i.e., adding independent zero-mean Gaussian noise  $n_{i,j} \sim \mathcal{N}(0, \sigma^2)$  to  $x_{i,j}$ ), followed by a sparsification probability of  $1 - A / B$  as in ternary  $(x_{i,j}, A, B)$ , which gives  $Z_{i,j}^{Gauss} = \frac{B}{A} (x_{i,j} + n_{i,j})$  with probability  $A / B$  and  $Z_{i,j}^{Gauss} = 0$ , otherwise. It can be observed that  $\mathbb{E}[Z_{i,j}^{Gauss}] = x_{i,j}$  and  $Var(Z_{i,j}^{Gauss}) = \frac{B}{A}\sigma^2 + (\frac{B}{A} - 1)x_{i,j}^2$ . Therefore, the Gaussian mechanism yields an unbiased estimator with a variance of  $\frac{B}{A}\sigma^2 d + (\frac{B}{A} - 1)\|x_i\|_2^2$ . By utilizing the post-processing property, it can be shown that the above Gaussian mechanism is  $\frac{2\sqrt{dc}}{\sigma}$ -GDP [15], and the communication overhead is  $(\log_2(d) + 32)\frac{A}{B}d$  bits in expectation.
Discussion: It can be observed that for SQKR, with a given privacy guarantee  $\epsilon$ -LDP, the variance (i.e., MSE) depends on  $k$  (i.e., the communication overhead). When  $e^{\epsilon} \ll 2^{k}$  (which corresponds to the high privacy regime), the variance grows rapidly as  $k$  increases. For the proposed ternary stochastic compressor, it can be observed that both the privacy guarantee (in terms of  $\mu$ -GDP) and the variance depend on  $AB$ . Particularly, with a given privacy guarantee  $\mu < \sqrt{4dr / (1 - r)}$  for  $r = A / B$ , the variance is given by  $(4d / \mu^{2} + 1)C^{2} - ||x_{i}||_{2}^{2}$ , which remains the same regardless of the communication overhead. In this sense, we essentially remove the dependency of accuracy on the communication overhead and therefore break the three-way tradeoff between communication
Figure 4: For the left figure, we set  $k = 10$  and derive the corresponding variance for SQKR, based on which  $A$  and  $B$  for the ternary stochastic compressor are computed such that they have the same communication overhead and MSE in expectation. The middle and right figures show the tradeoff between  $\mu$ -GDP and MSE. For the middle figure, we set  $\sigma \in \{\frac{2}{5}, \frac{1}{2}, \frac{2}{3}, 1, 2, 4, 6, 8, 10\}$  for the Gaussian mechanism, given which  $A$  and  $B$  are computed such that  $AB = c^2 + \sigma^2$  and the sparsity ratio is  $A / B$ . For the right figure, we set  $A \in \{5c, 10c, 20c, 30c\}$  and  $A / B \in \{0.2, 0.4, 0.6, 0.8, 1.0\}$ , given which the corresponding  $\sigma$ 's are computed such that  $AB = c^2 + \sigma^2$ .
overhead, privacy, and accuracy. $^{1}$  This is mainly realized by accounting for privacy amplification by sparsification. At a high level, when fewer coordinates are shared (which corresponds to a larger privacy amplification and a larger MSE), the ternary stochastic compressor introduces less ambiguity to each coordinate (which corresponds to worse privacy protection and a smaller MSE) such that both the privacy guarantee and the MSE remain the same. Since we use different differential privacy measures from [13] (i.e.,  $\mu$ -GDP in this work and  $\epsilon$ -DP in [13]), we focus on the comparison between the proposed ternary stochastic compressor and the Gaussian mechanism (which is order-optimal in most parameter regimes, see [30]) in the following discussion and present the detailed comparison with SQKR in the experiments in Section 7.
Let  $AB = c^2 + \sigma^2$ , it can be observed that the  $f$ -DP guarantee of the ternary compressor approaches that of the Gaussian mechanism as  $d$  increases, and the corresponding variance is given by  $Var(BZ_{i,j}) = \sigma^2 + c^2 - x_{i,j}^2$ . When  $A = B$ , i.e., no sparsification is applied, we have  $Var(BZ_{i,j}) - Var(Z_{i,j}^{Gauss}) = c^2 - x_{i,j}^2$ . Specifically, when  $x_{i,j} \in \{-c, c\}, \forall 1 \leq j \leq d$ , the ternary compressor demonstrates the same  $f$ -DP privacy guarantee and variance as that for the Gaussian mechanism, i.e., the improvement in communication efficiency is obtained for free (in the large  $d$  regime). When  $B > A$ , we have  $Var(BZ_{i,j}) - Var(Z_{i,j}^{Gauss}) = (1 - \frac{B}{A})\sigma^2 + c^2 - \frac{B}{A}x_{i,j}^2$ , and there exists some  $B$  such that the ternary compressor outperforms the Gaussian mechanism in terms of both variance and communication efficiency. It is worth mentioning that the privacy guarantee of the Gaussian mechanism is derived by utilizing the post-processing property. We believe that sparsification brings improvement in privacy for the Gaussian mechanism as well, which is, however, beyond the scope of this paper.
**Optimality:** It has been shown that, for  $k$ -bit unbiased compression mechanisms, there is a lower bound of  $\Omega(C^2 d / k)$  in MSE [40]. For the proposed ternary compressor, the MSE and the communication cost are given by  $O(ABd)$  and  $A(\log(d) + 1)d / B$  bits, respectively. Let  $k = A(\log(d) + 1)d / B$ , it achieves an MSE of  $O(A^2 d^2 (\log(d) + 1) / k)$ . Since  $A > c = C / \sqrt{d}$ , the MSE of the ternary compressor is given by  $O(C^2 d(\log(d) + 1) / k)$ , which implies that it is order-optimal up to a factor of  $\log(d)$ . Note that the factor of  $\log(d)$  is used to represent the indices of coordinates that are non-zero, which can be eliminated by allowing for shared randomness between the users and the server.
In this section, we examine the performance of the proposed ternary compressor in the case of distributed mean estimation. We follow the set-up of [9] and generate  $N = 1000$  user vectors with dimension  $d = 250$ , i.e.,  $x_{1}, \ldots, x_{N} \in \mathbb{R}^{250}$ . Each local vector has bounded  $l_{2}$  and  $l_{\infty}$  norms, i.e.,  $\|x_{i}\|_{2} \leq C = 1$  and  $\|x_{i}\|_{\infty} \leq c = \frac{1}{\sqrt{d}}$ .
Fig. 4 compares the proposed ternary stochastic compressor with SQKR and the Gaussian mechanism. More specifically, the left figure in Fig. 4 compares the privacy guarantees (in terms of the tradeoff between type I and type II error rates) of the ternary stochastic compressor and SQKR given the
In practice, utilizing the closed-form expressions of the MSE and the privacy guarantee  $\mu$ , one may readily obtain the corresponding  $A$  and  $B$  for any given privacy/MSE and communication cost specifications.
same communication overhead and MSE. It can be observed that the proposed ternary stochastic compressor outperforms SQKR in terms of privacy preservation, i.e., given the same type I error rate  $\alpha$ , the type II error rate  $\beta$  of the ternary stochastic compressor is significantly larger than that of SQKR, which implies better privacy protection. For example, for SQKR with  $\epsilon = 2$ , given type I error rate  $\alpha = 0.5$ , the type II error rate of the attacker is around  $f^{SQR}(\alpha) = 0.068$ , while the ternary compressor attains  $f^{ternary}(\alpha) = 0.484$ . Given the same MSE and communication cost as that of SQKR with  $\epsilon_{SQR} = \{1,2,5\}$ , if we translate the privacy guarantees of the ternary compressor from  $f$ -DP to  $\epsilon$ -DP via Lemma 1 (we numerically test different  $\epsilon$ 's such that  $f^{ternary}(\alpha) \geq \max \{0,1 - \delta - e^{\epsilon}\alpha, e^{-\epsilon}(1 - \delta - \alpha)\}$  holds for  $\delta = 0$ ), we have  $\epsilon_{ternary} = \{0.05,0.2,3.9\}$  for the ternary compressor, which demonstrates its effectiveness. The middle and right figures in Fig. 4 show the tradeoff between MSE and DP guarantees for the Gaussian mechanism and the proposed ternary compressor. Particularly, in the middle figure, the tradeoff curves for the ternary compressor with all the examined sparsity ratios overlap with that of the Gaussian mechanism with  $A / B = 1$  since they essentially have the same privacy guarantees, and the difference in MSE is negligible. For the Gaussian mechanism with  $\frac{A}{B} < 1$ , the MSE is larger due to sparsification, which validates our discussion in Section 6. In the right figure, we examine the MSEs of the proposed ternary compressor with various  $A$ 's and  $B$ 's. It can be observed that the corresponding tradeoff between MSE and privacy guarantee matches that of the Gaussian mechanism well, which validates that the improvement in communication efficiency for the proposed ternary compressor is obtained for free.
8 Limitation
The main results derived in this paper are for the scalar case, which are extended to the vector case by invoking the central limit theorem. In this case, the privacy guarantees derived in Theorem 5 are tight only in the large  $d$  regime. Fortunately, in applications like distributed learning,  $d$  corresponds to the model size (usually in the orders of millions for modern neural networks). Moreover, despite that the privacy-accuracy tradeoff of the proposed ternary compressor matches that of the Gaussian mechanism which is order-optimal in  $(\epsilon, \delta)$ -DP, the optimality of the proposed ternary compressor in the  $f$ -DP regime needs to be further established.
In this paper, we derived the privacy guarantees of discrete-valued mechanisms with finite output space in the lens of  $f$ -differential privacy, which covered various differentially private mechanisms and compression mechanisms as special cases. Through leveraging the privacy amplification by sparsification, a ternary compressor that achieves better accuracy-privacy-communication tradeoff than existing methods is proposed. It is expected that the proposed methods can find broader applications in the design of communication efficient and differentially private federated data analysis techniques.
Richeng Jin was supported in part by the National Natural Science Foundation of China under Grant No. 62301487, in part by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LQ23F010021, and in part by the Ng Teng Fong Charitable Foundation in the form of ZJUSUTD IDEA Grant No. 188170-11102. Zhonggen Su was supported by the Fundamental Research Funds for the Central Universities Grants. Zhaoyang Zhang was supported in part by the National Natural Science Foundation of China under Grant No. U20A20158, in part by the National Key R&D Program of China under Grant No. 2020YFB1807101, and in part by the Zhejiang Provincial Key R&D Program under Grant No. 2023C01021. Huaiyu Dai was supported by the US National Science Foundation under Grant No. ECCS-2203214. The views expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.
[1] D. Wang, S. Shi, Y. Zhu, and Z. Han, "Federated analytics: Opportunities and challenges," IEEE Network, vol. 36, no. 1, pp. 151-158, 2021.
Breaking the Communication-Privacy-Accuracy Tradeoff with  $f$ -Differential Privacy: Supplementary Material
A Tradeoff Functions for a Generic Discrete-Valued Mechanism
We consider a general randomization protocol  $\mathcal{M}(\cdot)$  with discrete and finite output space. In this case, we can always find a one-to-one mapping between the range of  $\mathcal{M}(\cdot)$  and a subset of  $\mathbb{Z}$ . With such consideration, we assume that the output of the randomization protocol is an integer, i.e.,  $\mathcal{M}(S) \in \mathbb{Z}_{\mathcal{M}} \subset \mathbb{Z}, \forall S$ , without loss of generality. Given the randomization protocol and the hypothesis testing problem in (2), we derive its tradeoff function as a function of the type I error rate in the following lemma.
Lemma 2. For two neighboring datasets  $S$  and  $S'$ , suppose that the range of the randomized mechanism  $\mathcal{R}(\mathcal{M}(S)) \cup \mathcal{R}(\mathcal{M}(S')) = \mathbb{Z}_{\mathcal{M}}^U = [\mathcal{Z}_L^U, \ldots, \mathcal{Z}_R^U] \subset \mathbb{Z}$  and  $\mathcal{R}(\mathcal{M}(S)) \cap \mathcal{R}(\mathcal{M}(S')) = \mathbb{Z}_{\mathcal{M}}^I = [\mathcal{Z}_L^I, \ldots, \mathcal{Z}_R^I] \subset \mathbb{Z}$ . Let  $X = \mathcal{M}(S)$  and  $Y = \mathcal{M}(S')$ . Then,
Case (1) If  $\mathcal{M}(S)\in [\mathcal{Z}_L^I,\mathcal{Z}_L^I +1,\ldots ,\mathcal{Z}_R^U ],\mathcal{M}(S')\in [\mathcal{Z}_L^U,\mathcal{Z}_L^U +1,\ldots ,\mathcal{Z}_R^I ]$  and  $\frac{P(Y = k)}{P(X = k)}$  is a decreasing function of  $k$  for  $k\in \mathbb{Z}_{\mathcal{M}}^{I}$ , the tradeoff function in Definition 2 is given by
$$
\beta_ {\phi} ^ {+} (\alpha) = \left\{ \begin{array}{l l} P (Y \geq k) + \frac {P (Y = k) P (X <   k)}{P (X = k)} - \frac {P (Y = k)}{P (X = k)} \alpha , \\ \quad i f \alpha \in \left(P (X <   k), P (X \leq k) \right], k \in \left[ \mathcal {Z} _ {L} ^ {I}, \mathcal {Z} _ {R} ^ {I} \right]. \\ 0, \quad i f \alpha \in \left(P (X <   \mathcal {Z} _ {R} ^ {I} + 1), 1 \right]. \end{array} \right. \tag {16}
$$