SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Under the condition of Lemma 1, the minimax risk of univariate mean estimation under  $f$ -FLDP is  $O(n^{-\frac{k - 1}{k}})$  by Corollary 1, which is the same rate for  $\epsilon$ -LDP. Here, we note that the assumption of Lemma 1 is quite general. Fig. 1b shows some examples of trade-off functions satisfying Lemma 1. It can be seen that Lemma 1 requires the slope at  $x = 0$  of the trade-off function should not be too steep. Nonetheless, note that there exists a trade-off function requiring  $\kappa < 1$  in order to make  $c_{f,\kappa}$  finite. For example,  $f(x) = 1 - x^{\frac{\kappa}{1 + \kappa}}$  is a trade-off function with  $\delta_f(t) = \frac{\kappa^\kappa}{(1 + \kappa)^{1 + \kappa}} t^{-\kappa}$  for  $t > \frac{\kappa}{1 + \kappa}$ . Therefore,  $\int t^{\kappa - 1} \delta_f(t) dt$  does not converge, and neither does  $c_{f,\kappa}$ . Thus, there exist some mechanisms that Corollary 1 cannot guarantee the minimax rate equivalent with the optimal rate under  $\epsilon$ -LDP. In other words, they may enjoy better minimax rates. In the supplementary material, it is shown that  $G_\mu$  satisfies Lemma 1 for any  $\mu > 0$ .
Combining it with Corollary 1, we can bound the minimax risk of univariate mean estimation under  $\mu$ -GLDP. Denote  $M_{\mu}$  for a family of  $\mu$ -GLDP mechanisms.
Corollary 2. The minimax risk of univariate mean estimation under  $\mu$ -GLDP mechanism is bounded as follows:
$$
c _ {0} \left(n e ^ {\frac {1}{2} \mu^ {2}}\right) ^ {- \left(1 - \frac {1}{k}\right)} \leq \mathcal {R} _ {n} \left(\theta \left(\mathcal {P} _ {k}, | \cdot | _ {2} ^ {2}, M _ {\mu}\right)\right) \leq c _ {1} (n \mu^ {2} (4 + \mu^ {2}) ^ {- 1}) ^ {- \left(1 - \frac {1}{k}\right)},
$$
for  $n > c_2$  where  $c_0, c_1 > 0$  are constants depending only on  $k$ , and  $c_2$  is a constant depending only on  $\mu$  and  $k$ .
We present the following  $\mu$ -GLDP algorithm that can be shown to be minimax optimal, as it achieves the rate of  $O(n^{-\frac{k - 1}{k}})$ . See the supplementary material for a proof and empirical comparison of other LDP mechanisms including the minimax optimal  $\epsilon$ -LDP mechanism in Duchi et al. (2018).
1.  $T = \left[n\left(1 + \frac{4}{\mu^2}\right)^{-1}\right]^{\frac{1}{2k}}$
2.  $M_{i}(X_{i}) = \max \{-T, \min \{X_{i}, T\} \} + \epsilon_{i}$  where  $\epsilon_{i} \sim N\left(0, \frac{4T^{2}}{\mu^{2}}\right)$ .
3.  $\hat{\theta}\left(M(X_1,\ldots ,X_n)\right)\coloneqq \frac{1}{n}\sum_{i = 1}^{n}M_i(X_i).$
The fact that  $\mu$ -GLDP and  $\epsilon$ -LDP have the same minimax rate might be unexpected to some, especially considering that the former is a more relaxed concept than the latter. In the following lemma, we present an interesting inequality that can shed light on their connection.
Lemma 2. Let  $M: \mathcal{X} \longrightarrow \mathcal{P}(\mathcal{Z})$  be a locally private mechanism taking only one data for its input. For  $P_1, P_2 \in \mathcal{P}(\mathcal{X})$ , denote  $m_i$  for density of  $M(P_i)$  for  $i = 1, 2$ . If  $M$  is  $f$ -FLDP, then for  $a \geq 2$ ,
$$
\mathbb {P} _ {m _ {2}} \left(\frac {m _ {1} (Z)}{m _ {2} (Z)} > a\right) \leq \delta_ {f} (a - 1).
$$
Lemma 2 implies that, for sufficiently small  $\delta_f(e^\epsilon - 1)$ , an  $f$ -FLDP mechanism behaves like  $\epsilon$ -LDP with high probability  $(1 - \delta_f(e^\epsilon - 1))$ . Moreover, this lemma provides a probabilistic interpretation for  $(\epsilon, \delta)$ -LDP: the probability that the likelihood ratio between output distributions exceeds  $e^\epsilon + 1$  is bounded by  $\delta$ . In the DP literature (Mironov, 2017; Meiser, 2018),  $(\epsilon, \delta)$ -DP is often informally explained as "an  $\epsilon$ -DP with probability  $1 - \delta$ ," which can cause confusion. In this regard, here we provide an accurate statement specifying the respective roles of  $\epsilon$  and  $\delta$ .
In addition, the  $\mu$ -GLDP mechanism possesses a similar contraction ability to  $\epsilon$ -LDP. The term contraction of a mechanism is generally used to describe the reduction of distance measures between distributions that is altered by a mechanism. In Duchi et al. (2018), it was shown that  $D_{kl}\left(M(P_1) || M(P_2)\right)$ , the Kullback-Leibler (K-L) divergence between privatized distributions under  $\epsilon$ -LDP, is bounded by  $2(e^{\epsilon} - 1)^{2} \| P_1 - P_2\|_{TV}^{2}$ . We establish a similar bound under  $f$ -FLDP. Following the notations in Lemma 2, the following theorem describes the contraction of  $f$ -FLDP mechanism in terms of K-L divergence.
Theorem 2. For given  $0 \leq \kappa \leq 1$  and a trade-off function  $f$ , if  $\int_0^\infty t^{\kappa - 1} \delta_f(t) dt$  is finite, then
$$
D _ {k l} ^ {s y} (m _ {1} \| m _ {2}) := D _ {k l} (m _ {1} \| m _ {2}) + D _ {k l} (m _ {2} \| m _ {1}) \leq c _ {f, \kappa} \frac {\| P _ {1} - P _ {2} \| _ {T V} ^ {1 + \kappa}}{(1 - \| P _ {1} - P _ {2} \| _ {T V}) ^ {\kappa}}
$$
holds for any  $f$ -FLDP mechanism  $M: \mathcal{X} \longrightarrow \mathcal{P}(\mathcal{Z})$  and distributions  $P_{1}$  and  $P_{2}$  over  $\mathcal{X}$ .
In the case of  $\mu$ -GLDP, the bound comes down to  $O\left(\|P_1 - P_2\|_{TV}^2\right)$  since  $\int_0^\infty t^{\kappa-1}\delta_f(t)dt$  is finite for  $\kappa = 1$ . Because  $f$ -FLDP includes  $\epsilon$ -LDP, our result is more general than Duchi et al. (2018) and further gives an insight into why  $\epsilon$ -LDP and  $\mu$ -GLDP have indistinguishable privatizing power. Our analysis also improves the bound found by Asoodeh et al. (2021), which is expressed as a constant multiple of  $D_{kl}$  ( $P_1\|P_2$ ) thus could be unbounded for point distributions. Our result bounds the divergence by total variation between  $P_1$  and  $P_2$ , which is finite for any two arbitrary distributions. See the supplementary material for an elaborated illustration.
3.2 Assouad's Bound for Nonparametric Density Estimation
This section aims to derive the minimax risks of private nonparametric density estimation under  $f$ -FLDP and  $\mu$ -GLDP. While Le Cam's method is effective for many problems, it may not be suitable for high-dimensional structured problems. In such cases, Assouad's method offers a solution by reformulating the estimation problem as a multiple-binary hypothesis testing problem. Given the parameter space  $\Theta$ , there exists a map  $V: \Theta \longrightarrow \{-1, +1\}^d$  and a family of distributions  $\{P_v\}_{v \in \{-1, +1\}^d}$  for  $d \in \mathbb{N}$  such that
$$
\Phi (\rho (\theta , \theta (P _ {v}))) \geq 2 \eta \sum_ {j = 1} ^ {d} 1 \{[ V (\theta) ] _ {j} \neq v _ {j} \}
$$
for every  $v \in \{-1, +1\}^d$ . We say  $\{P_v\}_{v \in \{-1, +1\}^d}$  induces a  $2\eta$ -Hamming separation under loss  $\Phi \circ \rho$ . Assouad's method establishes a lower bound of minimax risk by assuming not the worst case but a randomly selected case in  $\{P_v\}_{v \in \{-1, +1\}^d}$ . We present the Assouad's bound under  $f$ -FLDP mechanisms.
Theorem 3 (Private Assouad's bound). For given  $0 \leq \kappa \leq 1$  and a trade-off function  $f$ , if  $\int_0^\infty t^{\kappa - 1} \delta_f(t) dt$  is finite and a set of distributions  $\{P_v\}_{v \in \{-1, +1\}^d} \subset \mathcal{P}$  induces  $2\eta$ -Hamming separation under  $\Phi \circ \rho$ , then
$$
\mathcal {R} _ {n} \left(\theta (\mathcal {P}), \Phi \circ \rho , M _ {f}\right) \geq d \eta \left[ 1 - \sqrt {\frac {n c _ {f , \kappa}}{d} \sum_ {j = 1} ^ {d} \frac {\| P _ {+ j} - P _ {- j} \| _ {T V} ^ {1 + \kappa}}{\left(1 - \| P _ {+ j} - P _ {- j} \| _ {T V}\right) ^ {\kappa}}} \right] \tag {2}
$$
where  $P_{\pm j} = \frac{1}{2^{d - 1}}\sum_{v:v_j = \pm 1}P_v$
We can use Theorem 3 to obtain the lower bound of minimax risk of private nonparametric density estimation. The true density is assumed to be in the elliptical Sobolev space, as defined below:
Definition 3.1 (Elliptical Sobolev space). For a given orthonormal basis  $\{\phi_j\}_{j=1}^{\infty}$  of  $L^2([0,1])$ , smoothness parameter  $\beta > \frac{1}{2}$ , and radius  $r > 0$ , the elliptical Sobolev space is the following set:
$$
\mathcal {F} _ {\beta} [ r ] = \left\{h \in L ^ {2} [ 0, 1 ] \Bigg | h = \sum_ {j = 1} ^ {\infty} \theta_ {j} \phi_ {j}, \sum_ {j = 1} ^ {\infty} j ^ {2 \beta} \theta_ {j} ^ {2} \leq r ^ {2} \right\}.
$$
If we use the trigonometric basis, then  $\mathcal{F}_{\beta}$  represents a set of distributions with the bounded norm of  $\beta$ -derivatives with coinciding derivatives of degree  $< \beta$  at endpoints of  $[0,1]$  (Tsybakov, 2009). Applying Theorem 3, we can obtain the following lower bound.
Corollary 3 (Nonparametric density estimation). For given  $0 \leq \kappa \leq 1$  and a trade-off function  $f$ , if  $\int_0^\infty t^{\kappa - 1} \delta_f(t) dt$  is finite, then
$$
\mathcal {R} _ {n} \left(\theta (\mathcal {F} _ {\beta} [ r ]), \| \cdot \| _ {2} ^ {2}, M _ {f}\right) \geq r ^ {\frac {2}{\beta + 1}} c _ {1} (n c _ {f, \kappa}) ^ {- \frac {2 \beta}{(\beta + 1) (1 + \kappa)}}
$$
for  $n > c_{2}$  where  $c_{1},c_{2} > 0$  are constant depends on  $\beta$
It is known that the minimax risk in the non-private setting is  $O\left(n^{-\frac{2\beta}{2\beta + 1}}\right)$  (Tsybakov, 2009) and  $O\left((n\epsilon^2)^{-\frac{2\beta}{2\beta + 2}}\right)$  under  $\epsilon$ -LDP (Duchi et al., 2018). Thus according to Corollary 3, the minimax rate of  $f$ -FLDP with  $\kappa = 1$  is the same as that of  $\epsilon$ -LDP. As in Section 3.1, we can obtain the bounds of minimax risk of nonparametric density estimation under  $\mu$ -GLDP, using Lemma 1.
Corollary 4. The minimax risk of nonparametric density estimation under  $\mu$ -GLDP is bounded as follows:
$$
c _ {0} r ^ {\frac {2}{\beta + 1}} \left(n e ^ {\frac {1}{2} \mu^ {2}}\right) ^ {- \frac {2 \beta}{2 \beta + 2}} \leq \mathcal {R} _ {n} \left(\theta (\mathcal {F} _ {\beta} [ r ]), \| \cdot \| _ {2} ^ {2}, M _ {\mu}\right) \leq c _ {1} r ^ {\frac {2}{\beta + 1}} (n \mu^ {2}) ^ {- \frac {2 \beta}{2 \beta + 2}}
$$
for  $n > c_2$  where  $c_0, c_1 > 0$  are constants depending only on  $\beta$ , and  $c_2$  is a constant depending only on  $\mu$  and  $\beta$ .
Note that the minimax rate in Corollary 4 is essentially the same as that of  $\epsilon$ -LDP. However, under  $\mu$ -GLDP, one can find the optimal mechanism in a more straightforward way. This is in contrast to the discussion in Duchi et al. (2018) that the Laplacian mechanism, the canonical mechanism under  $\epsilon$ -LDP, cannot achieve the optimal minimax rate. They instead proposed a complicated mechanism to achieve optimality. Our minimax optimal mechanism and density estimator are presented below:
Suppose that  $n$  i.i.d. data  $X_{1},X_{2},\ldots ,X_{n}$  drawn from  $h\in \mathcal{F}_{\beta}[r]$  are available.
1. Set  $d = \left\lfloor (0.5n\mu^2 r^2)^{\frac{1}{2\beta + 2}} \right\rfloor$  where  $\lfloor x \rfloor$  is the largest integer not exceeding  $x$ .
2.  $M_{i}(X_{i}):= (\phi_{1}(X_{i}),\ldots ,\phi_{d}(X_{i}))^{T} + \epsilon$  where  $\epsilon \sim N(\mathbf{0},\frac{2d}{\mu^2}\mathbf{I}_{d\times d})$
3. Define a function  $\hat{h}_i:[0,1]\longrightarrow \mathbb{R}$  as  $\hat{h}_i\coloneqq \sum_{j = 1}^d [M_i(X_i)]_j\phi_j$
4. Repeat 2 and 3 for  $i = 1, \dots, n$ .
5.  $\hat{h} = \frac{1}{n}\sum_{i = 1}^{n}\hat{h}_i$  is the obtained estimator.
4 Empirical Comparison of Minimax Optimal Mechanisms
In this section, we compare the optimal mechanisms for nonparametric density estimation under  $\epsilon$ -LDP,  $\mu$ -GLDP, and non-private mechanisms. The simulation on univariate mean estimation is provided in the supplementary material. Our proposed mechanism in Section 3.2 under  $\mu$ -GLDP is compared with the mechanism in Duchi et al. (2018) and the non-private mechanism, which is obtained by using  $d = \left\lfloor (nr^2)^{\frac{1}{2\beta + 1}}\right\rfloor$  with  $\epsilon = 0$  in our  $\mu$ -GLDP algorithm.
The true underlying density is given as the density of  $\mathrm{Beta}(5,5)$  distribution. We set  $\epsilon \in \{0.4, 0.8, 1.6, 3.2, 6.4\}$  for  $\epsilon$ -LDP and  $\mu \in \{0.5, 1\}$  for  $\mu$ -GLDP. We vary the sample size from  $n = 100$  to  $n = 2000$ . Estimation errors are calculated by integrated mean squared errors. We repeat the experiment 100 times for each mechanism.
(a) Density estimates
(b) Mean squared errors
Figure 2: (a)  $95\%$  confidence intervals calculated from private density estimates over 100 repetitions for three LDP and non-private mechanisms. (b) The mean squared errors with one-standard errors of density estimates for various LDP mechanisms and sample sizes.
The trade-off functions in Fig. 1a show that 6.4-LDP is very close to the non-private setting and 1-GLDP is between 0.4-LDP and 6.4-LDP almost everywhere in terms of privacy levels. However, Fig. 2a, displaying the  $95\%$  point-wise confidence intervals obtained from estimated densities by different methods, shows that estimates under 1-GLDP are more in agreement with the true density than 6.4-LDP. The performance under 0.4-LDP is much worse than the others because it is the most private mechanism. The superiority of 1-GLDP can also be seen in terms of mean squared errors. In Fig. 2b, 1-GLDP performs better than all other  $\epsilon$ -LDPs.
The utility of a private mechanism in this density estimation essentially depends on how well it approximates the estimates of Fourier coefficients that would be obtained from the original data. This task requires an efficient mechanism to estimate a multi-dimensional coefficient vector for each observation. The optimal mechanism for this task under  $\epsilon$ -LDP transforms a vector of non-private Fourier coefficient estimates into a binomial vector, which limits the support in order to reduce variance and achieve privacy. However, due to the inherent variance introduced by discretization, the mean squared error is bounded by a constant multiple of  $\left(\frac{e^{\epsilon} + 1}{e^{\epsilon} - 1}\right)^2$ . Explicitly, the optimal error bound of the  $\epsilon$ -LDP density estimation mechanism in Duchi et al. (2018) can be derived as follows:
$$
\mathcal {R} _ {\epsilon} \leq (\beta + 1) \left(\frac {\beta}{\sqrt {2 \pi e}} n \left(\frac {e ^ {\epsilon} + 1}{e ^ {\epsilon} - 1}\right) ^ {- 2}\right) ^ {- \frac {2 \beta}{2 \beta + 2}} r ^ {\frac {2}{\beta + 1}}
$$
where  $\mathcal{R}_{\epsilon}$  is the expected error obtained from the  $\epsilon$ -LDP density estimation mechanism in Duchi et al. (2018). Hence, even with a lenient privacy constraint (i.e., a large  $\epsilon$ ), the estimation error may not decrease significantly. In contrast, our  $\mu$ -GLDP optimal mechanism generates an estimator by adding noise with decreasing variance as  $\mu$  tends to infinity. Again, the explicit optimal error bound is given
3We use  $\beta = 3$  and  $r = 408.8979$  for the proposed mechanism in Section 3.2
$$
\mathcal {R} _ {\mu} \leq (\beta + 1) \left(0. 5 \beta n \mu^ {2}\right) ^ {- \frac {2 \beta}{2 \beta + 2}} r ^ {\frac {2}{\beta + 1}} + O \left(n ^ {- \frac {2 \beta + 1}{2 \beta + 2}}\right)
$$
where  $\mathcal{R}_{\mu}$  is the expected error obtained from our  $\mu$ -GLDP density estimation mechanism. The comparison of the coefficients of  $n^{-\frac{2\beta}{2\beta + 2}}$ , which can be interpreted as the cost of privacy, gives the asymptotic error ratio:
$$
\frac {\mathcal {R} _ {\mu}}{\mathcal {R} _ {\epsilon}} \approx \left(\sqrt {\frac {\pi e}{2}} \mu^ {2} \left(\frac {e ^ {\epsilon} + 1}{e ^ {\epsilon} - 1}\right) ^ {2}\right) ^ {- \frac {2 \beta}{2 \beta + 2}},
$$
which exceeds 1 when  $\mu = 1$ . Therefore, the estimation under  $\epsilon$ -LDP is likely to produce higher errors than the estimation under 1-GLDP for any  $\epsilon$ . This is also evidenced by the result in Fig. 2a. The fact that private estimation error under  $\epsilon$ -LDP does not approach non-private estimation error implies that  $\epsilon$ -LDP adds more noise than necessary to high-dimensional data compared to  $\mu$ -GLDP. This highlights the limitation of  $\epsilon$ -LDP and the advantage of  $\mu$ -GLDP in achieving privacy for multi-dimensional data, which can be practically beneficial for addressing other private estimation problems.
The present work on  $f$ -FLDP establishes the private Le Cam and Assouad bounds reflecting the continuous nature of privacy constraints. As the trade-off function  $f$  becomes more (less) private, the supremum value of  $\kappa$  satisfying the conditions of Corollaries 1 and 3 increases (decreases), leading to the  $\epsilon$ -LDP (non-private, respectively) minimax rate in terms of  $n$ . This is in contrast to the somewhat counter-intuitive minimax lower bounds derived under  $\epsilon$ -LDP, whose utility never achieves that of non-private estimation even as  $\epsilon$  approaches infinity. Our simulation study in Section 4 shows that  $\mu$ -GLDP can offer similar or better privacy level than  $\epsilon$ -LDP, while achieving higher accuracy in private estimation. This suggests that  $\mu$ -GLDP may be a better option for achieving both privacy and accurate estimation. The results on the contraction inequality reported in Theorem 2 and in the supplementary material provide additional support for this claim based on the privatizing characteristics of the mechanisms. In summary, this work shows that  $\mu$ -GLDP is theoretically comparable in terms of both utility and privatizing ability while enjoying the better practical performance. This supports the suggestion made by Dong et al. (2022) that we use G(L)DP as a reliable standard for (L)DP both theoretically and empirically.
Possible limitations of this work are as follows: Although we establish lower bounds that connect privacy and utility in a continuous manner through  $\kappa \in [0,1]$ , the suggested optimal mechanisms only achieve the minimax rates for  $\kappa = 1$ . It will be desirable to identify private mechanisms and estimators that can achieve minimax rates between the non-private and  $\epsilon$ -LDP minimax rates, if such mechanisms exist. Additionally, the condition in Theorems 1 and 3 does not hold for all possible trade-off functions. Also, the bounds in Corollaries 1 and 3 require  $\int t^{\kappa -1}\delta_f(t)dt$  to be finite, which restricts the potential application of the theory, even though most commonly used privatization mechanisms, such as the Gaussian and Laplace mechanisms, satisfy this condition. Nevertheless, these bounds only coincide with the true minimax rates for only limited categories of privacy. Hence, we plan to explore whether there exists an optimal mechanism that enjoys a better minimax rate for mean and density estimation. Additionally, our theoretical minimax bound for estimations under  $\mu$ -GLDP is tight in terms of the sample size  $n$ , but it notably lacks tightness with respect to the privacy parameter  $\mu$ . Our lower bound converges to a finite value when  $\mu$  goes to 0 incompatible with the upper bound. Hence, further efforts are required to establish a rigorous minimax rate in terms of privacy constraints for local FDP. Finally, our results in Section 3.1 are restricted to univariate mean estimation, and extending them to high-dimensional estimation under non- $\epsilon$ -LDP would be a non-trivial task.
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No.2022-0-00937, Solving the problem of increasing the usability and usefulness of synthetic data algorithms for statistical data.) The work of Jeongyoun Ahn was partially supported by the National Research Foundation of Korea (NRF-2021R1A2C1093526, RS-2023-00218231). The work of Cheolwoo Park was partially supported by the National Research Foundation of Korea (NRF-2021R1A2C1092925).
Mironov, I. (2017). Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263-275.
Rohde, A. and Steinberger, L. (2020). Geometrizing rates of convergence under local differential privacy constraints. The Annals of Statistics, 48(5):2646 - 2670.
Sweeney, L. (2000). Simple demographics often identify people uniquely. Health (San Francisco), 671:1-34.
Sweeney, L. (2002). Achieving k-anonymity privacy protection using generalization and suppression. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(05):571-588.
Tang, J., Korolova, A., Bai, X., Wang, X., and Wang, X. (2017). Privacy loss in Apple's implementation of differential privacy on MacOS 10.12. arXiv preprint arXiv:1709.02753.
Thakurta, A. G., Vyrros, A. H., Vaishampayan, U. S., Kapoor, G., Freudiger, J., Sridhar, V. R., and Davidson, D. (2017). Learning new words. Granted US Patents, 9,645,998 [Online] Available: https://www.google.com/patents/US9645998.
Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation, 2009. Springer Series in Statistics.
Wang, S., Luo, X., Qian, Y., Du, J., Lin, W., and Yang, W. (2022). Analyzing preference data with local privacy: Optimal utility and enhanced robustness. IEEE Transactions on Knowledge and Data Engineering, pages 1-14.
Wasserman, L. and Zhou, S. (2010). A statistical framework for differential privacy. Journal of the American Statistical Association, 105(489):375-389.