SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuheng Ma $^{1}$  Han Zhang $^{1}$  Yuchao Cai $^{2}$  Hanfang Yang $^{13*}$
<sup>1</sup>School of Statistics, Renmin University of China
$^{2}$ Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente
<sup>3</sup>Center for Applied Statistics, Renmin University of China
{yma,hanzhang0816,hyang}@ruc.edu.cn
y.cai@utwente.nl
We propose conducting locally differentially private (LDP) estimation with the aid of a small amount of public data to enhance the performance of private estimation. Specifically, we introduce an efficient algorithm called Locally differentially Private Decision Tree (LPDT) for LDP regression. We first use the public data to grow a decision tree partition and then fit an estimator according to the partition privately. From a theoretical perspective, we show that LPDT is  $\varepsilon$ -LDP and has a minimax optimal convergence rate under a mild assumption of similarity between public and private data, whereas the lower bound of the convergence rate of LPDT without public data is strictly slower, which implies that the public data helps to improve the convergence rates of LDP estimation. We conduct experiments on both synthetic and real-world data to demonstrate the superior performance of LPDT compared with other state-of-the-art LDP regression methods. Moreover, we show that LPDT remains effective despite considerable disparities between public and private data.
Differential privacy (DP) [25] is a widely-used technique to protect sensitive information, like in medical trials [20], recommendation systems [49], and census data sharing [3]. Local differential privacy (LDP) [38, 24], a variation of DP, has gained particular attention, especially among industry experts [27, 35]. Unlike DP, LDP assumes data is privatized before being sent to a central collector. However, LDP models need more data to be accurate compared to DP [24], and many common techniques in data analysis such as standardization [10] and tree partition [59] are harder with LDP. This brings challenges to tasks such as density density estimation [23], mean estimation [24], and Gaussian estimation [36].
Fortunately, in some scenarios, private estimation performance can be enhanced with an additional public dataset [4, 7]. The public dataset can be either in-distribution, consisting of data from users who agree to share their personal information [6], or out-of-distribution, such as data from another source [34]. From a central DP perspective, an increasing amount of research has focused on leveraging public data to facilitate private learning, where public data mainly serves two purposes. On one hand, the knowledge learned from public data is implicitly transferred into the private model. Empirical investigations have demonstrated the effectiveness of pretraining on public data and fine-tuning privately on sensitive data [63, 43, 41, 61]. By gradient pre-conditioning with a subspace computed by public data, [65, 62, 37] managed to reduce the required amount of noise in differentially private gradient descent and accelerate its convergence. Through unlabeled public data, [45, 46] fed knowledge privately into student models. On the other hand, on public data, we can
conduct procedures that would be infeasible without access to the raw private data. For example, [10] used parameters computed by public data to standardize private data, which can augment the sample complexity of private mean estimation. Recently, [56] employed unlabeled public data to estimate the leading eigenvalue of the covariance matrix, resulting in an improved sample complexity for generalized linear models with non-interactive local differential privacy.
The paper focuses on the problem of non-parametric regression with LDP. While regression has been extensively studied in the central setting [50, 1, 14], the LDP case remains rarely explored. A notable reason is that most gradient-based methods [50, 1] are prohibited. In order to protect privacy, each data holder needs to compute the gradient of parameters locally, which requires a large amount of memory, computing power, and communication capacity on the terminal machine [52]. [29] proposed to impose Laplace noise on the data directly to provide privacy. However, this method is known to converge slowly [28] and suffer from the curse of dimensionality. More recently, [9, 33] investigated histogram-based approaches. Though theoretically optimal, histograms may perform poorly in practice, especially when the dimension of feature space is large. Thus, both methods proposed in [9, 33] face challenges when applied to real-world problems.
Under such background, using the idea of borrowing public data information, we propose an LDP non-parametric regression algorithm called the Locally differentially Private Decision Tree (LPDT) that achieves both optimal convergence rate and superior empirical performance. We first create a tree partition on the public dataset using the proposed max-edge rule. According to the partition, each data holder encodes the private data and releases the encoding which is processed using the proposed privacy mechanism. Finally, the curator aggregates the information in each partition cell and outputs a decision tree estimator. LPDT is advantageous from at least two perspectives: (i) LPDT integrates both benefits to leverage public data. It enables adaptive partitioning procedures which can eliminate some redundant cells and can transfer information from public data to private estimation through the tree partition. (ii) It inherits the merit of the decision tree model, such as interpretability, efficiency, stability, extensiveness to multiple feature types, and resistance to the curse of dimensionality.
We summarize our contributions. (i) For the first time, we propose to use public data in locally differentially private non-parametric regression. (ii) We propose a novel LDP regression algorithm called the locally differentially private decision tree that achieves theoretical optimality while maintaining satisfying practical performance. (iii) Under mild assumptions on the similarity between the distribution of public and private data, we establish the optimal convergence rate of LPDT, whereas the supremum of excess risk of LPDT without public fails to converge to zero. This demonstrates the theoretical advantage of incorporating public data. (iv) In experiments, we compare LPDT with other existing non-parametric LDP regression methods using both synthetic and real-world datasets. Our results demonstrate the overwhelming performance of LPDT, which illustrates the empirical improvement brought by public data. Moreover, we show that LPDT performs well even in the presence of significant disparities between public and private data.
This section is dedicated to the methodology of LPDT. In Section 2.1, we first present notations and preliminaries related to regression problems, followed by a recap of the definition of local differential privacy. Next, we introduce our hybrid privacy mechanism for general partition-based estimation in Section 2.2. In Section 2.3, we propose our partition rule. Finally, in Section 2.4, we provide a comprehensive description of LPDT.
Notations For any vector  $x$ , let  $x^i$  denote the  $i$ -th element of  $x$ . Recall that for  $1 \leq p < \infty$ , the  $L_p$ -norm of  $x = (x^1, \ldots, x^d)$  is defined by  $\| x \|_p \coloneqq (|x^1|^p + \dots + |x^d|^p)^{1/p}$ . Throughout this paper, we use the notation  $a_n \lesssim b_n$  and  $a_n \gtrsim b_n$  to denote that there exist positive constants  $c$  and  $c'$  such that  $a_n \leq c b_n$  and  $a_n \geq c' b_n$ , for all  $n \in \mathbb{N}$ . In addition, we denote  $a_n \asymp b_n$  if  $a_n \lesssim b_n$  and  $b_n \lesssim a_n$ . Let  $a \vee b = \max(a, b)$  and  $a \wedge b = \min(a, b)$ . Besides, for any set  $A \subset \mathbb{R}^d$ , the diameter of  $A$  is defined by  $\mathrm{diam}(A) := \sup_{x, x' \in A} \| x - x' \|_2$ . Let the standard Laplace random variable have the continuous probability density function  $p(x) = \frac{1}{2} e^{-|x|}$  for  $x \in \mathbb{R}$ .
Regression is to predict the value of an unobserved output variable  $Y$  based on the observed input variable  $X$ , based on a dataset  $D := \{(X_1, Y_1), \ldots, (X_n, Y_n)\}$  consisting of  $n$  i.i.d. observations drawn from an unknown probability measure  $P$  on  $\mathcal{X} \times \mathcal{Y} = [0, 1]^d \times [-M, M]$ . The density function of  $P$  is denoted as  $s$ . In addition, we have a public dataset  $D^{pub} := \{(X_1^{pub}, Y_1^{pub}), \ldots, (X_{n_q}^{pub}, Y_{n_q}^{pub})\}$  drawn from distribution  $Q$  on  $\mathcal{X} \times \mathcal{Y}$  with sample size  $n_q$ . Its density function is denoted as  $q$ .
It is legitimate to consider the least square loss  $L: \mathcal{X} \times \mathcal{Y} \times \mathcal{Y} \to [0, \infty)$  defined by  $L(x, y, f(x)) := (y - f(x))^2$  for our target of regression. Then, for a measurable decision function  $f: \mathcal{X} \to \mathcal{Y}$ , the risk is defined by  $\mathcal{R}_{L,\mathrm{P}}(f) := \int_{\mathcal{X} \times \mathcal{Y}} L(x, y, f(x)) d\mathrm{P}(x, y)$ . The Bayes risk, which is the smallest possible risk with respect to  $\mathrm{P}$  and  $L$ , is given by  $\mathcal{R}_{L,\mathrm{P}}^* := \inf \{\mathcal{R}_{L,\mathrm{P}}(f) | f: \mathcal{X} \to \mathcal{Y} \text{ measurable}\}$ . The function that achieves the Bayes risk is called Bayes function, namely,  $f^*(x) := \mathbb{E}\big(Y | X = x\big)$ .
Definition 2.1 (Local Differential Privacy). Given data  $\{(X_i,Y_i)\}_{i = 1}^n$ , each  $(X_{i},Y_{i})$  is mapped to a piece of privatized information  $s_i$  which is a random variable on  $\mathcal{S}$ . Let  $\sigma (\mathcal{S})$  be the  $\sigma$ -field on  $\mathcal{S}$ .  $s_i$  is drawn conditional on  $(X_{i},Y_{i})$  via the distribution R  $(S\mid X_i = x,Y_i = y)$  for  $S\in \sigma (\mathcal{S})$ . Then the mechanism R provides  $\varepsilon$ -local differential privacy ( $\varepsilon$ -LDP) if
$$
\sup  \left\{\frac {\operatorname {R} \left(S \mid X _ {i} = x , Y _ {i} = y\right)}{\operatorname {R} \left(S \mid X _ {i} = x ^ {\prime} , Y _ {i} = y ^ {\prime}\right)} \mid S \in \sigma (\mathcal {S}), \text {a n d} x, x ^ {\prime} \in \mathcal {X}, y, y ^ {\prime} \in \mathcal {Y} \right\} \leq e ^ {\varepsilon}.
$$
This formulation of local privacy is widely adopted [23, 9]. In contrast to central DP where the likelihood ratio is taken with respect to some statistics of all data, LDP requires individuals to guarantee their own privacy by considering the likelihood ratio with respect to each  $(X_{i},Y_{i})$ . Once the view  $s$  is provided, no further processing can reduce the deniability about taking a value  $(x,y)$  since any outcome  $s$  is nearly as likely to have come from some other initial value  $(x^{\prime},y^{\prime})$ .
2.2 Privacy mechanism for tree partition
This section focuses on the hybrid privacy mechanism for general tree partitions. We first introduce the standard regression tree and then present our privacy mechanism based on the random response and Laplacian mechanism.
For index set  $\mathcal{I}$ , let  $\pi = \{A_j\}_{j \in \mathcal{I}}$  be any tree partition of  $\mathcal{X}$  with  $\cup_{j \in \mathcal{I}} A_j = \mathcal{X}$  and  $A_i \cap A_j = \emptyset$ ,  $i \neq j$ . For any  $x \in \mathcal{X}$ , let the cell containing  $x$  be  $A(x)$ . A population decision tree regressor with partition  $\pi$  is defined as
$$
\bar {f} _ {\pi} (x) = \sum_ {j \in \mathcal {I}} \mathbf {1} \{x \in A _ {j} \} \frac {\int_ {A _ {j}} f ^ {*} \left(x ^ {\prime}\right) d P \left(x ^ {\prime}\right)}{\int_ {A _ {j}} d P \left(x ^ {\prime}\right)}. \tag {1}
$$
Here, we let  $0 / 0 = 0$  by definition. To get a empirical estimator given the data set  $D = \{(X_1,Y_1),\ldots ,(X_n,Y_n)\}$ , we estimate the numerator and the denominator of (1) respectively. To estimate the denominator, each sample  $(X_{i},Y_{i})$  contributes a one-hot vector  $U_{i}\in \{0,1\}^{|\mathcal{I}|}$  where the  $j$ -th element of  $U_{i}$  is  $\mathbf{1}\{X_i\in A_j\}$ . Then an estimation of  $\int_{A_j}d\mathrm{P}(x)$  is  $\frac{1}{n}\sum_{i = 1}^{n}U_i^j$ , which is the number of samples in  $A_{j}$  divided by  $n$ . Analogously, an estimation of  $\int_{A_j}f^* (x)d\mathrm{P}(x)$  is  $\frac{1}{n}\sum_{i = 1}^{n}Y_{i}\cdot U_{i}^{j}$ . Combining the pieces, a decision tree regressor is defined as
$$
f _ {\pi} (x) = \sum_ {j \in \mathcal {I}} \mathbf {1} \{x \in A _ {j} \} \frac {\sum_ {i = 1} ^ {n} Y _ {i} \cdot U _ {i} ^ {j}}{\sum_ {i = 1} ^ {n} U _ {i} ^ {j}}. \tag {2}
$$
In other words,  $f_{\pi}(x)$  estimates  $f(x)$  by the average of the responses in the cell  $A(x)$ . In the non-private setting, each data holder prepares  $U_{i}$  and  $Y_{i}$  according to the partition  $\pi$  and sends it to the curator. Then the curator aggregates the transmission following (2).
To protect the privacy of each data, we propose to estimate the numerator and denominator of the population regression tree using a privatized method. Specifically, given  $U_{i}^{j}$ , we independently sample  $\tilde{U}_{i}^{j}$  using the random response technique [57]
$$
\tilde {U} _ {i} ^ {j} = \left\{ \begin{array}{l l} U _ {i} ^ {j} - \frac {1}{1 + e ^ {\varepsilon / 4}} & \text {w i t h p r o b a b i l i t y} \frac {e ^ {\varepsilon / 4}}{1 + e ^ {\varepsilon / 4}} \\ 1 - U _ {i} ^ {j} - \frac {1}{1 + e ^ {\varepsilon / 4}} & \text {w i t h p r o b a b i l i t y} \frac {1}{1 + e ^ {\varepsilon / 4}}. \end{array} \right. \tag {3}
$$
Since  $\mathbb{E}_{\mathrm{R}}\left[\frac{1}{n}\sum_{i = 1}^{n}\tilde{U}_i^j\right] = \frac{e^{\varepsilon / 4} - 1}{e^{\varepsilon / 4} + 1}\frac{1}{n}\sum_{i = 1}^{n}U_i^j$ , we take  $\frac{e^{\varepsilon / 4} + 1}{e^{\varepsilon / 4} - 1}\frac{1}{n}\sum_{i = 1}^{n}\tilde{U}_i^j$  as the estimator of  $\int_{A_j}d\mathrm{P}(x)$ . To privatize  $Y_{1},\dots ,Y_{n}$ , we use the standard Laplace mechanism [25]. Namely, we let
$$
\tilde {Y} _ {i} = Y _ {i} + \frac {4 M}{\varepsilon} \xi_ {i} \tag {4}
$$
where  $\xi_{i}$  are i.i.d. standard Laplace random variables. Similarly,  $\frac{e^{\varepsilon/4} + 1}{e^{\varepsilon/4} - 1}\frac{1}{n}\sum_{i=1}^{n}\tilde{Y}_{i}\cdot\tilde{U}_{i}^{j}$  can be used to estimate  $\int_{A_j}f^*(x)dP(x)$ . Then using the privatized information  $(\tilde{U}_i,\tilde{Y}_i), i = 1,\dots,n$ , we define the locally differentially private decision tree regressor as
$$
f _ {\pi} ^ {\mathrm {D P}} (x) = \sum_ {j \in \mathcal {I}} \mathbf {1} \{x \in A _ {j} \} \frac {\sum_ {i = 1} ^ {n} \tilde {Y} _ {i} \cdot \tilde {U} _ {i} ^ {j}}{\sum_ {i = 1} ^ {n} \tilde {U} _ {i} ^ {j}}. \tag {5}
$$
Compared to [9, 33] which used the Laplacian mechanism to protect both  $U_{i}$  and  $Y_{i}$ , our mechanism (3) considers the fact that  $U$  is a binary vector. When  $|\mathcal{I}|$  is large, (3) can be more efficient than the Laplace mechanism which has a heavier-tailed distribution [23, 24].
2.3 Max-edge partition with variance reduction
While our privacy mechanism applies to any tree partition, it can be challenging to use general partitions such as the original CART [11] for theoretical analysis. Following the heuristic of [15], we propose a new splitting rule called the max-edge partition rule using the variance reduction criterion. This rule is amenable to theoretical analysis and can also achieve satisfactory practical performance. Given public dataset  $\{(X_{i}^{pub},Y_{i}^{pub})\}_{i = 1}^{n_q}$ , the partition rule is stated as follows:
- Let  $A_0^1 \coloneqq [0, 1]^d$  be the initial rectangular cell and  $\pi_0 \coloneqq \{A_0^j\}_{j \in \mathcal{I}_0}$  be the initialized cell partition.  $\mathcal{I}_0 = \{1\}$  stands for the initialized index set. In addition, let  $p \in \mathbb{N}$  represent the maximum depth of the tree and let  $n_l$  represent the minimum sample size in each leaf. These parameters are fixed beforehand by the user and possibly depend on  $n$ .
- Suppose we have obtained a partition  $\pi_{i - 1}$  of  $\mathcal{X}$  after  $i - 1$  steps of the recursion. Let  $\pi_i = \emptyset$ . In the  $i$ -th step, for each  $A_{i - 1}^j \in \pi_{i - 1}$ ,  $j \in \mathcal{I}_{i - 1}$ , suppose it is  $\times_{\ell = 1}^{d}[a_{\ell}, b_{\ell}]$ . We choose the edge to be split among the longest edges. The index set of longest edges is defined as
$$
\mathcal {M} _ {i - 1} ^ {j} = \left\{k \mid | b _ {k} - a _ {k} | = \max  _ {\ell = 1, \dots , d} | b _ {\ell} - a _ {\ell} |, k = 1, \dots , d \right\}.
$$
- Assume we split along the  $\ell$ -th dimension for  $\ell \in \mathcal{M}_{i-1}^j$ ,  $A_{i-1}^j$  is then partitioned into a left sub-cell  $A_{i-1}^{j,0}(\ell)$  and a right sub-cell  $A_{i-1}^{j,1}(\ell)$  along the midpoint of the chosen dimension, where  $A_{i-1}^{j,0}(\ell) = \left\{x \mid x \in A_{i-1}^j, x^\ell < \frac{a_\ell + b_\ell}{2}\right\}$  and  $A_{i-1}^{j,1}(\ell) = A_{i-1}^j / A_{i-1}^{j,0}(\ell)$ . Then the dimension to be split is chosen using the variance reduction criterion:
$$
\underset {\ell \in \mathcal {M} _ {i - 1} ^ {j}} {\arg \min } \sum_ {i = 1} ^ {n _ {q}} \left(Y _ {i} ^ {p u b} - f _ {\pi_ {i - 1} \cup A _ {i - 1} ^ {j, 0} (\ell) \cup A _ {i - 1} ^ {j, 1} (\ell) / A _ {i - 1} ^ {j}} \left(X _ {i} ^ {p u b}\right)\right) ^ {2}. \tag {6}
$$
- Once  $\ell$  is selected, We count the number of samples in the sub-cells  $\sum_{i=1}^{n} \mathbf{1}(X_i^{pub} \in A_{i-1}^{j,k}(\ell)), k = 0,1$ . If either of the cells contains fewer than  $n_l$  samples, the splitting is pruned and we let  $\pi_i = \pi_i \cup A_{i-1}^j$ . Otherwise, let  $\pi_i = \pi_i \cup \{A_{i-1}^{j,0}(\ell), A_{i-1}^{j,1}(\ell)\}$ .
The complete process is presented in Algorithm 2 in the appendix. For each grid, the partition rule selects the midpoint of the longest edges that achieves the largest variance reduction. This procedure continues until there are not enough samples contained in any leaf node, or the depth of the tree reaches its limit.
2.4 Decision Tree with local differential privacy
With these preparations, we finally present the full procedure of LPDT in Algorithm 1.
Algorithm 1: Locally differentially private decision tree (LPDT)
Input: Private data  $D = \{(X_{i},Y_{i})\}_{i = 1}^{n}$ , public data  $D^{pub} = \{(X_{i}^{pub},Y_{i}^{pub})\}_{i = 1}^{n_{q}}$
Parameters: Depth  $s$ , minimum leaf sample size  $n_l$
Curator create tree partition  $\pi$  following max-edge rule in Section 2.3 on public data  $D^{pub}$ .
Data holders of  $D$  create privatized information (3) and (4) according to  $\pi$ .
Curator aggregates the privatized information and compute  $f_{\pi}^{\mathrm{DP}}$  by (5).
Output: The LPDT estimator  $f_{\pi}^{\mathrm{DP}}$ .
In this section, we present our theoretical results and related comments. We first provide the  $\varepsilon$ -LDP guarantee of LPDT in Section 3.1. In Section 3.2, we establish the optimal convergence rate of LPDT with max-edge partition and the excess risk lower bound of LPDT without public data. Finally, we discuss the complexity of LPDT in Section 3.3.
3.1 Privacy guarantee for LPDT
Theorem 3.1. Let  $\pi = \{A_j\}_{j \in \mathcal{I}}$  be any partition of  $\mathcal{X}$  with  $\cup_{j \in \mathcal{I}} A_j = \mathcal{X}$  and  $A_i \cap A_j = \emptyset, i \neq j$ . Then the privacy mechanism  $\mathrm{R}(\tilde{U}, \tilde{Y} | X, Y)$  defined in (3) and (4) is  $\varepsilon$ -LDP. Consequently, the LPDT estimator  $f_{\pi}^{\mathrm{DP}}$  in Algorithm 1 is  $\varepsilon$ -LDP.
3.2 Convergence rate of LPDT
We first present the necessary assumptions on the distribution  $\mathrm{P}$  and  $\mathrm{Q}$ .
Assumption 3.2. Let  $\alpha \in (0,1]$ . Assume the true regression function  $f^{*}:\mathcal{X}\to \mathbb{R}$  is  $\alpha$ -Hölder continuous, i.e. there exists a constant  $c_{L} > 0$  such that for all  $x_{1},x_{2}\in \mathcal{X}$ ,  $|f^{*}(x_{1}) - f^{*}(x_{2})|\leq c_{L}\| x_{1} - x_{2}\|^{\alpha}$ . Also, assume that the density function of  $\mathrm{P}$  is bounded, i.e.  $p(x)\leq \bar{c}$  for some  $\bar{c} >0$ .
Assumption 3.3. We assume that there exists some constant  $\tau >1$  such that for all cells  $A\in \pi$  there holds  $\tau^{-1}\int_{A}d\mathrm{Q}_{X}(x)\leq \int_{A}d\mathrm{P}_{X}(x)\leq \tau \int_{A}d\mathrm{Q}_{X}(x)$
Assumption 3.2 is a standard condition widely used in non-parametric statistics. Assumption 3.3 depicts the similarity between the distribution of public data and private data. It is also a mild assumption and requires only the probabilities in each cell under  $\mathrm{P}_X$  and  $\mathrm{Q}_X$  to be similar. When  $p(x)$  and  $q(x)$  are both bounded from 0, this assumption is satisfied. Alternatively, it suffices to require that  $p(x) / q(x)$  is upper and lower bounded.
Theorem 3.4. Let  $f_{\pi}^{\mathrm{DP}}$  be the LPDT estimator in Algorithm 1. Suppose Assumption 3.2 and 3.3 hold. Then, for  $n_q \gtrsim n^{\frac{d}{2\alpha + 2d}}$ , if we set  $s \asymp \log n\varepsilon^2$  and  $n_l \asymp n_q / 2^s$ , there holds
$$
\mathcal {R} _ {L, \mathrm {P}} (f _ {\pi} ^ {\mathrm {D P}}) - \mathcal {R} _ {L, \mathrm {P}} ^ {*} \lesssim \left(\frac {\log n}{n \varepsilon^ {2}}\right) ^ {\frac {\alpha}{\alpha + d} \wedge \frac {1}{3}}
$$
with probability  $1 - 2 / n_q^2 -5 / n^2$  with respect to  $\mathrm{P}^n\otimes \mathrm{Q}^{n_q}\otimes \mathrm{R}^n$  where  $\mathbb{R}^n$  is the joint distribution of privacy mechanisms in (3) and (4).
Note that we only require  $n_q \gtrsim n^{\frac{d}{2\alpha + 2d}}$ , which means the sample size of public data can be much smaller than private data. As illustrated in [33], the minimax convergence rate over Hölder function space is  $(n(e^{\varepsilon} - 1)^{2})^{-\frac{\alpha}{\alpha + d}}$ , indicating that LPDT attains optimal rate when  $\alpha / (\alpha + d) \leq 1/3$ . In the case  $\alpha / (\alpha + d) > 1/3$ , or equivalently  $2\alpha > d$ , LPDT achieves fast yet sub-optimal convergence rate  $n^{-\frac{1}{3}}$ . Note that  $2\alpha > d$  only when  $d = 1$  and  $\alpha \geq 1/2$ , which rarely occurs. The next statement shows that LPDT fails without public data.
Theorem 3.5. Let  $f_{\pi}^{\mathrm{DP}}$  be the LPDT estimator in Algorithm 1 and  $\mathcal{P}$  be the class of distributions satisfying Assumption 3.2. For  $n_q = 0$  i.e. there is no public data, for any choice of  $s$ ,  $n_l$ , and  $\varepsilon$ , there holds
$$
\sup  _ {\mathrm {P} \in \mathcal {P}} \left(\mathbb {E} \left[ \mathcal {R} _ {L, \mathrm {P}} \left(f _ {\pi} ^ {\mathrm {D P}}\right) \right] - \mathcal {R} _ {L, \mathrm {P}} ^ {*}\right) \gtrsim 1.
$$
Under the same hypothesis function space, the supremum of excess risk of LPDT does not even converge without public data. Together with Theorem 3.4, this shows that the prior information contained in public data can greatly enhance the quality of the private estimation.
We compare our results with those of others. LPDT converges faster than deconvolution-based method [29] whose rate is  $n^{-\frac{2\alpha}{2\alpha + 5d}}$  [28]. As for histogram-based methods, [9] achieves the optimal rate only when the density function is lower bounded, which is a strong condition. To avoid the condition, [33] derived an ad hoc estimator by adding a regularization to the marginal density estimation. LPDT takes another approach to avoid the condition. It does not apply any regularization or truncation on the estimator in each cell. Instead, as long as Assumption 3.3 holds, the low-density regions can be identified and treated with larger cells automatically by the parameter  $n_l$ . As a sacrifice, the large cells restrict the approximation ability of LPDT and the convergence rate is no more than  $n^{-1/3}$ . In addition, our theoretical results hold in the sense of "with high probability", which is more closely related to practical needs than "in expectation" as addressed in [9, 33].