Title: Decision Tree for Locally Private Estimation with Public Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Local differential privacy

non-parametric regression

decision tree

public data
Abstract: We propose conducting locally differentially private (LDP) estimation with the aid of a small amount of public data to enhance the performance of private estimation. Specifically, we introduce an efficient algorithm called Locally differentially Private Decision Tree (LPDT) for LDP regression. We first use the public data to grow a decision tree partition and then fit an estimator according to the partition privately.  From a theoretical perspective, we show that LPDT is $\varepsilon$-LDP and has a mini-max optimal convergence rate under a mild assumption of similarity between public and private data, whereas the lower bound of the convergence rate of LPDT without public data is strictly slower, which implies that the public data helps to improve the convergence rates of LDP estimation. We conduct experiments on both synthetic and real-world data to demonstrate the superior performance of LPDT compared with other state-of-the-art LDP regression methods. Moreover, we show that LPDT remains effective despite considerable disparities between public and private data.

Decision Tree for Locally Private Estimation with Public Data

Abstract

We propose conducting locally differentially private (LDP) estimation with the aid of a small amount of public data to enhance the performance of private estimation. Specifically, we introduce an efficient algorithm called Locally differentially Private Decision Tree (LPDT) for LDP regression. We first use the public data to grow a decision tree partition and then fit an estimator according to the partition privately. From a theoretical perspective, we show that LPDT is ε -LDP and has a minimax optimal convergence rate under a mild assumption of similarity between public and private data, whereas the lower bound of the convergence rate of LPDT without public data is strictly slower, which implies that the public data helps to improve the convergence rates of LDP estimation. We conduct experiments on both synthetic and real-world data to demonstrate the superior performance of LPDT compared with other state-of-the-art LDP regression methods. Moreover, we show that LPDT remains effective despite considerable disparities between public and private data.

1 Introduction

Differential privacy (DP) [25] is a widely-used technique to protect sensitive information, like in medical trials [20], recommendation systems [49], and census data sharing [3]. Local differential privacy (LDP) [38, 24], a variation of DP, has gained particular attention, especially among industry experts [27, 35]. Unlike DP, LDP assumes data is privatized before being sent to a central collector. However, LDP models need more data to be accurate compared to DP [24], and many common techniques in data analysis such as standardization [10] and tree partition [59] are harder with LDP. This brings challenges to tasks such as density density estimation [23], mean estimation [24], and Gaussian estimation [36].

Fortunately, in some scenarios, private estimation performance can be enhanced with an additional public dataset [4, 7]. The public dataset can be either in-distribution, consisting of data from users who agree to share their personal information [6], or out-of-distribution, such as data from another source [34]. From a central DP perspective, an increasing amount of research has focused on leveraging public data to facilitate private learning, where public data mainly serves two purposes. On one hand, the knowledge learned from public data is implicitly transferred into the private model. Empirical investigations have demonstrated the effectiveness of pretraining on public data and fine-tuning privately on sensitive data [63, 43, 41, 61]. By gradient pre-conditioning with a subspace computed by public data, [65, 62, 37] managed to reduce the required amount of noise in differentially private gradient descent and accelerate its convergence. Through unlabeled public data, [45, 46] fed knowledge privately into student models. On the other hand, on public data, we can

conduct procedures that would be infeasible without access to the raw private data. For example, [10] used parameters computed by public data to standardize private data, which can augment the sample complexity of private mean estimation. Recently, [56] employed unlabeled public data to estimate the leading eigenvalue of the covariance matrix, resulting in an improved sample complexity for generalized linear models with non-interactive local differential privacy.

The paper focuses on the problem of non-parametric regression with LDP. While regression has been extensively studied in the central setting [50, 1, 14], the LDP case remains rarely explored. A notable reason is that most gradient-based methods [50, 1] are prohibited. In order to protect privacy, each data holder needs to compute the gradient of parameters locally, which requires a large amount of memory, computing power, and communication capacity on the terminal machine [52]. [29] proposed to impose Laplace noise on the data directly to provide privacy. However, this method is known to converge slowly [28] and suffer from the curse of dimensionality. More recently, [9, 33] investigated histogram-based approaches. Though theoretically optimal, histograms may perform poorly in practice, especially when the dimension of feature space is large. Thus, both methods proposed in [9, 33] face challenges when applied to real-world problems.

Under such background, using the idea of borrowing public data information, we propose an LDP non-parametric regression algorithm called the Locally differentially Private Decision Tree (LPDT) that achieves both optimal convergence rate and superior empirical performance. We first create a tree partition on the public dataset using the proposed max-edge rule. According to the partition, each data holder encodes the private data and releases the encoding which is processed using the proposed privacy mechanism. Finally, the curator aggregates the information in each partition cell and outputs a decision tree estimator. LPDT is advantageous from at least two perspectives: (i) LPDT integrates both benefits to leverage public data. It enables adaptive partitioning procedures which can eliminate some redundant cells and can transfer information from public data to private estimation through the tree partition. (ii) It inherits the merit of the decision tree model, such as interpretability, efficiency, stability, extensiveness to multiple feature types, and resistance to the curse of dimensionality.

We summarize our contributions. (i) For the first time, we propose to use public data in locally differentially private non-parametric regression. (ii) We propose a novel LDP regression algorithm called the locally differentially private decision tree that achieves theoretical optimality while maintaining satisfying practical performance. (iii) Under mild assumptions on the similarity between the distribution of public and private data, we establish the optimal convergence rate of LPDT, whereas the supremum of excess risk of LPDT without public fails to converge to zero. This demonstrates the theoretical advantage of incorporating public data. (iv) In experiments, we compare LPDT with other existing non-parametric LDP regression methods using both synthetic and real-world datasets. Our results demonstrate the overwhelming performance of LPDT, which illustrates the empirical improvement brought by public data. Moreover, we show that LPDT performs well even in the presence of significant disparities between public and private data.

2 Methodology

This section is dedicated to the methodology of LPDT. In Section 2.1, we first present notations and preliminaries related to regression problems, followed by a recap of the definition of local differential privacy. Next, we introduce our hybrid privacy mechanism for general partition-based estimation in Section 2.2. In Section 2.3, we propose our partition rule. Finally, in Section 2.4, we provide a comprehensive description of LPDT.

2.1 Preliminaries

Notations For any vector x , let x i denote the i -th element of x . Recall that for 1 ≤ p < ∞ , the L p -norm of x = ( x 1 , . . . , x d ) is defined by ∥ x ∥ p := ( | x 1 | p + · · · + | x d | p ) 1 /p . Throughout this paper, we use the notation a n ≲ b n and a n ≳ b n to denote that there exist positive constant c and c ′ such that a n ≤ cb n and a n ≥ c ′ b n , for all n ∈ N . In addition, we denote a n ≍ b n if a n ≲ b n and b n ≲ a n . Let a ∨ b = max( a, b ) and a ∧ b = min( a, b ) . Besides, for any set A ⊂ R d , the diameter of A is defined by diam( A ) := sup x,x ′ ∈ A ∥ x -x ′ ∥ 2 . Let the standard Laplace random variable have the continuous probability density function p ( x ) = 1 2 e -| x | for x ∈ R .

Regression is to predict the value of an unobserved output variable Y based on the observed input variable X , based on a dataset D := { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } consisting of n i.i.d. observations drawn from an unknown probability measure P on X × Y = [0 , 1] d × [ -M,M ] . The density function of P is denoted as s . In addition, we have a public dataset D pub := { ( X pub 1 , Y pub 1 ) , . . . , ( X pub n q , Y pub n q ) } drawn from distribution Q on X × Y with sample size n q . Its density function is denoted as q .

It is legitimate to consider the least square loss L : X ×Y ×Y → [0 , ∞ ) defined by L ( x, y, f ( x )) := ( y -f ( x )) 2 for our target of regression. Then, for a measurable decision function f : X → Y , the risk is defined by R L, P ( f ) := ∫ X×Y L ( x, y, f ( x )) d P( x, y ) . The Bayes risk, which is the smallest possible risk with respect to P and L , is given by R ∗ L, P := inf {R L, P ( f ) | f : X → Y measurable } . The function that achieves the Bayes risk is called Bayes function, namely, f ∗ ( x ) := E ( Y | X = x ) .

Definition 2.1 ( Local Differential Privacy ) . Given data { ( X i , Y i ) } n i =1 , each ( X i , Y i ) is mapped to a piece of privatized information s i which is a random variable on S . Let σ ( S ) be the σ -field on S . s i is drawn conditional on ( X i , Y i ) via the distribution R( S | X i = x, Y i = y ) for S ∈ σ ( S ) . Then the mechanism R provides ε -local differential privacy ( ε -LDP) if

This formulation of local privacy is widely adopted [23, 9]. In contrast to central DP where the likelihood ratio is taken with respect to some statistics of all data, LDP requires individuals to guarantee their own privacy by considering the likelihood ratio with respect to each ( X i , Y i ) . Once the view s is provided, no further processing can reduce the deniability about taking a value ( x, y ) since any outcome s is nearly as likely to have come from some other initial value ( x ′ , y ′ ) .

2.2 Privacy mechanism for tree partition

This section focuses on the hybrid privacy mechanism for general tree partitions. We first introduce the standard regression tree and then present our privacy mechanism based on the random response and Laplacian mechanism.

̸

For index set I , let π = { A j } j ∈I be any tree partition of X with ∪ j ∈I A j = X and A i ∩ A j = ∅ , i = j . For any x ∈ X , let the cell containing x be A ( x ) . A population decision tree regressor with partition π is defined as

Here, we let 0 / 0 = 0 by definition. To get a empirical estimator given the data set D = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } , we estimate the numerator and the denominator of (1) respectively. To estimate the denominator, each sample ( X i , Y i ) contributes a one-hot vector U i ∈ { 0 , 1 } |I| where the j -th element of U i is 1 { X i ∈ A j } . Then an estimation of ∫ A j d P( x ) is 1 n ∑ n i =1 U j i , which is the number of samples in A j divided by n . Analogously, an estimation of ∫ A j f ∗ ( x ) d P( x ) is 1 n ∑ n i =1 Y i · U j i . Combining the pieces, a decision tree regressor is defined as

In other words, f π ( x ) estimates f ( x ) by the average of the responses in the cell A ( x ) . In the nonprivate setting, each data holder prepares U i and Y i according to the partition π and sends it to the curator. Then the curator aggregates the transmission following (2).

To protect the privacy of each data, we propose to estimate the numerator and denominator of the population regression tree using a privatized method. Specifically, given U j i , we independently sample ˜ U j i using the random response technique [57]

Since E R [ 1 n ∑ n i =1 ˜ U j i ] = e ε/ 4 -1 e ε/ 4 +1 1 n ∑ n i =1 U j i , we take e ε/ 4 +1 e ε/ 4 -1 1 n ∑ n i =1 ˜ U j i as the estimator of ∫ A j d P( x ) . To privatize Y 1 , · · · , Y n , we use the standard Laplace mechanism [25]. Namely, we let

where ξ i are i.i.d. standard Laplace random variables. Similarly, e ε/ 4 +1 e ε/ 4 -1 1 n ∑ n i =1 ˜ Y i · ˜ U j i can be used to estimate ∫ A j f ∗ ( x ) d P( x ) . Then using the privatized information ( ˜ U i , ˜ Y i ) , i = 1 , · · · , n , we define the locally differentially private decision tree regressor as

Compared to [9, 33] which used the Laplacian mechanism to protect both U i and Y i , our mechanism (3) considers the fact that U is a binary vector. When |I| is large, (3) can be more efficient than the Laplace mechanism which has a heavier-tailed distribution [23, 24].

2.3 Max-edge partition with variance reduction

While our privacy mechanism applies to any tree partition, it can be challenging to use general partitions such as the original CART [11] for theoretical analysis. Following the heuristic of [15], we propose a new splitting rule called the max-edge partition rule using the variance reduction criterion. This rule is amenable to theoretical analysis and can also achieve satisfactory practical performance. Given public dataset { ( X pub i , Y pub i ) } n q i =1 , the partition rule is stated as follows:

Let A 1 0 := [0 , 1] d be the initial rectangular cell and π 0 := { A j 0 } j ∈I 0 be the initialized cell partition. I 0 = { 1 } stands for the initialized index set. In addition, let p ∈ N represent the maximum depth of the tree and let n l represent the minimum sample size in each leaf. These parameters are fixed beforehand by the user and possibly depend on n .

Suppose we have obtained a partition π i -1 of X after i -1 steps of the recursion. Let π i = ∅ . In the i -th step, for each A j i -1 ∈ π i -1 , j ∈ I i -1 , suppose it is × d ℓ =1 [ a ℓ , b ℓ ] . We choose the edge to be split among the longest edges. The index set of longest edges is defined as

Assume we split along the ℓ -th dimension for ℓ ∈ M j i -1 , A j i -1 is then partitioned into a left sub-cell A j, 0 i -1 ( ℓ ) and a right sub-cell A j, 1 i -1 ( ℓ ) along the midpoint of the chosen dimension, where A j, 0 i -1 ( ℓ ) = { x | x ∈ A j i -1 , x ℓ < a ℓ + b ℓ 2 } and A j, 1 i -1 ( ℓ ) = A j i -1 /A j, 0 i -1 ( ℓ ) . Then the dimension to be split is chosen using the variance reduction criterion:

Once ℓ is selected, We count the number of samples in the sub-cells ∑ n i =1 1 ( X pub i ∈ A j,k i -1 ( ℓ )) , k = 0 , 1 . If either of the cells contains fewer than n l samples, the splitting is pruned and we let π i = π i ∪ A j i -1 . Otherwise, let π i = π i ∪ { A j, 0 i -1 ( ℓ ) , A j, 1 i -1 ( ℓ )) } .

The complete process is presented in Algorithm 2 in the appendix. For each grid, the partition rule selects the midpoint of the longest edges that achieves the largest variance reduction. This procedure continues until there are not enough samples contained in any leaf node, or the depth of the tree reaches its limit.

2.4 Decision Tree with local differential privacy

With these preparations, we finally present the full procedure of LPDT in Algorithm 1.

Algorithm 1: Locally differentially private decision tree (LPDT)

Input: Private data D = { ( X i , Y i ) } n i =1 , public data D pub = { ( X pub i , Y pub i ) } n q i =1 Parameters: Depth s , minimum leaf sample size n l .

Curator create tree partition π following max-edge rule in Section 2.3 on public data D pub .

Data holders of D create privatized information (3) and (4) according to π .

Curator aggregates the privatized information and compute f DP π by (5).

Output:

The LPDT estimator f .

DP π

3 Theoretical results

In this section, we present our theoretical results and related comments. We first provide the ε -LDP guarantee of LPDT in Section 3.1. In Section 3.2, we establish the optimal convergence rate of LPDT with max-edge partition and the excess risk lower bound of LPDT without public data. Finally, we discuss the complexity of LPDT in Section 3.3.

3.1 Privacy guarantee for LPDT

Theorem 3.1. Let π = { A j } j ∈I be any partition of X with ∪ j ∈I A j = X and A i ∩ A j = ∅ , i = j . Then the privacy mechanism R( ˜ U, ˜ Y | X,Y ) defined in (3) and (4) is ε -LDP. Consequently, the LPDT estimator f DP π in Algorithm 1 is ε -LDP.

3.2 Convergence rate of LPDT

We first present the necessary assumptions on the distribution P and Q .

Assumption 3.2. Let α ∈ (0 , 1] . Assume the true regression function f ∗ : X → R is α -Hölder continuous, i.e. there exists a constant c L > 0 such that for all x 1 , x 2 ∈ X , | f ∗ ( x 1 ) -f ∗ ( x 2 ) | ≤ c L ∥ x 1 -x 2 ∥ α . Also, assume that the density function of P is bounded, i.e. p ( x ) ≤ c for some c > 0 .

Assumption 3.3. We assume that there exists some constant τ > 1 such that for all cells A ∈ π , there holds τ -1 ∫ A d Q X ( x ) ≤ ∫ A d P X ( x ) ≤ τ ∫ A d Q X ( x ) .

Assumption 3.2 is a standard condition widely used in non-parametric statistics. Assumption 3.3 depicts the similarity between the distribution of public data and private data. It is also a mild assumption and requires only the probabilities in each cell under P X and Q X to be similar. When p ( x ) and q ( x ) are both bounded from 0, this assumption is satisfied. Alternatively, it suffices to require that p ( x ) /q ( x ) is upper and lower bounded.

Theorem 3.4. Let f DP π be the LPDT estimator in Algorithm 1. Suppose Assumption 3.2 and 3.3 hold. Then, for n q ≳ n d 2 α +2 d , if we set s ≍ log nε 2 and n l ≍ n q / 2 s , there holds

with probability 1 -2 /n 2 q -5 /n 2 with respect to P n ⊗ Q n q ⊗ R n where R n is the joint distribution of privacy mechanisms in (3) and (4) .

Note that we only require n q ≳ n d 2 α +2 d , which means the sample size of public data can be much smaller than private data. As illustrated in [33], the minimax convergence rate over Hölder function space is ( n ( e ε -1) 2 ) -α α + d , indicating that LPDT attains optimal rate when α/ ( α + d ) ≤ 1 / 3 . In the case α/ ( α + d ) > 1 / 3 , or equivalently 2 α > d , LPDT achieves fast yet sub-optimal convergence rate n -1 3 . Note that 2 α > d only when d = 1 and α ≥ 1 / 2 , which rarely occurs. The next statement shows that LPDT fails without public data.

Theorem 3.5. Let f DP π be the LPDT estimator in Algorithm 1 and P be the class of distributions satisfying Assumption 3.2. For n q = 0 i.e. there is no public data, for any choice of s , n l , and ε , there holds

̸

Under the same hypothesis function space, the supremum of excess risk of LPDT does not even converge without public data. Together with Theorem 3.4, this shows that the prior information contained in public data can greatly enhance the quality of the private estimation.

Wecompare our results with those of others. LPDT converges faster than deconvolution-based method [29] whose rate is n -2 α 2 α +5 d [28]. As for histogram-based methods, [9] achieves the optimal rate only when the density function is lower bounded, which is a strong condition. To avoid the condition, [33] derived an ad hoc estimator by adding a regularization to the marginal density estimation. LPDT takes another approach to avoid the condition. It does not apply any regularization or truncation on the estimator in each cell. Instead, as long as Assumption 3.3 holds, the low-density regions can be identified and treated with larger cells automatically by the parameter n l . As a sacrifice, the large cells restrict the approximation ability of LPDT and the convergence rate is no more than n -1 / 3 . In addition, our theoretical results hold in the sense of "with high probability", which is more closely related to practical needs than "in expectation" as addressed in [9, 33].

Besides these advantages, we also discuss the benefit of public data for removing the range parameter [10, 9]. Consider the example from [33], where the convergence rate is given by ( r 2 d n nε 2 ) α α + d . When the set X = × d j =1 [ a j , b j ] is unknown, it becomes necessary to create a histogram partition over × d j =1 [ -r n , r n ] , introducing an additional factor of r 2 d n . However, with publicly available data, we can approximate the range of the j -th dimension using ̂ a j = min i X j i and ̂ b j = max i X j i . Subsequently, we can perform min-max scaling on each data point from × d j =1 [ ̂ a j , ̂ b j ] to map it into × d j =1 [0 , 1] , and then train an LPDT on × d j =1 [0 , 1] . Any x that falls outside the range × d j =1 [0 , 1] is predicted as 0 . We demonstrate that by this approach, Theorem 3.4 holds with a probability of at least 1 -d/n 2 q . See derivations in Section C.6 in the appendix.

3.3 Complexity analysis

We demonstrate that LPDT is an efficient method. We first consider the average computation complexity of LPDT. The training stage consists of two parts. The partition procedure takes O ( sn q d ) time and the computation of (5) takes O ( sn ) time. From the proof of Theorem 3.4, we know that 2 s ≍ ( nε 2 / log n ) -d 2 α +2 d . Thus the training stage complexity is around O ( n log nε 2 + n q d log nε 2 ) . Since each prediction of the decision tree takes O ( s ) time, the test time for each test instance is around O (log nε 2 ) . As for storage complexity, since LPDT only requires the storage of the tree structure and the prediction value at each node, the space complexity of LPDT is O ( ( nε 2 / log n ) -d 2 α +2 d ) . In short, LPDT is an efficient method with a small number of parameters.

We compare the complexities of LPDT with other LDP regression methods in Table 1. Notably, [29] is inefficient due to its unacceptable test and space complexity. Also, the dominant term of training complexity of [9] is O ( nd log nε 2 ) . When d is large, we can choose a small n q such that LPDT yields a strictly lower complexity than [9]. In addition, although [9] enjoys the same order of space complexity as LPDT, the memory of histogram-based methods suffers from the curse of dimensionality in practice. Since the storage of O ( h -d ) values is required, even h = 1 / 2 requires allocating an array of size 2 d , which is problematic for large d . In contrast, LPDT can resist the curse of dimensionality by only splitting along the relevant features and keeping a small number of nodes.

4 Experiments

In the experiments, we first validate our theoretical findings with synthetic data in Section 4.1. Then, in Section 4.2, we show the superior performance of LPDT on real-world datasets with identically distributed public data. In Section 4.3, we apply LPDT to Chicago taxi data to show that LPDT

performs well even with considerable differences between private and public data. Also, we analyze the influence of the distribution shift between private and public data on LPDT.

Splitting rule Note that most tree methods design their partition rules based on the information gained from the data. To boost the performance of LPDT, we also incorporate the variance reduction scheme from the original CART [11] to the tree construction. We denote the LPDT estimator using the max-edge partition rule with variance reduction in Section 2.3 as LPDT-M and denote the estimator using the standard variance reduction rule in [11] as LPDT-V. Since Theorem 3.1 holds for any partition, LPDT-V is also ε -LDP.

Experiment setup We choose the privacy budget ε ∈ [0 . 5 , 8] , covering commonly seen magnitudes of privacy budgets from low to high privacy regimes. We compare LPDT-M and LPDT-V with the following methods: ( i ) Private Histogram (PHIST) [9]. ( ii ) Adjusted Private Histogram (APHIST) [33]. ( iii ) Deconvolution Kernel (DECONV) [29]. Introduction to the methods and all implementation details are presented in Appendix D.1. We employ 5-fold cross-validation for parameter selection, and techniques for tuning parameters under LDP are discussed in Section D.2. The evaluation metric is the mean squared error (MSE).

4.1 Synthetic experiments

Necessity of public data To demonstrate intuitively why public data is essential for LPDT, we first visualize its estimation on a synthetic model, Y = sin(16 X ) + ϵ where X ∼ N (0 . 5 , 0 . 025) and ε ∼ N (0 , 1) . In this case, the marginal distribution is highly imbalanced with the majority of samples located in the middle part of [0 , 1] and a few samples on the sides. For ϵ = 8 , we fit two LPDT models: one with 500 public data and 7,000 private data, and another with 8,000 private data.

As shown in Figure 1(a), without public data, LPDT struggles with the imbalanced marginal. The grids on the side produce unstable predictions since only a few samples fall into them. As a result, LPDT tends to decrease depth s to stabilize its estimation. This leads to underfitting in the middle so that the predicted curve fails to capture the variation of the ground truth. In contrast, with the aid of public data, LPDT solves the issue as shown in Figure 1(b). For the mid-

0.0

0.2

0.4

0.6

0.8

1.0

x

-2.5

-1.0

0.5

2.0

3.5

y

Samples

f

(

x

)

LPDT

(a) Without public data

0.0

0.2

0.4

0.6

0.8

1.0

x

-2.5

-1.0

0.5

2.0

3.5

y

Samples

f

(

x

)

LPDT

(b) With public data

dle zone where samples are abundant, LPDT creates small grids to enlarge approximation capacity. Meanwhile, it prunes the grids on the sides to ensure stability. Even with fewer private data, the MSE of LPDT is reduced from 1.19 to 1.08 thanks to the additional public data. In summary, the experiment provides empirical evidence supporting the theoretical findings in Theorems 3.4 and 3.5, which highlight the necessity of public data for the effective performance of LPDT.

Parameter analysis of depth s We conduct experiments to investigate the selection of partition depth s in terms of MSE. We generate 6,000 training samples, 2,000 test samples, and 2,000 public samples following the synthetic model described above. We pick ε ∈ { 3 , 4 , 6 , 8 } and s ∈ { 2 , 3 , 4 , 5 , 6 } . For each pair of s and ε , we plot the 20 times averaged MSE versus s . The result is displayed in Figure 2(a). Apparently, for each ε , as s increases, MSE first decreases until s reaches a certain value. Then MSE begins to increase as s grows. This further confirms the trade-off observed in Theorem 3.4. Moreover, the depth s at which the test error is minimized increases as ε increases. This is compatible with theory since the optimal choice of s ≍ log nε 2 is monotonically increasing with respect to ε .

Parameter analysis of minimum leaf sample size n l We conduct experiments to investigate the choice of n l in terms of MSE. Following the same generating scheme, we choose ε ∈ { 3 , 4 , 6 , 8 } and plot MSE of LPDT versus n l for n l ∈ { 15 , 25 , · · · , 135 } . In Figure 2(b), the relation between MSE and n l is U-shaped under each ε , which indicates that a properly chosen n l is necessary as stated in

2

3

4

5

6

s

1

2

3

4

5

6

MSE

=3

=4

=6

=8

20

40

60

80

100

120

140

nl

1.0

1.5

2.0

2.5

3.0

MSE

=3

=4

=6

=8

(b)

n

l

- MSE

0

1

2

3

4

5

6

7

8

9

10

1.1

1.3

1.5

1.7

1.9

2.1

MSE

LPDT

DECONV

PHIST

APHIST

(c)

ε

- MSE

Theorem 3.4. Furthermore, LPDT achieves the best MSE for n l ∈ [35 , 75] when ε = 4 , 6 , 8 , while the minimum MSE occurred at n l = 125 when ε = 3 . This finding is compatible with Theorem 3.4 which states that the optimal choice of n l is monotonically decreasing with respect to ε .

Our parameter analyses indicate that decreasing ε favors smaller values of s and larger values of n l . These choices lead to a decision tree partition with fewer grids. In summary, when facing higher levels of privacy demand, LPDT cuts down the number of grids to stabilize its estimation.

Privacy utility trade-off We analyze how privacy budget ε influences the quality of prediction. Under the same setup, we evaluate LPDT and other methods for ε ∈ { 0 . 5 , 1 , 2 , 3 , · · · , 10 } with 50 repetitions. The results are displayed in Figure 2(c). To show significance, we plot the (0 . 1 , 0 . 9) quantiles as confidence intervals. When ε increases, the MSE of LPDT decreases much faster than the other methods. Note that the MSE of both PHIST and APHIST remains high, suggesting that their performances are limited by the histogram instead of the privacy mechanism.

4.2 Real data comparison with identically distributed public data

Experiment setup We conduct experiments on 12 real datasets, each repeated 50 times with a ratio of 1:7:2 for public data, training data, and testing data in each trial. The dataset details and pre-processing steps are summarized in Appendix D.3. To ensure significance, we adopt the Wilcoxon signed-rank test [58] with a significance level of 0.05 to check if a result is significantly better. For better comparison, we also train a decision tree (denoted as DT) on the original training data with no privacy protection, whose result will serve as a lower bound.

Table 2: Average MSE over real data sets for LDP regression methods. The best results are bolded and the second best results are underlined. The marked results with significance towards the rest results are marked with ∗ . Due to memory limitation, PHIST and APHIST are corrupted on two datasets which are marked with -.

Performance of accuracy and running time The representative results for ε = 2 , 6 are displayed in Table 2. Results of ε = 0 . 5 , 1 , 4 , 8 is in Appendix D.4. It can be seen that LPDT-M and LPDT-V both significantly outperform their competitors. All methods achieve a higher MSE than DT, while the results for LPDT-M and LPDT-V are reasonably close to DT. Due to memory limitations, PHIST and APHIST fail on two datasets. We also compare the total running time in Table 6 in Appendix D.4. In general, both LPDT-M and LPDT-V achieve less running time than PHIST and APHIST, and are significantly faster than DECONV.

(a)

s

- MSE

With the private and public distributions being identical, another alternative is to use solely public data to fit a decision tree. We train such decision trees on public data ( n q = 0 . 1 n ). It is compared with the decision tree trained on private data and LPDT with ε = 6 in Table 7 in Appendix. The results show that when n q ≪ n and ε is large, training a decision tree on public data is worse than using LPDT on most of the datasets.

The power of public data We conduct experiments to show the utility gain brought by public data. We take red wine dataset as an example. With n q = 100 public data, we run LPDT (with D P ) and PHIST (without D P ). For δ ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } ,

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.6

0.7

0.8

0.9

1.0

1.1

MSE

With

D

P

Without

D

P

we train each model on δ · n samples with 20 repetitions. The result is in Figure 3. PHIST with 1,100 samples achieves the same MSE as LPDT with 660 samples. The result shows that, with a small amount of public data, LPDT achieves the same performance with much fewer samples.

4.3 Real data comparison with non-identically distributed public data

We apply LPDT to the Chicago Taxi Dataset , a collection of taxi trips in Chicago provided by the Differential Privacy Temporal Map Challenge and contains sensitive information [21]. We use the fare of each trip as labels and other information as features. Then we regard trips paid by PR card and credit card as public data and private data, respectively. In Appendix D.5, we show the two parts are distributed differently. After preprocessing, the dataset has 101 features with 2,150,565 samples in private data and 24,436 samples in public data.

Performance We report the averaged MSE of LPDT over 20 repetitions for ε = 0 . 5 , 1 , 2 , 4 , 6 , and 8 . As a comparison to LPDT, we train two decision trees separately using the public and private datasets with no privacy protection. As for the comparison methods, DECONV fails due to the large sample size while PHIST and APHIST fail due to the dimensionality. However, after reducing the dimensionality by retaining only the continuous features, we are able to apply PHIST and APHIST. The results are displayed in Table 3. A first observation is that the decision tree trained on public data yields considerably worse results than the decision tree trained on private data. This suggests that relying solely on public data leads to biased predictions. Learning solely from public data achieves an MSE higher than LPDT for all three values of ε . Even for ε = 2 , LPDT significantly outperforms histogram-based methods with ε = 8 . This indicates that LPDT remains effective even when substantial disparities exist between the distributions of public and private data.

How does non-identically distributed public data help? It is counterintuitive that non-identically distributed public data can benefit private estimation. We show the logic behind using such public data by investigating the first split feature in the tree partition. LPDT identifies whether a trip ends in Zone 32 as an important feature for predicting fare and initiates the recursive partitioning process by splitting along this feature. Figure 4 illustrates that trips ending in this zone generally have lower fares for both public and private data, although the actual fares differ significantly between the two datasets. This observation demonstrates that despite having different distributions, public and private data may exhibit similar patterns, thereby allowing the partition created on public data to still be effective on private data. Following this line of reasoning, to determine whether public data is suitable for a specific private task, we can examine whether the qualitative relationships between labels and features remain consistent across both datasets. Whether a dataset can be utilized as public data can be of independent interest as in [32].

Analysis of distribution shift In Table 3, the MSE of LPDT reduces as ε increases but remains higher than the MSE achieved by training a decision tree directly on private data. The performance

public

private

10

15

20

25

fare

out

in

0.1

0.3

0.5

0.7

0.9

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

MSE

= 3

= 4

= 6

= 8

(a) Wine data

0.1

0.3

0.5

0.7

0.9

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

MSE

(b) Chicago taxi data

gap can be attributed to the distribution shift between private data and public data. In the following, we investigate how the difference between the two datasets affects the performance of LPDT. Besides the Chicago taxi data, we also adopt White Wine and Red Wine data in Section 4.2 as private and public data, respectively. The datasets contain the same variables but are distributed differently, as is investigated in transfer learning literature [48]. On both datasets, we combine part of the private samples with public data and perform the partition procedure on the combined dataset, with the portion of public data denoted as δ . When δ is small, there is less difference between data used for partition and training. We report the average MSE of LPDT for δ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } after 20 repetitions. Figure 5 shows that a small δ leads to a lower MSE for both datasets and all values of ε . Thus, LPDT is more powerful when the public and private data are distributed similarly, which also justifies the necessity of Assumption 3.3.

5 Conclusion

This paper addresses the challenge of effectively performing LDP regression given both public data and private data by introducing the locally private decision tree. Due to the novel idea of leveraging public data, LPDT is accurate, efficient, and interpretable. Theoretically, we establish the privacy guarantee and optimal convergence rate of LPDT. In experiments, we show the superior performance of LPDT regardless of the disparities between private and public data.

6 Limitations and broader impact

LPDT addresses the challenge of effectively performing LDP regression given both public data and private data. Future work may explore methodologies for incorporating public data in private learning using other models, such as linear models, and investigate the theoretical advantages of using public data in private learning. One limitation of this paper is that the measure of similarity between the public and private distribution considers only the marginal distribution. One may consider the similarities with respect to the regression function, potentially following transfer learning literature [13]. Moreover, in practice, public data may have a different distribution than private data, which is damaging to the learning process. The drawback yields the significance of public data selection or public data quality test such as [32, 60]. In addition, the theoretical analysis deal with n q ≳ n d 2 α +2 d and n q = 0 , while the intermediate zone for n q remains unexplored.

Acknowledgment

The authors would like to thank the reviewers for their constructive comments, which led to a significant improvement in this work. The project is supported by the Special Funds of the National Natural Science Foundation of China (Grant No. 72342010) and was supported in part by the National Key R&D Program of China (2022YFB2702401). This research was supported by Public Computing Cloud, Renmin University of China.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC

conference on computer and communications security , pages 308-318, 2016.

Faroudja Abid and Nouma Izeboudjen. Predicting forest fire in algeria using data mining techniques: Case study of the decision tree algorithm. In International Conference on Advanced Intelligent Systems for Sustainable Development , pages 363-370. Springer, 2020.

John M Abowd. The us census bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2867-2867, 2018.

Noga Alon, Raef Bassily, and Shay Moran. Limits of private learning with access to public data. Advances in neural information processing systems , 32, 2019.

MAltosole, G Benvenuto, M Figari, and U Campora. Real-time simulation of a cogag naval ship propulsion system. Proceedings of the institution of mechanical engineers, part M: journal of engineering for the maritime environment , 223(1):47-62, 2009.

Brendan Avent, Yatharth Dubey, and Aleksandra Korolova. The power of the hybrid model for mean estimation. Proceedings on Privacy Enhancing Technologies , (4):48-68, 2020.

Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan Ullman, and Steven Wu. Private query release assisted by public data. In International Conference on Machine Learning , pages 695-703. PMLR, 2020.

Sergei N. Bernstein. The Theory of Probabilities . Gastehizdat Publishing House, Moscow, 1946.

Thomas B Berrett, László Györfi, and Harro Walk. Strongly universally consistent nonparametric regression and classification with privatised data. Electronic Journal of Statistics , 15:2430-2453, 2021.

Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. In Advances in Neural Information Processing Systems , 2022.

L Breiman, JH Friedman, R Olshen, and CJ Stone. Classification and regression trees. 1984.

Thomas F Brooks, D Stuart Pope, and Michael A Marcolini. Airfoil self-noise and prediction. Technical report, 1989.

T Tony Cai and Hongming Pu. Transfer learning for nonparametric regression: Non-asymptotic minimax analysis and adaptive procedure. arXiv preprint arXiv:0000.0000 , 2022.

T Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics , 49(5):2825-2850, 2021.

Yuchao Cai, Yuheng Ma, Yiwei Dong, and Hanfang Yang. Extrapolated random tree for regression. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 3442-3468. PMLR, 23-29 Jul 2023.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms . MIT press, 2022.

Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems , 47(4):547-553, 2009.

Felipe Cucker and Ding-Xuan Zhou. Learning Theory: An Approximation Theory Viewpoint . Cambridge University Press, 2007.

Fida Kamal Dankar and Khaled El Emam. Practicing differential privacy in health care: A review. Trans. Data Priv. , 6(1):35-67, 2013.

Marie Douriez, Harish Doraiswamy, Juliana Freire, and Cláudio T Silva. Anonymizing nyc taxi data: Does it matter? In 2016 IEEE international conference on data science and advanced analytics (DSAA) , pages 140-148. IEEE, 2016.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

John C Duchi, Michael I Jordan, and Martin J Wainwright. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438. IEEE, 2013.

John C Duchi, Michael I Jordan, and Martin J Wainwright. Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association , 113(521):182-201, 2018.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60. IEEE, 2010.

Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security , pages 1054-1067, 2014.

Jianqing Fan and Young K Truong. Nonparametric regression with errors in variables. The Annals of Statistics , pages 1900-1925, 1993.

Farhad Farokhi. Deconvoluting kernel density estimation and regression for locally differentially private data. Scientific Reports , 10(1):21361, 2020.

Wei Gao and Zhi-Hua Zhou. Towards convergence rate analysis of random forests for classification. Advances in neural information processing systems , 33:9300-9311, 2020.

Evarist Giné and Richard Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models . Cambridge University Press, 2021.

Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. arXiv preprint arXiv:2303.01256 , 2023.

László Györfi and Martin Kroll. On rate optimal private regression under local differential privacy. arXiv preprint arXiv:2206.00114 , 2022.

Jihun Hamm, Yingjun Cao, and Mikhail Belkin. Learning privately from multiparty data. In International Conference on Machine Learning , pages 555-563. PMLR, 2016.

Apple Inc. Differential privacy technical overview. Technical Report Apple Inc., 2017.

Matthew Joseph, Janardhan Kulkarni, Jieming Mao, and Steven Z Wu. Locally private gaussian estimation. Advances in Neural Information Processing Systems , 32, 2019.

Peter Kairouz, Monica Ribero Diaz, Keith Rush, and Abhradeep Thakurta. (nearly) dimension independent private erm with adagrad rates via publicly estimated subspaces. In Conference on Learning Theory , pages 2717-2746. PMLR, 2021.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. Extremal mechanisms for local differential privacy. Advances in neural information processing systems , 27, 2014.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385. PMLR, 2015.

Michael R. Kosorok. Introduction to Empirical Processes and Semiparametric Inference . Springer Series in Statistics. Springer, New York, 2008.

Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Pascal Massart. Concentration Inequalities and Model Selection , volume 1896 of Lecture Notes in Mathematics . Springer, Berlin, 2007.

Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973 , 2022.

Warwick J Nash, Tracy L Sellers, Simon R Talbot, Andrew J Cawthorn, and Wes B Ford. The population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report , 48:p411, 1994.

Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semisupervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations , 2017.

Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations , 2018.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Noam Segev, Maayan Harel, Shie Mannor, Koby Crammer, and Ran El-Yaniv. Learn on source, refine on target: A model transfer learning framework with random forests. IEEE transactions on pattern analysis and machine intelligence , 39(9):1811-1824, 2016.

Hyejin Shin, Sungwook Kim, Junbum Shin, and Xiaokui Xiao. Privacy enhanced matrix factorization for recommendation with local differential privacy. IEEE Transactions on Knowledge and Data Engineering , 30(9):1770-1782, 2018.

Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing , pages 245-248. IEEE, 2013.

Ingo Steinwart and Andreas Christmann. Support Vector Machines . Springer Science & Business Media, 2008.

Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Considerations for differentially private learning with large-scale public pretraining. arXiv preprint arXiv:2212.06470 , 2022.

Pınar Tüfekci. Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods. International Journal of Electrical Power &Energy Systems , 60:126-140, 2014.

Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes . Springer Series in Statistics. Springer-Verlag, New York, 1996.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Di Wang, Huangyu Zhang, Marco Gaboardi, and Jinhui Xu. Estimating smooth GLM in non-interactive local differential privacy model with public unlabeled data. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132 of Proceedings of Machine Learning Research , pages 1207-1213. PMLR, 16-19 Mar 2021.

Stanley L Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965.

Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics , pages 196-202. Springer, 1992.

Xiaotong Wu, Lianyong Qi, Jiaquan Gao, Genlin Ji, and Xiaolong Xu. An ensemble of random decision trees with local differential privacy in edge computing. Neurocomputing , 485:181-195, 2022.

Da Yu, Sivakanth Gopi, Janardhan Kulkarni, Zinan Lin, Saurabh Naik, Tomasz Lukasz Religa, Jian Yin, and Huishuai Zhang. Selective pre-training for private fine-tuning. arXiv preprint arXiv:2305.13865 , 2023.

Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations , 2021.

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning , pages 12208-12218. PMLR, 2021.

Fang Zhou, Q Claire, and Ross D King. Predicting the geographical origin of music. In 2014 IEEE International Conference on Data Mining , pages 1115-1120. IEEE, 2014.

Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. In International Conference on Learning Representations , 2021.

Appendix

In this appendix, we provide the detailed results for the additional methodology (Appendix A), the error analysis of the main theoretical results (Appendix B), the full proof of all theoretical results (Appendix C), and details as well as additional results of experiments (Appendix D).

A Methodology of locally differentially private decision tree

A.1 Algorithm of max-edge partition with variance reduction rule

Algorithm 2: Max-edge Partition Rule with Variance Reduction

B Error analysis

We rely on the following decomposition.

where f π , f π and f DP π are defined in (1), (2), and (5), respectively. Loosely speaking, the first error term quantifies the depravation brought by adding privacy noises to the estimator, which we call the privatized error. The second term corresponds to the expected estimation error brought by the randomness of the data, which we call the sample error. The last term is called approximation error, which arises due to the limited approximation capacity of piecewise constant functions. The following three lemmas provide bounds for each of the three errors.

Lemma B.1 ( Bounding of Privatised Error ) . Let f DP π be the LPDT estimator in Algorithm 1. Suppose f π is the decision tree regressor defined in (2) . Suppose π is generated by Algorithm 2. Let Assumption 3.2 and 3.3 hold. Then, if we take n l ≍ n q / 2 s , there holds

with probability P n ⊗ Q n q ⊗ R n at least 1 -1 /n 2 -4 /n 2

Lemma B.2 ( Bounding of Sample Error ) . Suppose f π and f π are the decision tree regressor and the population decision tree regressor defined in (2) and (1) , respectively. Let Assumption 3.2, and 3.3 hold. Then, if we take n l ≍ n q / 2 s , there holds

with probability P n ⊗ Q n q at least 1 -1 /n 2 q -1 /n 2 .

Lemma B.3 ( Bounding of Approximation Error ) . Suppose f π is the population decision tree regressor defined in (1) . Let Assumption 3.2 holds. Then there holds

C Proofs

C.1 Proof of Theorem 3.1

Proof of Theorem 3.1 . For each conditional distribution R ( ˜ U i , ˜ Y i | X i = x, Y i = y ) , we can compute the density ratio as

( i ) The first part is characterized by the random perturbation mechanism. Since the conditional density is identical if x and x ′ belongs to a same A j , we have

The right-hand side can be computed as

By definition, u and u ′ are one-hot vectors and differ at most on two entries. Without loss of generality, assume they differ on the first two elements. Also, the procedure in (3) yields that e -ε/ 4 ≤ P [ ˜ U j i | U j i = 0] / P [ ˜ U j i | U j i = 0] ≤ e ε/ 4 . As a result, (9) becomes

( ii ) For the second part, there holds

Bringing (10) and (11) into (8) yields the desired conclusion.

C.2 Proof of Theorem 3.4

Proof of Theorem 3.4 . Bringing Proposition B.1, B.2, and B.3 into the decomposition (7), we have

holds with probability P n ⊗ Q n q ⊗ R n at least 1 -2 /n 2 q -5 /n 2 . When (2 αs/d ) ∧ p = 2 αs/d , by choosing 2 s ≍ ( nε 2 / log n ) d/ (2 α +2 d ) , i.e. s ≍ log nε 2 , we obtain R L, P ( f DP π ) -R L, P ( f π ) ≲ ( nε 2 / log n ) -α α + d . When (2 αs/d ) ∧ s = s , by choosing 2 s ≍ ( nε 2 / log n ) 1 / 3 , i.e. s ≍ log nε 2 , we obtain R L, P ( f DP π ) -R L, P ( f π ) ≲ ( nε 2 / log n ) -1 / 3 . Note that 2 αs/d ≥ s if and only if α α + d ≥ 1 / 3 . As a result, we get

with probability P n ⊗ Q n q ⊗ R n at least 1 -2 /n 2 q -5 /n 2 .

C.3 Proof of Theorem 3.5

Proof of Theorem 3.5. We consider such P c ∈ P such that for each d P c,X ( x ) = c for some constant c and x ∈ [0 , 1 / 2] , and d P c,Y | X (0) = 1 . Then for such P c , the Bayes function is f ( x ) = 0 and R ∗ L, P c = 0 . For the excess risk of f DP π , we have

We first take expectation with respect to ξ . There holds

For standard Laplace random variable ξ , there holds E ξ ξ 2 = 2 . Each ˜ U j i takes either 1 -1 e ε/ 4 +1 or -1 e ε/ 4 +1 , which implies ( ˜ U j i ) 2 ≥ 1 ( e ε/ 4 +1) 2 . Together, (13) becomes

Bringing this into (12) leads to

Next, we take the expectation of ˜ U j i with respect to both P and P R . Specifically, we have

For any n , there exists some P c with c < 1 / (2 √ n ) . For such P c , we have

since when n l = 0 , each grid has volumn 2 -s . When t < 1 /n , we have 1 / √ nt ≥ 1 ≥ 2 · 2 -s -1 for any s , which implies 1 / √ nt ≥ 2 · √ n ∫ A j d P c,X ( x ) . Bring this into (15), we have

Using the central limit theorem, we know

for t ≤ 1 /n . Then, (16) becomes

This together with (14) yields

C.4 Proofs of results in Section B

To prove lemmas in Section B, we first present several technical results. Their proof will be found in Section C.4

Lemma C.1. Suppose ζ i , i = 1 , · · · , n are independent random variables such that a i ≤ ζ i ≤ b i . Then there holds

for any t > 0 .

Lemma C.2. Suppose ξ i , i = 1 , · · · , n are independent standard sub-exponential random variables with parameters ( ν, β ) [55, Definition 2.9]. Then there holds

for any t > 0 . Moreover, a standard Laplace random variable is sub-exponential with parameters ( √ 2 , 1) .

Lemma C.3. Let π be a partition generated from Algorithm 2. Suppose Assumption 3.2 holds. Then for any D = { ( X 1 , Y 1 ) , · · · , ( X n , Y n ) } drawn i.i.d. from P and any x ∈ X , there holds

with probability P n at least 1 -1 /n 2 .

Lemma C.4. Let π be a partition generated from Algorithm 2. Suppose Assumption 3.2 holds. Then for any D = { ( X 1 , Y 1 ) , · · · , ( X n , Y n ) } drawn i.i.d. from P and any x ∈ X , there holds

with probability P n at least 1 -1 /n 2 .

Lemma C.5. Let π be a partition generated from Algorithm 2 using public data set D pub = { ( X pub 1 , Y pub 1 ) , · · · , ( X pub n q , Y pub n q ) } . Suppose Assumption 3.2 and 3.3 hold. Suppose n ≥ n q ≥ 2 s log 2 n . If we take n l = n q / 2 s , there holds

with probability 1 -1 /n 2 q . Moreover, the analogous conclusion holds for the empirical probability measure, i.e.

with probability 1 -1 /n 2 q -1 /n 2 .

Lemma C.6. Let π := { A j i , i ∈ { 1 , · · · , p } , j ∈ { 1 , · · · , 2 s } } = { A j , j ∈ I} be the variance reduction decision tree partition generated by Algorithm 2. Then for any A j i := × d i =1 [ a i , b i ] ∈ π , denote the depth of A j i as depth( A j i ) = i . Then for any A j i ∈ π , there holds

This implies that for any A j ∈ π , we have

Lemma C.7. Let π := { A j i , i ∈ { 1 , · · · , p } , j ∈ { 1 , · · · , 2 s } } = { A j , j ∈ I} be the variance reduction decision tree partition generated by Algorithm 2. Then for any A j i ∈ π , denote the depth of A j i as depth( A j i ) = i . Then there holds

Proof of Lemma B.1 . We intend to bound

For each j ∈ I and any x ∈ A j , the point-wise error can be decomposed as

using triangular inequality. We bound the three parts separately.

( i ) For the numerator of ( I ) , note that ˜ Y i -Y i = 4 M ε ξ i and ˜ U j i ∈ [ -1 , 1] , Lemma C.2 yields that, ( ˜ Y i -Y i ) ˜ U j i are sub-exponential random variables with parameter ( ε 2 √ 2 M , 1) and consequently

with probability at least 1 -1 /n 2 . For the denominator, applying Lemma C.1 yields

with probability at least 1 -2 /n 2 . Moreover, by Lemma C.5, we have

holds with probability at least 1 -1 /n 2 q -1 /n 2 . Then, we can guarantee that

with probability 1 -1 /n 2 q -3 /n 2 . This together with (17) yields

( ii ) For ( II ) , we first cancel the ∑ n i =1 U j i on both the numerator and the denominator. Then, applying Lemma C.1, we get

with probability 1 -2 /n 2 since | Y i | ≤ M . Combining this with (19), we get

( iii ) For ( III ) , note that U j i ∈ { 0 , 1 } , we have ∑ n i =1 Y i U j i ≤ M ∑ n i =1 U j i . Thus,

where the last inequality follows from (18) and (19). Combining this with (21) and (22), we have

with probability P n × Q n q × R at least 1 -4 /n 2 -1 /n 2 q for any x ∈ X . This directly implies the desired bound of R L, P ( f DP π ) -R L, P ( f π ) .

Proof of Lemma B.2 . We intend to bound

For any fixed j , we have

We bound two terms separately. For ( I ) , Lemma C.5 yields

with probability 1 -1 /n 2 q -1 /n 2 . For the numerator, Lemma C.4 yields

Together, we get ( I ) ≲ 2 s /n . Analogously, by Lemma C.3, we have

This together with (23) yields ( II ) ≲ 2 s /n . The bound of ( I ) and ( II ) together yields the desired conclusion.

Proof of Lemma B.3 . We intend to bound

For each x , if Assumption 3.2 holds, the point-wise error can be bounded by

Then we can bound the integral of point-wise error by the sum of the integral of errors on each A j , namely

Apply Lemma C.5 and C.6, we have

For each j , we can guarantee

since (2 α/d ) ∧ 1 ≤ 1 . Bringing (26) into (25), we get

By applying Lemma C.7, we immediately have

This together with (24) yields the desired conclusion.

C.5 Proofs of results in Section C.4

Proof of Lemma C.1 . The conclusion follows from Example 2.4 and Proposition 2.5 in [55].

Proof of Lemma C.2 . The conclusion follows from (2.18) in [55].

In the subsequent proof, we define the empirical measure D := 1 n ∑ n i =1 δ ( X i ,Y i ) given samples D = { ( X 1 , Y 1 ) , · · · , ( X n , Y n ) } , where δ is the Dirac function. To conduct our analysis, we first need to recall the definitions of VC dimension ( VC index ) and covering number , which are frequently used in capacity-involved arguments and measure the complexity of the underlying function class [54, 40, 31].

Definition C.8 (VC dimension) . Let B be a class of subsets of X and A ⊂ X be a finite set. The trace of B on A is defined by { B ∩ A : B ⊂ B} . Its cardinality is denoted by ∆ B ( A ) . We say that B shatters A if ∆ B ( A ) = 2 #( A ) , that is, if for every A ′ ⊂ A , there exists a B ⊂ B such that A ′ = B ∩ A . For n ∈ N , let

Then, the set B is a Vapnik-Chervonenkis class if there exists n < ∞ such that m B ( n ) < 2 n and the minimal of such n is called the VC dimension of B , and abbreviate as VC( B ) .

Since an arbitrary set of n points { x 1 , . . . , x n } possess 2 n subsets, we say that B picks out a certain subset from { x 1 , . . . , x n } if this can be formed as a set of the form B ∩ { x 1 , . . . , x n } for a B ∈ B . The collection B shatters { x 1 , . . . , x n } if each of its 2 n subsets can be picked out in this manner. From Definition C.8 we see that the VC dimension of the class B is the smallest n for which no set of size n is shattered by B , that is,

where ∆ B ( { x 1 , . . . , x n } ) = # { B ∩ { x 1 , . . . , x n } : B ∈ B} . Clearly, the more refined B is, the larger is its index.

To further bound the capacity of the function sets, we need to introduce the following fundamental descriptions of covering number which enables an approximation of an infinite set by finite subsets.

Definition C.9 (Covering Number) . Let ( X , d ) be a metric space and A ⊂ X . For ε > 0 , the ε -covering number of A is denoted as

where B ( x, ε ) := { x ′ ∈ X : d ( x, x ′ ) ≤ ε } .

To prove Lemma C.10, we need the following fundamental lemma concerning with the VC dimension of random partitions in Section 2.3, which follows the idea put forward by [30] of the construction of random forest. To this end, let p ∈ N be fixed and π p be a partition of X with number of splits s and π ( p ) denote the collection of all partitions π p .

Lemma C.10. Let ˜ A be the collection of all cells × d i =1 [ a i , b i ] in R d . The VC index of ˜ A equals 2 d +1 . Moreover, for all 0 < ε < 1 , there exists a universal constant C such that

Proof of Lemma C.10. The first result of VC index follows from Example 2.6.1 in [54]. The second result of covering number follows directly from Theorem 9.2 in [40].

Before we proceed, we list the well-known Bernstein's inequality that will be used frequently in the proofs. Lemma C.11 was introduced in [8] and can be found in many statistical learning textbooks, see e.g., [42, 19, 51].

Lemma C.11 (Bernstein's inequality) . Let B > 0 and σ > 0 be real numbers, and n ≥ 1 be an integer. Furthermore, let ξ 1 , . . . , ξ n be independent random variables satisfying E P ξ i = 0 , ∥ ξ i ∥ ∞ ≤ B , and E P ξ i 2 ≤ σ 2 for all i = 1 , . . . , n . Then for all τ > 0 , we have

Proof of Lemma C.3 . Let ˜ A be the collection of all cells × d i =1 [ a i , b i ] in R d . Applying Lemma C.10 with Q := (D X +P X ) / 2 , there exists an ε -net { ˜ A k } K k =1 ⊂ ˜ A with

such that for any j ∈ I p , there exist some k ∈ { 1 , . . . , K } such that

Since

we get

Consequently, by the definition of the covering number and the triangle inequality, for any j ∈ I p , there holds

Therefore, we get

For any fixed 1 ≤ k ≤ K , let the random variable ξ i be defined by ξ i := 1 { X i ∈ ˜ A k }-∫ ˜ A k d P X ( x ′ ) . Then we have E P X ξ i = 0 , ∥ ξ ∥ ∞ ≤ 1 , and E P X ξ 2 i ≤ ∫ ˜ A k d P X ( x ′ ) . According to Assumption 3.2, there holds E P X ξ 2 i ≤ c · 2 -s . Applying Bernstein's inequality in Lemma C.11, we obtain

with probability P n at least 1 -2 e -τ . Then the union bound together with the covering number estimate (28) implies that

with probability P n at least 1 -e -τ . Let τ = 2log n and ε = 1 /n . Then for any n > N 1 := (2 C ) ∧ (2 d +1) ∧ (4 e ) , we have τ +log(2 K ) = 2 log n +log(2 C )+log(2 d +1)+(2 d +1)log(4 e )+2 d log n ≤ (4 d +5)log n . Therefore, we have

with probability P n at least 1 -1 /n 2 . This together with (30) yields that

Proof of Lemma C.4 . Let ˜ A be the collection of all cells × d i =1 [ a i , b i ] in R d . Then there exists an ε -net { ˜ A k } K k =1 ⊂ ˜ A with K bounded by (28) such that for any j ∈ I , (29) holds for some k ∈ { 1 , . . . , K } . Consequently, by the definition of the covering number and the triangle inequality, for any j ∈ I p , there holds

where the last inequality follow from the condition Y ⊂ [ -M,M ] .

For any fixed 1 ≤ k ≤ K , let the random variable ˜ ξ i be defined by ˜ ξ i := 1 { X i ∈ ˜ A k } Y i -∫ ˜ A k f ∗ ( x ′ ) d P X ( x ′ ) . Then we have E P ˜ ξ i = 0 , ∥ ξ ∥ ∞ ≤ 1 , and E P ˜ ξ 2 i ≤ M 2 ∫ ˜ A k d P( x ′ ) . According to Assumption 3.2, there holds E P ˜ ξ 2 i ≤ M 2 · c · 2 -s . Applying Bernstein's inequality in Lemma C.11, we obtain

with probability P n at least 1 -2 e -τ . Similar to the proof of Lemma C.3, one can show that for any n ≥ N 1 , there holds

with probability P n at least 1 -1 /n 2 . This together with (32) yields that

Proof of Lemma C.5 . Since π is generated by Algorithm 2, the number of public samples in each of its leaf cell A j , j ∈ I p is no fewer than n l , i.e.

which implies n -1 q ∑ n i =1 1 { X pub i ∈ A j } ≥ 2 -s Hence, by Lemma C.3, there holds

with probability Q n q at least 1 -1 /n 2 q . By Assumption 3.3, for sufficiently large n with log n ≥ 12 · max( c, 1) · (4 d +5) , there holds

where we used condition n q ≥ 2 s log 2 n . For the opposite direction, notice that by Assumption 3.3 and 3.2, there holds

(35) and (36) together yields the first conclusion. For the second conclusion, again by Lemma C.3, we have

for j ∈ I p with probability P n at least 1 -1 /n 2 . Consequently, there holds

for sufficiently large n . Also,

for large n . In conclusion, we have

(38) holds when the first conclusion and (37) both hold, which yields a probability Q n q × P n at least 1 -1 /n 2 q -1 /n 2 .

Proof of Lemma C.6 . According to the max-edge partition rule, when the depth of the tree i is a multiple of dimension d , each cell of the tree partition is a high-dimensional cube with a side length 2 -i/d . On the other hand, when the depth of the tree s is not a multiple of dimension d , we consider the max-edge tree partition with depth ⌊ i/d ⌋ and ⌈ i/d ⌉ , whose corresponding side length of the higher dimensional cube is 2 -⌊ i/d ⌋ and 2 -⌈ i/d ⌉ . Note that in the splitting procedure of max-edge partition, the side length of each sub-rectangle decreases monotonically with the increase of s , so the side length of a random tree partition cell is between 2 -⌈ i/d ⌉ and 2 -⌊ i/d ⌋ . This implies that

Since i/d -1 ≤ ⌊ i/d ⌋ ≤ ⌈ i/d ⌉ ≤ i/d + 1 , we immediately get 2 -1 √ d · 2 -i/d ≤ diam( A j i ) ≤ 2 √ d · 2 -i/d .

Proof of Lemma C.7 . For binary tree partition π , we can grow π to a perfect binary tree π with depth s , meaning that all interior nodes have two children and all leaf nodes have the same depth s . Then, we can compute the number of leaf nodes on π . On one hand, a depths perfect binary tree has 2 s leaf nodes [17]. On the other hand, consider the perfect tree grown by π . Each leaf node A j , when grown to depth s , is the root node of a depth-( s -depth( A j )) perfect binary tree. Thus, the node A j with induce 2 s -depth( A j ) leaf nodes. Immediately, the total number of π can be computed as ∑ j ∈I 2 s -depth( A j ) . Matching the results of these two computations yields the desired conclusion.

C.6 Derivation of removing the range parameter

At present, we assume that P X = Q X , but the analogous conclusion holds as long as their density ratio is bounded. Let X be scaled to ˘ X . The excess risk can be decomposed by

Note that

Denote the CDF of X j as F j . Since

and similar result holds for b j , we have P ( X j / ∈ [ ̂ a j , ̂ b j ]) ≲ log n q n q with probability 1 -1 /n 2 q . Thus, there holds P ( X / ∈ × d j =1 [ ̂ a j , ̂ b j ])) ≲ log n q n q with probability 1 -d/n 2 q . Note that if we restrict n q ≳ ( nε 2 ) d ∨ 2 α 2 α +2 d (instead of n q ≳ n d 2 α +2 d as in the paper, which is a minor change), there holds P ( ˘ X / ∈ × d j =1 [0 , 1]) ≲ ( nε 2 ) -α α + d · log n . Thus, up to a log factor, Theorem 3.4 holds with probability 1 -d/n 2 q .

D Experiment details

D.1 Implementation details

All experiments are conducted on a machine with 72-core Intel Xeon 2.60GHz and 128GB of main memory. The code of LPDT is available on GitHub 2 . Each round of training and testing of LPDT may take less than a second for small data with thousands of samples. For large datasets such as Chicago taxi data, it may take minutes.

For LPDT-M and LPDT-V, we choose n l ∈ { 2 , 5 , 10 , 20 , 40 , 60 , 80 , 100 , 120 , 140 , 160 } and s ∈ { 1 , 2 , 3 , 4 } in Section 4.2. For large data in Section 4.3, we let s ∈ { 4 , 6 , 8 , 10 } . In addition, we add one more parameter adjusting the allocation of the privacy budget on the numerator and denominator. Specifically, let (3) and (4) be

as well as

for ρ ∈ [0 , 1] . In this case, the mechanisms are respectively ρε -LDP and (1 -ρ ) ε -LDP, which means the hybrid mechanism is still ε -LDP. We select ρ ∈ { 0 . 3 , 0 . 5 , 0 . 7 } .

Decision Tree (DT): For standard non-private decision trees, we use the implementation by Scikit-Learn [47]. We select max _ depth in { 1 , 2 , 3 , 4 } in Section 4.2. For large data in Section 4.3, we let max _ depth ∈ { 4 , 6 , 8 , 10 } .

Private Histogram (PHIST): We implement the Private Histogram proposed by [9] in Python. PHIST applies the Laplacian mechanism to privatize the estimation of marginal and joint probabilities for a cubic histogram partition with bandwidth h . In cells with a marginal probability less than t , estimation is truncated to 0. We let h ∈ { 1 / 4 , 1 / 3 , 1 / 2 , 1 } , resulting in the number of grids in each feature to be { 4 , 3 , 2 , 1 } . Due to memory limitations, we set h = 1 for datasets with d > 21 . We set the truncation parameter t ∈ { 0 . 01 , 0 . 05 } .

Adjusted Private Histogram (APHIST): We implement the Adjusted Private Histogram proposed by [33] in Python. APHIST modifies PHIST by replacing the privatized marginal probability with an average of marginal probability and volume of the cell. In cells with an averaged probability less than t , estimation is truncated to 0. We let h ∈ { 1 / 4 , 1 / 3 , 1 / 2 , 1 } , resulting in the number of grids in each feature to be { 4 , 3 , 2 , 1 } . Due to memory limitations, we set h = 1 for datasets with d > 21 . We set the truncation parameter t ∈ { 0 . 01 , 0 . 05 } .

Deconvolution kernel (DECONV): DECONV imposes noise to ( X i , Y i ) directly to provide privacy. It then treats the regression procedure as a measurement error problem and solves it conventionally using a deconvolution kernel. We implement DECONV proposed by [29] in Python. We set the bandwidth h ∈ { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 , 5 } .

D.2 Parameter tuning strategy

To disentangle the parameter tuning problem under local differential, we propose to leave part of the private data as a validation set, whose sample size is n v . For each set of parameters on the parameter grid, the model is partitioned on X pub and trained on X . Then the model is passed to each data holder in the validation set. The data holder computes the point-wise error | ̂ f ( x ) -f ( x ) | 2 for each model and returns a privatized value to the curator. By advanced composition [26, 39], for k parameter grids, we can guarantee the ε - privacy of x with a noise M 2 √ k ξ/ε where ξ is a standard Laplace random

M 2 k ξ i /ε . As long as kn -1 v ε -1 are sufficiently small, the validation procedure can be well conducted. As a result, the capacity of parameter grids is restricted by o ( n v ε ) . In many cases, a large amount of data can be acquired if LDP is posed. In such a scenario, we can assure n v to be large enough to cover the desired parameter grid.

D.3 Details of real data sets

We summarize the details of real data sets in Table 4, with the number of instances and features after pre-processing reported. Each feature is min-max scaled to the range [0 , 1] individually. We also present additional information of the data sets including the data source and the pre-processing details.

ABA: The Abalone dataset originally comes from biological research [44] and now it is accessible on UCI Machine Learning Repository [22]. ABA contains 4177 observations of one target variable and 8 attributes related to the physical measurements of abalone.

AIR: The Airfoil Self-Noise dataset on UCI Machine Learning Repository records the result of a series of aerodynamic and acoustic tests of airfoil blade sections conducted in an anechoic wind tunnel [12]. It comprises 1503 instances of 6 attributes including wind tunnel speeds and angles of attack.

ALG: The Algerian Forest Fires dataset on UCI Machine Learning Repository contains 244 instances of 11 attributes and 1 output attribute. The task is to predict the condition of forest fires in Algeria [2]. The attribute date is omitted when conducting regression in our experiments.

AQU: The QSAR aquatic toxicity dataset was used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas (fathead minnow) on a set of 908 chemicals. It contains 546 instances of 8 input attributes and 1 output attribute.

CON: The Concrete Compressive Strength dataset on UCI Machine Learning Repository contains 1030 instances of 8 input attributes and 1 output attribute. The task is to predict the concrete compressive strength which is a regression problem.

BUI: The Residential Building Data Set Data Set dataset on UCI Machine Learning Repository includes construction cost, sale prices, project variables, and economic variables corresponding to real estate single-family residential apartments in Tehran, Iran. It contains 372 instancesa of 103 input attributes and 2 output attributes.

CBM: The Condition Based Maintenance of Naval Propulsion Plants dataset [5] on UCI Machine Learning Repository was generated from a sophisticated simulator of Gas Turbines. It contains 11934 instances of 16 features.

CCP: The Combined Cycle Power Plant Data Set dataset [53] on UCI Machine Learning Repository contains 9568 data points. There are 4 features that can be used to predict the net hourly electrical energy output of the power plant.

CPU: The cpusmall dataset is from LIBSVM [16]. It contains 8192 instances, each with 12 attributes.

FIS: The QSAR fish toxicity dataset on UCI Machine Learning Repository was used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas (fathead minnow) on a set of 908 chemicals. It contains 908 instances of 7 features.

RED: This dataset contains the information on red wine of the Wine Quality dataset [18] on UCI Machine Learning Repository. There are 11 input variables to predict the output variable wine quality. 4898 instances are collected in the dataset.

WHI: This dataset also originates from the Wine Quality dataset [18] on UCI Machine Learning Repository. There are 11 features related to white wine to predict the corresponding wine quality.

HOU:The Housing-Boston dataset can be acquired from LIBSVM datasets of NTU, which is comprised of 506 observations with 13 features. The dataset is used to predict the price of a house in Boston.

MUS: The Geographical Original of Music dataset was built from a personal collection of 1059 tracks covering 33 countries/areas. The program MARSYAS [64] was used to extract audio features from the wave files. We used the default MARSYAS settings in single vector format (68 features) to estimate the performance with basic timbal information covering the entire length of each track.

D.4 Additional result of the real data experiment

We present the additional result of the real data experiment omitted in the main text due to page limitation.

D.5 Details of Chicago taxi dataset

The dataset used in this study was obtained from the Differential Privacy Temporal Map Challenge (DeID2), which aims to develop algorithms that preserve data utility while guaranteeing individual privacy protection. The dataset contains quantitative and categorical information about taxi trips in Chicago, including time, distance, location, payment, and service provider. These features include the identification number of taxis ( taxi _ id ), time of each trip ( seconds ), the distance of each trip ( miles ), index of the zone where the trip starts ( pca ), index of the zone where the trip ends ( dca ), service provider ( company ), the method used to pay for the trip ( payment _ type ), amount of tips ( tips ), and amount of fares ( fare ).

To preprocess the data, we selected locations in the central region of the map data to ensure sufficient passenger pick-up and drop-off orders, and we chose the top 10 taxi companies with the highest number of orders. We then divided the dataset based on the payment feature to distinguish between

private and public data, which is in accordance with the fact that we only had access to specific data holders. Additionally, we converted all categorical features, including pca , dca , and company , into one-hot values, resulting in a 102-dimensional feature vector. Ultimately, we made predictions for the fares of the entire trip.

Figure 6 shows a boxplot of the prediction feature in the Chicago taxi dataset. As we can see from the plot, the public data and private data differ in distribution, which we referred to as a distribution shift in the main text.

public

private

0

5

10

15

20

25

30

35

fare

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a scheme for training decision trees on a combination of public and private data. By using local differential privacy the algorithm guarantees that adversaries learn little about the private data from the outcomes of the algorithm. Under assumptions and a specific splitting rule for the algorithm's public part, the authors prove properties about the convergence and generalization of the algorithm. The algorithm is empirically evaluated on 15 real and 1 synthetic dataset and compared against private histograms and deconvolution.\n\nSTRENGTHS:\n- The paper is written in clear English.\n- Substantial theoretical analysis.\n- Empirical evaluation on many (15) datasets shows that this method outperforms private histograms and deconvolution.\n- The effect of different hyperparameter settings was clearly explored.\n\nWEAKNESSES:\nMajor:\n- Assumption 3.2 is key to deriving error bounds but I am unsure if this holds. Decision trees are discontinuous models that are often used to model discontinuous data.\n- Sections 2 and 3 are very dense in mathematical notation and use complicated phrasing to say simple things. E.g. section 2.2 spends a full page that is hard to read on explaining that: an estimated regression tree predicts the mean of samples in a leaf and that the private tree predicts it based on private estimates of reaching the leaf (U) and the sample value (Y), privatized using randomized response and Laplace mechanism.\n- Only relatively high values of $\\epsilon$ were explored. While it is debatable what is a good value, to the best of my knowledge $\\epsilon < 1$ is generally considered 'good privacy'.\n\nMinor:\n- Although I understand the 'max-edge partition with variance reduction' was introduced to prove theoretical properties it suffers from data that is non-uniformly distributed or contains useless features.\n    - Useless features: the algorithm will never choose the same feature to split on twice unless depth > d (as a result of splitting on the largest distance). This means that if there are few informative features these will be split on only once.\n    - Non-uniform data: the splits are determined on the midpoint between minimum and maximum values of a feature instead of looking at how data is distributed in the feature. This means data with e.g. long tails will cause a split to create one leaf with almost all data and one leaf with almost none.\n- The algorithm and analyses are based on continuous features but the datasets used for evaluation also contain categorical features. E.g. 'sex' in Abalone.\n- Limitations and broader impact have been moved to the appendix while these should be part of the main text.\n- The visualizations only show mean estimates, no standard deviations / standard errors.\n- It would be nice to compare to (global) differentially private decision trees to get an idea of the cost of decentralization in this task.\n\nQUESTIONS:\n1. Does assumption 3.2 hold when considering decision trees that make discontinuous partitions?\n2. Does the proposed algorithm still outperform existing work when considering better privacy protection with $\\epsilon < 1$?\n3. Can the theoretical guarantees be applied to categorical data (where distances such as 'diam' do not make sense)?\n\nLIMITATIONS:\nThe limitations are moved to the appendix with the only limitation discussed being assumption 3.3.\n\nDepending on the answer to question 1 I believe assumption 3.2 needs to be added to the limitations and a discussion on the choices of $\\epsilon$ should be added. The limitations should be given in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies nonparametric regression under local differential privacy (LDP) constraints with the aid of public data. The paper proposes an algorithm, locally differentially private decision tree (LPDT), which uses public data to construct the splitting criteria for the decision tree and private data to compute the regressed values for each leaf node. The paper shows that under certain assumptions, with a small amount of public data, the algorithm can achieve a near-optimal convergence rate for decision tree regression under LDP constraints. The paper also demonstrates the effectiveness of the algorithm through experiments.\n\nSTRENGTHS:\n1. The paper studies an interesting problem, which has not been studied before to the best of my knowledge. The paper also proposes an algorithm that achieves an optimal convergence rate under reasonable assumptions. \n\n2. The paper shows that without public data, decision tree learning will have a nontrivial risk with any amount of private data. This suggests the importance of public data for decision tree regression under LDP constraints.\n\nWEAKNESSES:\n1. The importance of public data has been established for different learning tasks recently, especially under the central notion of differential privacy. Extending a similar observation to decision tree learning under local DP constraints is interesting, but less surprising.\n\n2. The similarity assumption between P and Q, Assumption 3.3, looks weird. The assumption doesn't pose any requirement on how y is correlated with x. Imagine the case when P and Q have the same marginal on the feature space while the correlation between X and Y are completely different under P and Q. It would be hard to learn useful information about how x can be used to predict y from public data, and hence the structure of a good decision tree. I am not sure whether I have missed other important assumptions.\n\n3. Another baseline to compare (theoretically, and empirically) would be to only use public data to train the decision tree. How would the proposed method compare against this baseline? E.g., in the experiment in Section 4.2.\n\nQUESTIONS:\nSee the questions above.\n\nAdditional questions:\n1. In Theorem 3.4, how does the error depend on tau? Could you comment on the dependence on tau as well?\n2. Figure 4(b), when delta = 0, and eps is large, does the proposed method recover the non-private training rate?\n3. For LDP estimation, the dependence on eps can be improved to inverse exponential in eps when eps is large, e.g. in Feldman et al 2022. Can the rate in Theorem 3.4 improve to a similar dependence on eps as well? It seems the considered eps's in the experimental section are mostly large.\n\n[Feldman et al 2022] Vitaly Feldman, Jelani Nelson, Huy Nguyen, Kunal Talwar. Private Frequency Estimation via Projective Geometry\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies non-parametric estimation under local differential privacy with public data. The authors propose a locally private decision tree algorithm and show that it is min-max optimal under some regimes. The authors also test the algorithm on real and synthetic datasets.\n\nSTRENGTHS:\n1. The idea to leverage public data in LDP estimation is practically relevant. \n2. The experiment results are extensive. It includes both synthetic and real-world data and discussion on the effect of various parameters. Empirically the algorithm outperforms existing algorithms in most experiment settings.\n3. The authors provide a thorough theoretical analysis of the proposed algorithm in terms of sample complexity and computation cost.  The proposed algorithm has improved time complexity over prior works.\n\nWEAKNESSES:\n1. The proposed algorithm does not seem to improve too much in terms of sample complexity with public data. We normally hope that using public data should improve the performance somehow (e.g. in reference [8], with public data the prior bound on the mean can be removed), but it seems that the algorithm merely attains the optimal bound without public data only when $\\varepsilon\\lesssim 1$. Existing algorithms that do not use public data can also achieve the same sample complexity.\n2. Given the first point, the fact that the proposed algorithm does not work without public data appears to be a significant disadvantage. While LPDT has advantages in time complexity and empirically outperforms existing methods, it comes at the expense of additional resource of public data. It is thus in some sense unfair to compare with existing methods that do not use public data.\n\n#### Some minor problems\n\n1. The lower bound only matches the upper bound in Theorem 3.4 when $\\varepsilon\\lesssim 1$ (since $e^{\\varepsilon}-1\\simeq \\varepsilon$ only with small $\\varepsilon$), but all experiment results are for large $\\varepsilon\\ge 2$.\n2. The notation $p$ is used first for the density function and then again used for decision tree depth. Please consider changing one of them to avoid confusion.\n\n#### Update\n- Increased my score to 5 after the authors added new results for small $\\varepsilon$ and introduced a new algorithm that matches the lower bound for all ranges of $\\varepsilon$.\n- Increased to 6 after the authors resolved my question for removing range parameters.\n\nQUESTIONS:\nCan existing algorithms be easily modified to take advantage of public data? I feel it is best to fully investigate the possibility of leveraging existing algorithms before inventing something entirely new. Extension based on prior algorithms also serve as a necessary baseline to compare against.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a locally differentially private (LDP) decision-tree (DT) regressor algorithm that takes advantage of public data for utility improvement.\nThe proposed algorithm, LPDT, uses Randomized Response (RR) and the Laplace mechanism to protect the tree partition step.\nThis paper also introduces a new splitting rule named \"max-edge partition rule\" by using the variance as a reduction criterium.\nTheoretical results were presented for convergence rates, training/testing time and space complexity, with important gains in comparison with the state-of-the-art.\nExperimental results were provided with synthetic and many real-world datasets to validate the proposed approach, with significative gains over the state-of-the-art.\n\nSTRENGTHS:\nThe motivation behind using Locally Differentially Private (LDP) methods for decision trees is well-founded. The idea of using public data is new and has proven to bring advantages in many fields (e.g., vision, languages).\nTheoretical and experimental validation were presented showing advancement over the state-of-the-art.\n\nWEAKNESSES:\nThe experiments only considered medium privacy regimes \\epsilon >= 2. The paper would greatly benefit from including high privacy regimes \\epsilon <=1 and the utility gain of using public data.\n\nQUESTIONS:\n- Why only presenting results for medium to high privacy regimes?\n- In lines 73 & 74 \"we show that LPDT performs well even in the presence of significant disparities between public and private data.\", it is unclear how \"performs well\" is concise with the results of Fig. 4. For instance, training only on private date led to MSE=0.8 (Table 3). Using only \\alpha=1 as public data, with \\epsilon=3 (medium regime), the MSE~3.5 is ~4.5x larger. How worse would that be in high privacy regimes?\n\nLIMITATIONS:\nAlthough using public data can lead to utility improvements, the authors did not discuss the limitation of having such data in real-world applications. Indeed, following my last question on \"data heterogeneity\", the learning process could be really damaged if public and private data have different distributions. I recommend the authors to further discuss this paper's limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}