SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and similar result holds for  $b^{j}$ , we have  $\mathbb{P}(X^{j} \notin [\widehat{a^{j}}, \widehat{b^{j}}]) \lesssim \frac{\log n_{q}}{n_{q}}$  with probability  $1 - 1 / n_{q}^{2}$ . Thus, there holds  $\mathbb{P}(X \notin \times_{j=1}^{d} [\widehat{a^{j}}, \widehat{b^{j}}]) \lesssim \frac{\log n_{q}}{n_{q}}$  with probability  $1 - d / n_{q}^{2}$ . Note that if we restrict  $n_{q} \gtrsim (n\varepsilon^{2})^{\frac{d\vee 2\alpha}{2\alpha + 2d}}$  (instead of  $n_{q} \gtrsim n^{\frac{d}{2\alpha + 2d}}$  as in the paper, which is a minor change), there holds  $\mathbb{P}(\check{X} \notin \times_{j=1}^{d}[0,1]) \lesssim (n\varepsilon^{2})^{-\frac{\alpha}{\alpha + d}} \cdot \log n$ . Thus, up to a log factor, Theorem 3.4 holds with probability  $1 - d / n_{q}^{2}$ .
D Experiment details
D.1 Implementation details
All experiments are conducted on a machine with 72-core Intel Xeon 2.60GHz and 128GB of main memory. The code of LPDT is available on GitHub<sup>2</sup>. Each round of training and testing of LPDT may take less than a second for small data with thousands of samples. For large datasets such as Chicago taxi data, it may take minutes.
- For LPDT-M and LPDT-V, we choose  $n_l \in \{2, 5, 10, 20, 40, 60, 80, 100, 120, 140, 160\}$  and  $s \in \{1, 2, 3, 4\}$  in Section 4.2. For large data in Section 4.3, we let  $s \in \{4, 6, 8, 10\}$ . In addition, we add one more parameter adjusting the allocation of the privacy budget on the numerator and denominator. Specifically, let (3) and (4) be
$$
\tilde {U} _ {i} ^ {j} = \left\{ \begin{array}{l l} U _ {i} ^ {j} - \frac {1}{1 + e ^ {\rho \varepsilon / 2}} & \text {w i t h p r o b a b i l i t y} \frac {1}{1 + e ^ {\rho \varepsilon / 2}} \\ 1 - U _ {i} ^ {j} - \frac {1}{1 + e ^ {\rho \varepsilon / 2}} & \text {w i t h p r o b a b i l i t y} \frac {1}{1 + e ^ {\rho \varepsilon / 2}}. \end{array} \right.
$$
$$
\tilde {Y} _ {i} = Y _ {i} + \frac {2 M}{(1 - \rho) \varepsilon} \xi_ {i}
$$
for  $\rho \in [0,1]$ . In this case, the mechanisms are respectively  $\rho \varepsilon$ -LDP and  $(1 - \rho)\varepsilon$ -LDP, which means the hybrid mechanism is still  $\varepsilon$ -LDP. We select  $\rho \in \{0.3, 0.5, 0.7\}$ .
- Decision Tree (DT): For standard non-private decision trees, we use the implementation by Scikit-Learn [47]. We select max_depth in  $\{1,2,3,4\}$  in Section 4.2. For large data in Section 4.3, we let max_depth  $\in$ $\{4,6,8,10\}$ .
- Private Histogram (PHIST): We implement the Private Histogram proposed by [9] in Python. PHIST applies the Laplacian mechanism to privatize the estimation of marginal and joint probabilities for a cubic histogram partition with bandwidth  $h$ . In cells with a marginal probability less than  $t$ , estimation is truncated to 0. We let  $h \in \{1/4, 1/3, 1/2, 1\}$ , resulting in the number of grids in each feature to be \{4, 3, 2, 1\}. Due to memory limitations, we set  $h = 1$  for datasets with  $d > 21$ . We set the truncation parameter  $t \in \{0.01, 0.05\}$ .
- Adjusted Private Histogram (APHIST): We implement the Adjusted Private Histogram proposed by [33] in Python. APHIST modifies PHIST by replacing the privatized marginal probability with an average of marginal probability and volume of the cell. In cells with an averaged probability less than  $t$ , estimation is truncated to 0. We let  $h \in \{1/4, 1/3, 1/2, 1\}$ , resulting in the number of grids in each feature to be \{4, 3, 2, 1\}. Due to memory limitations, we set  $h = 1$  for datasets with  $d > 21$ . We set the truncation parameter  $t \in \{0.01, 0.05\}$ .
- Deconvolution kernel (DECONV): DECONV imposes noise to  $(X_{i},Y_{i})$  directly to provide privacy. It then treats the regression procedure as a measurement error problem and solves it conventionally using a deconvolution kernel. We implement DECONV proposed by [29] in Python. We set the bandwidth  $h\in \{0.02,0.05,0.1,0.2,0.5,1,5\}$ .
D.2 Parameter tuning strategy
To disentangle the parameter tuning problem under local differential, we propose to leave part of the private data as a validation set, whose sample size is  $n_v$ . For each set of parameters on the parameter grid, the model is partitioned on  $X^{pub}$  and trained on  $X$ . Then the model is passed to each data holder in the validation set. The data holder computes the point-wise error  $|\widehat{f}(x) - f(x)|^2$  for each model and returns a privatized value to the curator. By advanced composition [26, 39], for  $k$  parameter grids, we can guarantee the  $\varepsilon$ -privacy of  $x$  with a noise  $M^2\sqrt{k}\xi/\varepsilon$  where  $\xi$  is a standard Laplace random variable. Then, the estimated validation mean squared error is  $\frac{1}{n_v}\sum_{i=1}^{n_v}\left(\widehat{f}(X_i) - f^*(X_i)\right)^2 + M^2\sqrt{k}\xi_i/\varepsilon$ . As long as  $\sqrt{k}n_v^{-1}\varepsilon^{-1}$  are sufficiently small, the validation procedure can be well conducted. As a result, the capacity of parameter grids is restricted by  $o(n_v\varepsilon)$ . In many cases, a large amount of data can be acquired if LDP is posed. In such a scenario, we can assure  $n_v$  to be large enough to cover the desired parameter grid.
$^{2}$ https://github.com/Karlmyh/LPDT
D.3 Details of real data sets
We summarize the details of real data sets in Table 4, with the number of instances and features after pre-processing reported. Each feature is min-max scaled to the range [0, 1] individually. We also present additional information of the data sets including the data source and the pre-processing details.
Table 4: Description of real datasets
DATASET	n	d	DATASET	n	d
ABA	4177	8	CPU	8192	12
AIR	1503	6	FIS	908	6
ALG	244	12	HOU	506	13
AQU	546	8	MUS	1059	68
BUI	372	105	WHI	4898	12
CBM	11934	16	RED	4898	12
CCP	9568	4	CON	1030	8
ABA: The Abalone dataset originally comes from biological research [44] and now it is accessible on UCI Machine Learning Repository [22]. ABA contains 4177 observations of one target variable and 8 attributes related to the physical measurements of abalone.
AIR: The Airfoil Self-Noise dataset on UCI Machine Learning Repository records the result of a series of aerodynamic and acoustic tests of airfoil blade sections conducted in an anechoic wind tunnel [12]. It comprises 1503 instances of 6 attributes including wind tunnel speeds and angles of attack.
ALG: The Algerian Forest Fires dataset on UCI Machine Learning Repository contains 244 instances of 11 attributes and 1 output attribute. The task is to predict the condition of forest fires in Algeria [2]. The attribute date is omitted when conducting regression in our experiments.
AQU: The QSAR aquatic toxicity dataset was used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas (fathead minnow) on a set of 908 chemicals. It contains 546 instances of 8 input attributes and 1 output attribute.
CON: The Concrete Compressive Strength dataset on UCI Machine Learning Repository contains 1030 instances of 8 input attributes and 1 output attribute. The task is to predict the concrete compressive strength which is a regression problem.
BUI: The Residential Building Data Set Data Set dataset on UCI Machine Learning Repository includes construction cost, sale prices, project variables, and economic variables corresponding to real estate single-family residential apartments in Tehran, Iran. It contains 372 instancesa of 103 input attributes and 2 output attributes.
CBM: The Condition Based Maintenance of Naval Propulsion Plants dataset [5] on UCI Machine Learning Repository was generated from a sophisticated simulator of Gas Turbines. It contains 11934 instances of 16 features.
CCP: The Combined Cycle Power Plant Data Set dataset [53] on UCI Machine Learning Repository contains 9568 data points. There are 4 features that can be used to predict the net hourly electrical energy output of the power plant.
CPU: The cpusmall dataset is from LIBSVM [16]. It contains 8192 instances, each with 12 attributes.
FIS: The QSAR fish toxicity dataset on UCI Machine Learning Repository was used to develop quantitative regression QSAR models to predict acute aquatic toxicity towards the fish Pimephales promelas (fathead minnow) on a set of 908 chemicals. It contains 908 instances of 7 features.
RED: This dataset contains the information on red wine of the Wine Quality dataset [18] on UCI Machine Learning Repository. There are 11 input variables to predict the output variable wine quality. 4898 instances are collected in the dataset.
WHI: This dataset also originates from the Wine Quality dataset [18] on UCI Machine Learning Repository. There are 11 features related to white wine to predict the corresponding wine quality.
HOU: The Housing-Boston dataset can be acquired from LIBSVM datasets of NTU, which is comprised of 506 observations with 13 features. The dataset is used to predict the price of a house in Boston.
MUS: The Geographical Original of Music dataset was built from a personal collection of 1059 tracks covering 33 countries/areas. The program MARSYAS [64] was used to extract audio features from the wave files. We used the default MARSYAS settings in single vector format (68 features) to estimate the performance with basic timbal information covering the entire length of each track.
D.4 Additional result of the real data experiment
We present the additional result of the real data experiment omitted in the main text due to page limitation.
Table 5: Additional results for average MSE over real data sets for LDP regression methods. The best results are bolded and the second best results are underlined. The marked results with significance towards the rest results are marked with  $*$ . Due to exceeding the memory limit, PHIST and APHIST are corrupted on two data sets which are marked with - .
DT	ε = 0.5	ε = 1
LPDT-M	LPDT-V	APHIST	PHIST	DECONV	LPDT-M	LPDT-V	APHIST	PHIST	DECONV
ABA	5.67e+0	1.10e+1	1.07e+1	1.12e+1	1.16e+1	1.05e+7	1.06e+1	1.01e+1*	1.06e+1	1.07e+1	3.06e+6
AIR	2.26e+1	3.79e+2	1.29e+2*	2.10e+2*	3.10e+2	1.19e+6	1.67e+2	1.17e+2*	9.25e+1*	1.26e+2	5.64e+7
ALG	2.12e-2	2.59e-1*	2.59e-1*	2.74e-1	2.83e-1	8.81e+2	2.40e-1*	2.40e-1*	2.56e-1	2.59e-1	1.73e+2
AQU	1.92e+0	1.35e+1	3.01e+0*	3.68e+0*	4.06e+0	6.02e+4	3.06e+0	3.63e+0	2.99e+0*	3.11e+0	9.55e+3
BUI	1.75e+5	3.54e+6*	1.70e+6*	-	-	1.20e+9	2.46e+6*	1.54e+6*	-	-	3.30e+9
CBM	4.08e-27	6.96e+0	6.64e+0*	6.96e+0*	6.99e+0	9.69e+6	2.46e+0*	1.90e+0*	6.91e+0	6.92e+0	1.58e+6
CCP	2.19e+1	4.52e+2*	4.89e+2*	6.79e+2	9.31e+2	1.23e+10	4.72e+2	4.72e+2	4.15e+2*	4.82e+2	1.43e+7
CON	9.38e+1	4.56e+2	4.56e+2	3.24e+2*	3.38e+2*	3.33e+8	3.08e+2	3.42e+2	2.99e+2	3.03e+2	3.18e+7
CPU	2.15e+1	3.56e+2	3.56e+2	3.56e+2	3.68e+2	6.61e+7	3.41e+2	9.37e+1*	3.41e+2	3.43e+2	9.37e+7
FIS	1.07e+0	3.66e+0	3.66e+0	2.62e+0*	2.81e+0*	1.68e+5	2.48e+0	2.48e+0	2.24e+0*	2.30e+0	1.57e+4
HOU	2.11e+1	1.05e+2*	1.04e+2	1.08e+2	1.17e+2	2.34e+5	8.56e+1*	8.56e+1*	9.00e+1*	9.32e+1	1.55e+7
MUS	3.00e+2	6.21e+2*	6.21e+2*	-	-	2.18e+5*	4.26e+2*	4.26e+2*	-	-	1.29e+6*
RED	4.76e-1	8.78e-1*	8.94e-1*	1.09e+0	1.30e+0	2.74e+6	7.67e-1*	7.04e-1*	8.01e-1	8.69e-1	1.15e+5
WHI	5.77e-1	8.77e-1*	8.69e-1*	9.45e-1	1.03e+0	1.28e+9	8.68e-1	8.48e-1	8.32e-1*	8.56e-1	1.58e+6
DT	ε = 4	ε = 8
LPDT-M	LPDT-V	APHIST	PHIST	DECONV	LPDT-M	LPDT-V	APHIST	PHIST	DECONV
ABA	5.67e+0	9.46e+0*	9.54e+0*	2.00e+1	1.05e+1	1.14e+1	7.96e+0*	6.94e+0*	2.07e+1	1.05e+1	1.05e+1
AIR	2.26e+1	4.67e+1*	4.15e+1*	1.52e+3	5.28e+1	4.85e+1	4.45e+1*	3.47e+1*	1.64e+3	4.87e+1	4.71e+1
ALG	2.12e-2	2.40e-1*	2.42e-1*	2.59e-1	2.48e-1	2.03e+3	2.43e-1	2.46e-1	2.64e-1	2.46e-1	2.54e-1
AQU	1.92e+0	2.81e+0	2.77e+0*	4.54e+0	2.85e+0	5.23e+0	2.71e+0*	2.63e+0*	4.86e+0	2.83e+0	2.85e+0
BUI	1.75e+5	1.49e+6*	1.35e+6*	-	-	3.90e+7	1.40e+6*	1.28e+6*	-	-	1.38e+7
CBM	4.08e-27	9.34e-1*	5.10e-1*	7.45e+0	6.48e+0	1.41e+5	7.08e-1*	3.77e-2*	4.49e+0	2.00e+0	6.44e+0
CCP	2.19e+1	9.08e+1*	6.78e+1*	2.20e+4	3.34e+2	2.86e+2	8.22e+1*	4.76e+1*	2.26e+4	3.26e+2	2.10e+2
CON	9.38e+1	2.70e+2*	2.54e+2*	4.06e+2	2.97e+2	3.93e+2	2.34e+2*	2.04e+2*	4.21e+2	2.96e+2	2.91e+2
CPU	2.15e+1	3.05e+2*	8.36e+1*	9.65e+2	3.40e+2	3.93e+5	3.01e+2*	4.85e+1*	1.02e+3	3.40e+2	3.42e+2
FIS	1.07e+0	1.90*	2.07e+0	3.47e+0	2.17e+0	2.40e+0	1.55e+0*	1.65e+0*	3.67e+0	2.15e+0	2.17e+0
HOU	2.11e+1	7.80e+1*	7.37e+1*	1.18e+2	8.27e+1	1.31e+3	7.32e+1*	7.02e+1*	1.26e+2	8.19e+1	1.27e+2
MUS	3.00e+2	3.33e+2*	3.46e+2*	-	-	8.03e+4	3.21e+2*	3.26e+2*	-	-	2.25e+4
RED	4.76e-1	7.13e-1	6.92e-1	3.63e+0	7.19e-1	4.03e+2	6.14e-1*	5.79e-1*	3.88e+0	7.09e-1	6.94e-1
WHI	5.77e-1	7.39e-1*	7.50e-1*	4.34e+0	8.05e-1	1.88e+4	6.93e-1*	6.40e-1*	4.50e+0	8.02e-1	8.21e-1
D.5 Details of Chicago taxi dataset
The dataset used in this study was obtained from the Differential Privacy Temporal Map Challenge (DeID2), which aims to develop algorithms that preserve data utility while guaranteeing individual privacy protection. The dataset contains quantitative and categorical information about taxi trips in Chicago, including time, distance, location, payment, and service provider. These features include the identification number of taxis (taxi_id), time of each trip (seconds), the distance of each trip (miles), index of the zone where the trip starts (pca), index of the zone where the trip ends (dca), service provider (company), the method used to pay for the trip (payment_type), amount of tips (tips), and amount of fares (fare).
To preprocess the data, we selected locations in the central region of the map data to ensure sufficient passenger pick-up and drop-off orders, and we chose the top 10 taxi companies with the highest number of orders. We then divided the dataset based on the payment feature to distinguish between
Table 6: Average time over real data sets for LDP regression methods. Due to exceeding the memory limit, PHIST and APHIST are corrupted on two data sets which are marked with -.
DT	LPDT-M	LPDT-V	APHIST	PHIST	DECONV
ABA	5.34e-3	1.78e-2	3.23e-2	2.70e-2	2.71e-2	1.47e+0
AIR	2.63e-3	1.13e-2	1.76e-2	9.68e-3	9.58e-3	1.29e-1
ALG	1.79e-3	2.02e-3	2.81e-3	6.10e-3	5.55e-3	1.17e-2
AQU	2.21e-3	3.38e-3	5.21e-3	7.17e-3	5.62e-3	3.12e-2
BUI	3.87e-3	4.09e-3	2.61e-2	-	-	1.35e-1
CBM	2.58e-2	8.98e-2	1.23e-1	1.04e-1	1.04e-1	2.76e+1
CCP	8.11e-3	6.28e-2	6.98e-2	4.65e-2	4.70e-2	4.42e+0
CON	3.28e-3	6.38e-3	1.01e-2	7.45e-3	7.14e-3	9.06e-2
CPU	1.52e-2	3.99e-2	6.40e-2	5.86e-2	5.80e-2	9.75e+0
FIS	3.00e-3	4.53e-3	7.63e-3	6.64e-3	6.01e-3	5.53e-2
HOU	2.52e-3	3.37e-3	5.30e-3	5.21e-3	5.54e-3	5.03e-2
MUS	6.24e-3	6.64e-3	4.94e-2	-	-	6.49e-1
RED	3.49e-3	9.01e-3	1.41e-2	1.17e-2	1.18e-2	2.65e-1
WHI	7.16e-3	1.68e-2	4.26e-2	3.29e-2	3.29e-2	2.70e+0
Table 7: Additional results for average MSE over real data sets for decision trees. The best results are bolded.
DT	DT-pub	PDT-M	PDT-V
ABA	5.67e+0	7.93e+0	8.38e+0	7.34e+0
AIR	2.26e+1	4.28e+1	4.49e+1	3.60e+1
ALG	2.12e-2	2.59e-1	2.44e-1	2.46e-1
AQU	1.92e+0	2.42e+0	2.73e+0	2.67e+0
BUI	1.75e+5	1.39e+6	1.44e+6	1.31e+6
CBM	4.11e-27	1.57e+0	7.62e-1	1.23e-1
CCP	2.19e+1	7.21e+1	8.42e+1	5.18e+1
CON	9.38e+1	2.21e+2	2.44e+2	2.13e+2
CPU	2.10e+1	2.92e+2	3.02e+2	6.15e+1
FIS	1.07e+0	1.48e+0	1.65e+0	1.76e+0
HOU	2.11e+1	5.58e+1	7.43e+1	7.10e+1
MUS	3.00e+2	3.38e+2	3.27e+2	3.27e+2
RED	4.76e-1	5.69e-1	6.75e-1	6.12e-1
WHI	5.76e-1	7.10e-1	7.03e-1	6.61e-1
private and public data, which is in accordance with the fact that we only had access to specific data holders. Additionally, we converted all categorical features, including  $pca$ ,  $dca$ , and company, into one-hot values, resulting in a 102-dimensional feature vector. Ultimately, we made predictions for the fares of the entire trip.
Figure 6 shows a boxplot of the prediction feature in the Chicago taxi dataset. As we can see from the plot, the public data and private data differ in distribution, which we referred to as a distribution shift in the main text.
Figure 6: Boxplot of fare in Chicago taxi dataset