Analysis of differentially private synthetic data: a general measurement error approach
Authors: Yangdi Jiang
Yi Liu
Xiaodong Yan
Anne-Sophie Charest
Linglong Kong
Bei Jiang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Measurement Error Model
Differential Privacy
Regression
Statistical Inference

Abstract: Differential private (DP) synthetic datasets have been receiving significant attention from academia, industry, and government. However, little is known about how to perform statistical inference using DP synthetic datasets. Naive approaches that do not take into account the induced uncertainty due to DP mechanism will result in biased estimators and invalid inferences. In this paper, we present a general class of bias-corrected DP estimators with valid asymptotic confidence intervals for parameters in regression settings, by establishing the connection between additive DP mechanisms and measurement error models. Our simulation shows that when the sample covariance between DP noises and data is close to zero, our estimator is far superior to the widely used sufficient statistic perturbation algorithm, and the CIs can achieve better coverage when comparing to the naive CIs obtained from ignoring the DP mechanism.

Under review as a conference paper at ICLR 2023
ANALYSIS OF DIFFERENTIALLY PRIVATE SYNTHETIC DATA: A MEASUREMENT ERROR APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Differential private (DP) synthetic datasets have been receiving significant atten- tion from academia, industry, and government. However, little known about how to perform statistical inference using DP synthetic datasets. Naive approaches that do not take into account the induced uncertainty due to the DP mechanism will result in biased estimators and invalid inferences. In this paper, present a class of MLE-based easy-to-ime to--lmmmmm bias-corrected DP estimators with valid asymp- totic confidence intervals (CI) for parameters in regression settings, by establish- ing the connection between additive DP mechanisms and measurement error mod- els. Our simulation shows that our estimator has comparable performance to the widely used sufficient statistic perturbation (SSP) algorithm in some scenarios but with the advantage of releasing synthetic dataset and obtaining statistically valid asymptotic Cls, which can achieve better coverage when compared to the naive CIs obtained by ignoring the DP mechanism.
INTRODUCTION
Differential privacy (DP) is a mathematically rigorous definition that quantifies privacy risk. It builds on the idea of releasing "privacy-protected" query results, such as summary statistics, using randomized responses. In recent years, the use of differential privacy has quickly gathered popularity as it promises that there will be no additional privacy harm for any individual whether the said individual's ' data belongs in a private dataset or not, and therefore encourage data sharing.
One important characteristic of DP is its composition property (Dwork & Roth 2014). That is, to avoid the scenario where the same analysis rerun and averaging away the noises from the randomized responses, the composition property indicates that running the same analysis twice will have double the amount of privacy risk as running the analysis once. The data provider often set total amount of privacy risk/budget allowed, commonly referred to as the privacy budget, and each analysis from researchers uses portion of the privacy budget. Once the total privacy budget is exhausted, any new analysis would not be possible unless the data provider decides to increase the total privacy budget and thus take on more privacy risk. This could be problematic as it limits the number of analyses that researchers can run, which result in the dataset not being fully explored. In consequence, it diminishes the probability of serendipitous discovery and amplifies the odds of being tricked by unanticipated data problems (Evans et al. WWrking Paper).
To address the problem above, various methods of releasing differentially private synthetic datasets (Lu (2016); Bowen & Liu 2 Gambs al. (202T)) have been proposed. Using the post- processing property of DP, any analysis on the DP synthetic dataset will be differentially private without the additional cost of privacy budget. Therefore, by releasing DP synthetic dataset, it cir- cumvents the problem of running out of privacy budget. Here we will mention few notable meth- ods of generating DP synthetic datasets. In general, the methods of generating DP datasets can be categorized into the parammeretrcc method and the parametric method. For the non non-parametric methods, the DP dataset constructed based on the empirical distribution of the data. The simplest approach would be directly adding Laplace or Gaussian noises to the confidential dataset. For the parametric methods, the DP dataset is constructed based on a parametric distribution/model of the data. Using the robust and flexible model of vine copula, Gambs et al. (2021) draw the DP synthetic dataset from the DP trained vine copula model. From the Bayesian perspective. (0116) proposes generating DP synthetic dataset by drawing samples from the DP version of the posterior predictive

---

Under review as a conference paper at ICLR 2023
distribution. For more comprehensive overview of different DP dataset generation methods, refer to Bowen & Liu 2020).
Dwork & Roth (2014) characterizes differential privacy a definition of privacy tailored to the
problem of Devaacyppreservngg data analysis. However, for a statistician, the goal of statistical infer- ence is often important data analysis. Under the framework of differential privacy, the methods for making statistical inferences are under-explored. Fortunately, the interest in statistical inference under differential privacy has been rising recently including the works like Sheffid (2017) and Bar-] rientos et al. (2019). To make statistical inferences, statistical models need to be specified. With different differential privacy algorithms, it comes different statistical models. It turns out that addi- tive mechanisms, like the Laplace mechanism or Gaussian mechanism, give statistical models that are naturally related to the measurement error models. In other words, each additive mechanism can be viewed as some variation of the measurement error model, and therefore, the tools from the measurement error models can be used to make inferences in the differential privacy setting.
In this paper, we generate DP synthetic dataset by adding DP noises directly to the confidential dataset through the Gaussian mechanism. We choose this method due to its simplicity, and more importantly, it allows to establish the connection to the theory of measurement error as we will see in section 3.T. Using the established tool in the theory of measurement error, we then derive an MLE-based DP bias-corrected estimator and asymptotic confidence interval for our param- eter of interest. Therefore, by establishing a connection to measurement error, we will he able to develop statistical inference under the differential privacy setting. To demonstrate the usefulness of this connection, we study statistical inference under the linear regression setting while preserv- ing differential privacy. In particular, we derive DP consistent estimator and asymptotic confidence interval for the regression coefficient.
Related work As one of the most common statistical models, linear regression has been studied before in differential privacy literature. One of the widely used methods for obtaining a DP estimator for the regression coefficient is through the perturbation of sufficient statistics (Dwork et al. [014]
Sheffei 2017] Wang 2018). It's commonly used due to its simplicity and is closely related to the classical ordinary least square method. Motivated by Dwork & Lei| (2009), Alabi et al. (2022) shows that algorithms based on a robust estimator, such as median-basse estimator, perform better compared to the classical ordinary least square estimator on small sample cases. Similar to our work, Charest & Nombo (2020) uses simulation extrapolation (SIMEX), technique from the literature on measurement error (Carroll et al. (206)). to obtain a DP estimator for the regression coefficient. However, what differs from our work is that there is no mention of constructing confidence intervals in Charest & Nombo 2020). Agarwal et al. 202 D: Agarwal & Singh (2021) also mentioned the
connection between the measurement error model and differentially private mechanism. Differing from our work. Agarwal et al. (202T) focused on the setting where only covariates are perturbed with differentially private noises and on the goal of learning a predictive linear model using principal component regression. Similar to our work, Agarwal & Singh (202T) use the connection to make inferences the regression coefficient under more general and less structured setting, but the methodology is much more involved compared to our more simplistic approach. Lastly, Evans & King (2022) also uses the connection to obtain a consistent estimator of the regression coefficient, but without any mention of the confidence interval.
Using the Johnson-Lindenstrenn least square estimator and derived DP asymptotic confidence intervals for the regression coefficients. Differing from the additive DP noises used in our work, a random projection could potentially limit the usefulness of the synthetic dataset for other types of analysis. Instead of obtaining confidence intervals for the regression coefficient, [Barrientos et al. (2019) studies the DP hypothesis testing for the regression coefficient by perturbing the t-statistic. However, since the approach achieves DP through randomizing the t-statistics, each hypothesis testing will cost a portion of the total privacy budget and the total privacy budget can be exhausted quickly.
transform (Blocki et al. 2012). Sheffet (017) studies DP ordinary
Lastly, from the Bayesian perspective, Bernstein & Sheldon (009)) studies the DP Bayesian linear regression, which requires prior distribution for the regression coefficient, through releasing private sufficient statistics and thus is imperilled to the problem of privacy budget running out as described above.
2

---

Under review as a conference paper at ICLR 2023
Structure of the paper In section 2 state the necessary concepts related to differential privacy and measurement error. In section 3 we establish the connection between differential privacy and measurement error and working under regression setting, derive DP consistent estimator and DP asymptotic confidence interval for regression coefficient /3 using the tool from measurement error framework. In section [4] we conduct simulation to examine the performance of our DP estimator against the widely used sufficient statistics perturbation method (SSP) and show that the performance of our estimator is comparable
SSP estimator in some scenarios while being outperformed in others. Furthermore, we look at the coverage of our DP confidence interval compared with the naive Cls obtained from ignoring the DP noises, and demonstrate the issue with naive inference as not only are the naive Cls centred at the wrong value, but they also have shorter length than would be obtained with the true data (Carroll et al. 2006).
2 PRELIMINARIES
2.1 DIFFERENTIAL PRIVACY
We begin by going through some basics regarding differential privacy. The central idea around differential privacy that it gives the assurance that any sequence of query responses is "essentially" equally likely occur, independent of the presence absence of any individual (Dwork & Roth (2144). Naturally, the most basic concept differential privacy is the notion of neighhouring datasets where two datasets differ by one individual record.
Definition 2.1 (Neighboring datasets). Two datasets of the same dimension (same numbers of columns and rows) are called neighbouring datasets if they only differ in exactly one row/individual record.
In this paper, we are only concerned with approximate differential privacy, which is natural relax- ation of the original definition of E-differential privacy.
Definition 2.2 (Approximate differential privacy (Dwork al. 2006a b)). A randomized algorithm M (s, d)differenni private if for all (measurable) set S and for all neighburing datasets X and X
P(M(X) S) S exp(s))(M((X) E SS)+
One of the most important properties of differential privacy is its immunity to post-processing. That is, without any additional information the confidential dataset, it's impossible to make function of the output of a differentially private algorithm M any less differentially private. More precisely,
Proposition 2.1 (Post-processing property (Dwork & Roth 2014)). Let M be randomized algo- rithu that is (e, 8) diifferenntilll private. Let be an arbitrary randomized mapping with domain within the range of M. Then M is (c, -)dfffferentiall private.
Another fundamental notion in differential privacy is the idea of a query. A query is function to be applied to the dataset (Dwork & Roth (2014). Naturally, to achieve the same degree of privacy protection, different queries will likely require different amount of noise perturbation. To quantify this, we need the concept of sensitivity:
Definition 2.3 (le sensitivity). The l2 sensitivity of a query function f is defined as DS maxx.x [XX) f(X))l2 where the max is taken over all possible pair of neighboring datasets X and X'
(c, difffrrretl privacy can be achieved through the application of Gaussian mechanism,
Definition 2.4 (Analytic Gaussian Mechanism (Balle & Wang 2018).. Let f X - Rd be a function with global L2 sensitivity A. For any : 0 ò E [0, 1,, the Gaussian output perturbation mechanism (x)
fxx + Z with Z ~ N 00 1) is (e, (E.8)-DP DD if and only if
80
60
T
<
2
A
2x
A
where T is the cumulative distribution function for standard normal random variable.
3

---

Under review as a conference paper at ICLR 2023
2.2 MEASUREMENT ERROR MODEL
In simplest terms, measurement error problems can be described as the problem of making infer- ences about statistical model in terms of variable Z that is directly observable. Instead, a surrogate variable W of is observed, and inference must be made through W instead. The statistical models and inference methods are called measurement error models (Stefanski 2000).
A measurement error model consists of two parts, the first is the error structure relating the surrogate W to the truth Z, and the second is data structure of variable Z (Carroll et al. 2006). As an example, consider the following measurement error model,
W X + Y = (X;
(la)
(Ib)
where the U is assumed to have mean zero, constant variance and is independent of X. Similarly, assumed have Gaussian distribution with mean and constant variance and is independent of and U.
Model [T) above is referred error-in-variable model, where the covariates measured with error in a regression setting. Eq. (Ta) describes the classical measurement error structure, in which only the true (unobserved) covariate X is measured with additive error. Eq, [b) describes the re- gression structure of the data Z = {X, Y.. It reduces to the familiar linear regression structure for g(X: B) 3.
Remark There are other types of error structures such as multiplicative error, but in this paper, we will restrict ourselves to only additive measurement error.
In measurement error literature, there is important distinction between the functional model where is not modelled and the structural model where X is modelled with a parametric distribution. For this paper, we will restrict our attention to structural modelling where X is assumed to have Gaussian distribution.
Remark
Under 11 with 3) XTB, B, one of the most well-known effects of the measurement error to bias the regression coefficient towards zero. This phenomenon commonly referred to as attenuation. More precisely, the OLS estimator obtained by regressing Y on the surrogate W is not a consistent estimator of B but instead of 3* = X where
0
<1
(2)
The attenuation factor À is referred to as the reliability ratio (Carroll et al. 2006). The larger the of the variance of the measurement error, the closer zero the attenuation factor will be. Therefore, when ignoring the measurement error. the naive method of regressing Y on W will result in severe underestimation of when the magnitude of measurement error is large.
Based on equation 2 we can obtain consistent estimator of B as
Sw 3 3 Bats/ / B.ot S., à?
where S. is the sample variance of the surrogate W and a is a consistent estimator of o
So far we have only discussed the scenario where only the explanatory variable X is measured with error, but the scenario where both explanatory and response variables are measured with error is much more beneficial to the differential privacy framework. Although there is admittedly less lit- erature on response measurement error, the extension is surprisingly easy in some cases. As noted in Carroll et al. (2006), for unbiased and homoscedastic response measurement error in linear re- gression, the response measurement error increases the variability of the fitted lines without causing bias. Furthermore, all hypothesis tests, confidence intervals, etc. remain perfectly valid albeit they are less powerful. These conclusions indicate that unbiased eттor in linear regression requires no special adjustments when extending to response measurement error. However, for the binary sponse, the measurement error becomes misclassification, which is no longer an unbiased error, and therefore special considerations required. As the first steps establish the connections between
4

---

Under review as a conference paper at ICLR 2023
differential privacy and the measurement enror model, will focus on the linear regression mea- surement eπror throughout this paper. Nonlinear regression such as logistic regression will be left for future directions.
3 DIFFERENTIAL PRIVACY MECHANISM AS MEASUREMENT ERROR MODEL
3.1 GAUSSIAN MECHANISM AS MEASUREMENT ERROR MODEL
Let's denote the private dataset by Z, then the analytic Gaussian mechanism (sec.
2.1 releases
differentially private dataset by adding a centred Gaussian noise U N (0,
z= U
(4)
Note that A denotes the sensitivity for the identity query function, that is, A == maxz, I Z'Il where || || denotes the Frobenius norm.
Refer back to section 2.2 it's easy to observe that equation can be viewed as the error structure between the surrogate variable and the true unobservable variable of a measurement error model. A key difference here is that commonly measurement error problems, the magnitude of the measurement eπror o. the variance of U, is unknown and has to be estimated. Fortunately, in the differential privacy setting, the variance of is purely determined by the privacy budget e, and the sensitivity A, and therefore it can be publicized and is assumed to be known.
Remark For unbounded variables, like Gaussian random variables, A will be so and the Gaus- sian mechanism will no longer work without additional procedures. To deal with unbounded pre- dictors, we simply clip the variable within a fixed interval. To disclose A, the fixed interval must be chosen before without seeing the confidential dataset.
3.2 STATISTICAL INFERENCE FROM MRASUREMENT ERROR PERSPECTIVE
Now equipped with the perspective that the Gaussian mechanism can be viewed as the mcasurement structure of measurement еπтоr model, we need the second component of the mcasurement error model, the data structure, to make statistical inferences.
Let's consider the regression where partition the private dataset Z {X,y) where X is the exploratory variable and y is the response variable. Furthermore, we assume a functional relation- ship between X and the expected value of y.
y 9 x)
(5)
where q MO0 Ti) and it's assumed to be independent of X.
Combined with equation we can write our measurement error model as the following,
y = (X: +
Ż U
where Ž = (X, and (Uz, u,)) Note the variance of U is assumed be known. When u, has zero variance, then it reduces to model [ error-in-variable model, in section 2.2
Under model (6) one of the classical methods for estimation is the maximum likelihood approach (Wansbeek & Meijer 2000) due to several nice properties such consistency and asymptotic nor- mality that the maximum likelihood estimator (MLE) enjoys. Since only Z are observed, the likeli- hood function to maximize comes from the marginal distribution of 2, which is simply multivariate normal distribution. For some function g, numerical analysis required to maximize the likelihood and closed-fom solution often does not exist. However, in this paper, will focus scenario that a closed-for solution exists. That is, we will focus on the case that gX;; () = X T, in which case eq. (6) reduces to the following.
such
y = X q
Ż = Z + U
5

---

Under review as a conference paper at ICLR 2023
Denotes
1 x x
0g = S. of (1+ |++1 [
B
x
o
y
71
where S. n-k
x
To obtain the limiting distribution of these estimators, we have the
2
following theorem.
Theorem 3.1 (Fuller (1987)) Let model 7 holds, that is, assume homoscedastic linear re- gression model, additive measurement enror structure and normally distributed predictor. Let
and let i (3 0q Then,
n ( 0) N(O,T),
where the submatrices I are
T3 -Ma 0 + M 1o + M
09 Var
I Bg -2,,
with = u B and M, 1 + Eg. Furthermore, The variance of the approximate distribution of can be estimated by
M + +M= + M Var( =n
where M2 - 2
The theorem above is not valid if a clipping process is applied to Z to ensure finite sensitivity. Therefore, the clipped interval needs to be sufficiently large minimize the impact the clipping effect.
Remark
Directly following the theorem above,
derive the result for the simple linear regression, Y Bo + X8 + q, which will be used in the simulation in section 4
Corollary 3.1.1 (Simply linear regression Then, the vector
(Fuller 1987)) )) Suppose o known, of > 0. and of 0.
30 30 31
Vn
~ N (0,Tr
B1
where the covariance matrix I' is.
Cov (2,
-12 0.00 + Cov Cov2(2,
+ t
0 + Cov (2,
σ. 0 + Cov (x,
r
Ux
oo
oo
Furthermore, Var (Bo,
is a consistent estimator where
[B 33
mean( (x)) Var (1
+ +
-mean(X* X) Var (3.)
Var
-mean(X) Var
(3.]
Var (3.)
1 S2S, + 3/00
Var
B1
1 (Sx - o?)²
where S.
mean y) 31 (x mean (x))
(n
Note that 3 is the MLE of (3, but o, is not the MLE of Og is used here because the limiting distribution be derived under less restrictive conditions than those used to obtain the maximum likelihood estimator (Fuller 1987)

---

Under review as a conference paper at ICLR 2023
Immediately following from the corollary above, derive an asymptotic confidence interval for 31 as follows,
Corollary 3.1.2 (Asymptotic confidence interval), The interval is defined as follows
Bi + 112/22n-21 Var (i)
where f1-a/2,n-2 denotes the 1 1-a/2 quantile of the student's distribution with df n is 1 1-a a asymptotically correct confidence interval for the regression coefficient B1.
4 SIMULATION AND RESULTS
In this section, we perform simulations to evaluate the performance of our estimator against the widely used SSP algorithm (Dwork et al., 2014, Sheffei 2017] Wang 2018 Alabi et al. 2022). As the result will show that estimator comparable to the SSP algorithm in some scenarios. Furthermore, we will obtain an asymptotic confidence interval for 31 without additional privacy cost, which is one of the advantages of our approach. Compared to the naive CI obtained by ignoring the DP noises, our CI does a much better job capturing the true value for B1.
4.1 METHOD
For this simulation, we assume the following simple linear regression model,
Y Bo + B1 Xx + 9
Additionally, we assume de -N 1) and X, ~ (0.
To conduct the simulation, we set the coefficients to be (8n,3))=(11 and then draw X,, 1,2,... n from (N001) 1) and the regression noises qt, = 1.2..... n. from (N,0 1). Once Y; S + B + 4t, 1, .......... ni is obtained, we clip Y. within the interval [-3, 3] to ensure a finite sensitivity A. The particular interval of [-3,3] 3] is chosen since the interval is relatively large that the effect of clipping will not have a big impact on the result.
First, we will obtain the point estimators for 30 and 81 using the SSP algorithm. To implement the SSP algorithm, we follow the DPSuffStat: algorithm in Alabi et al. (2022) with a few adjust- DP synthetie dataset described in section 3.1 v(1 (11-00 0)2 (3 (-3))² 37,, and then obtain the estimates described in section To compare the performance between these two estimators, we report their median absolute er- (MAE) for each combination of sample size E {500, 1000, 2000, 5000, 10 1093 and privacy budget E {0.1, 0.5, 1,5] while setting 5 = 1/nn.
ments
To obtain our estimator, we first construct
with
3.2
Due to the post-processing property of DP, any statistics derived from the DP synthetic dataset will remain differentially private and won't 't incur any additional privacy risk. Therefore, the asymptotic CI describes corollary 3.1.2 is differentially private. Similarly, the naive CI obtained by ignoring the DP noises is differentially private as well. To compare our asymptotic CI with the naive CI, we report their relative frequencies of capturing the true value of B, out of the 1000 trials.
Lastly, the normal distribution assumption of the covariates might not be realistic in practice. There- fore, we rerun the simulation described above but with X, drawn from Unif(0, 1) instead to evaluate the performance of our method under a different setting.
4.2 RESULT
Table shows the MAE results between our DP estimator (bottom value) and the SSP estimator (top value). As can observe from the table, one might expect when privacy budget sample size n increases, the MAE for both estimator decrease. However, the SSP estimator outperforms our SSP estimator except when both sample size and privacy budget are large, their performances are similar. The lower performance of our estimator is due to the nature of the finite sample. In simulation,
First, the Gaussian mechanism is used instead of the Laplace mechanism for better comparison. Then, we extend the algorithm accommodate different clipping intervals for X, and Y.-

---

Under review as a conference paper at ICLR 2023
Table 1: MAE result for uniformly/normally distributed predictor. The top value within each cell indicates the MAE for the SSP algorithm, and the bottom value within each cell indicates the MAE for our estimator without applying the Gram-Schmil process.
Gaussian
Uniform
8 = 0.1
e 0.5
8
E =
5
8
0.5
8
8 5
2.139
0.592
0.342
0.148
6.497
1.884
1.040
0.301
n = 500
7.105
2.871
2.029
1.117
5.400
2.504
2.039
1.757
1.218
0.334
0,205
0.132
3,938
1.001
0.568
0.183
1000
5.929
2.476
1.862
1.018
4.557
2.310
1.889
1.735
0.699
0.200
0.140
0.127
2.212
0.557
0.307
0.121
n 2000
5.026
2.185
1.652
0.838
3.963
2.114
1.837
1.652
0.324
0.131
0.127
0.128
0.984
0.244
0.147
0.090
n 5000
3,976
1.876
1,508
0.681
3.218
1.893
1.801
1.596
0.198
0.128
0.129
0.128
0.551
0.138
0.097
0.080
T = 104
3.401
1,711
1.392
0.574
2.858
1.815
1.805
1.481
0.128
0.128
0.128
0.128
0.088
0.075
0.074
0.074
10
2.048
1.339
1,111
0.276
1.978
1.803
1.726
1.151
the sample covariance between DP noises and data often non-negligible when the sample size is small relative to the amount of noises injected. This results in poor estimation of ag which leads to the poor performance of our estimator. Although our estimation performance is worse than the SSP method, it's still comparable in some scenarios (the combination of a small privacy budget and small sample size or the combination of large privacy budget and a large sample size). Furthermore, our approach allows the release of synthetic dataset and more importantly, provides the method to obtain confidence interval without additional privacy budget. We will discuss the performance of our confidence interval next.
Figure [ show the coverage probabilities and margin of eтror of our confidence intervals (DP) under normally distributed and uniformly distributed covariate X. For comparison, the naive CIs derived from the synthetic dataset (naive) and non-DP CIs (non-DP) derived from the confidential dataset are also plotted. As shown both figures, the coverage of our CIs relatively close to the nominal level (90%, indicated by the dotted lines). In comparison, the coverage of the naive CIs never captures the true value of 31 even though they are much narrower in comparison. This highlights the importance of considering DP noises when making statistically valid inferences. The reason behind the terrible coverage of the naive Cls, as explained in Carroll et al.] (2006). is because the variance the naive estimator be smaller than the true data estimator when the privacy budget is small (or DP noises are large), which results in a more precise, but biased estimator.
CONCLUSION
In this paper, we established a connection between DP mechanisms and measurement error mod- els. Applying the tools from the measurement eтror framework, developed statistical inference under linear regression while preserving differential privacy. In particular, we derived DP consis- tent estimator and DP asymptotic conference interval for the regression coefficients. To evaluate the performance of the estimator, we compared it to the widely used SSP method and demonstrated our estimator has comparable performance in some scenarios but has the advantage of obtaining statistically valid asymptotic confidence intervals without additional privacy cost. Furthermore, by comparing the coverage between our asymptotic CIs and naive Cls, we illustrated the importance of incorporating the DP mechanism into the inference method to ensure valid statistical inference.
For future directions, some theoretical works on the comparison between our estimator and the SSP estimator could be interesting direction. Similarly, the extension of the theorem 3.] to accommodate the clipping might be a fruitful path to pursue. Furthermore, there are still many tools from the measurement emrr literature yet to be utilized. One of the obvious next steps would be to extend the linear regression setting to the more general generalized linear model setting such as logistic regression. We hope this paper will motivate future works to explore more the connection
8

---

Under review as a conference paper at ICLR 2023
eps: 0.5
eps: 0.5
coverage probability
0.75
C
margin of error
2
CI
DP
0.50
Naive
1
Naive
0.25
Non-DP
Non-DP
0.00
0
sample size
sample size
eps: 5
eps: 5
coverage probability
0.75
CI
margin of error
CI
DP
DP
0.50
1.0
Naive
Naive
Non-DP
0.5
Non-DP
0.00
0.0
sample size
sample size
eps: 0.5
eps: 0.5
coverage probability
C
margin of error
Cl
2-
DP
DP
0.50
Naive
1
Naive
0.25
non-DP
non-DP
0.00
sample size
sample size
eps: 5
eps: 5
coverage probability
0.75
margin of error
2-
Cl
0.50
DP
DP
Naive
1
Naive
0.25
non-DP
non-DP
0.00
0
sample size
sample size
Figure The coverage probabilities comparison between our DP confidence intervals, naive CIs and non-DP Cls various sample sizes. The top 4 plots for the normally distributed covariate, and the bottom 4 plots are for the uniformly distributed covariate. Note the horizontal axis in logarithmic scale with sample size 500, 1000, 2000, 5000, 10 10a. The dotted line indicates the nominal CI level of 0.9.
between differential privacy and measurement error, and to develop statistical inference under the differential privacy setting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anish Agarwal and Rahul Singh. Causal inference with corrupted data: Measurement error, missing values, discretization, and differential privacy. 2021. doi: 10.48550/ARXX222100027206 URL
hitps://arxiv. org/ abs 200. 02780
Anish Agarwal, Devavrat Shah, Dennis Shen, and Dogyoon Song. On robustness principal com- ponent regression. Journal of the American Statistical Association, 1166536:1731--7455 2021.
Daniel Alabi, Audra McMillan, Jayshree Sarathy, Adam Smith, and Salil Vadhan. Differentially private simple linear regression. Proceedings on Privacy Enhancing Technologies, 2022:184 204, 04 2022. doi: 00124788popets-2222-0000
Borja Balle and Yu-Xiang Wang. Improving the Gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. Jennifer Dy and Andreas Krause (eds.), Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pp. 394-403, PMLR, 10-15 Jul 2018. URL https:
//proceedings ml prw.s.essssp...... v80 ba wawlellalllll.lla html
Andrés F. Barrientos, Jerome Reiter, Ashwin Machanavajhalaa and Yan Chen. Differentially private significance tests for regression coefficients. Journal of Computational and Graphical Statisticy, 28(2):440-453: 2019, doi: 10100000608000.00000 26000000181138888) URL https://doo org/ 1tt.:/100100011111111111
2018 1538881
Garrett Bernstein and Daniel Sheldon. Differentially private bayesian linear regres- sion. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httpa///////eeeaaaoooa////////// 150f2acab6610288a222211 httpa//aa2227710c2000-333-7-0000 29722004662230023311133333323233 pdff
Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet. The johnson-lindenttraus trans- form itself preserves differential privacy. 2012 IEEE 53rd Annual Symposium Foundations of Computer Science, pp. 410-419, 2012.
Claire McKay Bowen and Fang Liu. Comparative study of differentially private data synthesis methods. Statistical Science, 35(2), may 2020. doi: 10.1214/19-sts44.. URL /attps://dol.
st
Raymond J. Carroll, David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu. Measure- ment Error in Nonlinear Models: A Modern Perspective. Chapman and Hal//CRC, 2nd edition, 2006.
Anne-Sophic Charest and Leila Nombo. Analysis of differeetialll--rrivte microdata using SIMEX. In Josep Domingo-Ferre and Krishnamurty Muralidhar (eds.), Privacy in Statistical Databases, pp. 109-120 120, Cham, 2020. Springer International Publishing. ISBN 978-3-030-57521-2.
Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. Proceedings of the Forty- First Annual ACM Symposium Theory of Computing, STOC '09, pp. 37 371-380, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585062. doi: 10.1145/
1536414.1536466. UtL /ttp:///../..../////////////..// 1et..o..111111111 1536466
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor Comput. Sci., 9(3 4):211 407, aug 2014. ISSN 1551-305X. doi: 10.1561/040000004..
URL 2ttp:/00....01.0....55.50.......
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Serge Vaudenay (ed.). Advances in Cryptology EUROCRYPT 2006, pp. 486-503, Berlin, Heidelberg, 2006a. Springer Berlin Heidelberg. ISBN 978-3-540-34547-3.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, pp. 265-284, Berlin, Heidelherg. 2006b. Springer Berlin Heidelherg. ISBN 978-3-540-32732-5.
10

---

Under review as a conference paper at ICLR 2023
Cynthia Dwork, Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Analyze gauss: Optimal bounds for privacy-preserving principal component analysis. In Proceedings of the Forty- Sixth Anmual ACM Symposium Theory of Computing, STOC 14, pp. 11-20, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 978 4781255077 doi: 10.1 10.11449/9919996255999 2591 883
URL.tttss://ddb /ttp://66.....111111111111111199
Georgina Evans and Gary King. Statistically valid inferences from differentially private data re- leases, with application to the facebook urls dataset. Political Analysis, pp. 1-21, 2021 2022.
URL httph:ttp///c...t. htpp:://www.ddeo wtt.d/..og//1111///ppaaaaa.. 2022.1
Georgina Evans, Gary King, Margaret Schwenzfeier, and Abhradeep Thakurta. Statistically valid inferences from privacy protected data. Working Paper.
Wayne A. Fuller. Measurement Error Models. Wiley, 1987.
Sébastien Gambs, Frédéric Ladouceur, Antoine Laurent, and Alexandre Roy-Gaumond. Growing synthetic data through differeetially--rivate vine copulas. Proceedings on Privacy Enhancing Technologies, (3):122-144) 11122-141,-0 2021.
Fang Liu. Model-based differentially private data synthesis and statistical inference in multiply synthetic differentially private data, 2016.
wwwpxii..rg/ab///10000 08052
Or Sheffet. Differentially private ordinary least squares. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICMI' 17, pp. 3105-3114, JMLR.org, 2017.
L. A. Stefanski. Measurement error models. Journal of the American Statistical Association, 95(452):1353-13355 2000. ISSN 01621459. URL http:/ www. jtt.tsstrgg..........oo....s..te ort.oooorrgsttabll.o..
2669787
Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty- Thirty-Fourth Conference Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 93-103. AUAI Press, 2018. URL http: /aaai. org/ ua 201
htwpaaaa11111ooeeeeeee w/wcceed htss/apeerss. pdd
Tom Wansbeek and Erik Meijer. Measurement Error and Latent Variables in Econometrics. Elsevier Science B. V. 2000.


Reviews:
[{"review_id": "mah9B_wgTIF", "reviewer": "Reviewer_cBN5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses a connection between differentially private linear regression and statistical inference under measurement error. The main technical contribution is to use this connection to apply prior results from the measurement error model to DP linear regression.", "summary_of_the_review": "The contribution of the paper is to apply prior results from the measurement error model to differentially private linear regression, so it does not introduce new tools, techniques, or proofs. The empirical results are unconvincing due to the unrealistic assumption on orthogonality of noise/data covariances.", "strength_and_weaknesses": "## Strengths\n- The paper is clearly written.\n- Empirical experiments show an improvement in terms of estimating confidence intervals.\n\n## Weaknesses\nI primarily have the following questions/concerns:\n1) The empirical study suggests that there is an improvement in average error only when the covariance matrices of the data and the added noise are orthogonal. However, this does not seem to be a realistic scenario, and I fail to see what conclusion is drawn from this experiment. Can you clarify why it's reasonable to assume the covariance matrices to be orthogonal?\n2) There is no theoretical comparison to prior methods in terms empirical or population risk. In particular, different variants of the sufficient statistics perturbation method have been studied under different scenarios (Gaussian design, fixed, bounded and unbounded designs). It will be important to understand how the proposed method compares to existing methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kFK32IbRWr", "reviewer": "Reviewer_ffBc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper uses the DP synthetic data, generated by adding Gaussian noise (Gaussian mechanism) to the sensitive data, to model the analysis using the measurement error approach. Simulations show that the proposed method works well for linear regression.", "summary_of_the_review": "I hope the authors can answer my questions regrading the implications in real world scenarios, I am open to increasing my score after rebuttal.", "strength_and_weaknesses": "The proposed method draws interesting parallels between the measurement error theory and DP synthetic data and shows how one can account for the other to generate good estimators. I like the idea, but I think the limitations of the proposed method need more discussion. Such as, how applicable is it in the real world? especially when the input data to a regression model does not follow any nice properties as ensured by simulations. What happens when sensitivity is large (i.e. it is not possible to constrain data in [0,1] without destroying utility)? \n\nI do understand that the proposed method is claimed to be the first to draw this connection, so empirical evaluation comes in second, but I do think it requires more extensive experimentation. Especially, comparing (either via experiments or by including a good discussion), on the advantage of using this compared to generating synthetic data using say TableGAN, and then doing a normal regression.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cSy_-iE5tjW", "reviewer": "Reviewer_cuVk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of statistical inferences from differential private synthetic datasets. The authors use the tools of measurement error models to derive an estimator that is bias-corrected. Experiments are performed that compare the proposed estimator to SSP (sufficient statistic perturbation), which show that the superiority of SSP in practice, but a better performance of the proposed estimator if the sample covariance was 0.", "summary_of_the_review": "The paper proposes a connection between DP and measurement error models. While I think it is a new connection, the results seem limited and underperform the proposed baseline. I am not knowledgeable about measurement error models so I might miss some aspect of it, but overall I would lean towards reject.", "strength_and_weaknesses": "Strengths\n- The paper tackles an interesting problem, namely statistical inference using DP synthetic datasets\n- The paper makes the connection to measurement error models which is to the best of my knowledge new\n\nWeaknesses\n- The proposed estimator seems to be worse than the baseline in practical scenarios\n\n\nTypos\n- The title ends with \"conference submissions\"\n- Missing \\ref in Definition 2.4", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Ipon2dyxRv", "reviewer": "Reviewer_3s3F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the standard error-in-variables model: Y=X’b+q and tilde{X}=X+U. The authors impose that X, U, and q are homoscedastic Gaussians with fixed dimensions. Then the standard attenuation bias formula can be inverted to correct for the attenuation bias, if the variance of q is known. The authors quote estimation and inference results from Fuller (1987). The stated contribution is twofold: \n1) establishing the connection between measurement error models and additive differential privacy mechanisms\n2) using this connection to repurpose known inference results to the differential privacy setting\n\nMy primary dissatisfaction is the framing of these results in the context of previous works which make one or both of these points.", "summary_of_the_review": "The main result is essentially an interpretation of the standard attenuation bias formula. The shortcomings in terms of the clarity, the gap between theory and practice, and the inadequate framing limit the significance of the contribution. I will improve the score if these shortcomings are addressed.", "strength_and_weaknesses": "The strength is that the proposed solution is very simple. The weaknesses are described as actionable items below", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject