Simulation-based Bayesian Inference from Privacy Protected Data
Authors: Yifei Xiong
Nianqiao Ju
Sanguo Zhang
Venue: Accepted by TMLR ()

Abstract: Many modern statistical analysis and machine learning applications require training models on sensitive user data. Under a formal definition of privacy protection, differentially private algorithms inject calibrated noise into the confidential data or during the data analysis process to produce privacy-protected datasets or queries. However, restricting access to only privatized data during statistical analysis makes it computationally challenging to make valid statistical inferences. In this work, we propose simulation-based inference methods from privacy-protected datasets. In addition to sequential Monte Carlo approximate Bayesian computation, we adopt neural conditional density estimators as a flexible family of distributions to approximate the posterior distribution of model parameters given the observed private query results. We illustrate our methods on discrete time-series data under an infectious disease model and with ordinary linear regression models. Illustrating the privacy-utility trade-off, our experiments and analysis demonstrate the necessity and feasibility of designing valid statistical inference procedures to correct for biases introduced by the privacy-protection mechanisms.

Published in Transactions on Machine Learning Research (03/2025)
Simulation-based Bayesian Inference from Privacy Protected Data
Yifei Xiong Deportment of Statistics Purdue Universily
ziong (innggg9@@uuu.e.cm
Nianqiao Phyllis Ju
nianqiao Daprrduc..ppruuuccccom eduu
Department of Statistics Purdue University
Sanguo Zhang
sgzhang@ucas..oomm
School of Mathematical Sciences University of Chinese Academy of Sciences
Reviewed on OpenReview: https: // openreview. net/ forum? id- SB7J2ND445
Abstract
Many modern statistical analysis and machine learning applications require training models on sensitive user data. Under formal definition of privacy protection, differentially private algorithms inject calibrated noise into the confidential data or during the data analysis process to produce privacy-protected datasets queries. However, restricting access to only privatized data during statistical analysis makes it computationally challenging to make valid statistical inferences. In this work, we propose simulation-based inference methods from vrivacyprrotecter datasets. In addition to sequential Monte Carlo approximate Bayesian computation, we adopt neural conditional density estimators as a flexible family of distributions to approximate the posterior distribution of model parameters given the observed private query results. We illustrate methods discrete time-series data under infectious disease model and with ordinary linear regression models. Illustrating the privacy-utility trade-off, our experiments and analysis demonstrate the necessity and feasibility of designing valid statistical inference procedures to correct for biases introduced by the privacy-protection mechanisms.
1 Introduction
Motivation. Many Al systems require collecting and training massive amounts of personal information (such as income, disease status, location, purchase history, etc.). Despite unprecedented data collection efforts by companies, governments, researchers, and other agencies, oftentimes, data collectors have to lock the data inside their own databases due privacy concerns. Differential privacy (DP) provides a mathematical definition for the protection of individual data (Drechsler al., 2024). Under this framework, crivacy-protectin procedures (i..,, DP algorithms) have enabled data collectors, such as tech companies, the US Census Bureau, and social scientists, to share research data in wide variety of settings while protecting the privacy of individual users. Privacy rescarchers typically collect confidential data and then inject calibrated random noise into the confidential data to achieve the desired levels of privacy protection. Some algorithms aim for DP data analysis, resulting in DP optimizations (Arora et al., 2023; Bassily et al., 2021), approximations (Chaudhuri al., 2013; Bie al., 2023), or predictions (Rho al., 2023); while other algorithms produce DP datasets (or descriptive statistics), which enables data sharing across research teams and entities. Examples of the latter include the 2020 US Census (Abowd et al., 2022; Gong et. al., 2022;

---

Published in Transactions on Machine Learning Research (03/2025)
Drechsler, 2023), the Facebook URL dataset (Evans & King, 2023), and New York Airbub Open Data (Guo & Hu, 2023). Our work tackles this data-sharing regime: we aim to make valid statistical inferences with privatized data, and we place a special focus on using complex models such as contimuous-time Markov jump processes.
Although the DP data-sharing regime corrupts confidential information in order to satisfy privacy, since the probabilistic design of such mechanisms can be publicly known, in principle analysts can still conduct reliable estimation and uncertainty quantification by accounting for bias and noises introduced during privatization. However, in practice, valid inference based privatized data is challenge that requires the revision of existing statistical methods designed originally for confidential data (Foulds et al., 2016). Even for well-understood procedures such as ordinary linear regression and generalized linear models, adding the extra layer of privacy protection has introduced new theoretical and methodological questions in statistics (Cai et al., 2021; Alabi & Vadhan, 2022; Li et al., 2023; Barrientos et al., 2019).
However, for complex models, even the confidential data likelihood functions are intractable or time-consuming to compute. Accounting for privacy noise top of that is a formidable challenge. Without developing valid inference procedures under this regime, we must either make biased estimations or restrict ourselves to simple models. In this work, we propose methods to estimate the parameters of complex models that underlic privacy-protected data.
Related works. There is a fast-growing literature on statistical inference under differential privacy. Early work by Williams & McSherry (2010) first proposed using Bayesian inference to handle DP noise. Since then, various Bayesian methods have been developed for privatized data. Markov Chain Monte Carlo (MCMC) methods have been proposed in some specific models and priors, such as exponential family distributions (Bernstein & Sheldon, 2018), Bayesian linear regression (Bernstein & Sheldon, 2019) and generalized linear models (Kulkarni et al., 2021). As for generic algorithms, Ju et al. (2022) proposed data- augmentation MCMC strategy to overcome the intractable marginal likelihood resulting from privatization, and Gong (2022) derived point estimates of the posterior distribution using the Expeetation-Maximiztt algorithm. Several frequentist inference methods have also been developed. Karwa et al. (2015) employed a parametric bootstrap method to construct confidence intervals for the model parameters of the log-lincar model. Awan & Wang (2023) proposes simulation-bant inference methods for hypothesis testing and confidence intervals, for frequentist. inference.
To the best of our knowledge, only three papers (Waites & Cummings, 2021; Lee al., 2022; Su et al., 2023) have incorporated normalizing flows (Kobyzev et al., 2020; Papamakarios et al., 2021) and DP, and both works design DP-versions of normalizing flow. In contrast, our work uses flow-based mothods as neural density estimation tool to analyze DP-protected data.
Our contributions. In this work, we propose several likelihood-free inference methods that make statistical inferences from privacy-protecte data. First, we highlight that sequential Monte Carlo approximate Bayesian ionputation (SMC-ABC) can be used for this purpose, which improves the current practice of using ABC (Gong, 2022). Next, we propose sequential private posterior estimation (SPPE) and sequential private likelihood estimation (SPLE), two soquential neural density cstimation methods to approximate the private data posterior distribution with normalizing flows. Unlike likelihood- based methods to learn from private data, SMC-ABC, SPPF, and SPLE require only simulations from a generative model for confidential data, and hence are applicable to more models. We demonstrate the efficiency of our methods on infectious discase model using synthetic and several real discase outbreak data. SPPE and SPLE require fewer numbers of simulations from the data model to achieve the same inference results compared with SMC-ABC. We also propose a privacy mechanism for the release of the infection curve. Our experiments also demonstrate the privacy-utility trade-off in linear regression, where we compared the proposed likelihood-free methods with data augmentation MCMC, a likelihood- based inference method. The code is available on GitHub'.
hhttpe://github http://tttttccc////feee-iiio/oao based- BaaaddBByesiia--TTeeeeeee Iww.rrneeeeeerrrrneeeeee Pro.arr-riiivvccc-otttteeeeeeee
2

---

Published in Transactions on Machine Learning Research (03/2025)
2 Background and challenges
Let E - be the model parameter and z = (.1,-.., ) E X" represent the confidential database, where n is the sample size. We model the database with some likelihood function f(x 0). This confidential data model can be 'simulator' whose likelihood can be impossible to compule. In the confidential data setting, our interest to approximate the posterior distribution x(0) x²) x (0)f(22) 0) where x" is the observed dataset and π (0) is the prior. In this section, we first review some key approaches for likelihood-free inference in the confidential data setting and then describe output perturbation mechanisms to generate differentially private queries Sdp- Finally, we explain why learning 0 given observed sap is challenging. We will present new methods for likelihood-ffee inference in the privatized data setting in Section 3.
2.1 Likelihood-free free inference for confidential data
Likelihood-free inference methods require only the ability to generate data from simulator model, instead of evaluating ils likelihood.
Approximate Bayesian Computation (ABC) and scquential Monte Carlo ABC (SMC-ABC).
ABC-based methods circunvent likelihood evaluations with simulations. In its most basic form, a set of parameters are sampled from the prior g(X) x00) and a data set x(0) is simulated from the model for each 00), The samples ((0) such that 2 is close to the observed data x" provide an approximation to the posterior (( z²). It is also common to choose some summary statistics s and to retain samples according to distance between each 8(x(+) and s(x"). The efficiency of ABC is controlled by the closeness between the prior distribution, which can be viewed as its proposal distribution, and the posterior distribution, which is the target distribution. SMC-ABC (Sisson et al., 2007; Beaumont et al., 2009) improves ABC by specifying a sequence of intermediate target distributions between the prior and the posterior, and thus the g( samples can gradually evolve towards the target distribution. The sequential updates in SMC-ABC propagate and reweight the parameter samples with importance sampling.
Neural density estimation. In contrast to sampling-based approximations like ABC and SMC-ABC, we can pursuc density approximation of (0 | x²) with neural networks. We illustrate a neural cstimator for the posterior density =0 x)). Let (0 x)} denote the collection of densities representable by an architecture q and weight parameters d. We can train the neural estimator to minimize an ideal loss function
e = arg min Ep(e..) |- log (0 x)] = arg min / Soxx log loggg(0 x)] (0)f(x 0)d6dz.
This loss is an expected value of - log do under the joint distribution p(f, 2) (0)f 6). When f(x () is intractable, the expeclation is also intractable, and training relies on Monte Carlo approximations of the integral using samples ((0(5) (,,(())) In sequential neural density estimation, we can sequentially improve the Monte Carlo estimation of the loss function (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg et al., 2019). Later, in Section 3.4, we will use randomized quasi-Monte Carlo (RQMC) methods (Owen, 1997a;b) as a drop-in replacement for standard Monte Carlo to reduce variance. Other neural density approaches include approximating likelihood functions (Papamakarios et al., 2019) or likelihood ratios (Miller et al., 2022). We refer the readers to (Cranmer et al., 2020; Lueckmann al., 2021) for systematic reviews of neural density approximations.
A notable neural density estimation method is normalizing flows (NF) (Dinh et al., 2017; Papamakarios et al., 2017; Durkan et al., 2019; Papamakarios el al., 2021), which are adopted in our experiments. NT start with a simple base density (e.g. multivariate Normal) and push it through a sequence of invertible and differentiable transformations. The resulting distribution richer and more complex. We choose NF because of their expressive power and popularity in probabilistic modeling.
2.2 Differential privacy
Given confidential data v, let r be a randomized algorithm to produce a 'differentially private statistic' Sdp from æ. We also use SS(S a:) to denote the conditional density of the private output Sap given the
3

---

Published in Transactions on Machine Learning Research (03/2025)
confidential data Intuitively, procedure is private when perturbing individual's response in the dataset leads to only small change in the algorithm's outcome. This is characterized by the e-DP definition from Dwork et al. (2006), based on neighboring databases.
Definition 1 (c-D)). We say x, x' E X are 'neighboring' databases if they differ by one and only one data record. Denote this by d(x,,²) S A privacy mechanism with conditional density n satisfies c-DP if, for all possible values of sdp and for neighboring dalasets, the following probability ratio is bounded:
LA (ssa x) dsap S exp((e), A. C Range(n).
(1)
JA ((sap x') dsap
The parameter e referred to as the privacy loss budgel, and plays a pivotal role in determining the extent to which Sup discloses information about z: Larger values of e correspond to reduced privacy guarantees, whereas 0 signifies perfect privacy.
Outpul perturbation methods achieve privacy by first computing a query X - S (such mean, median, histogram, contingency tables) of the database and then releasing s(x) with added noise. To satisfy e-DP, the query must have finite sensitivity.
Definition (Global sensitivity (Nissim et al., 2007)). The Ipp sensitivity of a function 8, denoted Ap(s), is the maximum 4y-norm change in the function's value between neighboring databases i and x', namely
(p(s)
max d(z,z)==1
||s(x)) s(x')ll
A common output perturbation technique is the Laplace mechanism. Proposition 3 (Laplace mechanism (Dwork et al., 2006)). For real-valued query X + S, adding zero-centered Laplace noise with parameter A achieves e-DP.
Chaudhuri et al. (2013) provides multivariate version of the Laplace mechanism. Other mechanisms include the exponential mechanism and the Gaussian mechanism (Liu, 2018). There are also relaxations of c-D, such (e, 6DD and Gaussian DP (Dong al., 2022).
We exploit output perturbation on summary statistics in two ways. From an inference perspective, when s(x) is sufficient statistic for 6, the confidential data posterior T( x)) is fully characterized by s(x) and the prior. This is the rationale for choosing T( sap) as the target distribution and for the posterior from perturbed data to still informative about the model parameters. From a computation perspective, since many summary statistics are of lower dimension than the confidential data, this can facilitate efficient computations. In Scction 3.4, we leverage the low dimensionality sap by incorporating quasi-Monts Carlo techniques.
2.3 Bayesian inference with privatized queries
In the private data setting, our goal is to approximate the posterior distribution of 8 given privatized data 8dp- A key challenge with data analysis on privatized data via x =(0 sap) is the intractable private data marginal likelihood.
10)= f(sap 0) = ffc f(x ())ssap x)dr.
(2)
When the confidential data likelihood f(r 0) can be evaluated, Ju et al. (2022) proposed data-augmentattin MCMC algorithm to approximale the doubly-intractable private data posterior
<(0)
sap) x f(Sap ()).
(3)
The data augmentation strategy circumvents the intractability of evaluating Equation (2) by working with the joint posterior distribution p00,x x (ap) & f(r (f))((((sap)) x)) instead of the marginal posterior in Equation (3).
In this work, we study the challenging scenarios when the confidential likelihood f(r () is intractable. This situation is 'triply intractable', as there are thrce levels of intractability in f(x (), f(Sap ()), and T(0 sap).
Data augmentation MCMC, which includes an inference step to sample from x( x), is no longer applicable in the trrplyy intractable' setting.
4

---

Published in Transactions on Machine Learning Research (03/2025)
3 Proposed methods
We present methods for likelihood-free inference for the triply intractable posterior distribution x(0) sap). First and foremost, we recognize that SMC-ABC methods are viable solutions to approximate the private data posterior, as long as the privacy mechanism publicly known and can be replicated by the data analyst.
Next, under the same assumptions, we focus on two neural density approximation methods that leverage state-of-the-art simulation-basee inference methods and can be more efficient than the SMC-ABC baseline. We present two complementary approaches: a) Section 3.2: approximate the private data marginal likelihood (sap () in Equation (2), and b) Section 3.3: approximate x( sap) from Equation (3) directly.
3.1
Adapting SMC-ABC to the private data setting
A data analyst can create a (hierarchical) private data simulator for Sdp by combining the confidential data simulator x ~ f( () and a tractable output perturbation mechanism (o.g. the Laplace mechanism) with sdp ~ 77( | z). With this simulator for $dp, a rejection ABC algorithm can simulate parameter samples k=a..... from the prior, and then keep samples correspond to simulated private query d close to the observed value <ap. This insight has been exploited in Gong (2022), using ABC to approximate T( sap)- We point out that SMC-ABC can also be used for this purpose. We include a description of the algorithm in Appendix C for completeness.
Both SMC-ABC and ABC aim to target the exact posterior distribution, unlike neural density estimation methods which insist that the approximation comes from some parametric family 9a- For this reason, we will use SMC-ABC as baseline to test and validate our methods in Section 4.
3.2 Private data likelihood estimation
Our first neural density approximation strategy is to approximate the private data marginal likelihood (sap 0) with neural network denoted by (o(sap 0). When training (o(sap () f(sap (), we aim to minimize their average KT, divergence under the prior r(),, corresponding to minimizing
Bn(() [DKL (f(sap (sap [0)]]
(4)
After some derivations (in Appendix A), Equation (4) is equivalent to Epp(..ap) [- log (s 0) up to a constant independent of Φ. The expectation taken with respect to the prior and the private data generating process p(0, 6ap) = x0) s'(aap 0), which is int ractble. To facilitate computations, let 'ss write Equation (4) with respect the confidential data generating process, resulting in
(p) = Ep(0.2) |-
[ n(sap log ga (sap 0)dsap
(5)
With T arg min (p) the resulting private data likelihood estimation is ug(#ap | 0). Since Equation (5) is an expectation with respect to tractable distribution, we can approximate the integral with Monte Carlo methods, discussed in Section 3.4.
When the primary inference goal is the maximum likelihood estimator, one can approximate it with M = arg maxe (sap 8). Under the Bayesian paradigm, the posterior approximation of Equation (3) can be (f) x T (0)g 0). Quantitics such as posterior median, mean, and credible regions can be estimated accordingly.
3.3 Private data posterior estimation
Now we approximate the private data posterior =(0 sap) in Equation (3) directly with neural posterior
estimator (f Sap), bypassing the synthetic likelihood step in Section 3.2. To find (o(0 sap) as =(O sap), let us minimize their KL divergence
log (0 sap)
DKL (T(0 (ap) sap)) = (0leap))
qa ( sap)
5

---

Published in Transactions on Machine Learning Research (03/2025)
As shown in Section A, this is equivalent to
[- mrsap I x)) (o sap)dsap
(Ø) = Ep(0.r)
(6)
Many Bayesian problems use uninformative priors (0), which are dispersed in the paramcter space. As a result, when employing a naive Monte Carlo strategy that uses the prior as the proposal to approximate expectations in Equation (5) or Equation (6) has most of its samples falling in low-density regions, leading to inefficient estimation due to high variance. To address this issue, importance sampling utilizes a proposal distribution P change the bases of integration and thus reduce the variance of numerical integration. The automatic posterior transformation (APT) method (Greenberg et al., 2019) uses p(0, = f(0) |0) proposal and has (unnormalized) importance weights
p(0)
( sap) (0 sdp) x(0)
(7)
Based on this idea, we design a modified loss of Equation (6) that leverages the APT framework to improve efficiency in posterior estimation. Specifically, we propose the following loss function
(ppe- (6) = [x00,.) [- n(sap x) log (o sap)
(8)
and it is useful for the sequential training stralegy we introduce Section 3.5. The derivation of Equation (8) is Appendix A.
3.4 Nested RQMC estimators
We can inspect the general form of loss functions in Equations (5) and (8) with
((6) -Ep(0.2) [nteap x)g((ap: d)
(9)
Here g(Sap, 0) = log (0) sap) for PPE-A and g(sap, 0) = log go (sap 0) for PLE. Equation a double integral, where the outer expectation with respect to some proposal distribution p and the inner integral involves the privacy mechanism y) and the neural estimator qo.
With independent and identically distributed (i.i.d.) samples from the joint density (000))(1))) p(0, = f()) f), we can unbiasedly approximate Equation (9) with
[ n(sap I 1x(0) 8(0) dssap
Mo = Σ
(10)
The inner integrals I(0,x)) f (spp #)g(sap. 0)dsap can be approximated with standard Monte Carlo integration lechniques, as popular DP mechanisms such as Laplace and Gaussian mechanisms be easily simulated. Using M i..d. samples, the not-mmea--suuared-eroo (RMSE) to estimate I(0, x) approximations are typically on the order of (MM-1/2 due to the central limit theorem.
In many applications, the DP query result 8dp serves as a private descriptive statistic of a dataset x. This statistic is commonly embedded in a low-dimensional space S, where the dimension r == dim(S) significantly less than the dimension of the data space dim(X"). For the privacy mechanismss, we typically model the generation process as J (z,)(z)) + sap where u 2[00, 1]" is a uniform random variable from the r-dimensional hypercube. For example, the process Tuu, s(x) s(x) Av(o) sgn(u - 2) log 2 1 1 achieves e-DP for 1-dimensional queries.
Randomized quasi-Monte Carlo (RQMC) methods, as detailed in (Owen, 1997a;b), differ from traditional
Monte Carlo (MC) methods in that it generate correlated, low-discrepancy sequences (u(").. "",,...,,,,))) (M0) c [0. 1)" These sequences cover the parameter space more evenly than the pseudo-random sequences used by MC

---

Published in Transactions on Machine Learning Research (03/2025)
methods. This low-discrepancy feature helps to reduce the variance of the estimators, which is particularly useful in low-dimensional integration tasks (I. 'euyyer 2018). The estimator used in RQMC can be described as
M Σ 1=1
M Σ ==1
fRQMC 0.0 M
1
(z))
1 M Jo (v
(11)
The RQMC estimator is unbiased and has smaller RMSE compared to MC estimators. The +discrepannc of the point set {o(I:M), denoted by (*(((I:M) is of the order O(M My)) = O(M-1+5) for positive con- stant ó. If our neural approximation family (.) has bounded Hardy-Krause variation VHK (+)] (Aistleit- ner et al., 2017), then, according to Basu & Owen (2016), the mean squared error of the RQMC estimator in Equation (11) satisfies
( - 1(0,,))) S VHI((O,,) O(M
(12)
Thus, the RMSE of the RQMC estimator is of the order O(M-1+*), achieving faster convergence than an MC estimator with (MM-1/2). We verify this improvement in convergence from the RQMC estimator on the SIR model and linear regression example with the neural spline flow approximation family (Durkan et al., 2019), in Appendix E, Figure
3.5 Sequential neural estimations
This section presents sequential neural approximation methods privacy-protected data. Our central goal is to approximate the private data posterior distribution =(0 sap) given observed privatized data query Sdp- We summarize the Sequential Private Posterior Estimation (SPPE) algorithm in Algo.1 and the Sequential Private Likelihood Estimation (SPLE) in Algo.2.
Algorithm Sequential private-data posterior estimation (SPPE)
Input: observed privatized summary statistics &ap: neural estimation family (o sap), and confidential data simulator f(x 0)
Initialization: set (0) = T0), simulated data filtration Dn- = 0
for r= 1,2, ... R do
Sample ] 1= from (0) Simulate (+) f(- ((0) for each i
Update filtration D, D. U ((06)) x0() (00) i=
Update < arg min& (prg-A(d) using fRQMC Equation (11) on D,
Set proposal (0) (0(0) )pp)
end for return #(O sap) = ( app)
Both SPPE and SPLE use normalizing flows as the variational family to minimize some KL divergence, which takes the general form of Equation (9). We have designed their sequential training procedures be sample efficient, in the sense that training data generated during previous rounds are kept and used in subsoquent rounds.
In sequential approximation procedure, iteratively refine the neural approximations towards the target distribution. After the r-th training round, we incorporate the current neural density estimator de into the proposal distributtion of the next training round, using the automatic posterior transformation weights and loss functions described in Equations (7) and (8) respectively. Sequential training procedures can gradually move ds towards high-density regions of the private data posterior, and thus achieve good accuracy with fewer samples from the simulator, compared with non-sequential training procedures that use a fixed proposal distribution.
7

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm Sequential private-data likelihood estimation (SPLE)
observed privatized summary statistics *'p: neural estimation family (sap 00, and confidential data simulator f(x !)
Input:
Initialization: set (0) Tff), simulated data filtration Da {
for r 1, .... do
Sample [ from 1r-1
Simulate 200) i=
~ f(- ) for each i
Update filtration D, = Dr-1 U ((A(), (x(0))
Update o & arg ming (L) using fRQuC Equation (11) on D,
Set proposal (0) & ((0)gs
end for
return posterior estimation 7(0 sap) = (n( and likelihood estimation fo(sap () 90- (Sdp 10)
4 Applications
Here we illustrate our methods on the suscopibbeeinieetdlrrovvvv (SIR) model and linear regression. We include experiments on the Naïve Bayes log- log-linear model in the Appendix.
4.1 SIR model for disease spread
The SIR model is a time-series model that describes how an infections disease spreads a closed population. It is most often used as a deterministic ordinary differential equation (ODE), but can also be represented by a Markov jump process.
To the best of our knowledge, inference on privacy-protected data with the SIR model has not been studied in the literature. Our proposed methods are particularly suitable for this problem for two reasons. First, our methods are simulation-based and thus are applicable under the ODE model, when other likelihood-baset methods can no longer be applied. Second, in the SIR model, low-dimensional summary statistics can be very informative about model parameters. Then the RQMC methods discussed in Section 3.4 can provide efficiency and accuracy gains when evaluating the loss functions.
We describe stochastic SIR model in closed population with K people. As the disease spreads, the individuals progress through the three states: susceptíble, infected, and recovered. We use S(t), T(t), and R(tt) to denote the number of individuals within each compartment at time t. We make the following assumptions: (a) individuals are infected at rate 8 81
K resulting in a decrease of S by one and an increase of I by one, (b) infected individuals recover with a rate yT, leading to a decrease of T by one and an increase of R by one. The confidential data likelihood of this continuous- time Markov jump process is hard to compute. Our goal is to infer the infection and recovery rates = (8, 7), under initial conditions (S, I, R) = (K - 1,1,0.
Privatizing the infection curve. Here, we propose a mechanism to privatize the infection trajectory I(t)/K, which is the proportion of infected individuals at each t.
Proposition (DP infection trajectory). Consider a sequence of points (t1.... ,,t} in the time interval 1(0)))) independently.
[0, our privatized query be sáp (S1,..., ,8L) where each 8; Binomial (n, K+2m The mechanism generating Sap (si, ,1)) satisfies e-DP, with L.
This algorithm adds calibrated noise to the SIR process to produce Sap: differential private time series. It can probably protect each individual's infection status. We demonstrate that analysts can still make inferences about population parameters (8,7) by only knowing sdp; which retains information about the speed of discase spread.
Experiments on synthetic data. We illustrate the performance SPPE and SPLE on synthetic privatizod SIR model data. The data generating parameters are set to emulate a measles outbreak. We set L 10 time points with 1000 and m 1000, achieving e 10 differential privacy. Lower privacy loss budgets
8

---

Published in Transactions on Machine Learning Research (03/2025)
A.
Truc" posterior
Round:
Round:
Round: 10
B
-2.6
-2.8
MMD
SPPE
0.0
-1.0
-0.5 log
0.0
-05
N1.00
-1.0
-1.5
-2.0
SPL
-2.5
2
8
10
4
8
10
Number of Rounds
Number of J'Roundds
Figure 1: Inference STR model. A. Convergence of sequential posterior estimations given DP-protected infection trajectory. Each round entails N = 1000 simulations. B. Approximation accuracy by SPPE (orange) and SPLE (red) against the number of rounds, the error bars represent the mean with the upper and lower quartiles over 20 random trials.
(e = 1, 0.1) are explored in Appendix E.1.1 by adjusting m accordingly. We also describe prior specifications and implementation details in the Appendix.
Figure 1-A describes the convergence of the posterior approximations 50 spp)) towards the SMC-ABC (Beau- mont et al., 2009) baseline, requiring up to 5 x 10s simulations. We use SMC-ABC as the baseline because it does not resort to the variational approximations employed by SPPE and SPLE. Both SPPE and SPLE quickly adapt to meet the SMC-ABC results, with orders of magnitudes fewer simulations needed. After the first round of simulations, SPPE can identify the high probability region of T( sáp), while the SPLE posterior is still exploring the parameter space. See Appendix for the SPLE approximation after 1. After r= 5 rounds, both methods can concentrate around the posterior mean and have captured the posterior correlation (B. ! Sap). By inspecting marginal posterior histograms (Figure 6), we find that, in this example, the posterior approximated by SPPE is slightly more concentrated than those from SMC-ABC and SPLE.
To quantitatively evaluate the performance of our methods, we use the following metrics: (1) MMD (Gretton et al., 2012): maximum mean discrepancy between the neural estimated posteriors and SMC-ABC posterior; (2) C2ST (Topez-Paz & Oquab, 2016): classifier two sample tests; (3) NL.G:: negative log density at true parameters; and (4) LMD: log median distance from simulated to observed sdp. Smaller values indicate better performance for all four metries. In Figure 1-B, we compare the performance of these methods at various numbers of simulation samples/rounde After Round 5, SPLE and SPPE have similar accuracy. We also compare their runtime in Table 1 of the Appendix. To achieve MMD lower than 0.1, SPPE 6x faster and SPLE is 2x faster than SMC-ABC.
Real disease outbreaks. We apply our privacy mechanism and inference methods to several real infectious disease outbreaks: influenza, Ebola, and COVID-19. In Figure 2-A, compare the posterior distributions (B, 7 Sap) obtained from SMC-ABC, SPPE, and SPLE. The results are similar to synthetic data experiments: when SPPE and SPLE use the same computational resources, the SPPE posterior converges faster than SPLE. In Figure 2-B, we inspect the 95%-credible interval for the ratio Ro = B/7,, which is known as the basic reproduction number. Our Ro estimates using privatized data are consistent with common estimates of Ro for these discases (Eisenberg, 2020). Specifically, typical Rg ranges are (1.51, 2.53) for Ebola, (1.53355) for COVID-19, 19, and with the exception of the flu outbreak 20.9, which should be modeled by the SI model instead of SIR.
4.2 Bayesian linear regression
We demonstrate our methods on a linear regression model, and compare it to existing work on DP regression analysis like (Ju et al., 2022; Bernstein & Sheldon, 2019). We consider linear regression with n subjects and p predictors. Denote Fy Raxp as the design matrix without intercept terms, and let z = (1xx1. 2g) represent the design matrix with the intercept. Ordinary linear regression models assume that the outcomes
9

---

Published in Transactions on Machine Learning Research (03/2025)
A
"Tree postzror
-3.0
flu
Ebola-a
Ebola-b
Ebola-c
Covid-19
B
-2.5
-3.5
-3.0
-4.0%
Bbolo-
-2
-2.0
-1.5
-3.0
SWe
-3.5
-2.5
-3.0
-4.0
-2.5
-2.0
-1
-2
-1
-2.0
-1.5
-1.0
-25
109
-3.5
Covid-19
3.0
-4.0
-1.5
log B
log
log
log B
log
A/V
Figure 2: Inference on real infectious disease outbreaks. A. Visualization of the posterior distribution given private infection curve applied to flu, Ebola [in a) Guinea, b) Liberia, and c) Sierra Leone], and COVID-19 Clark County, Nevada. All experiments use a privacy level of e 10. B. Mean and 95% credible intervals for Ro B/7 with different methods in each dataset. Grey: SMC-ABC; orange: SPPE; red: SPLE. A non-DP baseline (black stars) is also shown, which is obtained from the solution of the corresponding ordinary differential equations.
y satisfy ylag ~ (1,3, o I.). Under the constraint of differential privacy, both outcomes y and predictors wi are subject to calibrated noise. In a Bayesian setting, we model the predictors with Zo. ~> NV Σ) for 1,2,... n independently. Our parameter of interest 3, which represents the (p 1)-dimension vector of regression coefficients. Our experiments assume that a, m, and E are fixed, to illustrate our algorithm. In practice, one can also estimate these parameters from data. Our setting is the same as that used in Ju et al. (2022).
Private sufficient statistics. We achieve e-DP on confidential data (z,y) by adding Laplace noise sufficient statistics. Achieving privacy requires finite & sensitivity on confidential data. As result, before adding noise for privacy, we first need to clamp the predictors and the responses, and then normalize them to take values in 1]. Let's denote the clamped confidential data as T and respectively. We then define the summary statistics of clamped data as š = (L G, 1 D. T t). We have refined the sensitivity analysis of Ju et al. (2022) to ((s) = (p² + 3p + 3). The privacy summary statistics &dp is achioved by adding independent Laplace(O, A1 (8)/c) noise to each entry s. This output perturbation mechanism satisfies e-DP. Details of clamping and sensitivity analysis are the supplementary materials.
Posterior approximations. We compare the 95% posterior credible intervals obtained by methods applicable to linear regression, including SMC-ABC, SPPE, SPLE, Data-augmentation MCMC (DA-MCMC) (Ju et al., 2022), Gibbs-SS (Bernstein & Sheldon, 2019), and RNPE (Ward et al., 2022). See Table 2, 3, and 4 for marginal posterior credible intervals of 3 sap at privacy levels e = 10, 1 and 0.1 respectively. Both tables are in Appendix E.2. We also use the Salmogorov-Smirnon test assess the similarity of empirical posterior marginal, shown in Figure 3 al privacy level of 10. Results from SPPE, SPLE, SMC-ABC, and DA-MCMC reach agreement the posterior marginals. However, results from Gibbs-SS and RNPE are qualitatively different from the other four methods, yielding biased approximations for fo and Bn respectively. Among the likelihood-free methods, SPLE attempts to approximate the likelihood function and is the most similar to DA-MCMC, our likelihood- based baseline.
The cost of privacy. Although it has been standard practice in many DP work (Bernstein & Sheldon, 2018; 2019; Ju et al., 2022; Gong, 2022) to achieve finite global sensitivity through clamping, this benefit of privacy protection comes at the cost of accuracy of the subsequent statistical analysis.
Our experiments demonstrate the utivacy-utility trade-off, which has also been demonstrated by related works on noise/ priveacavvcceearee inference. A naive plug-in estimator (plugging Sap as into the conjugate posterior 70 {)) gives the wrong posterior; See the second rows in Table 2. Besides, achieving privacy protection comes the cost of estimation accuracy: private data posterior =( 8ap) is different from the
10

---

Published in Transactions on Machine Learning Research (03/2025)
Test statistics for
Test statistics for
Test statistics for Pa
1.0
DM-MCMC
DA-MCMC
DA-MCMC
0.8
SMC-ABC
SMC-ARC
SMC-ABC
SPPE
SPPE
SPPE
0.6
0.4
Gibbs-55
Gbbs-55
Gibos-55
0.2
RNPE
RNPE
RNPE
0.0
DAMOMC SMC-AdC
spPe
Glbbe S5
ANFE
DAMCMC
SPPE
SPLE
Gidee SS
RNPP
D4 MAMCCC
SPPE
SPLEt
GibbreS5
RRPPE
SMC
SMC
Figure 3: Kolmogorov-Smirnov lest statistics welween approximations of posterior marginals, al = 10.
confidential data posterior T( I) even under a high loss budget of e = 10 (small privacy noise) setting; See the first rows in Table 2, 3, and 4. With small privacy noise, data corruption primarily comes from the clamping step. Evaluating this censoring bias is still an open problem in DP data analysis, with some recent. attempts Biswas et al. (2020); Evans et al. (2019); Covington al. (2021).
5 Discussion
this work, we propose three simulation-based inference methods to learn population parameters from privacy-protccted data: SMC-ABC, SPPE, and SPLE. The latter two are neural density cstimation methods. We have designed SPPE and SPLE to leverage stale-of-the-arl computational tools, such normalizing flows and randomized quasi-Monte Carlo, to be computationally efficient for complex data models. SPPE aims to approximate the posterior-data posterior, and SPLE approximates the posterior-data marginal likelihood.
We compare our methods to similar DP data analysis work that focuses on post processing of privacy-protected datasets (or their summary statistics). Compared with existing ABC-based analysis for DP data (Gong, 2022), SPPE and SPLE do not reject training samples and hence are more computationally efficient. Compared with DA-MCMC (Ju al., 2022), our method does not require that the confidential data likelihood can be evaluated easily. Our methods require only that one simulate from the prior distribution x (0) and the confidential model f( (). They also all scale lincarly with the sample size of the confidential database.
Our work contributes to the growing literature on statistical analysis with privatized data. In particular, our simulation-based inference framework can be applied to complex models with intractable likelihood functions and the resulting triply-intractable private data posterior. We hope our methods can catalyze DP-protected data sharing between data collectors and analysts. Our experiments and analysis demonst the necessity and feasibility of designing valid statistical inference procedures to correct for biases introduced by privacy-protoctinn mechanisms. We advocate for increases in both sharing privacy-protecte data by collectors and using valid inference procedures.
We acknowledge the limitations of the present work and point out future directions. Our methods leverage the fact that popular DP mechanisms can be efficiently achieved with random number generators. In some tasks, such as DP principle component analysis (Chandhuri et al., 2013), the privacy mechanism is actually intractable to simulate from but density easy to evaluate. Our method not applicable to this type of DP mechanism. Additionally, since our method scales linearly in the sample size of the confidential database, it might not be ideal for massive datasets, such as the Facebook URL dataset (Evans & King, 2023), which concerns millions of users. It is of interest to develop methods that scale sub-linearly in sample size. Finally, in Appendix D, we show the algorithms scale lincarly with the dimension of the summary statistics and quadratically with the number of nodes per layer in the neural network. In our experiments, we found that using at most layers and 50 nodes can lead to satisfactory performance. precise understanding of how the convergence rate depends on the sample size of the confidential data and the dimensionality of the summary statistics remains be established and is likely to vary on a case- by-cse basis for the choice neural density imatooss
11

---

Published in Transactions on Machine Learning Research (03/2025)
Acknowledgments
We sincerely appreciate the valuable comments and suggestions from anonymous referees. The work of the third author was supported by the National Natural Science Foundation of China No.12171454 and Fundamental Research Funds for the Central Universities.
References
John Abowd, Robert Ashmead, Ryan Cumings-Menooe Menon, Simson Garfinkel, Micah Heineck, Christine Heiss, Robert Johns, Daniel Kifer, Philip Leclere, Ashwin Machanavajhhala, Brett Moran, William Scxton, Matthew Spence, and Pavel Zhuravlev. The 2020 Census Disclosure Avoidance System TopDown Algorithm. Harvard Data Science Review, (Special Issue 2), Jun 24 2022. https:/
wwtp://wwss...ss.seeeeeeeeeeeeee Tevz36li.
Christoph Aistleitner, Florian Pausinger, Anne Marie Svane, and Robert F Tichy, On functions of bounded variation. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 162, pp. 405-418. Cambridge University Press, 2017.
Daniel Alabi and Salil Vadhan. Hypothesis testing for differentially private linear regression. Advances in Neural Information Processing Systems, 35:14196-14209 2022.
Raman Arora, Raef Bassily, Tomás González, Cristóbal A Guzmán, Michael Menart, and Enayat Ullah.
Faster rates of convergence to stationary points in differentially private optimization. In International Conference on Machine Learning, pp. 1060-1092. PMLR, 2023.
Jordan Awan and Zhanyu Wang. Simulation-baseel, finite-sample inference for privatized data, 2023.
Andrés F Barrientos, Jerome P Reiter, Ashwin Machanavaijhala, and Yan Chen. Differentially private significance tests for regression coefficients. Journal of Compulational and Graphicul Stalistics, 28(2): 440-453, 2019.
Raef Bassily, Cristóbal A Guzmán, and Michael Menart, Differentially private stochastic optimization: new results in convex and non-convex scttings. In A. Beygelzimer, Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: https://openrevier net/forumiid-Ra- -id
Kinjal Basu and Art B Owen. Transformations and Hardy-Krause Krause variation. SIAM Journal on Numerical Analysis, 54(3):1946-1966, 2016.
Mark Beaumont, Jean-Marie Cormuet, Jean-Michel Marin, and Christian P Robert. Adaptive approximate Bayesian computation. Biometrika, 96(4):983 -99-9990,,,,,, 2009.
Garrett Bernstein and Daniel R Sheldon. Differentially private Bayesian inference for exponential families. Advances in Newral Information Processing Systems, 31, 2018.
Garrett Bernstein and Daniel R Sheldon. Differentially private Bayesian linear regression. Advances in Neural Information Processing Systems, 32, 2019.
Alex Bie, Gautam Kamath, and Guojun Zhang. Private GANs, Revisited. arXio preprint arXi:22302.0200 2023.
Sourav Biswas, Yihe Dong, Gautam Kamath, and Jonathan Ullman. Coinpress: Practical private mean and covariance estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14475-14485. Cur- ran Associates, Inc., 2020. URL Pwppss-aappppp..peee.ee..
www.eeddinnnn.nuiiss........ wwtpppprrccppepppccppppppppp hwtpp/ppe.plleeepppappppaaeeep filee//paper/222//ffee.
T Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Slatistics, 49(5:22825 2850, 2021.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Kamalika Chaudhuri, Anand D Sarwate, and Kaushik Sinha. A near-optimal algorithm for differentially- private principal components. Journal of Machine Learning Research, 14, 2013.
Christian Covington, Xi He, James Honaker, and Gautam Kamath. Unbiased statistical estimation and valid confidence intervals under differential privacy. arXiv preprint (XKi:2110.144400 2021.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48)330055 30062, 2020.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference Learning Representations, 2017. URL attps://arxiis wwtp:/t////pooe/e// 08803.88800008 pdf.
Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Sociely Series B: Stalistical Methodology, 84(1):3 37, 2022.
Jörg Drechsler. Differential privacy for government agencies are we there yet? Journal of the Americon Statistical Association, 118(541):761-773, 2023. doi: 10.1080/01621455) 25..2222.21211333.8. URL sttps://doi.. 15//10001880/11214555 2161385.
Jörg Drechsler, Daniel Kifer, Jerome Reiter, and Aleksaudra Slavković. Handbook of Sharing Confidential Data: Differential Privacy, Secure Multiparty Computation, and Synthetic Data. CRC Press, 2024.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, 32, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265 284. Springer, 2006.
Joseph Fisenberg. RO: How scientists quantify the intensity of an outbreak like coronavirus and its pandemic potential. https: //sph. umich. www....icuppp.....cco..o.. 2020. Accessed: 2023-10-08.
secientists-quuniifyyooom
htmm,,
Georgina Evans and Gary King. Statistically valid inferences from differentially private data releases, with application to the Facebook URLs dataset. Political Analysis, 31(1):1-21, 2023. doi: (0.1017/pan.204)
Georgina Evans, Gary King, Margaret Schwenzfeier, and Abhradeep Thakurta. Statistically valid inferences from privacy-protected data. American Political Science Review, pp. 1-16, 2019.
James Foulds, Joseph Geumlek, Max Welling, and Kamalika Chaudhuri. On the theory and practice of privacy-preservigg Bayesian data analysis. arXiv preprint (XXi:1603.07244, 2016.
Daniel Gillespie. Exact stochastic simulation of coupled chemical reactions. The Journal of Physical Chemistry, (1(55):2340-230) 1977.
Ruobin Gong. Exact inference with approximate computation for differentially private data via per- turbations. Journal of Privacy and Confidentialityy 12(2), Nov. 2022. doi: 10.29012/jpc.797. URL
wwwpey/oooiienniaaityy.. org/ index.ddxxxxx pttp://pcc/rttticce//eeee//////p
Ruobin Gong, Erica L. Groshen, and Salil Vadhan. Harnessing the known unknowns: Differential privacy and the 2020 census (co-editors' forward). Harvard Data Science Review, (Special Issue 2), 2022. URL https://áoii org/ wttg:/....6///9666811.1 cb06b469.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood- free inference. In International Conference on Machine Learning, pp. 2404-2414. PMLR, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test.. The Journal of Machine Learning Research, 13(1):723-777 773, 2012.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Shijic Guo and Jingehen Hu. Data privacy protection and utility preservation through Bayesian data synthesis: A case study on airbnb listings. The American Slatislicion, 77(2):192-200 2023. doi: 10.1080/00031305, 2022.2077440. URL https://doi.org/ wtt.////..00000000300000000
2077440.
Nianqiao Ju, Jordan Awan, Ruobin Gong, and Vinayak Rao. Data augmentation MCMC for Bayesian inference from privatized data. Advances in Newral Information Processing Systems, 35:12732-12743, 2022.
Vishesh Karwa, Dan Kifer, and Aleksandra B Slavković. Private posterior distributions from variational approximations. arXiv preprint ar Xrx:::1511.00 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transaclions on Pallern Analysis and Machine Intelligence, 43(11) (111)))3434977777 2020.
Tejas Kulkarni, Joonas Jälkö, Antti Koskela, Samuel Kaski, and Autti Honkela. Differentially private Bayesian inference for generalized lincar models. In International Conference on Machine Learning, pp. 5838-5849. PMLR, 2021.
Jaewoo Lee, Minjung Kim, Yonghyun Jeong, and Youngmin Ro. Differentially private normalizing flows for synthetic tabular data generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7345-7355 2022.
Sai Li, Linjun Zhang, T Tony Cai, and Hongzhe Li. Estimation and inference for high-dimensional generalized linear models with knowledge transfer. Journal of the American Statistical Association, pp. 12, 2023.
Fang Liu. Generalized Gaussian mechanism for differential privacy. IEEE Transactions on Knowledge and Data Engineering, 3144:7447-76, 2018.
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXvv.6610006545, 2016.
Man-Ma Lueckmann, Pedro J Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems, 30, 2017.
Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation-based inference. In International Conference on Artificial Intelligence and Statistics, pp. 343-351. PMLR, 2021.
Pierre L L'Ecuyer. Randomized quasi-Monte Carlo: An introduction for practitioners. Springer, 2018.
Benjamin Kurt Miller, Christoph Weniger, and Patrick Forré. Contrastive neural ratio estimation. arXiv preprint ar Xiv::22210 0tl::2210006177 70, 2022.
Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth scnsitivity and sampling in private data analysis. In Proceedings of the Thitty-Ninth Annual ACM Symposium on Theory of Computing, STOC '07 pp. 75 84, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595036318. doi: N0.1145/1250790.15500
URL https://doi. 0tt.o.o////...11/2555099990 1ew.111251099..1111 1250803.
Art B Owen. Monte Carlo variance of scrambled net quadrature. SIAM Journal on Numerical Analysis, 34 (5):1884-1910, 1997a.
Art В Owen. Scrambled net variance for integrals of smooth functions. The Annals of Statistics, 25(4): 1541-1562, 1997b.
George Papamakarios and Iain Murray. Fast e-free inference of simulation models with Bayesian conditional density estimation. Advances in Newral Information. Processing Systems, 29, 2016.
14

---

Published in Transactions on Machine Learning Research (03/2025)
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Aduances in Newral Information Processing Systems, 30, 2017.
George Papamakarios, David Sterratt, and Tain Murray. Sequential neural likelihood: Fast likelihood-fres inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 837-848. PMLR, 2019.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modcling and inference. J. Mach. Learn. Res., 22(1), jan 2021. ISSN 1532-4435.
Saeyoung Rho, Rachel Cummings, and Vishal Misra. Differentially private synthetic control. In International Conference on Artificial Intelligence and Statistics, pp. 1457-1491. PMLR, 2023.
Scott A Sisson, Yanan Fan, and Mark M Tanaka. Soquential Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 104(6):1760-177) 1766-1765, 2007.
Bingyue Su, Yu Wang, Daniele Schiavanzi, and Fang Liu. Differentially private normalizing flows for density estimation, data synthesis, and variational inference with application electronic health records. ar preprint arXtt:2302.05787 2023.
Chris Waites and Rachel Cummings. Differentially private normalizing flows for privacy-preserving density estimation. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, pp. 1000-1009, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384735, doi: 10.1146/3466102224462222 URL https://doii org/10 10 100.1111/346102. 3462625.
Daniel Ward, Patrick Cannon, Mark Beaumont, Matteo Fasiolo, and Sebastian Schmon. Robust neural posterior estimation and statistical model criticism. Advances in Neural Information Processing Systems, 35:33845-3388 33859, 2022.
Oliver Williams and Frank McSherry. Probabilistic inference and differential privacy. Advances in Neural Information Processing Systems, 23, 2010.
A Derivations for SPLE and SPPE objectives
Lemma 5. For privale data likelihood eslimation, minimizing the average KI. divergence
DKL (f(sap ( (9)laa 0)) under the prior (f), is equivalent to minimizing the objective function
(PLE(() = =Ex(0.2) 1 [-]n(aan ((sap xx) log go (s 0)dsap
(13)
With respect to the joint distribution x0) x (z)) 0), the objective function still has the form
(p)E( = (xp(022) [- ((sap xx) log (s 00dssp
(14)
15

---

Published in Transactions on Machine Learning Research (03/2025)
Proof. Note that
(o) [DKL (f(sap (s)p [8)]
l (0)
| [(aap] 0) (log (sap 0) log go 0)) dsap d0
/f ((0)f(sap) 0) log go (sdp) 1 d)dsapd0
C1
sxe
Ci
=(0)[(sap x) 0) log (s(p O)dsapdddx
Sxxxxx
C1 /1 exxx p(0,2) | m(sap x)) log 0)dsdp dAdr
[. (0(Bap xx) log 9o (sup
C1 Ep(0..)
0)dsap
where C1 = sssxe (sa) ()) log (sap O)dsap is a constant unrelated to ø. Furthermore, if are sampled from some proposal p(0), we still have
Ex(e) [DKL. (S(spp (>)p 0))
= =/l axe
p(0)f(sup 0) log (sap 0)dsapd0 - Ex((2)) ( n(sap x) log (s(pp) dd)ssp
Lemma 6. For private data posterior estimation, minimizing the average KL divergence
DEL (p(0) sap)llaa(0) sap)) with respect to the marginal evidence p(sap) = Sg (0) 0)d0, is equivalent to minimizing the objective function
(PrE(Ø) Ey((0.0) [-(n(s n(sap a)) log (0 sap ddspp
(15)
With respect to the joint distribution x(0,x)) x f(0)f(x 0), then the objective function has the form
(ppE-A(0) Epp(,.)) [-] n(sap x)) log (0 sap)
(16)
where
(0 sap) = (0 8dp) p(0)
((0) d0.
2 7sapp ()) = 20 (sp))
(17)
T(0) (ssap 5)¹
T(())
Proof. We have
Eplend) [DEL (T(0) sap))lga(() sap))]
/ p(sap) T(0 sap) (log ग (0) sap) log (0 sap) do doap
C2 - // (T 8ap) log (0 sap)dsapp00
= C2 - -11 T(0)n(sap x))(x 0) log (o sap dódd
Cr 1/ x(0,x) Moon x)) log (0 sanudsap dOdr
C2 Ex((.0) If ((sapp 2o (0 sap)
16

---

Published in Transactions on Machine Learning Research (03/2025)
where C2 Sfexe (( sap) log (( sdp) is constant unrelated to é. Furthermore, if 0 are sampled from some proposal p(0), then
Bp(sar) [DRL. (# sap san) 16 sap))]
= =Il p(sap)+(0 sdp) log (()) sap dsspddd - Bp((.z) Ig n(sap x)) log ( sap]dsap (18)
where =(0 |8pp) is called proposal posterior (Greenberg et al., 2019), which satisfied
p(sap) / ((0) 0)d0. #(0 sup) T(0 sap) ((0)p(sap)
p(0)p(sap)
Based on Prop. 1 in the work of Papamakarios & Murray (2016), minimizing Equation (18) results in the convergence of (o(0) sap) to #(0) sap) and yo(0 5ap) to x(0) | sap)-
B Proof of Proposition
Proposition 4. Consider a sequence of L points {li,-.. .1..3 in the time inderval [0, T) our privalized query independently. The mechanism generating can be 5dp = (81, ... ,8.) where each 8; Binomial (1, K +2m Sdp
I(t:)+m
(. ... 8L) , satisfies c-DP, with L.
Proof. Denote the numbers infectious as I(I) 10... K} and its neighbors I(t), note that |z(() - (())-T(1) s holds for all E [0, because a change of the infection status of any one of the K individuals will at most increase decrease (t) by only 1. We examine the following density ratio:
(n-x))
(*)
[[t))m
K-I(())+ K42m
p(8: I(4))
K42m
p(s: I(ta))
(.)
[it+)mm
K I(t))mm
(n-#;)
K+2m
K+2m
[(4)) +m)
8,
K I(l;) +m (n-aa)
i(t.) +m/
K I(t:) +
= H,
This expression can be analyzed under three distinct cases. In the first case, when I(4) > 144), we have
K +m (n-81)
T(l.) I((:) + 11, leading H: S
K K-J(t,)+m
S I+m
)".
In the second case, if
K-I(t)+m
K- K-I(t))m
m
I(t) < [(4), then 7(4)) [(4) 1, and we obtain H; <
[(t,)m
<
[(,))m 7(t)+m
< [+m Lastly, when
[(t.)
T(I,) T(b), H, equals to Thus, p()l)((s)) S p(x))((t)) m
(+mm) < exp (.) Now for 8dp (81, sL), we have
p(sap (I(t))t [0,21}) p(sap [(tt), ((tt)) < exp p(sap (T(O)( E (D)) p(sap T(41) ... [((r.)
m D
which gives the mechanism generating Sip (s satisfies eDDP, with m L. m
C Description of the SMC-ABC algorithm
Algorithm 3 presents the extension of the SMC-ABC algorithm (Sisson al., 2007; Beaumont et al., 2009) to posterior estimation from the privatized data settings, by incorporating the private data simulator f(x and the privatized algorithm 77.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm Sequential Monte Carlo Approximate Bayesian Computation (SMC-ABC)
Input: observed privatized summary statistics sap: discrepancy function P, thresholds {ee)s transition kernels { simulator f(x 0), and privatized algorithm 7) for t = 1,2, ..... T do
for i = ...,. ..... do if t= then
Sample ~*
~ (0) independently
else
Sample 0r from the previous population (0!-)) } with weights (W(1)
Perturb Or using transition kernel: 0 (2 0*)
end if
repeat
Generate x² ~ f(x )0)) Generate sip ~ (sddp x²²)
until p(sap= app S Et
Set of9) 0+*
Compute weights:
1.
if =1=1 1
w(())
ift
Ko 0
end for Normalize the weights so that Ei= W.(:)
=1
Compute effective sample size: ESS EN (W( if ESS E then
Resample (0(²) with weights (W())) to obtain new population
Set weights (W! = 1/N}
end if end for return posterior estimation (0)) w N
D Computational Complexity Analysis
In this section, analyze the computational complexity of the proposed (S)PLE and (S)PPE algorithms in detail. We begin by examining the computational cost of neural spline flows, which serve as the core density estimators in our framework.
Computational complexity for neural spline flows. Denote the neural spline flow (Durkan et al., 2019) as (o(v =) with trainable parameters Φ. Let nı represent the number of flow layers, and each layer containing n, hidden layers of n, units to generate the parameters of K knots for monotonic spline. Assuming the input dimensions of y and are dy and dz, respectively. Each flow layer processes the concatenated input (y, 2) and outputs (3K- 1) parameter for constructing the monotonic spline for each dimension of (soc Appendix A.1 in Durkan al. (2019) for details).
The computational complexity of evaluating gradient one sample, i.e., Voqo(yi 21), is then given by
ONSF O (mu [(d2 + dy nnn + Kdyn..)
(19)
Computational complexity for PLE. To minimize the PLE loss (PLE(0), the gradient with respect & is computed over mini-batches of size B. For each pair (0,, 24) in a mini-batch. the integral I(0;, 2;) from Equation (12) is estimated using M pseudo-samples (u(1) ...., m(M) generated via. RQMC. ".(,)....
18

---

Published in Transactions on Machine Learning Research (03/2025)
Considering a neural spline flow with ny layers, each containing n4 hidden layers with na units, the computa- tional cost for processing single mini-batch is then
CPLE = 0 (m:BM [0dim(0) + dim(s))n, + n K (s) + ndim (ndim
(X)BM log M),
(20)
where the last term represents the complexity of generating an RQMC-based estimator with M samples. Specifically, the factor M log M accounts for the cost of generating RQMC sequence, and dim(x) ndim (X)
represents the dimension of the confidential data E X with sample size n.
Computational complexity for PPE. For the PPE method, the roles of the parameters 0 and summary statistics Sdp are reversed compared to PLE. To estimate the normalization constant Z ((sdp= Φ) in Equation (17), we using the atomic proposal method (Greenberg et al., 2019), approximated as
na (ssap: の) = (o(0) Sap) (() na Σ
((0) do
1
go(0; sdpp) T(,))²
where na is the number of atoms, and 10;1/== are subsamples under the current mini-batch, which can be regarded as the samples from proposal p(0) (see Appendix A.2 in Greenberg et al. (2019) for details). The computational complexity for PPE is then given by
CPPE = O (m (d) + dim(ss)m, + +nam + Kdim(s)m,amu + vudim()BBi log M). (21)
Both PLE and PPE scale linearly with the posterior dimension dim(0). Notably, while they also scale linearly with the confidential data sample size n, this scaling is decoupled from the complexity of the flow structure, which directly models the conditional density between the summary statistics s (or sap) and the model parameters 0, rather than the high-dimensional confidential data itself.
E
Experimental details
The training and inference processes of the methods were primarily implemented using the Pytorch package in Python.
Experimental setup. We employed neural spline flows (Durkan et al., 2019) as the conditional density estimator, consisting of 8 layers. Following the flow structure scttings (Lucckmann et al., 2021), each layer consists of two residual blocks with 50 units and ReLU activation function, with 10 bins in each monotonic piecewise rational-quadratie transforms, and the tail bound was set to 5. Through empirical evaluation, we found that 8 layers provided sufficient flexibility for posterior estimation. Appendix E.4 compares 5-layer and 10-layer models, showing that 5 layers slightly degrade performance, while 10 layers perform similarly to 8 layers but with higher computational cost.
In the training process, the number of samples simulated each round is = 1000 and there are rounds in total. In each round of training, we randomly select 5% of the newly generated samples as validation data. According to the early stop criterion proposed by Papamakarios et al. (2019), stop training if the value of loss validation data does not decrease after 20 epochs in single round. For stochastic gradient descent oplimizer, we choose the Adam (Kingma & Ba, 2014) with the batchsize of 100, the learning rale of 5 x 1 and the weight decay is 10-4
E.1
SIR model
E.1.1
Detailed results synthetic data
In our experiments, we use the Gillespie algorithm (Gillespie, 1977) to simulate the whole process over a duration of 160 time units and record the populations at intervals of 1-time units. The prior distribution of is set A (log 0.4, 0.5) and prior distribution 7 set A 0. 0.125, 0.2). The value of K is configured 1000000, while N is set to 1000 for the number of observations.
19

---

Published in Transactions on Machine Learning Research (03/2025)
To publish the privatized data about the infection process, we select the infectious group [(t) at L 10 evenly-spaced points in time [0, T), with the privacy parameters set to 1000 and m = 1000, which satisfied e-DP with E = 10. The ground truth parameters are
0 = (ex)(- exp(-3)),
and the observed private statistic Sap simmlated from the model with ground truth parameters 0* is up (0.0010, 0.0310, 0.6140, 0.2630, 0. 0.1230, 0.0470, 0.0180, 0.0090, 0.0050, 0.0030).
Figure 4 illustrates the convergence of the approximate posterior by SPPE and SPLE in each round, and compares our results with the SMC-ABC method, where we performed simulations up to 5 109 times for the SMC-ABC method to generate the near exact "Truc' posterior. The tolerance parameter 6 in SMC-ABC selected as e, = 0.5 x 0.7 where i represents the iteration round, and the discrepancy function p was chosen as the l2 norm. Figure 5 depicts the results of the SMC-ABC method under the same performance metrics. Our method, after 10 rounds or 10 simulations, achieved a similar performance as the SMC-ABC method with approximately 10s simulations. The computational time costs comparison between our methods and SMC-ABC, as rllustrated in Table 1, also reveals that our approaches require siguificantly fewer simulationss, resulting in substantially lower simulation time expenditures compared to the SMC-ABC method. However, methods require extra time for the training of normalizing flows, duration dependent the flow's complexity. Overall, both SPPE and SPLE allain a low MMD more rapidly than SMC-ABC.
Table 1: Computational Cost achieve MMD 0.1 in the SIR Model (Mean + Standard Deviation)
Method SMC-ABC SPPE
Simulation Time (min)
Network Training Time (min)
Total
Number of Simulations
115.38 1 1.51
115.38 t 1.51
82.85
1.03
2.65 1.03
15.73 + 7.67 45.98 14.46
18.38 8.63
2.12 0.71
SPLE
7.11 1.01
53.09
15.26
5.40 0.77
True'
Round:
Round:
Round:
Round:
Round:
Round:
Roand: 7
Round:
Round:
Round:
SPPE
lng
SPL B
-2
-2
-2
-2
-2
-7
-2
0
Figure Detailed convergence sequential posterior estimations given DP-protected infection trajectory under the SIR model. Each round entails N- 1000 simulations. Orange: SPPE; red: SPLE; grey: SMC-ABC.
0.6 *
1.0
MMD
0.4
0.9
0.8
0.2
0.7
0.0,
x
0.6
X
X
-1.0
1.0
NLOG
LMD
1.5
0.0
-2.0
X
-0.5
X
X
-2.5
X
16'
10 10a
2 10a
10*
10 10t
1010
Number Simulations
Number of Simulations
Figure 5: Approximation accuracy by SMC-ABC on the SIR model against the number of simulations.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Finally, we investigate the marginal posterior distributions T( sap) and xτ(> sap) in Figure 6, and conclude that SPPE and SPLE perform similarly well. In this example, the posterior approximated by SPPE is slightly more concentrated than those from SMC-ABC and SPLE.
"True" posterior
7.5
200
5.0
100
7.5
SPPE
5.0
100
2.5
0.0
SPLE
200
0.3
0.2
2.5
0.1
0.0
0.0
0.6
0.8
1.0
0.05
0.06
10
15
20
Figure 6: Marginal posterior histograms of 8,7, and 8/y in SIR model on synthetic data. Grey: SMC-ABC;
orange: SPPE; red: SPL.E. The vertical lines indicate true data generating parameters, set mimic a measles outbreak.
Task: SIR model
Task: linear regression
10a
10-1
10-2
RMSE
10-3
10-4
10-5
MC Slope: -0.5039 -0,9682
MC Slope: 0.4931
ROMC -0.8857
10a
104
102
10a
Number of samples
Number of samples
Figure Rate of convergence of MC and RQMC: The z-axis represents the number of samples M used for integral estimation, and the y-axis shows the logarithmic value of the noo-mman-squar--eroon (RMSE). The results indicate that the RMSE of the MC method is approximately O(M while the RMSE of the RQMC method is approximately O(M-1).
Finally, we evaluate the impact of the privacy loss budget on the performance of SPPE and SPLE. By setting the parameter m to either 10 or 105, we can adjust the privacy loss budget to = or < = 0.1, respectively. Figure illustrates the postcrior approximation accuracy under different values. SPPE demonstrates rapid and stable convergence, while SPLE requires approximately 5 training rounds to converge.
E.1.2 Inference on real disease outbreaks
We applied our inference methods (SPPE and SPLE) to analyze several real infectious disease outbreaks, namely influenza, Ebola, and COVID-19. For these experiments, we chose the privacy parameters M = 100, 100, and 10, which result. in an overall privacy loss budget 10 according Proposition
influenza outbreak. We utilized the dataset from boarding school, obtained from https: apppo/eettrrr///aa//eeaaaaaaargp
html. html. The total population in the school was K 763.
21

---

Published in Transactions on Machine Learning Research (03/2025)
A
B
0.4
CST
0.8
0.9
MMD
04
0.0
0.8
0.7
0.0
0.7
1.5
2
0.5
ALOG
1.0
LMD
-1.0
NT.00
-1.0
-1.
-2.0
-2.0
-0.5
-2.5
10
Number of Rounds 10
Number of Rounds
Number Rounds
Number of Rounds
Figure 8: Approximation accuracy by SPPE (orange) and SPLE (red) against the number of rounds with A. Privacy loss budget E == B. Privacy loss budget 6 = 0.1. The error bars represent the mean with the upper and lower quartiles.
We considered a daily time interval, and the observed private statistic sap is fp (0.0010, 0.0039, 0.0105, 0.0367, 0.0996, 0.2910, 0.3840, 0.3368, 0.3106, 0.2516).
Ebola outbreak in West Africa, 2014. We analyzed datasets from three regions: a) Guinea, b) Liberia, and c) Sierra Leone. The dataset source is from https: who www.wnttotte.eeeeeeeeeeeeee bett/gho/dtt//nnn
ebola-sitree. We assumed potential contact individuals of K 100, 000. We selected 9 equally spaced time intervals of 120 days starting from 03/31/2014 The resulting observed private statistic såp is follows
80 (a) = (0.0010, 0.0111, 0.0085, 0.0129, 0.0510, 0.0520, 0.0224, 0.0212, 0.0084, 0.0023).
(p. = (0.0010, 0.0007, 0.0019, 0.0664, 0.2579, 0.0742, 0.0610, 0.0542, 0.0172, 0.0003).
dd.(c) (0.0010, 0.0079, 0.0434, 0.2156, 0.2054, 0.0928, 0.0549, 0.0366, 0.0237, 0.0232).
COVID-19. We examined the COVID-19 dataset for Clark County, Nevada. See https://usafacts: / htt///ttte//eeeeee wwwpaeewwwuatt//eeeeeeeaeao/aeee county/. We assumed a potential contact population of K = 100, 000. We selected 9 equally spaced time intervals of days, starting from 09/07/2020 To obtain the number of currently infected individuals, we calculated the difference in the total confirmed cases with a time interval of days from the original dataset. The resulting observed private statistic sip is:
org/ hrg/vise wisuullizann www.aaiiinnoooovviirueeeeeeeeeee org////e.....aatooooooooorrrrrrr
S dp.covid = (0.0010, 0.0281, 0.0566, 0.0978, 0.2108, 0.2443, 0.2574, 0.0864, 0.0434, 0.0263).
The numerical results are presented and compared in Figure 4 of the main text.
E.2 Bayesian linear regression
Data generating parameters. the predictors with To, ~ Np E), where = In and m = (0.9, -1.17). The outcomes satisfy y Io ~ (2,3,, I,) where x = 2, and the prior for 3 is independent normal (0, We set the privacy loss budget 10 and the number of subjects n 100. The ground truth parameters are denoted as
Following the parameters settings in Ju et al. (2022), we model
0" (- (.79, 2.89, -0.66).
We simulate the summary statistics & from the model using the ground truth parameters 0*, resulting in
00.374
1.0000
0.0938
-0.1270
- 0.0629
0.2499,
0.0938
0.0180
-0.0094
0.0299
-0.1270
-0.0094
0.0280
corresponding vector form
E (-0.3742, -0.0629, 0..0999, 0.2499, 0.0938, -0.1270, 0.0180, -0.0094, 0.0280).
22

---

Published in Transactions on Machine Learning Research (03/2025)
Furthermore, the observed private statistic Sáp is given by
0.3824
1.0000
0.0988
-0.1385
såp= =
-0.0667
0.2720,
0.0988
0.0219
-0.0229
0.0320
0.1385
-0.0229
0.0341
its corresponding vector form is
dp, vec (-0.3824, -0.0667, 0.0320, 0.2720, 0.0988, -..1385 0.0219, -0.0229, 0.0341).
Experimental results. In Figure 9, we present comparison of the performance of the SPPE and SPLE methods across different metrics as the number of simulation rounds increases. The SPLE method demonstrates faster convergence in this task. Figure 10 displays the posterior after 10 rounds, where both the SPPE and SPLE methods achieve results that close to the near exact posterior obtained by the SMC-ABC method.
0.8
MMD
0.2
C2ST
0.7
0.1
0.6
0.0
-225
5
N1.00
2.30
NT
-2.35
3
-2.40
-2.4
Number of Rounds
Number of Rounds
Figure 9: Approximation accuracy by SPPE (orange) and SPLE (red) on the Bayesian linear regression model against mumber rounds, the error bars represent the mean with the upper and lower quartiles.
-2.5 0.0
2.5
-2.5 0.0
-2.5 0.0
-2.5 0.0
2.5
-2.5 0.0 2.5
Po
Ag
-2.5
-255
Bi
Fi
B.
-2.5
0.0
0.0
8,
Fr
R1
Figure 10: Posterior comparison on the Bayesian linear regression model. Grey: SMC-ABC; orange: SPPE; red: SPLE. The vertical lines and black dots indicate true data generating parameters.
To further characterize the utivacy-utility trade-off, we compare the private data posterior distributions under several levels of privacy loss budget, in Figure 11. The underlying confidential data the same for each e = 0.1, 0.3, 1, 10, 30, 100. For each privacy loss level, we simulate one private summary sap(e), and SPPE and SPLE to approximate the private data posterior л( sdpit). The two methods yield very
23

---

Published in Transactions on Machine Learning Research (03/2025)
similar results. More interestingly, use the same confidential data Ju et al. (2022) and the posterior distributions in Figure 11 follow similar trend with that in Figure 2 of Ju et al. (2022). For larger privacy loss budget (smaller noise), confidential data are mainly corrupted by clamping, and the proposed methods can elevate the effect of this censoring bias, as [ sap) is closer to the confidential data expectation E(Ø z). As e gets closer to 0 (near perfect privacy), the privacy mechanism has injected so much noise into Sdp that. the posterior distribution more dispersed, and little information about B can be learned based on observing 9dp-
$2
100
Figure 11: Marginal posterior histograms of 0 (Bo, 82) given Sap generated with several levels of privacy loss budget on the same confidential data z. Orange: SPPE; red: SPLE. The dash-dotted horizontal lines indicate the confidential data posterior means, and the dotted lines indicate prior means.
Finally, we compare the posterior means and 95% credible intervals obtained by different methods under various privacy loss budgets, shown in Tables 3, and 4. When larger noise is added, the posterior from all methods converge toward the prior of the parameters. Additionally, the naive posterior approximation fails to compute results due to the non-positive definiteness of the covariance matrix. The correspouding Kolmogorov-Sminng test results, presented in Figure 12, shows that RNPE exhibits a larger bias compared to other methods, while the results from DA-MCMC, SMC-ABC, SPPE, SPLE, and Gibbs-SS are closely aligned.
Table Estimated posterior mean and 95% credible intervals for the linear regression example using various methods. Here privacy loss budget is set 10.
B1
92
Confidential Posterior given #
-2.15 (-2.68, -1.61)
-2.79
(-3.08, -2.50)
-0.83 (-1.08, -0.58)
Naive posterior approximation
-4.63 (-5.04, -4.22)
-6.23
(-6.57, -5.90)
-5.10 (-5.40, -4.79)
DA-MCMC
-0.62 (-2.50, 0.99)
-2.72
(-3.74, -0.96)
0.54 (-1.06, 2.46)
SMC-ABC
-0.59 (-2.28, 0.93)
-2.44
-0.36)
0.85 (-0.88, 2.77)
SPPE
-0.51 (-2.25, 1.07)
-2.40
-0.30)
0.90 (-0.89, 2.85)
SPLE
-0.64 (-2.31, 0.96)
-2.61
-0.98)
0.64 (-0.93, 2.48)
Gibba-SS
-0.46 (-2.22, 1.39)
-0.50 (-2.08, 1.28)
0.41 (-1.68, 2.13)
RNPE
-1.40 (-3.59, 0.94)
-2.55 (3.344, 0.51)
0.29 (-2.11, 2.97)
Table 3: Estimated posterior mean and 95% credible intervals for the linear regression example using various methods. Here privacy loss budget is set to 1.
So
B1
32
Confidential Posterior given
-2.15 (-2.68, -1.61)
-2.79
(-3.08, -2.50)
-0.83
(-1.88, -0.58)
Naive posterior approximation
MCMC
-1.00 (-2.75, 0.85)
-0.96 (-2.66, 0.89)
1.23 (-0.76, 2.88)
SMC-ABC
-0.80 (-2.74, 1.17)
-0.83 (-2.64, 1.23)
0.90 (-1.00, 2.58)
SPPE
-0.93 (-2.80, 1.04)
-0.90 (-2.62, 0.97)
1.11 (-0.71, 2.72)
SPLE
-0.89 (-2.69, 1.06)
-1.04 (-2.82, 0.91)
1.25 (-0.55, 2.88)
Gibbs-SS
-0.92 (-2.72, 0.94)
-0.67 (-2.21, 1.04)
1.04 (-0.62, 2.47)
RNPE
-1.37 (-3.11, 2.07)
-0.51 (-3.01, 2.53)
1.34 (-1.77, 3.76)
24

---

Published in Transactions on Machine Learning Research (03/2025)
Table Estimated posterior mean and 95% credible intervals for the linear regression example using various methods. Here privacy loss budget is set 0.1.
9
81
32
Confidential Posterior given
-2.15 (-2.68, -1.61)
-2.79
(-3.08,
-2.50)
-0.83
(-1.08, -0.58)
Naive posterior approcimation
DA-MCMC
-0.12 (-1.98, 1.77)
-0.13 (-2.13, 1.88)
0.13 (-1.91, 2.12)
SMC-ABC
-0.09 (-1.97, 1.93)
-0.10 (-2.09, 1.95)
0.11 (-1.90, 2.22)
SPPE
-0.12 (-2.13, 1.73)
-0.12 (-2.11, 1.83)
0.14 (-1.81, 2.09)
SPLE
-0.04 (-2.01, 1.99)
-0.06 (-2.00, 1.85)
0.12 (-1.86, 2.09)
Gibbe-SS
-0.05 (-1.94, 1.95)
-0.01 (-1.95, 1.97)
0.21 (-1.62, 1.89)
RNPE
-0.22 (-3.56, 3.16)
0.13 (-3.12, 3.51)
0.34 (-3.12, 3.61)
A
Test statistics for Po
Test statistits for Pt
Test statistics for fz
1.0
UA-NCMC
UA-MCMC
UA M-NOMC
SMC-ARC
SMC-ABC
SMC.ABC
SPPE
SPPE
SPPE
0.6
SPLE
Gibbs-S5
Gibbs-55
INPE
RNNE
NNNe
DAMCMC
SMC ABc
SPPE
SPLE
Gilber 55
RHPE
OKACCIC
SMC-4DC
SPPE
SPLE
Gibbx-Ss SS
RIPE
SMC MMC-
SPLE
Gilbs 58
Rime
B
Test statistics for P
Test statistics for fr
Test statistics fr
SMC-AIC
SMC-ABCC
SWC-ABC
SPPE
SPPE
SPPE
SPL
SPLF
SPLE
DAMOMCC
SMC-
MMC
SPPE
SPLE
Gálss
Ripe
MAMMMM
SAVE
SPLE
Gibss ss RAPe
DA De-MEMe
SMC
SPLE
Gábis st
Rome
Figure 12: Kolmogorov-Smirnov test statistics between approximations of posterior marginals. A. Privacy
loss budget = 1. B. Privacy loss budget = 0.1.
E.3 Naïve Bayes lin-linear model
Model description. The naive Bayes lon-linear model is a commonly used approach for modeling categorical data (Karwa al., 2015). The input feature-vector, denoted as x = (æ1, ,Tx), consists of K features, each taking values in the range {1, J+). The output class, denoted as y, represents the target calegory and (akes values in {1,2, ... /). The model assumees that the conditional probability of the input given the output, denoted p(z v), can be factorized as the product of individual feature probabilities: p(rx v). The model parametors are ptj- which represents the probability p(** = j y == i), with prior (pi.,... "pp," ~ Dirichlet(a,,, "... Dirichlet(a,,....
p(x 8) = IT IK=IP(E
".. " ,af,,, for all and k; and pt = ply = i), with prior
(p......p))
We assume that (nf ......ntt. .ntt))) Multinomial(n pp "...... .P....) for all i and & and (nI,.... ..) Multinomial(7;pi, Pi, P..... Pr). Here nř represents the counts #(y = i = j). One sufficient statistics of the model is the proportion of the counts rf. == nn where n = ELe Σ/ nt, for all To protect the privacy of the dataset, Laplace noise et is added to the proportion of the counts, resulting in the privatized proportion m rhe + et When et Laplace(O, 2K) the released private statistic fm ba.. satisfied eDDP.
25

---

Published in Transactions on Machine Learning Research (03/2025)
In our simulation, set a.. a; for all i,j, k and 100, with / 2, K and Jj 2 for all k. The privacy loss budget 10. The ground truth parameters are
Pi,1 = 0.3887, Pi,2 = 0.6113, p 0.7537, pî, 0.2463. P1, 0.6534, p2,2 0.3466, PI. 0.5834, p2.2 0.4166,
P1 0.8489, p2 0.1511.
and the observed private statistics simulated from the model with ground truth paramcters are
ri,1 0.3275, r1.2 0.4520, ri 0.5862, rī 0.1827,
r 0.1293, 12.2 0.0858, r 0.1288, r 0.0954.
Experimental results. Figure 13 illustrates the performance of the SPPE and SPLE methods across four different metrics, and Figure 14 shows the marginal posterior histograms after 10 rounds, methods stabilize in performance after round 3.
0.04
0.66
MMD
0.03
0.64
0.02
0.62
0.01
0.60
0.00
0.58
10
-1888
NLOG
-199
-1.92
1.94
-10
2
10
4
8
10
Number of Rounds
Number of Rounds
Figure 13: Approximation accuracy by SPPE (orange) and SPLE (red) the log- ligel model against the number of rounds, the error bars represent the mean with the upper and lower quartiles.
10
10
5.
5.0
7.5
7.5
2.5
2.5
0.0
10-
5.0
5.
0.0
0.2
9.5
0.50
0.75
0.50
0.25
0.2
0.55 0.0 0.5
0.5 020 0.5
0.5 0200.5
015 031005
0.5 0.50
Pl.
Pi:
Figure 14: Marginal posterior histograms of the log-lineat model. Grey: SMC-ABC; orange: SPPE; red: SPLE. The vertical lines indicate true data generating parameters.
E.4 Neural spline flows with different layers
To evaluate the impact of varying the number of layers in the neural spline flow (NSF) model, we conducted additional experiments using configurations with and 10 layers, in addition to the existing 8-layer setup.
The results are shown in Figure 15. For the Bayesian linear regression task, the posterior approximation performance with 5-layer and 10-layer configurations comparable the 8-layer NSF. This indicates that.
26

---

Published in Transactions on Machine Learning Research (03/2025)
5-layer NSF possesses sufficient flexibility represent the posterior or likelihood. However, for the SIR model, the SPLE method with the 5-layer configuration converges slightly slower compared to the 8-layer NSF, indicating that fewer layers may marginally affect convergence in some cases. Besides, as analyzed in Appendix D, increasing the number layers also results in a lincar increase in computational cost.
A
1.0
0.5
1.0
0.6
MMD
MMD
0.4
0.8
0.2
0.2
0.7
0.0
0.6
0.0
-0.5
-05
-1.0
NL.0
-1.0
-2.5
-2
Number of Rounds
Number of Rounds
Number of Rounds
Number of Rounds
0.3
MMD
0.2
MMD
0.2
C2ST
0.7
0.1
0.6
0.1
0.6
0.0
-22
LMD
-2.3
$000
-244
2.5
10
6
10
10
10
Number Rounds
Number of Rounds
Number of Rounds
Number of Roundd
Figure 15: Comparison of approximation performance for NSF models with different layer configurations by SPPE (orange) and SPLE (red). A. 5-layer NSF on SIR model. B. 10-layer NSF on SIR model. C. 5-layer NSF on Bayesian linear regression model. D. 10-layer NSF on Bayesian linear regression model.
F Statement on Computing Resources
Our numerical experiments were conducted on computer equipped with four GeForce RTX 2080 Ti graphics cards and a pair of 14-core Intel E5-2690 v4 CPUs.
27


Reviews:
[{"review_id": "ecrdn8tqJ9", "reviewer": "Reviewer_76WG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The general problem is interesting and important.\n* The proposed methods seem to work for some problems for which the currently existing noise-aware methods are not applicable (intractable likelihood).\n\n\n### Weaknesses\n\n* The writing in general could be improved; most seriously, it takes some effort to try and distinguish novel contributions from existing work, as much of the actual background material is interleaved with the contributions in the Methods section (eg importance sampling to reduce variance on p4, basics of RQMC on p5)\n* It is not entirely clear what some of the claimed contributions mean (see Requested changes for specifics).\n* There is no mention of releasing the source code.\n* Various other issues (see Requested changes for details).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on learning noise-aware posterior distributions from differentially private query results, when the likelihood function is not readily available but a simulator is. The authors propose to use normalizing flows to approximate either the (intractable) marginal likelihood, or the noise-aware posterior directly. They experimentally show that their method can achieve similar performance as the baselines (sequential Monte Carlo ABC method for SIR model on synthetic and real epidemic data, existing noise-aware methods for linear regression with simple synthetic data), while saving on the computation cost (compared to SMC-ABC).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify writing in general; preferably introduce all background material in the corresponding section, not under Methods (introducing eg SMC-ABC, normalizing flows in the Background would make this clearer).\n2. Contributions: mention explicitly in what sense the proposed methods are claimed to provide efficiency and utility (and preferably add ref to corresponding Sec that backs up the claim).\n3. Clarify how the proposed method scales to higher dimensional $s_{dp}$ DP summary statistics (eg, based on the discussion in Sec3.3 & Sec4.1, it is not clear if the focus on low-dimensional queries is a fundamental limitation of the approach or simply a convenience).\n4. Clarify how the SMC-ABC baseline (or proposed method?) is done.\n5. Experimental details: mention how the hyperparameters have been chosen for each method (grids, Bayesian optimization, something else), also report the DP parameters clearly for all experiments in the main paper (SIR models).\n6. Should add some missing references to the discussion in Related work:\n\t* Williams & McSherry 2010: Probabilistic inference and DP \n\t\t* first proposed using Bayesian inference to handle DP noise (should mention, no need to compare experimentally)\n\t* Kulkarni et al. 2021: DP Bayesian Inference for GLMs\n\t\t* propose method for noise-aware GLMs (should mention, no need to compare experimentally)\n\t* Lee et al. 2022: DP Normalizing Flows for Synthetic Tabular Data Generation\n\t\t* also combine normalizing flows with DP (should mention, no need to compare experimentally)\n7. p6 just before Sec 4: \"...achieve good accuracy with fewer samples from the simulator.\" Fewer samples compared to what, what is the concrete baseline here?", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kHF75lschS", "reviewer": "Reviewer_u4vF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses a significant and seemingly novel problem, contributing to an important area of research.\n- The introduction and privatization of the SIR model is a valuable and noteworthy addition.\n- The study demonstrates strong empirical results, which lend credibility to the proposed methods.\n\nWeaknesses:\n\n- The article’s writing could be improved, as it is challenging to follow for readers unfamiliar with the presented concepts.\n- The title does not effectively convey the core problem studied in the paper.\n- The choice of $\\epsilon = 10$ in the simulations is questionable, as this value is unusually high.\n- The scalability of the proposed method with increasing dimensionality has not been thoroughly explored by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose three simulation-based methods for likelihood-free inference to estimate population parameters from data protected by privacy measures. These include Sequential Monte Carlo for Approximate Bayesian Computation, as well as two techniques based on neural density estimation: Sequential Private-Data Posterior Estimation and Sequential Private-Data Likelihood Estimation. They validate these methods with numerical experiments, applying them to the Susceptible-Infected-Recovered (SIR) model—augmented with a privacy mechanism—and to Bayesian linear regression.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Critical) Can you update the title to reflect the problem that is being studied ? \n- Can you add mode details on the background on MCMC methods and normalizing flows ? The paper treats the two communities that it targets itself to (i.e. Differential Privacy and simulation-based statistics) differently, first by taking the time to detail many basic facts for people from differential privacy, but then by assuming a lot of implicit prior knowledge on the simulations parts.\n- Is it possible to have the results for experiments with realistic $\\epsilon$s ? Between $0.1$ and $1$ for instance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nm1HWYRVHP", "reviewer": "Reviewer_Yzv4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I believe that the results from this paper could benefit privacy researchers: Assuming that the data have already been privatized by a DP mechanism, the proposed method presents us a logical way to approximate the Bayesian posterior.\n- Demonstrations of the use of these methods for real world applications are appreciated.\n- The experimental results indicate that the proposed methods (SPPE and SPLE) can accurately approximate the baseline (AMC-ABC), though SPPE seems to perform better overall. \n \nWeaknesses:\n- From what I understand, the Bayesian inference is the main theme for this paper, but the paper's title does not reflect this. At least, Bayesian inference should has its own subsection in the background section. Authors could argue that this work applies in non-Bayesian setting as well, in which case I recommend the authors to provide at least one example of such setting.\n- The presentation in Section 3 (Methods) are confusing: In the first paragraph, the authors wrote \"We present two complementary\napproaches...\" without telling the readers which subsections refer to these two approaches---this leaves the readers to do some pattern matching of the two approaches to the three subsections (excluding the algorithm section).\n- The SMC-ABC method is the main method, but is barely talked about in this paper. In some places, the method is referred to as the \"baseline\"; in the experiment, it is referred to as \"'True' posterior\". (I was quite confused about this term until I read the experiment details in the Appendix \"..we performed simulations up to 5 × 10^5 times for the SMC-ABC method to generate the near exact ‘True’ posterior\"...). If SMC-ABC is one of the main methods proposed to solve the problem, the authors should run its algorithm with the same simulation steps as those of SPPE and SPLE and show the results in the main paper.\n- The discussions of the experiments results are quite short. Specifically, why is SPPE generally better than SPLE? Under what condition does SPLE perform relatively well? Does the number/width of layers affect the approximation quality?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is concerned with statistical analysis of the data that has been privatized with privacy-protected data e.g. those obtained from differential privacy (DP) techniques. The authors' method assumes that the DP technique must be known and gives rise to the conditional density $\\eta(s_{\\text{dp}} | x)$ where $x$ is the private data and $s_{\\text{dp}}$ is the DP output. As with typical Bayesian inference, the goal is to evaluate the posterior likelihood $\\pi(\\theta | s_{\\text{dp}}) \\propto f(s_{\\text{dp}} | \\theta) \\pi(\\theta)$ where $f(s_{\\text{dp}} | \\theta) = \\int f(x | \\theta) \\eta(s_{\\text{dp}} | x) \\ dx$. The obstacle here is the intractability of $\\pi(\\theta | s_{\\text{dp}}), f(s_{\\text{dp}} | \\theta)$ and possibly $f(x | \\theta)$. To this end, the authors propose to approximate the posterior using sequential Monte Carlo approximate Bayesian computation (SMC-ABC), Sequential private-data posterior estimation (SPPE) and Sequential private-data likelihood estimation (SPLE).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n- I would recommend this paper if the authors could address all of my concerns in the weaknesses, as doing so would increase the readability and the credibility of the paper.\n- Note that $\\pi(\\theta | s) = \\pi(\\theta | x)$ when $s$ is a sufficient statistics, and that's why we are interested in calculating $\\pi(\\theta | s)$. The authors should add a brief discussion about this point.\n- Maybe I couldn't find it, but what is the value of $\\epsilon$ in the first experiment?\n\nMinor changes:\n- Can the authors add some explanation on why they choose the neural spline flows for the approximations? I am curious if there are any other models that the authors have tried but resulted in poor approximation quality.", "broader_impact_concerns": "There is no ethical implications as far as I am concerned.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have answered all reviewer concerns and all reviewers thus consider the paper to be acceptable to TMLR.

As a minor formatting point for the camera ready version, I would encourage the authors to refer to equations in the paper as "Equation (3)" instead of "equation 3", as is standard practice. Please also check the bibliography for consistency, inclusion of publication forums and correct capitalisation of terms derived from proper names (e.g. Bayesian, Gaussian) and abbreviations (e.g. GAN).
Decision: Accept (poster)