Title: On Consistent Bayesian Inference from Synthetic Data
Venue: NeurIPS 2023
Accepted: no
Keywords: synthetic data

Bayesian inference

Bernstein-von Mises theorem

differential privacy
Abstract: Generating synthetic data, with or without differential privacy, has attracted significant attention as a potential solution to the dilemma between making data easily available, and the privacy of data subjects. Several works have shown that consistency of downstream analyses from synthetic data, including accurate uncertainty estimation, requires accounting for the synthetic data generation. There are very few methods of doing so, most of them for frequentist analysis. In this paper, we study how to perform consistent Bayesian inference from synthetic data. We prove that mixing posterior samples obtained separately from multiple large synthetic datasets converges to the posterior of the downstream analysis under standard regularity conditions when the analyst's model is compatible with the data provider's model. We show experimentally that this works in practice, unlocking consistent Bayesian inference from synthetic data while reusing existing downstream analysis methods.


Abstract

Generating synthetic data, with or without differential privacy, has attracted significant attention as a potential solution to the dilemma between making data easily available, and the privacy of data subjects. Several works have shown that consistency of downstream analyses from synthetic data, including accurate uncertainty estimation, requires accounting for the synthetic data generation. There are very few methods of doing so, most of them for frequentist analysis. In this paper, we study how to perform consistent Bayesian inference from synthetic data. We prove that mixing posterior samples obtained separately from multiple large synthetic datasets converges to the posterior of the downstream analysis under standard regularity conditions when the analyst's model is compatible with the data provider's model. We show experimentally that this works in practice, unlocking consistent Bayesian inference from synthetic data while reusing existing downstream analysis methods.

1 Introduction

Synthetic data has the potential of opening privacy-sensitive datasets for widespread analysis. The 15 idea is to train a generative model with real data, and release synthetic data that has been generated 16 from the model. The synthetic data does not contain records from real people, and ideally it preserves 17 the population-level properties of the real data, making it useful for analysis. Privacy preservation can 18 be guaranteed with differential privacy (DP) (Dwork et al. 2006b), which offers provable protection 19 of privacy. 20

The most convenient and straightforward way for downstream analysts to analyse synthetic data 21 is using the same method that would be used with real data. However, ignoring the additional 22 stochasticity arising from the synthetic data generation will yield biased results and overconfident 23 uncertainty estimates (Raghunathan et al. 2003; Räisä et al. 2023; Wilde et al. 2021). This is especially 24 problematic under DP, which requires adding extra noise, which will be ignored if the synthetic data 25 is treated like real data. This problem creates the need for noise-aware analyses that account for the 26 synthetic data generation. 27

When the downstream analysis is frequentist, it is possible to account for the synthetic data generation 28 when multiple synthetic datasets are generated and analysed (Raghunathan et al. 2003). Recent work 29 has extended this to DP synthetic data (Räisä et al. 2023), which allows generating multiple synthetic 30 datasets without compromising on privacy. These methods reuse the analysis method for the real 31 data, and only require using simple combining rules to combine the results from the analyses on each 32 synthetic dataset, making them simple to apply. 33

For Bayesian downstream analyses, Wilde et al. (2021) have shown that the analyst can use additional 34 samples of public real data to correct their analysis. However, their method requires targeting a 35 generalised notion of the posterior (Bissiri et al. 2016) and needs the additional public data for 36 calibration. Ghalebikesabi et al. (2022) propose a correction using importance sampling to avoid the 37

need of public data, but only prove convergence to a generalised posterior and do not clearly address 38 the noise-awareness of the method. 39

In the context of missing data, Gelman et al. (2014) have proposed inferring the downstream posterior 40 of a Bayesian analysis by imputing multiple completed datasets, inferring the analysis posterior for 41 each completed dataset separately, and mixing the posteriors together. We study the applicability 42 of this method to synthetic data, aiming the bring the simplicity of the frequentist methods using 43 multiple synthetic datasets to Bayesian downstream analysis. 44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

Contributions

We study inferring the downstream analysis posterior by generating multiple synthetic datasets, inferring the analysis posterior for each synthetic dataset as if it were the real dataset, and mixing the posteriors together. We find that in this setting, the synthetic datasets also need to be larger than the original dataset.

We prove that when the Bernstein-von Mises, or a similar theorem, applies, this method converges to the true posterior as the number of synthetic datasets and the size of the synthetic datasets grow. Under stronger assumptions, we prove a convergence rate for this method in the synthetic dataset size, which we expect to match the rate that usually applies in the Bernstein-von Mises theorem (Hipp and Michel 1976). These are presented in Section 3.

We evaluate this method with two examples in Section 4: non-private univariate Gaussian mean estimation, and differentially private Bayesian logistic regression. In the first example, we use the tractability of the model to derive further theoretical properties of the method, and in both examples, we verify that the method works in practice through experiments.

1.1 Related Work

Generating synthetic data to preserve privacy was, as far as we know, originally proposed by Liew 60 et al. (1985). Rubin (1993) proposed accounting for the synthetic data generation in frequentist 61 downstream analyses by adapting multiple imputation (Rubin 1987), which involves generating 62 multiple synthetic datasets, analysing each of them, and combining the results with so called Rubin's 63 rules (Raghunathan et al. 2003; Reiter 2002). Recently, Räisä et al. (2023) have shown that multiple 64 imputation also works when the synthetic data is generated under DP when the data generation 65 algorithm is noise-aware in a certain sense. 66

Wilde et al. (2021) study downstream Bayesian inference from DP synthetic data by considering 67 the analyst's model to be misspecified, and targeting a generalised notion of the posterior (Bissiri 68 et al. 2016) to deal with the misspecification, which makes method their more difficult to apply than 69 standard Bayesian inference. They also assume that the analyst has additional public data available to 70 calibrate their method. 71

Ghalebikesabi et al. (2022) use importance sampling to correct for bias with DP synthetic data, 72 and have Bayesian inference as an example application. However, they also target a generalised 73 variant (Bissiri et al. 2016) of the posterior instead of the noise-aware posterior we target, and they do 74 not evaluate uncertainty estimation, so the noise-awareness of their method is not clear. 75

Weare not aware of any existing work adapting multiple imputation for Bayesian downstream analysis 76 in the synthetic data setting. In the missing data setting without DP, where multiple imputation was 77 originally developed (Rubin 1987), Gelman et al. (2014) have proposed sampling the downstream 78 posterior by mixing samples of the downstream posteriors from each of the multiple synthetic datasets. 79 We find that this is not sufficient in the synthetic data setting, and add one extra component: our 80 synthetic datasets are larger than the original dataset. We compare the two cases in more detail in 81 Supplemental Section F, and in particular explain why large synthetic datasets are not needed in the 82 missing data setting. 83

Noise-aware DP Bayesian inference is critical for taking into account the DP noise in synthetic data, 84 but only a few works address this even without synthetic data. Bernstein and Sheldon (2018) present 85 an inference method for simple exponential family models. Their approach was extended to linear 86 models (Bernstein and Sheldon 2019) and generalised linear models (Kulkarni et al. 2021). Recently, 87 Ju et al. (2022) developed an MCMC sampler that can sample the noise-aware posterior using a noisy 88

2 Background on Bayesian Inference 90

Bayesian inference is a paradigm of statistical inference where the data analyst's uncertainty in a 91 quantity Q after observing data X is represented using the posterior distribution p ( Q | X ) (Gelman 92 et al. 2014). The posterior is given by Bayes' rule: 93

where p ( X | Q ) is the likelihood of observing the data X for a given value of Q , and p ( Q ) is the 94 analyst's prior of Q . Computing the denominator is typically intractable, so analysts often use 95 numerical methods to sample p ( Q | X ) (Gelman et al. 2014). 96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

where A is any measurable set.

As a slight abuse of notation, we allow the arguments of TV( · , · ) to be random variables, probability distributions, or probability density functions interchangeably. We list some properties of total variation distance that we use in Lemma A.1 in the Supplement.

Now we can state the theorem.

Theorem 2.2 (Bernstein-von Mises (van der Vaart 1998)) . Let n denote the size of the dataset X n . Under regularity conditions stated in Condition A.4 in Supplemental Section A.2, for true parameter value Q 0 , the posterior ¯ Q ( X n ) ∼ p ( Q | X n ) satisfies as n → ∞ for some µ ( X n ) and Σ , that do not depend on the prior, where the convergence in probability is over sampling X n ∼ p ( X n | Q 0 ) .

3 Bayesian Inference from Synthetic Data

When the downstream analysis is Bayesian, and the analyst has access to non-DP synthetic data, they would ultimately want to obtain the posterior p ( Q | X,I A ) of some quantity Q given real data X , where I A denotes the background knowledge such as priors of the analyst. In the DP case, the exact posterior is unobtainable, so we assume that X is only available through a noisy summary ˜ s (Ju et al. 2022; Räisä et al. 2023), so the posterior is p ( Q | ˜ s, I A ) . To unify these notations, we use Z to denote the observed values, so Z = X in the non-DP case, Z = ˜ s in the DP case, and the posterior of interest is p ( Q | Z, I A ) . We summarise these random variables and their dependencies in Figure 1, and give an introduction to DP in Supplemental Section A.3.

In order to introduce the synthetic data into the posterior of interest, we can decompose the posterior 126 as 127

where we abuse notation by using X ∗ as the variable to integrate over, so inside the integral X ∗ is 128 not a random variable. The decomposition in (4) means that we could sample p ( Q | Z, I A ) by first 129

Bernstein-von Mises Theorem It turns out that in many typical settings, the prior's influence on the posterior vanishes when the dataset X is large. A basic example of this is the Bernstein-von Mises theorem (van der Vaart 1998), which informally states that under some regularity conditions, the posterior approaches a Gaussian that does not depend on the prior as the size of the dataset increases.

A crucial component of the theorem, and also our theory, is the notion of total variation distance between random variables, which is used to measure the difference between two random variables or probability distributions.

Definition 2.1. The total variation distance between random variables (or distributions) P 1 and P 2

is

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

θ : data generating model parameters

X : real data

X ∗ : hypothetical data

Z : observed summary of X ( Z = X without DP)

X Syn : synthetic data, X Syn ∼ p ( X ∗ | Z, I S )

Q : estimated quantity in downstream analysis

I S : synthetic data generator's background infor- mation

I A : analyst's background information

θ

Q

X

X

∗

Z

X

Syn

sampling the synthetic data from the posterior predictive X Syn ∼ p ( X ∗ | Z, I A ) , and then sampling Q ∼ p ( Q | Z, X ∗ = X Syn , I A ) .

/negationslash

Note that the random variable X ∗ represents a hypothetical real dataset that could be obtained if more data was collected, as seen in Figure 1, and it is not the synthetic dataset. The synthetic dataset X Syn is a sample from the conditional distribution of X ∗ given Z . For this reason, p ( Q | Z, X ∗ , I A ) = p ( Q | Z, I A ) . To make our notation less cluttered, we write p ( · | X ∗ , · ) in place of p ( · | X ∗ = X Syn , · ) in probabilities when the meaning is clear.

There are still two major issues with the decomposition in (4):

Sampling p ( Q | Z, X ∗ , I A ) requires access to Z , which defeats the purpose of using synthetic data.

X ∗ needs to be sampled conditionally on the analyst's background information I A , while the synthetic data provider could have different background information I S .

To solve the first issue, in Section 3.2 we show that if we replace p ( Q | Z, X ∗ , I A ) inside the integral of (4) with p ( Q | X ∗ , I A ) , the resulting distribution converges to the desired posterior,

in total variation distance as the size of each synthetic data set X ∗ grows. It should be noted that many such synthetic data sets will be needed to account for the integral over X ∗ .

The second issue is known as congeniality in the multiple imputation literature (Meng 1994; Xie and Meng 2016). We look at congeniality in the context of Bayesian inference from synthetic data in Section 3.1, and find that we can obtain p ( Q | Z, I A ) under appropriate assumptions on the relationship between I A and I S .

Exactly sampling the LHS of (5) requires generating a synthetic dataset for each sample of p ( Q | Z, I A ) , 150 which is not practical. However, we can perform a Monte-Carlo approximation for p ( Q | Z, I A ) by 151 generating m synthetic datasets X Syn 1 , . . . , X Syn m ∼ p ( X ∗ | Z, I A ) , drawing multiple samples from 152 each of the p ( Q | X ∗ = X Syn i , I A ) , and mixing these samples, which allows us to obtain more than 153 one sample of p ( Q | Z, I A ) per synthetic dataset. We look at some properties of this in Supplemental 154 Section E, but we use the integral form in (5) in the rest of our theory. 155

3.1 Congeniality 156

157

158

159

In the decomposition (4) of the analyst's posterior, X ∗ should be sampled conditionally on the analyst's background information I A , while in reality the synthetic data provider could have different background information I S .

160

161

162

163

164

165

166

167

168

A similar distinction has been studied in the context of missing data (Meng 1994; Xie and Meng 2016), where the imputer of missing data has a similar role as the synthetic data generator. Meng (1994) found that Rubin's rules implicitly assume that the probability models of both parties are compatible in a certain sense, which Meng (1994) defined as congeniality .

As our examples with Gaussian distributions in Section 4.1 and Supplemental Section C.2 show, some notion of congeniality is also required in our setting. However, because we study synthetic data instead of imputation, and Bayesian instead of frequentist downstream analysis, we need a different formal definition. As the analyst only makes inferences on Q , it suffices that both the analyst and synthetic data generator make the same inferences of Q :

169

for all X ∗ and 170

171

172

Definition 3.1. The background information sets I S and I A are congenial for observation Z if

In the non-DP case, (7) is redundant, as it is implied by (6), but in the DP case, both are needed, as the parties may draw different conclusions on X given Z = ˜ s .

Combining congeniality and (5), 173

where the convergence is in total variation distance as the size of X ∗ grows. In the following, we 174 assume congeniality, and drop I A and I S from our notation. 175

3.2 Consistency Proof 176

To recap, we want to prove that the posterior from synthetic data, 177

converges in total variation distance to p ( Q | Z ) as the size n of X ∗ n grows. We prove this in 178 Theorem 3.4, which requires that both p ( Q | Z, X ∗ n ) and p ( Q | X ∗ n ) approach the same distribution as 179 n grows. We formally state this in Condition 3.2. In Lemma 3.3, we show that Condition 3.2 is a 180 consequence of the Bernstein-von Mises theorem (Theorem 2.2) under some additional assumptions, 181 so we expect it to hold in typical settings. 182

183

To make the notation more compact, let ¯ Q + n ∼ p ( Q | Z, X ∗ n ) , and let ¯ Q n ∼ p ( Q | X ∗ n ) .

184

185

186

187

188

189

Condition 3.2. For all Q there exist distributions D n such that

as n →∞ , where the convergence in probability is over sampling X ∗ n ∼ p ( X ∗ n | Z, Q ) .

Theorem 2.2 implies Condition 3.2 with some additional assumptions:

Lemma 3.3. If the assumptions of Theorem 2.2 (Condition A.4) and the following assumptions:

(1) Z and X ∗ are conditionally independent given Q ; and

(2) p ( Z | Q ) > 0 for all Q , hold for the downstream analysis for all Q 0 , then Condition 3.2 holds. 190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

Proof. The full proof is in Supplemental Section B.1. Proof idea: when Z and X ∗ are conditionally independent given Q , so p ( Q | Z, X ∗ ) can be equivalently seen as the result of Bayesian inference with observed data X ∗ and prior p ( Q | Z ) . As the only difference to p ( Q | X ∗ ) is the prior, the Bernstein-von Mises theorem implies that both p ( Q | Z, X ∗ ) and p ( Q | X ∗ ) converge in total variation distance to the same distribution.

Assumption (1) of Lemma 3.3 will hold if the downstream analysis treats its input data as an i.i.d. sample from some distribution. Assumption (2) holds when the likelihood is always positive, and in the DP case when the density of the privacy mechanism is positive everywhere, which is the case for common DP mechanisms like the Gaussian and Laplace mechanisms (Dwork and Roth 2014).

Next is the main theorem of this work: (5) holds under Condition 3.2.

Theorem 3.4. Under congeniality and Condition 3.2, TV( p ( Q | Z ) , ¯ p n ( Q )) → 0 as n →∞ .

Proof. The full proof is in Supplemental Section B.1. Proof idea: the proof consists of three steps. The first two are in Lemma B.1 and the third is in Lemma B.2 in the Supplement. The first step is showing that TV( ¯ Q n , ¯ Q + n ) P - → 0 when X ∗ n ∼ p ( X ∗ n | Z, Q ) for fixed Z and Q . This is a simple consequence of the triangle inequality and Condition 3.2, as total variation distance is a metric. In the second step, we show that TV( ¯ Q n , ¯ Q + n ) P - → 0 also holds when X ∗ n ∼ p ( X ∗ n | Z ) . In the final step, we show that this implies the claim.

3.3 Convergence Rate

Under stronger regularity conditions, we can get a convergence rate for Theorem 3.4. The regularity conditions depend on uniform integrability:

Definition 3.5. A sequence of random variables X n is uniformly integrable if

Now we can state the regularity conditions for a convergence rate O ( R n ) :

Condition 3.6. There exist distributions D n such that for a sequence R 1 , R 2 , · · · > 0 , R n → 0 as n →∞ , are uniformly integrable when X ∗ n ∼ p ( X ∗ n | Z ) .

Note that X ∗ n ∼ p ( X ∗ n | Z ) conditions on Z , not Q and Z like in Condition 3.2. We prove that Condition 3.6 is met in univariate Gaussian mean estimation for R n = 1 √ n in Theorem D.1 in the Supplement. This is the same rate that commonly applies in the Bernstein-von Mises theorem (Hipp and Michel 1976).

Condition 3.6 implies an O ( R n ) convergence rate:

Theorem 3.7. Under congeniality and Condition 3.6, TV( p ( Q | Z ) , ¯ p n ( Q )) = O ( R n ) .

Proof. The full proof is in Supplemental Section B.2. Proof idea: first, we prove the uniform integrability of 1 R n TV( ¯ Q n , ¯ Q + n ) when X ∗ n ∼ p ( X ∗ n | Z ) by using the triangle inequality and properties of uniform integrability. Second, we prove that this implies the claimed convergence rate.

4 Examples

In this section, we present two examples of downstream inference from synthetic data at a high level. 227 First, we demonstrate univariate Gaussian mean estimation. Second, we have logistic regression on 228 a toy dataset, with DP synthetic data. In the first example, we use the tractability of the model to 229 derive additional theoretical properties, and in both examples, we experimentally verify our theory. 230

)

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=16,font=/AAAAAU+DejaVuSans>GLYPH<c=31,font=/AAAAAU+DejaVuSans>    GLYPH<c=25,font=/AAAAAU+DejaVuSans>GLYPH<c=23,font=/AAAAAU+DejaVuSans>"GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=17,font=/AAAAAU+DejaVuSans>GLYPH<c=31,font=/AAAAAU+DejaVuSans>$GLYPH<c=30,font=/AAAAAU+DejaVuSans>!"  GLYPH<c=25,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=29,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=20,font=/AAAAAU+DejaVuSans>

GLYPH<c=22,font=/AAAAAU+DejaVuSans> GLYPH<c=27,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=30,font=/AAAAAU+DejaVuSans>GLYPH<c=23,font=/AAAAAU+DejaVuSans>GLYPH<c=25,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=14,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=9,font=/AAAAAU+DejaVuSans>

GLYPH<c=10,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAU+DejaVuSans>

GLYPH<c=12,font=/AAAAAU+DejaVuSans>

GLYPH<c=13,font=/AAAAAU+DejaVuSans>

GLYPH<c=18,font=/AAAAAU+DejaVuSans>GLYPH<c=30,font=/AAAAAU+DejaVuSans>GLYPH<c=23,font=/AAAAAU+DejaVuSans>GLYPH<c=31,font=/AAAAAU+DejaVuSans>    GLYPH<c=25,font=/AAAAAU+DejaVuSans>GLYPH<c=23,font=/AAAAAU+DejaVuSans>"GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=17,font=/AAAAAU+DejaVuSans>GLYPH<c=31,font=/AAAAAU+DejaVuSans>$GLYPH<c=30,font=/AAAAAU+DejaVuSans>!"  GLYPH<c=25,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=29,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=20,font=/AAAAAU+DejaVuSans>

GLYPH<c=22,font=/AAAAAU+DejaVuSans> GLYPH<c=27,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=30,font=/AAAAAU+DejaVuSans>GLYPH<c=23,font=/AAAAAU+DejaVuSans>GLYPH<c=25,font=/AAAAAU+DejaVuSans>

GLYPH<c=15,font=/AAAAAU+DejaVuSans>GLYPH<c=30,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=28,font=/AAAAAU+DejaVuSans>%!"GLYPH<c=4,font=/AAAAAU+DejaVuSans>!GLYPH<c=3,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAV+DejaVuSans-Oblique>

&

GLYPH<c=7,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=7,font=/AAAAAU+DejaVuSans>

GLYPH<c=5,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=4,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAU+DejaVuSans>

GLYPH<c=17,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>"GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=19,font=/AAAAAU+DejaVuSans> GLYPH<c=31,font=/AAAAAU+DejaVuSans>#GLYPH<c=27,font=/AAAAAU+DejaVuSans>GLYPH<c=24,font=/AAAAAU+DejaVuSans>GLYPH<c=25,font=/AAAAAU+DejaVuSans> GLYPH<c=4,font=/AAAAAU+DejaVuSans>!GLYPH<c=3,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAV+DejaVuSans-Oblique>

&

GLYPH<c=7,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=7,font=/AAAAAU+DejaVuSans>

GLYPH<c=5,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAU+DejaVuSans>

GLYPH<c=21,font=/AAAAAU+DejaVuSans>GLYPH<c=27,font=/AAAAAU+DejaVuSans>"GLYPH<c=26,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>!%GLYPH<c=30,font=/AAAAAU+DejaVuSans>GLYPH<c=8,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>GLYPH<c=24,font=/AAAAAU+DejaVuSans>GLYPH<c=22,font=/AAAAAU+DejaVuSans>"GLYPH<c=22,font=/AAAAAU+DejaVuSans>GLYPH<c=3,font=/AAAAAU+DejaVuSans>

GLYPH<c=9,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=8,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAAAU+DejaVuSans>

GLYPH<c=11,font=/AAAAAV+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAU+DejaVuSans>

GLYPH<c=31,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=7,font=/AAAAAY+DejaVuSans>

GLYPH<c=10,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>GLYPH<c=8,font=/AAAAAY+DejaVuSans>

GLYPH<c=7,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=14,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=7,font=/AAAAAY+DejaVuSans>

GLYPH<c=10,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=15,font=/AAAAAY+DejaVuSans>GLYPH<c=8,font=/AAAAAY+DejaVuSans>

GLYPH<c=7,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=14,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=12,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=7,font=/AAAAAY+DejaVuSans>

GLYPH<c=10,font=/AAAAAY+DejaVuSans>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=13,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>GLYPH<c=8,font=/AAAAAY+DejaVuSans>

GLYPH<c=7,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=16,font=/AAAAAY+DejaVuSans>

GLYPH<c=14,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>GLYPH<c=11,font=/AAAAAY+DejaVuSans>

GLYPH<c=17,font=/AAAAAY+DejaVuSans>GLYPH<c=24,font=/AAAAAY+DejaVuSans>GLYPH<c=19,font=/AAAAAY+DejaVuSans>GLYPH<c=23,font=/AAAAAY+DejaVuSans>GLYPH<c=27,font=/AAAAAY+DejaVuSans>GLYPH<c=25,font=/AAAAAY+DejaVuSans>GLYPH<c=26,font=/AAAAAY+DejaVuSans>GLYPH<c=4,font=/AAAAAY+DejaVuSans>GLYPH<c=25,font=/AAAAAY+DejaVuSans>GLYPH<c=3,font=/AAAAAY+DejaVuSans>

GLYPH<c=9,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=28,font=/AAAAAY+DejaVuSans>

GLYPH<c=6,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=8,font=/AAAAAY+DejaVuSans>

GLYPH<c=5,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=4,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAY+DejaVuSans>

GLYPH<c=18,font=/AAAAAY+DejaVuSans>GLYPH<c=22,font=/AAAAAY+DejaVuSans>GLYPH<c=26,font=/AAAAAY+DejaVuSans>GLYPH<c=21,font=/AAAAAY+DejaVuSans>GLYPH<c=3,font=/AAAAAY+DejaVuSans>GLYPH<c=25,font=/AAAAAY+DejaVuSans>GLYPH<c=27,font=/AAAAAY+DejaVuSans>GLYPH<c=24,font=/AAAAAY+DejaVuSans>GLYPH<c=9,font=/AAAAAY+DejaVuSans>GLYPH<c=3,font=/AAAAAY+DejaVuSans>GLYPH<c=20,font=/AAAAAY+DejaVuSans>GLYPH<c=19,font=/AAAAAY+DejaVuSans>GLYPH<c=26,font=/AAAAAY+DejaVuSans>GLYPH<c=19,font=/AAAAAY+DejaVuSans>GLYPH<c=3,font=/AAAAAY+DejaVuSans>

GLYPH<c=9,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=8,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAAAY+DejaVuSans>

GLYPH<c=11,font=/AAAAAZ+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAAAY+DejaVuSans>

Supplemental Section C, contains more detailed descriptions of the examples, and some additional 231 results. Supplemental Section D proves an O ( 1 √ n ) convergence rate for Theorem 3.4 in the Gaussian 232 mean estimation case. Our code is in the supplementary material. 233

234

4.1 Non-private Gaussian Mean Estimation

Our first example is very simple: the analyst infers the mean µ of a Gaussian distribution with known 235 variance from synthetic data that has been generated from the same model. The posteriors for this 236 setting can be found in Supplemental Section A.4. To differentiate the variables for the analyst and 237 data provider, we use bars for the data provider (like ¯ σ 2 0 ) and hats for the analyst (like ˆ σ 2 0 ) . 238

When the synthetic data is generated from the known variance model with known variance ¯ σ 2 k , we 239 sample from the posterior predictive p ( X ∗ | X ) as 240

N n X ∗ denotes a Gaussian distribution over n X ∗ i.i.d. samples. 241

When downstream analysis is the model with known variance ˆ σ 2 k , we have 242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

Now, using µ ∗ to denote a sample from the mixture of posteriors from synthetic data ¯ p n ( µ ) in (9), we show in Supplemental Section C.1 that

as n X ∗ →∞ , so µ ∗ asymptotically has the same mean and variance as the downstream posterior distribution p ( µ | X ) on the real data.

We test the theory with a numerical simulation in Figure 2. We generated the real data X of size n X = 100 by i.i.d. sampling from N (1 , 4) . Both the analyst and data provider use N (0 , 10 2 ) as the prior. The data provider uses the correct known variance ( ¯ σ 2 k = 4 ), and the analyst either uses the correct known variance ( ˆ σ 2 k = 4 ), or a too small known variance ( ˆ σ 2 k = 1 ), which is an example of uncongeniality.

In the congenial case in the left panel of Figure 2, both parties have the same posterior given the real data X , and the mixture of posteriors from synthetic data is very close to that. In the uncongenial case in the right panel, where the analyst underestimates the variance, the parties have different posteriors given X , but the mixture of synthetic data posteriors is still close to the data provider's posterior.

In Figure 3, we examine the convergence of the mixture of posteriors from synthetic data under congeniality. We see that setting n X ∗ = n X is not enough, as the mixture of posteriors is significantly wider than the analyst's posterior. The synthetic dataset needs to be larger than the original, with n X ∗ = 5 n X already giving a decent approximation and n X ∗ = 20 n X a rather good one. In Figure S1 in the Supplement, we also examine the effect of m on the mixture of synthetic data posteriors, and see that m must also be sufficiently large, otherwise the method produces very jagged posteriors.

4.2 Differentially Private Logistic Regression

(

)GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=10,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=10,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=20,font=/AAAABC+DejaVuSans>

GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=31,font=/AAAABC+DejaVuSans>GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=28,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>!GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=31,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=10,font=/AAAABC+DejaVuSans>

GLYPH<c=11,font=/AAAABC+DejaVuSans>

GLYPH<c=15,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=29,font=/AAAABC+DejaVuSans> GLYPH<c=27,font=/AAAABC+DejaVuSans>!"

GLYPH<c=14,font=/AAAABC+DejaVuSans>GLYPH<c=30,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=26,font=/AAAABC+DejaVuSans>GLYPH<c=26,font=/AAAABC+DejaVuSans>GLYPH<c=27,font=/AAAABC+DejaVuSans>GLYPH<c=23,font=/AAAABC+DejaVuSans>GLYPH<c=27,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=29,font=/AAAABC+DejaVuSans>!GLYPH<c=13,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

)GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

)GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

)GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=9,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=12,font=/AAAABC+DejaVuSans>

GLYPH<c=20,font=/AAAABC+DejaVuSans>

GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=31,font=/AAAABC+DejaVuSans>GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=28,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>!GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=31,font=/AAAABC+DejaVuSans>

GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=10,font=/AAAABC+DejaVuSans>

GLYPH<c=11,font=/AAAABC+DejaVuSans>

GLYPH<c=15,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=29,font=/AAAABC+DejaVuSans> GLYPH<c=27,font=/AAAABC+DejaVuSans>!"

GLYPH<c=14,font=/AAAABC+DejaVuSans>GLYPH<c=30,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=26,font=/AAAABC+DejaVuSans>GLYPH<c=26,font=/AAAABC+DejaVuSans>GLYPH<c=27,font=/AAAABC+DejaVuSans>GLYPH<c=23,font=/AAAABC+DejaVuSans>GLYPH<c=27,font=/AAAABC+DejaVuSans>GLYPH<c=25,font=/AAAABC+DejaVuSans>GLYPH<c=29,font=/AAAABC+DejaVuSans>!GLYPH<c=13,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=8,font=/AAAABC+DejaVuSans>

GLYPH<c=21,font=/AAAABC+DejaVuSans>"GLYPH<c=29,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>GLYPH<c=24,font=/AAAABC+DejaVuSans>GLYPH<c=22,font=/AAAABC+DejaVuSans>!GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>

GLYPH<c=7,font=/AAAABD+DejaVuSans-Oblique>

GLYPH<c=6,font=/AAAABD+DejaVuSans-Oblique>

GLYPH<c=4,font=/AAAABC+DejaVuSans>

GLYPH<c=4,font=/AAAABD+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAABC+DejaVuSans>

GLYPH<c=21,font=/AAAABC+DejaVuSans>"GLYPH<c=29,font=/AAAABC+DejaVuSans>GLYPH<c=7,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>GLYPH<c=24,font=/AAAABC+DejaVuSans>GLYPH<c=22,font=/AAAABC+DejaVuSans>!GLYPH<c=22,font=/AAAABC+DejaVuSans>GLYPH<c=3,font=/AAAABC+DejaVuSans>

GLYPH<c=7,font=/AAAABD+DejaVuSans-Oblique>

GLYPH<c=4,font=/AAAABC+DejaVuSans>

GLYPH<c=5,font=/AAAABD+DejaVuSans-Oblique>

#

GLYPH<c=8,font=/AAAABD+DejaVuSans-Oblique>

GLYPH<c=5,font=/AAAABC+DejaVuSans>

GLYPH<c=19,font=/AAAABC+DejaVuSans>GLYPH<c=30,font=/AAAABC+DejaVuSans>GLYPH<c=29,font=/AAAABC+DejaVuSans>GLYPH<c=6,font=/AAAABC+DejaVuSans>GLYPH<c=15,font=/AAAABC+DejaVuSans>GLYPH<c=20,font=/AAAABC+DejaVuSans>

GLYPH<c=15,font=/AAAABC+DejaVuSans>GLYPH<c=20,font=/AAAABC+DejaVuSans>

GLYPH<c=6,font=/AAAABC+DejaVuSans>

GLYPH<c=16,font=/AAAABC+DejaVuSans>GLYPH<c=17,font=/AAAABC+DejaVuSans>GLYPH<c=18,font=/AAAABC+DejaVuSans>

'

Our second example is logistic regression on a simple 3-d binary toy dataset, ( n X = 2000 ), with DP 263 synthetic data, under the same setting as used by Räisä et al. (2023) for frequentist logistic regression. 264 We change the downstream task to Bayesian logistic regression to evaluate our theory. 265

Under DP, Z is a noisy summary ˜ s of the real data. We need synthetic data sampled from the posterior 266 predictive p ( X ∗ | ˜ s ) , which is exactly what the NAPSU-MQ algorithm of Räisä et al. (2023) provides. 267 In NAPSU-MQ, ˜ s is the values of user-selected marginal queries with added Gaussian noise. We used 268 the open-source implementation of NAPSU-MQ 1 by Räisä et al. (2023), and describe NAPSU-MQ 269 in Supplemental Section A.3. 270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

Because of the simplicity of this model, it is possible to use the exact posterior decomposition (4) as a baseline, by using p ( X | ˜ s ) instead of p ( X ∗ | ˜ s ) to generate synthetic data. We give a detailed description of this process in Supplemental Section C.5. We have also included the DP-GLM algorithm (Kulkarni et al. 2021) that does not use synthetic data, and the non-DP posterior from the real data as baselines. We obtained the code for DP-GLM from Kulkarni et al. (2021) upon request.

Figure 4 compares the mixture of posteriors from synthetic data from (9) that uses p ( Q | X ∗ ) , with n X ∗ /n X = 20 and m = 400 synthetic datasets, to the baselines. The posterior from (9) is very close to the posterior from (4). The DP-GLM posterior that does not use synthetic data is somewhat wider. The privacy bounds are /epsilon1 = 1 , δ = n -2 X = 2 . 5 · 10 -7 .

We ran the experiment 100 times and also with /epsilon1 = 0 . 1 and /epsilon1 = 0 . 5 , and plot coverages and widths of credible intervals in Figure S4 in the Supplement. With /epsilon1 = 1 and /epsilon1 = 0 . 5 , the coverages are accurate and DP-GLM consistently produces wider intervals. With /epsilon1 = 0 . 1 , the mixture of synthetic data posteriors likely needs more and larger synthetic datasets to converge, as it produced wider and slightly overconfident intervals for one coefficient.

5 Discussion

Synthetic data are often considered as a substitute for real data that are sensitive. Since the data generation process is based on having access to the Z , one might ask why is the synthetic data needed in first place. Why cannot we simply perform the downstream posterior analysis directly using Z ? Our analysis allows Z to be an arbitrary, even noisy, representation of the data, and it might be difficult for the analyst to place a model for such generative process for Q . In most applications, the analyst does have a model for Q arising from the data. Therefore using the synthetic data as a proxy for the Z allows the analyst to use existing models and inference methods to perform the analysis.

Limitations A clear limitation of mixing posteriors from multiple synthetic datasets is the computational cost of analysing many large synthetic datasets, which may be substantial for more complex Bayesian downstream models, where even a single analysis can be computationally expensive. However, the separate analyses can be run in parallel. We also expect the downstream posteriors of different synthetic datasets to be similar to each other, so it should be possible to use the information gained from sampling a few of them to speed up sampling the others.

Under DP, we need noise-aware synthetic data generation, which limits the settings in which the method can currently be applied. However, if new noise-aware methods are developed in the future, the method can immediately be used with them.

To recover the analyst's posterior, the method requires congeniality, which basically requires the analyst's prior to be compatible with the data provider's. However, the method was still able to recover the data provider's posterior in the Gaussian example, suggesting that the data provider's prior information overrides the analyst's prior information. This suggests an interesting area of future research: analysis methods that override the data provider's prior. An importance sampling approach similar to that of Ghalebikesabi et al. (2022) could provide one approach. This observation also raises interesting questions on whether truly general and objective synthetic data generation is possible.

Conclusion We considered the problem of consistent Bayesian inference of downstream analyses 309 using multiple, potentially DP, synthetic datasets, and studied an inference method that mixes the 310 posteriors from multiple large synthetic datasets. We proved, under general and well-understood 311 regularity conditions of the Bernstein-von Mises theorem, that the method is asymptotically exact as 312 the sizes of the synthetic datasets grow. We also derived a convergence rate under stricter regularity 313 conditions. We studied the method in two examples: non-private Gaussian mean estimation and 314 DP logistic regression. In the former, we were able to use the analytically tractable structure of the 315 setting to derive additional properties of the method, including a convergence rate without additional 316 assumptions. In both settings, we experimentally validated our theory, and showed that the method 317 works in practice. This fills a major gap in the synthetic data analysis literature, unlocking consistent 318 Bayesian inference while reusing existing downstream analysis methods. 319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

References

Balle, B. and Y.-X. Wang (2018). 'Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising'. In: Proceedings of the 35th International Conference on Machine Learning . Vol. 80. Proceedings of Machine Learning Research. PMLR, pp. 394-403.

Bernstein, G. and D. Sheldon (2018). 'Differentially Private Bayesian Inference for Exponential Families'. In: Advances in Neural Information Processing Systems . Vol. 31, pp. 2924-2934.

Bernstein, G. and D. Sheldon (2019). 'Differentially Private Bayesian Linear Regression'. In: Advances in Neural Information Processing Systems . Vol. 32, pp. 523-533.

Billingsley, Patrick. (1995). Probability and Measure . 3rd ed. Wiley Series in Probability and Mathematical Statistics. New York, NY: Wiley.

Bissiri, P. G., C. C. Holmes, and S. G. Walker (2016). 'A General Framework for Updating Belief Distributions'. In: Journal of the Royal Statistical Society. Series B, Statistical Methodology 78.5, pp. 1103-1130.

Duane, S., A. D. Kennedy, B. J. Pendleton, and D. Roweth (1987). 'Hybrid Monte Carlo'. In: Physics letters B 195.2, pp. 216-222.

Dwork, C. (2008). 'Differential Privacy: A Survey of Results'. In: International Conference on Theory and Applications of Models of Computation . Springer, pp. 1-19.

Dwork, C., K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor (2006a). 'Our Data, Ourselves: Privacy Via Distributed Noise Generation'. In: Advances in Cryptology - EUROCRYPT . Vol. 4004. Lecture Notes in Computer Science. Springer, pp. 486-503.

Dwork, C., F. McSherry, K. Nissim, and A. D. Smith (2006b). 'Calibrating Noise to Sensitivity in Private Data Analysis'. In: Third Theory of Cryptography Conference . Vol. 3876. Lecture Notes in Computer Science. Springer, pp. 265-284.

Dwork, C. and A. Roth (2014). 'The Algorithmic Foundations of Differential Privacy'. In: Foundations and Trends in Theoretical Computer Science 9.3-4, pp. 211-407.

Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin (2014). Bayesian Data Analysis . Third edition. Chapman & Hall/CRC Texts in Statistical Science Series. Boca Raton: CRC Press.

Ghalebikesabi, S., H. Wilde, J. Jewson, A. Doucet, S. Vollmer, and C. Holmes (2022). 'Mitigating Statistical Bias within Differentially Private Synthetic Data'. In: Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence . PMLR, pp. 696-705.

Gilks, W. R., N. G. Best, and K. K. C. Tan (1995). 'Adaptive Rejection Metropolis Sampling Within Gibbs Sampling'. In: Journal of the Royal Statistical Society Series C: Applied Statistics 44.4, pp. 455-472.

Hipp, C. and R. Michel (1976). 'On the Bernstein-v. Mises Approximation of Posterior Distributions'. In: The Annals of Statistics 4.5, pp. 972-980.

Hoffman, M. D. and A. Gelman (2014). 'The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.' In: Journal of Machine Learning Research 15.1, pp. 1593-1623.

Ju, N., J. Awan, R. Gong, and V. Rao (2022). 'Data Augmentation MCMC for Bayesian Inference from Privatized Data'. In: Advances in Neural Information Processing Systems . Vol. 35, pp. 1273212743.

Kelbert, M. (2023). 'Survey of Distances between the Most Popular Distributions'. In: Analytics 2.1, pp. 225-245.

Kulkarni, T., J. Jälkö, A. Koskela, S. Kaski, and A. Honkela (2021). 'Differentially Private Bayesian Inference for Generalized Linear Models'. In: Proceedings of the 38th International Conference on Machine Learning . Vol. 139. Proceedings of Machine Learning Research. PMLR, pp. 5838-5849.

Liew, C. K., U. J. Choi, and C. J. Liew (1985). 'A Data Distortion by Probability Distribution'. In: ACM Transactions on Database Systems 10.3, pp. 395-411.

Meng, X.-L. (1994). 'Multiple-Imputation Inferences with Uncongenial Sources of Input'. In: Statistical Science 9.4.

Neal, R. M. (2011). 'MCMC Using Hamiltonian Dynamics'. In: Handbook of Markov Chain Monte Carlo . Chapman & Hall / CRC Press.

Raghunathan, T. E., J. P. Reiter, and D. B. Rubin (2003). 'Multiple Imputation for Statistical Disclosure Limitation'. In: Journal of Official Statistics 19.1, p. 1.

Räisä, O., J. Jälkö, S. Kaski, and A. Honkela (2023). 'Noise-Aware Statistical Inference with Differentially Private Synthetic Data'. In: Proceedings of The 26th International Conference on Artificial Intelligence and Statistics . PMLR, pp. 3620-3643.

Reiter, J. P. (2002). 'Satisfying Disclosure Restrictions with Synthetic Data Sets'. In: Journal of 377 Official Statistics 18.4, p. 531. 378

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys . New York: John Wiley \& 379 Sons. 380

381

382

Rubin, D. B. (1993). 'Discussion: Statistical Disclosure Limitation'. In: Journal of Official Statistics 9.2, pp. 461-468.

383

384

385

386

387

van der Vaart, A. W. (1998). Asymptotic Statistics . Repr. 2000. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.

Wilde, H., J. Jewson, S. J. V ollmer, and C. Holmes (2021). 'Foundations of Bayesian Learning from Synthetic Data'. In: The 24th International Conference on Artificial Intelligence and Statistics . Vol. 130. Proceedings of Machine Learning Research. PMLR, pp. 541-549.

Xie, X. and X.-L. Meng (2016). 'Dissecting Multiple Imputation from a Multi-Phase Inference 388

Perspective: What Happens When God's, Imputer's and Analyst's Models Are Uncongenial?' In: 389 Statistica Sinica . 390

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the use of synthetic data $X^{sync} $ created from a model $p(X^{sync}|Z, I_S)$ on a further Bayesian analysis where the analyst has access to $p(Q|X^{sync})$ and to $p(X^{sync}| Z)$, where $Q$ is the parameter. \n the paper is based on equality (4) \n$ p(Q|Z) = \\int p(Q|Z.X^*)p(X^*|Z)dx^*$ where $X^*$ essentially represents $X^{sync}$ and $Z$ is either the real data or a differentially private version of the data and the idea is to replace \n$\\int p(Q|Z.X^*)p(X^*|Z)dx^*$ by $p_n(Q) = \\int p(Q|X^*)p(X^*|Z)dx^*$ which is fully accessible by the Bayesian and then show that the latter is close to the former. \n\nTo do that the authors assume that \n$p(Q|Z,X^*_n)$ and $p(Q|X^*_n)$ are both close in total variation to the same seq of distribution say $D_n$ in probability when $X^*_n $ follows $p(X^*_n|Z,Q_0)$ and that given $Q$ $X^*$ and $Z$ are independent. \n\nThen the authors treat 2 toy examples a Gaussian and a logistic regression example and run some simulations to illustrate.\n\nInthe above presentation I am not mentioning the fact that the Bayesian models can be different from the generating model, which is treated but quickly pushed by assuming that this is not a problem.\n\nSTRENGTHS:\nThe paper is well motivated and it is an important problem. If the results are correct then the paper is relevant and interesting.\n\nWEAKNESSES:\nI am not sure the results are correct. From the presentation I don't understand the author's eq (4) or rather their comment which says that \n$p(Q|X^*, Z) $ is different from $p(Q|Z)$. The reason is that the authors do not explain what is the generating model for $X^*$ (In their paper the authors sometimes use $X^{sync}$ and sometimes $X^*$ as if they were the same, so I gather that they represent the same thing , but in the DAG of fig 1, they are not the same at all. \n\nThe $X^* = X^{sync}$ is distributed from $p(X^*|Z)$, which does not depend on $Q$. Hence unless the authors clarify this point then their result are not valid. \n\nThe authors consider two toy examples but in neither of them do they check tht the theoretical setup considered before is valid. For instance in the Gaussian example the generating model for the synthetic data is \n$ X^* \\sim p( \\cdot | X) = \\int p(\\cdot | \\mu) \\pi(\\mu| X) d\\mu$, i.e. the posterior predictive density. \nThe relation (4) writes as $\\pi(\\mu|X) = \\int p(\\mu | X, x^*)p(x^*|X)dx^*$ but the model $p(\\mu | X, x^*)$ is not defined. The authors seem to consider that  $x^* , X|\\mu$ are iid but this is not possible because $\\mu$ is unknown and it does not correspond to their Gaussian example. \n\nThere are a number of other results which seem dubious to me. See below.\n\nQUESTIONS:\n1. In condition 3.2 the authors write for all Q but do not mention $Z$ while the distribution depends on $Z$. Is the condition almost sure in Z? in probability ? The same is true for condition 3.6.\n\n2. Lemma 3.3 : What does it mean \" hold for the downstream analysis for all $Q_0$ \" ? condition (1) says given $Q$. Are $Q$ and $Q_)$ the same?\n3. eq (17) says that they have the same mean and variance but not that they have the same limiting distribution. Why don't  the authors verify the assumptions for this toy example. Surely if these assumptions don't hold for that one they will never hold. \n\n4. in the supplement equation (204) : in my version of Asymptotic statistics there is not corollary 2.3 but a continuous mapping Theorem. I imagine that the authors are referring to the Lehman - Scheffe Theorem which states that if the sequence of probability densities $f_n$ converges pointwise to a probability density $f$ then it converges in $L_1$. Hoewever here the sequence is also random and the convergence is pointwise almost surely and I don't see the argument which allows to glue all these sets (for each Q) of probabiulity 1 to apply Lehman Scheffe. In other words the sets of proba 1 may differ from one $Q$ to another. \n\n5. Minor comments: The authors recall in the call fairly trivial probability results which they should quote and recall possibly in the supplement and free this space to better explain their setup.\n\nLIMITATIONS:\nThe authors are conscious of some of the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper works on performing consistent Bayesian inference from synthetic data under DP. The authors propose a solution that involves mixing posterior samples from multiple large synthetic datasets, proving that this technique converges to the posterior of downstream analysis under specific conditions. This was established through experimentation involving non-private Gaussian mean estimation and DP logistic regression.\n\nSTRENGTHS:\nThe paper offers a unique and engaging exploration of Bayesian Inference in the context of Synthetic Data, providing a fresh perspective in a field predominantly characterized by frequentist analysis.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\nThe motivation behind the research is not explicitly stated. Could you clarify the unique benefits that Bayesian Inference offers in this context? How does it enhance the study or application beyond the capabilities of other methodologies (frequentist)?\n\nThe rationale for using Synthetic Data, specifically Synthetic Data without DP, is also vague. This choice seems to offer no additional solid protection under this setting. When applying DP, why choose to release Synthetic Data instead of the DP summary directly?\n\nThe paper's primary theoretical contribution is not evidently defined. The claim that the distribution of Synthetic Data can be arbitrarily close to the original distribution as the sample size 'n' approaches infinity appears to be a trivial expectation. Could you elaborate on this aspect more?\n\nThere has been prior discussion on the topic of Bayesian Inference from Synthetic Data, for example, as seen in reference [1]. Could you specify what new insights or advancements your study brings to the table, beyond the contributions of these previous works?\n\n\n\n[1] Wilde, H., Jewson, J., Vollmer, S., & Holmes, C. (2021, March). Foundations of Bayesian learning from synthetic data. In International Conference on Artificial Intelligence and Statistics (pp. 541-549). PMLR.\n\nLIMITATIONS:\nWhile the study explores the concept of Synthetic Data, its impetus is not distinctly articulated, leading to ambiguity regarding the problem the authors aim to address. The theoretical contribution appears to be weak.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by Bayesian approaches for performing multiple imputation of missing data, this paper investigates the applicability of similar strategies for the analysis of synthetic data. Namely, the paper proposes inferring the downstream posterior of a Bayesian analysis by: generating multiple synthetic datasets; inferring the analysis posterior for each synthetic dataset; and mixing the posteriors together. (Interestingly, the paper finds that, contrary to the missing data imputation context, in the synthetic data case this strategy requires the synthetic datasets to be larger than the original dataset.)  \n\nThe paper provides theory showing that under the regularity conditions of the Bernstein-von Mises theorem (augmented by the additional conditions presented in Lemma 3.3), and assuming the congenial conditions in Definition 3.1, then the proposed strategy will approximate the data provider posterior distribution as the number of synthetic datasets and the synthetic dataset sizes increase. (The paper also proves a convergence rate result under stronger assumptions.)\n\nThe method is evaluated using two simple examples: (i) non-private univariate Gaussian mean estimation (when the variance is assumed to be known); and (ii) differentially private logistic regression.\n\nSTRENGTHS:\nThis is an interesting paper. It addresses an important topic with an approach that appears to be novel and sound.\n\nWEAKNESSES:\nOne limitation of the proposed approach appears to be its reliance on the congeniality assumption (which we should not expect to hold in general). While the paper uses a simple example to illustrate that the method was still able to recover the data provider’s posterior when congeniality was violated, the paper needs to provide more extensive evidence of the robustness of the proposed approach w.r.t. violations of this assumption (as, in practice, it seems that the usefulness of the proposed approach for data analysis will depend on how robust the method is to violations of congeniality).\n\nMore specifically, the paper shows that for the toy problem of Gaussian mean estimation (with known variance) the mixture of posteriors converges to the data provider’s posterior even when the analyst’s variance is different from the data provider’s variance (right panel of Figure 2). However, for this example we have that the posterior distribution for the mean is already Gaussian in the finite sample setting to begin with. Providing additional examples where the posterior distribution of the quantity of interest is not Gaussian in the finite sample setting, but where the mixture of posteriors approximates the data provider’s posterior when congeniality is violated would provide more convincing illustrative examples. Perhaps, one simple example is the problem of Gaussian variance (or precision) estimation with known means. In this case, the paper could assess the robustness w.r.t. congeniality violations by choosing different means for the data provider and data analyst.  The paper should provide additional examples along these lines.\n\nThe paper might also want to include some discussion about some practically important settings where the Bernstein-von Mises theorem does not hold, and where the proposed approach might not be applicable (e.g., for models where the number of parameters increases with the sample size).\n\nOther minor suggestions:\n\nLine 69: change “which makes method their more” to “which makes their method more”\n\nLine 302: change “To recover the analyst’s posterior …” to “To recover the data provider’s posterior …”\n\nQUESTIONS:\nSee suggestions above.\n\nLIMITATIONS:\nYes, the paper addresses well the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies Bayesian inference based on synthetic datasets generated in a DP and non-DP setting. \n\nThe paper suggests a specific sampling approach for downstream Bayesian inference using synthetic DP and non-DP dataset. It contributes theoretical results on the convergence of the inferenced posterior (from synthetic dataset) to the true posterior showing that (under certain assumptions) it converges as the the of number of synthetic datasets and the size of the datasets increases. Additionally, a convergence rate is derived \nExperimental results are provided for non-DP Bayesian mean inference and DP Bayesian logistic regression showing that the inference approach works, along with examples of the effect of parameters influencing the convergence (including number of observations, samples and level of congeniality)\n\nSTRENGTHS:\n- Very timely and interesting topic; I enjoyed learning about the specific Bayesian+DP setting.\n- The theory is mostly well presented in the main paper (see suggestion/questions below). The narrative is relatively easy to follow.\n- The theory appears sound. I have not found any obvious issues; however I would need to rely on other reviewers (and perhaps later the community as a whole) to validate the many proofs in the supplementary.\n\nWEAKNESSES:\nThe following are question and comments; not necessarily weaknesses per se:\n\n  -  The balance between theory and experiments is generally very good for my taste, but I feel the experimental part (in the main paper) let the theory part down a bit. The initial experiments focus on intuition and basic insights which I string support; however once the basics have been presented it would had been helpful with an experiment which covers many more scenarios proving summaries of the performance (using TV, coverage, means/modes and variance as metrics), along the most relevant dimensions such as number of observations, level of \n  -  ... it would also have been interesting with a more realistic example (high dimensional) using a more complicated model to better motivate the paper. Have the authors validated the results on such an example?  \n  -  Figure 4 (right): It is not clear to me why the non-DP posterior does not mange to center its mode closer to the true parameter, is this an effect of the prior being centered on the true parameter combined with relatively few observations (the prior is not specified in the main text as far as I can tell?) - or other things?\n  -  Figure 1: I am slightly confused by the graphical model, probably because the nature and role of $\\theta$ is never really explained in detail. I hope the authors can clarify this (perhaps along with a detailed explanation the generative model in general)? For completeness, I would suggest including $I_a$ and $I_s$ in the figure as well. \n\nOverall, I am generally positive about the paper but the experimental parts misses an opportunity to convince me. I will opt for a borderline score until I get a chance to see the other reviews and the authors' response.\n\nQUESTIONS:\nIncluded above.\n\nLIMITATIONS:\nIncluded above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This work is sloving a interesting task, which infers the downstream analysis posterior using synthetic data. The work proved that the Bernstein-von Mises theroy applies, the method can converage to the ture posterio as the number of synthetic datasets. The experimental settings are under two examples, i.e. non-private univariate Gaussian
56 mean estimation and differentially private Bayesian logistic regression.

STRENGTHS:
1, The work is trying to solve an interesting task, which is infering the downstream analysis posterior using synthetic data.

2, The paper is well-writen and presented. 

3, The code is provided. So it will be helpful for the following work.

WEAKNESSES:
1. Since synthetic data is generated by models which are trained using real data. So why synthetic data can improve the consisten bayesian inference is not clear. I think the paper needs more discussion about differences bewteen the real data and synthetic data.

2, The synthetic data is a big topic. In the work, for me, it is not clear which synthetic data methods are used and how the synthetic data method is trained using real data.

3, The applications are missing. Is it possible to extend the proposed method or therory to some kind of real application.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-28