Fully Synthetic Data for Complex Surveys

Yajuan Si1

Joint work with Shirley Mathur2and Jerry Reiter3

University of Michigan 1

University of Washington 2

Duke University 3

The 2023 FCSM Research & Policy Conference

Oct 25, 2023

1/20
Outline

Introduction

Two Methods for Generating Fully Synthetic Survey Data

Inferences

Simulation Studies

Discussion

2/20
Fully synthetic data

▶ Synthetic data approaches have been proposed for data confidentiality and

privacy protection (Rubin, 1993).

▶ Releasing fully synthetic data sets can preserve confidentiality, since

identification of units and their sensitive data can be difficult when the

released data are not actual, collected values, where the agency

1. randomly and independently samples units from the sampling frame to

comprise each synthetic data set,

2. imputes the unknown data values for units in the synthetic samples using

models fit with the original survey data, and

3. releases multiple versions of these data sets to the public.

▶ Methods for inferences from these multiply-imputed data files have been

developed for a variety of statistical inference tasks.

3/20
Complex sample surveys

▶ Previous research on multiple imputation (MI) for missing data suggests

that imputation models should account for the survey design features,

such as stratification, clustering, and survey weights (Reiter et al., 2006).

▶ Similarly, when using multiple imputation for synthetic data, the models

also should account for the survey design (Mitra and Reiter, 2006;

Fienberg, 2010).

▶ The key challenge is properly incorporating weights in the synthesis

model, which relates to the long-standing debate about the role of survey

weights in model-based inferences (Pfeffermann, 1993, 2011; Little, 2004).

4/20
Existing methods

▶ Bayesian finite population inference approach conditional on design

features available for all population units

▶ Weighted finite population Bayesian bootstrap (WFPBB) (Dong et al.,

2014)

▶ Pseudo-likelihood approach (Pfeffermann, 1993; Savitsky and Toth, 2016;

Kim et al., 2021), in which each individual’s contribution to the likelihood

function of a synthesis model is raised to a power that is a function of the

survey weights

5/20
Potential concerns

▶ The Bayesian finite population inference approach, while theoretically

principled, requires completing full populations, which can be

cumbersome, and the availability of design variables for all records in the

population, which may not be the case in some surveys.

▶ The WFPBB releases (multiple copies of) individuals’ genuine data

records, which creates obvious disclosure risks.

▶ Synthesizing weights does not have a theoretical basis; thus, it is unclear

if this approach can adequately capture uncertainty from complex designs.

▶ Pseudo-likelihood approaches also may not estimate sampling variability

correctly (Williams and Savitsky, 2021), and it is not clear how easily they

can be implemented with machine learning synthesizers like classification

and regression trees, which are commonly used in practical synthetic data

projects.

6/20
New proposal (Mathur et al., 2023)

▶ We build on the WFPBB approach by first creating pseudo-populations

that account for the survey weights.

▶ We then take simple random samples (SRSs) from each

pseudo-population, estimate synthesis models from each SRS, and

generate draws from these models to create multiply-imputed, fully

synthetic public use files.

▶ We consider two synthetic survey data generation processes.

1. We generate multiple synthetic data sets from each SRS; we call this

Synrep-R.

2. We generate one synthetic data set from each SRS; we call this SynRep-1.

▶ For both approaches, we derive multiple imputation combining rules that

enable the estimation of variances.

7/20
Synrep-R

8/20
Synrep-1

9/20
Synthetic data generation process

1. Resample via Bayesian bootstrap: To inject sufficient sampling

variability, given the data from the “parent” sample D, we generate M

samples, (S

(1)

, . . . , S

(M)

), each of size n using independent Bayesian

bootstraps.

2. Use the WFPBB to make pseudo-populations: For each S

(m)

, we

construct an initial P ́olya urn using the set of {Yi, w

(m)

i

}. We then draw

N − n units using probabilities (p

(m)

1

, . . . , p

(m)

n ) determined from

p

(m)

i =

w

(m)

i − 1 + l

(m)

i,k−1

(N − n)/n

N − n + (k − 1)(N − n)/n , (1)

for the kth draw, k ∈ {1, . . . , N − n}, where l

(m)

i,k−1

is the number of

bootstrap selections of Yi among the elements present in the urn at the

k − 1 draw. The N − n draws combined with the data in S

(m)

comprise

one pseudo-population, P

(m)

. We repeat this for m = 1, . . . , M to create

Ppseudo = {P(m): m = 1, . . . , M}.

10/20
Synthetic data generation process (cont.)

3. Draw SRS from each pseudo-population: For m = 1, . . . , M, take a

simple random sample D(m) of size n from P

(m)

. Let

Dsrs = {D(m): m = 1, . . . , M}.

4. Generate synthetic data replicates: For m = 1, . . . , M, estimate a

synthesis model using D(m), and draw from the predictive distributions to

form synthetic data replicates using either Step 4a or Step 4b.

4a. SynRep-R: For m = 1, . . . , M, draw R > 1 synthetic replicates D

(m,r)

syn of

size n, where r = 1, . . . , R, using each D(m). We release

Dsyn = {D(m,r)

syn : m = 1, . . . , M; r = 1, . . . , R} including indicators of

which m each D

(m,r)

syn belongs to.

4b. SynRep-1: For m = 1, . . . , M, draw one synthetic data sample D

(m)

syn of

size n from each D(m). Release Dsyn = {D(m)

syn : m = 1, . . . , M}.

11/20
Combining rules for SynRep-R

For each D

(m,r)

syn , let q

(m,r)

syn be the point estimate of Q, and let v

(m,r)

syn be the

estimate of the variance associated with q

(m,r)

syn . The analyst needs to compute

the following quantities.

q ̄

(m)

syn =

XR

r=1

q

(m,r)

syn /R, q ̄syn =

XM

m=1

q ̄

(m)

syn /M (2)

bsyn =

XM

m=1

( ̄q

(m)

syn − q ̄syn)

2

/(M − 1) (3)

w

(m)

syn =

XR

r=1

(q

(m,r)

syn − q ̄

(m)

syn )

2

/(R − 1), w ̄syn =

XM

m=1

w

(m)

syn /M (4)

v ̄syn =

XM

m=1

XR

r=1

v

(m,r)

syn /MR (5)

Tr =



1 + M−1



bsyn − v ̄syn − w ̄syn/R. (6)

We compute approximate 95% intervals for Q as q ̄syn ± t0.975,M−1

√

Tr.

12/20
Combining rules for SynRep-1

The analyst computes each (q

(m)

syn , v

(m)

syn ) by acting is if D

(m)

syn is a SRS of size n

from the population. We require the following quantities for inferences.

q ̄syn =

X

M

m=1

q

(m)

syn /M (7)

bsyn =

X

M

m=1

(q

(m)

syn − q ̄syn)

2

/(M − 1) (8)

v ̄syn =

X

M

m=1

v

(m)

syn /M (9)

Tm =



1 + M−1



bsyn − 2 ̄vsyn. (10)

We compute approximate 95% intervals for Q as q ̄syn ± t0.975,M−1

√

Tm.

13/20
Repeated sampling properties

▶ We conduct simulation studies to compare the repeated sampling

performances of different inferential methods, estimating the finite

population mean.

▶ Alternative estimators include:

1. Pseudo-Pop as the procedure that uses a point estimator of Q ̄ and variance

estimator of (1 + 1/M)B computed with the WFPBB-generated

pseudo-populations (P

(1)

, . . . ,P

(M)

)

2. Pseudo-SRS as the procedure that uses a point estimator of q ̄ and variance

estimator of Raghunathan et al. (2003) computed with (D(1), . . . , D(M))

3. SRSsyn as the procedure that generates synthetic data by using the

unweighted sample mean and standard deviation as plug-in parameters in a

normal distribution

4. Direct using the unweighted sample mean and standard deviation from D,

i.e., ignoring the survey weights

5. HT as the Horvitz and Thompson (1952) estimator and its estimated

variance using D.

14/20
Under a PPS design with different (M, R) values

−0.5

0.0

0.5

M2R10 M5R10 M10R10 M50R10 M50R5

Relative bias (%)

0.95

1.00

1.05

M2R10 M5R10 M10R10 M50R10 M50R5

Ratio of est to emp variances

0.900

0.925

0.950

0.975

1.000

M2R10 M5R10 M10R10 M50R10 M50R5

Coverage rate

Pseudo−Pop Pseudo−SRS SynRep−R SynRep−1 Direct and SRSsyn are not plotted because they result in large biases (≈ 39%) and very low coverage rates (mostly close to 0). The

HT method is not plotted but results in unbiased estimates and near nominal coverage rates, as expected. 15/20
Issues with negative variances

Table: Percentage of negative variance estimates in the PPS simulation studies. When

M = 50, all variance estimates are positive.

(M, R) Method Percentage of negative variances (%)

M2R10 Pseudo-SRS 36

M2R10 SynRep-2 37

M2R10 SynRep-1 43

M5R10 Pseudo-SRS 11

M5R10 SynRep-2 13

M5R10 SynRep-1 21

M10R10 Pseudo-SRS 3

M10R10 SynRep-2 3

M10R10 SynRep-1 10

16/20
Under a SRS design

−0.5

0.0

0.5

M2R10 M5R10 M10R10 M50R10 M50R5

Relative bias (%)

0.95

1.00

1.05

M2R10 M5R10 M10R10 M50R10 M50R5

Ratio of est to emp variances

0.900

0.925

0.950

0.975

1.000

M2R10 M5R10 M10R10 M50R10 M50R5

Coverage rate

Pseudo−Pop Pseudo−SRS SynRep−R SynRep−1 Estimates from SynRep-R and SynRep-1 do not lose much efficiency relative to the

estimators from Pseudo-SRS.

17/20
Discussion

▶ SynRep-R and SynRep-1 represent a general strategy for constructing fully

synthetic data from complex samples: use the WFPBB to undo the

complex design, then replace the confidential values with synthetic data.

▶ There are many related topics worth further investigation, e.g.,

1. use model-based approaches to smooth the weights

2. extend to multivariate data and for various estimands of interest, e.g.,

subdomain means and regression coefficients

3. account for missing survey data

4. posterior sampling inference

5. comparison with other proposed methods for full synthesis with complex

samples

18/20
Acknowledgements

The work was funded by the U.S. National Science Foundation grant (SES

2217456) and a pilot project from the Michigan Center on the Demography of

Aging with funding from the National Institute on Aging (P30 AG012846).

19/20
References

Dong, Q., Elliott, M. R., and Raghunathan, T. E. (2014). A nonparametric method to generate synthetic populations to adjust for

complex sampling design features. Survey Methodology, 40(1):29–46.

Fienberg, S. E. (2010). The relevance or irrelevance of weights for confidentiality and statistical analyses. Journal of Privacy and

Confidentiality, 1:183–195.

Horvitz, D. G. and Thompson, D. J. (1952). A generalization of sampling without replacement from a finite university. Journal of the

American Statistical Association, 47(260):663–685.

Kim, H. J., Drechsler, J., and Thompson, K. J. (2021). Synthetic microdata for establishment surveys under informative sampling.

Journal of Royal Statistical Society, Series A, 184:255–281.

Little, R. J. (2004). To model or not to model? Competing modes of inference for finite population sampling inference for finite

population sampling. Journal of the American Statistical Association, 99:546–556.

Mathur, S., Si, Y., and Reiter, J. P. (2023). Fully synthetic data for complex surveys. https://arxiv.org/abs/2309.09115.

Mitra, R. and Reiter, J. P. (2006). Adjusting survey weights when altering identifying design variables via synthetic data. In

Domingo-Ferrer, J. and Franconi, L., editors, Privacy in Statistical Databases, pages 177–188. New York: Springer-Verlag.

Pfeffermann, D. (1993). The role of sampling weights when modeling survey data. International Statistical Review, 61(2):317–337.

Pfeffermann, D. (2011). Modelling of complex survey data: Why model? Why is it a problem? How can we approach it? Survey

Methodology, 37(2):115–136.

Raghunathan, T. E., Reiter, J. P., and Rubin, D. B. (2003). Multiple imputation for statistical disclosure limitation. Journal of Official

Statistics, 19:1–16.

Reiter, J. P., Raghunathan, T. E., and Kinney, S. K. (2006). The importance of modeling the survey design in multiple imputation for

missing data. Survey Methodology, 32:143–150.

Rubin, D. B. (1993). Discussion: Statistical disclosure limitation. Journal of Official Statistics, 9:462–468.

Savitsky, T. D. and Toth, D. (2016). Bayesian estimation under informative sampling. Electronic Journal of Statistics, 10(1):1677–1708.

Williams, M. R. and Savitsky, T. D. (2021). Uncertainty estimation for pseudo-Bayesian inference under complex sampling.

International Statistical Review, 89(1):72–107.

20/20