Title: Debiasing Synthetic Data Generated by Deep Generative Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep generative model

efficient influence function

inferential utility

synthetic data
Abstract: While synthetic data hold great promise for privacy protection, their statistical analysis poses significant challenges that necessitate innovative solutions. The use of deep generative models (DGMs) for synthetic data generation is known to induce considerable bias and imprecision into synthetic data analyses, compromising their inferential utility as opposed to original data analyses. This bias and uncertainty can be substantial enough to impede statistical convergence rates, even in seemingly straightforward analyses like mean calculation. The standard errors of such estimators then exhibit slower shrinkage with sample size than the typical 1 over root-$n$ rate. This complicates fundamental calculations like p-values and confidence intervals, with no straightforward remedy currently available. In response to these challenges, we propose a new strategy that targets synthetic data created by DGMs for specific data analyses. Drawing insights from debiased and targeted machine learning, our approach accounts for biases, enhances convergence rates, and facilitates the calculation of estimators with easily approximated large sample variances. We exemplify our proposal through a simulation study on toy data and two case studies on real-world data, highlighting the importance of tailoring DGMs for targeted data analysis. This debiasing strategy contributes to advancing the reliability and applicability of synthetic data in statistical inference.

Debiasing Synthetic Data Generated by Deep Generative Models

Abstract

While synthetic data hold great promise for privacy protection, their statistical analysis poses significant challenges that necessitate innovative solutions. The use of deep generative models (DGMs) for synthetic data generation is known to induce considerable bias and imprecision into synthetic data analyses, compromising their inferential utility as opposed to original data analyses. This bias and uncertainty can be substantial enough to impede statistical convergence rates, even in seemingly straightforward analyses like mean calculation. The standard errors of such estimators then exhibit slower shrinkage with sample size than the typical 1 over rootn rate. This complicates fundamental calculations like p-values and confidence intervals, with no straightforward remedy currently available. In response to these challenges, we propose a new strategy that targets synthetic data created by DGMs for specific data analyses. Drawing insights from debiased and targeted machine learning, our approach accounts for biases, enhances convergence rates, and facilitates the calculation of estimators with easily approximated large sample variances. We exemplify our proposal through a simulation study on toy data and two case studies on real-world data, highlighting the importance of tailoring DGMs for targeted data analysis. This debiasing strategy contributes to advancing the reliability and applicability of synthetic data in statistical inference.

1 Introduction

The concept of generating synthetic data as a means of privacy protection was initially introduced by Rubin (1993) within the framework of multiple imputation, a widely used technique for managing the statistical analysis of incomplete data sets. Since its inception, a substantial body of literature on synthetic data has emerged (Raghunathan et al., 2003; Raghunathan, 2021; Drechsler, 2011; Raab et al., 2016; Reiter, 2005), with a recent surge in interest propelled by advancements in deep generative modelling technology (Raghunathan, 2021; van Breugel et al., 2023; Wan et al., 2017; Yan et al., 2022; Nowok et al., 2016; Endres et al., 2022; Hernandez et al., 2022). In this work, we focus on tabular synthetic data and their inferential utility, which captures whether synthetic data can be used to obtain valid estimates and inference for a population parameter (Decruyenaere et al., 2024).

The substantial privacy protection potential offered by synthetic data is marred by significant challenges that undermine their inferential utility (Raab et al., 2016). Previous work by Decruyenaere et al. (2024) has shown that these challenges are much more severe when using deep generative models (DGMs), rather than parametric statistical models, which is why we focus on the former. First, standard confidence intervals and p-values obtained on synthetic data may drastically underestimate the uncertainty in synthetic data as these ignore the size of the original data. Indeed, synthetic data obtained via generators trained on small datasets will unsurprisingly deliver much worse quality than synthetic data obtained via generators trained on large datasets. This uncertainty in the generator must therefore be translated into analysis results, such as confidence intervals and p-values. Standard confidence intervals and p-values ignore this, as they do not distinguish whether the data are synthetic or real. Second, it is well known from the literature on plug-in estimation that data-adaptive methods (such as DGMs) cannot succeed to estimate all features of the data-generating distribution well (Bickel et al., 1993; van der Laan and Rose, 2011; Chernozhukov et al., 2018; Hines et al., 2022). These methods are designed to optimally balance bias and variance w.r.t. a chosen criterion, such as prediction error. However, they cannot guarantee that such an optimal trade-off is simultaneously made w.r.t. all possible discrepancy measures that exist, such as mean squared error in specific functionals (mean, variance, least squares projections...) of the observed data distribution. Such data-adaptive methods therefore leave non-negligible bias in estimators of such functionals, leading to excess variability, slow convergence, and confidence intervals that do not cover the truth at nominal level (and may even never contain the truth, even in large samples) (Decruyenaere et al., 2024).

Related work. While several approaches have been proposed to account for the uncertainty arising from synthetic data generation, we are not aware of strategies for generating and analysing DGMbased synthetic data that guarantee valid inference. Raghunathan et al. (2003) developed a framework inspired by the work on multiple imputation for missing data, by combining the results of multiple synthetic datasets, but this is not readily applicable to DGM-based synthetic data. Räisä et al. (2023) extended this work for differentially private (DP) synthetic data, acknowledging the additional DP noise, but continue to consider parametric (Bayesian) data generation strategies.

Our work instead focuses on obtaining valid inference from a single synthetic dataset, which is arguably more attractive for use by practitioners. Raab et al. (2016) derived alternative combining rules that reduce to the correction factor √ 1 + m/n for the standard error (SE) of an estimator in the case of inference from a single (non-DP) synthetic dataset of size m generated from an original dataset of size n . The method suggested by Awan and Cai (2020) to preserve efficient estimators in a single (DP or non-DP) synthetic dataset relies on generating data conditional on the estimate in the original data. Both procedures are only applicable to parametric generative models and therefore suffer from the same limitation as the aforementioned approaches. To enable Bayesian inference from a single DP synthetic dataset, Wilde et al. (2021) proposed a corrected analysis that relies on the availability of additional public data, while Ghalebikesabi et al. (2022) investigated importance weighting methods to remove the noise-related bias, but they do not study the impact on inference.

Contributions. Our work is the first to propose a generator-agnostic solution that mitigates the impact of the typical slower-than-√ n -convergence of estimators in synthetic data created by DGMs. As far as we are aware, our approach is thus the only one that provides some formal guarantees for (more) honest inference in this setting. In this paper, we show how the statistical bias in estimators can be removed, by adapting results on debiased or targeted machine learning (van der Laan and Rose, 2011; Chernozhukov et al., 2018) to the current setting where machine learning is not necessarily used in the analysis, but rather in the generation of synthetic data. Although we build upon ideas from existing work, our extension is non-trivial, since (1) previous work did not consider synthetic data; and (2) we demonstrate, with significant generality, how to mitigate the estimation errors in the DGMthat would otherwise propagate into the estimator calculated on synthetic data.

In Section 2 and 3, we propose a generator-agnostic debiasing strategy, directed towards the downstream statistical analysis of the synthetic data. As such, we obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting the generators. We illustrate this with a simulation study in Section 4, showing that the coverage of both the mean and linear regression coefficient estimators indeed improves. In Section 5, we further cement the utility of our debiasing strategy in a practical setting through two case studies. While the proposed strategy is generator-agnostic, we focus our analyses on two DGMs for tabular data: CTGAN and TVAE (Xu et al., 2019). Finally, Section 6 concludes with a discussion on our method and its limitations.

2 Notation and Set-Up

The aim of this paper is to use synthetic data in order to learn a specific functional θ ( . ) of the observed data distribution P . We formalise the problem of learning θ ( P ) based on synthetic data as follows. Suppose that, based on n independent (possibly high-dimensional) samples O 1 , ..., O n from P , we estimate the observed data distribution as ̂ P n . Here, ̂ P n may be obtained by fitting parametric models to the distribution of the observed data, or alternatively by training DGMs. Based on m independent (synthetic) random samples S 1 , ..., S m from ̂ P n , we then estimate θ ( P ) . For this, the data analyst (to whom ̂ P n is unknown) uses the m synthetic samples S 1 , ..., S m to approximate ̂ P n as ˜ P m , and obtains an estimator of θ ( P ) given by θ ( ˜ P m ) . We use P n to denote the empirical distribution of the observed data and ˜ P m to denote the empirical distribution of the synthetic data. We index expectations by the distribution under which they are taken; e.g., E P ( Y ) denotes the population expectation of Y .

Throughout, we assume that the parameter of interest is sufficiently smooth in the data-generating distribution so that rootn consistent estimators (i.e., with SEs shrinking at 1 over rootn rate) can be obtained. Formally, we assume the θ ( P ) is pathwise differentiable with efficient influence curve (EIC) ϕ ( ., P ) under the nonparametric model, as is satisfied for (most) standard statistical analyses. The EIC is a functional derivative of θ ( P ) w.r.t. the data-generating distribution P (in the sense of a Gateaux derivative), which has mean zero under P (Fisher and Kennedy, 2021; Hines et al., 2022).

In what follows, we illustrate our debiasing strategy via two examples. Here, we briefly outline some of their theoretical foundations we build upon. Appendix A.1 clarifies how these debiased estimators originate from their EIC. We further refer to them as debiased or EIC-based estimators.

Population mean. Adapting the traditional formulation of the population mean to the context of synthetic data, we choose ˜ P m = ˜ P m . As a result, we will study the large sample behaviour of the synthetic data sample average:

The EIC of this sample average is ϕ ( S, ̂ P n ) = S -θ ( ̂ P n ) .

Linear regression coefficient. Second, suppose we are interested in a specific coefficient θ ≡ θ ( P ) of some exposure A for the outcome Y in the linear model E P ( Y | A,X ) = θA + ω ( X ) , where X is a possibly high-dimensional vector of covariates and ω ( . ) is an unknown function. Our proposal allows ω ( X ) to correspond to a standard linear model in all covariates, but is more generally valid. Building upon the nonparametric definition of θ as given in Appendix A.1, we adjust it to obtain an estimator in the setting of synthetic data. Let Y , X and A be jointly or sequentially modelled by some generative model, from which a single synthetic dataset S i = ( ˜ Y i , ˜ A i , ˜ X i ) is sampled ( i = 1 , . . . , m ). We then consider the estimated regression coefficient of exposure A given by

where the nuisance parameters E ˜ P m ( A | ˜ X i ) and E ˜ P m ( Y | ˜ X i ) are estimated based on synthetic data. This EIC-based estimator coincides with the Maximum Likelihood Estimator (MLE) when least squares predictions for these nuisance parameters are used. Its EIC is (Vansteelandt and Dukes, 2022)

3 Methodology

In a first step, we establish an understanding of how much the estimate based on synthetic data θ ( ˜ P m ) differs from the population parameter of interest θ ( P ) . For this, we study the difference θ ( ˜ P m ) -θ ( P ) , for which we consider 2 von Mises expansions (i.e., functional Taylor expansions).

In Appendix A.2 we derive that

We now examine the eight terms of this equation and discuss why some may be negligible while other may introduce bias. First, R ( · ) are remainder terms, which can generally be shown to be small, but must be studied on a case-by-case basis; see Hines et al. (2022) for worked out examples. In order for these remainder terms to be o p ( m -1 / 2 ) and o p ( n -1 / 2 ) , we generally need that faster than n -1 / 4 convergence rates are obtained for the unknown functionals of ˜ P m and ̂ P n that appear in the EICs. It is unknown whether these convergence rates are attainable for DGMs; whether they are, will partly depend on the number of parameters in the DGM itself, the dimension of the data and the complexity of the observed data distribution. The simulation study in Section 4 will give further insight into this.

Second, the two empirical process terms (1) and (2) in the von Mises expansion can be shown to be o p ( m -1 / 2 ) and o p ( n -1 / 2 ) by Markov's inequality, under weak conditions. For (1) to converge to zero, we will need the difference between ˜ P m and ̂ P n to converge to zero (in L 2 ( ̂ P n ) at any rate), and the estimator to be calculated on a different part of the data than the one on which ˜ P m was estimated (Chernozhukov et al., 2018). For (2) to converge to zero, we will need ̂ P n (e.g., the DGM) to consistently estimate P (in L 2 ( P ) at any rate). In addition, it can be argued that the DGM needs to be trained on a different part of the data than that on which the debiasing step (see later) will be applied. In Appendix A.3, we elaborate on the necessity of sample splitting.

We thus foresee that the two remainder and two empirical process terms are negligible under certain conditions. Third, as elaborated in Appendix A.2, we show that the large sample behaviour of both

is standard, and well understood; in particular, these terms vary around zero with variance that can be estimated well. By contrast, the terms

need some further discussion. Term (5) generally fails to have mean zero because the synthetic data S i do not originate from the distribution ˜ P m . This term therefore induces a bias in θ ( ˜ P m ) that results from using data-adaptive estimates ˜ P m on the synthetic data; a similar term would appear if we instead analysed the real data. Term (6) likewise fails to have mean zero because the observed data O i do not originate from the distribution ̂ P n . Also this term thus induces a bias, now resulting from the use of a generative model to obtain ̂ P n . It may be large relative to (3) when DGMs are used, because of slow convergence of ̂ P n . It is precisely this term that causes estimators based on synthetic data to converge slowly with increasing sample size, as observed in Decruyenaere et al. (2024).

After identifying the two problematic terms, we now propose in a second step a targeting or debiasing strategy to remove these bias terms (5) and (6). As in van der Laan and Rose (2011) and Chernozhukov et al. (2018), we will remove bias term (5) by analysing the data with debiased estimators based on the EIC that ensure that this bias term then becomes zero. Novel to our proposal is that we will additionally shift the generated data to ensure that also bias term (6) becomes zero. This bias term depends on the EIC, which itself depends on the target parameter of interest. In the next two paragraphs, we discuss how this can be done for the two considered estimators. Note that the proposed strategy does not require any actual finetuning or retraining of the DGM. For a given parameter of interest, a mere post-processing of the synthetic samples, based on access to the DGM as well as

the original data it was trained on, allows eliminating the bias for a given parameter of interest. A graphical summary of the problem setting and our debiasing strategy can be found in Appendix A.4.

3.1 Population mean

For the population mean, the debiasing step with respect to term (5) is implicit since the traditional estimator as given in Section 2 is a debiased estimator. Therefore, the proposal to debias a given DGMwith respect to the population mean θ ( P ) = ∫ odP ( o ) amounts to first training the DGM and then augmenting the output for the variable of interest to ensure that bias term (6) is zero, or hence

In other words, the population mean of the synthetic data under the DGM should match the sample average of the real data. Here, θ ( ̂ P n ) can be approximated by generating a very large sample k (e.g., one million samples) based on the DGM and calculating their sample mean Y . The generative model must then be corrected, to ensure that this sample mean equals O . To obtain a debiased synthetic sample, we shift all samples generated by the given DGM by adding O -Y to the considered variable. Note that the sample average of such a set of m corrected synthetic samples will generally differ from O .

3.2 Linear regression coefficient

For linear regression coefficients, this section shows how the samples generated by a given DGM need to be adapted to eliminate bias term (6), written as follows (see Appendix A.5):

To compute this bias, we must generate, for each observed value X i , a very large sample of measurements of A and Y with the given level X i , based on the DGM. Then E ̂ P n ( Y | X i ) and E ̂ P n ( A | X i ) can be estimated as the sample average of those values for respectively Y and A . Further based on a very large DGM-generated sample k (e.g. one million samples), we calculate θ ( ̂ P n ) as follows:

Debiasing of the DGM can now be done by adding the product b { ˜ A i -E ̂ P n ( A | ˜ X i ) } to the synthetic outcome observations ˜ Y i generated by the DGM. An in-depth elaboration is provided in Appendix A.5. With the proposed shifting, we ensure that the debiasing with respect to term (6) is completed. We then proceed our analysis with these shifted synthetic observations and employ the EIC-based estimator of Section 2, which ensures that bias term (5) equals zero as well.

3.3 Properties

To summarise, when the remainder terms R ( . ) and the empirical process terms (1) and (2) are all o p ( m -1 / 2 ) and o p ( n -1 / 2 ) and, furthermore, the suggested debiasing approach is used so as to remove terms (5) and (6), then we expect that

In particular, the resulting estimator θ ( ˜ P m ) may even converge at rootn rates (under standard conditions of pathwise differentiability (Bickel et al., 1993; Hines et al., 2022)) and has an easy-tocalculate variance that acknowledges the uncertainty in the generation of synthetic data (provided that the statistical convergence of the generator is not too slow). In Appendix A.6, we show that the variance of θ ( ˜ P m ) may be approximated by

thereby generalising results known for parametric synthetic data generators (Raab et al., 2016; Decruyenaere et al., 2024), where the correction factor √ 1 + m/n was proposed. Thus, when m →∞ , the debiased estimator θ ( ˜ P m ) based on debiased synthetic data has the same distribution as when the real data were analysed. Therefore, the proposed debiasing strategy delivers analysis results that are asymptotically equivalent to those obtained from the same analysis on the real data, provided that n/m = o (1) . This means that results of the same quality and confidence intervals of the same expected length are then obtained, as will be illustrated in the case study in Section 5.2.

Since E { ϕ ( O,P ) 2 } is unknown, it merely remains to estimate it as

This is a consistent estimator when ˜ P m converges to ̂ P n as m goes to infinity, and moreover, ̂ P n converges to P as n goes to infinity. We note that this sample variance will be subject to bias that results from 'poor' tuning of the DGM. Removing this bias is not required, because this variance will be scaled by 1 /m +1 /n so that any bias becomes negligible in large samples. While the use of debiased estimators based on the EIC of E { ϕ ( O,P ) 2 } may potentially improve performance, this goes beyond the scope of this work.

Practical implications. Sections 3.1 and 3.2 show how bias term (6) is eliminated by shifting the synthetic variable of interest. We describe how bias term (5) is also removed by using debiased estimators, which we referred to as EIC-based estimators (see Section 2). As mentioned earlier, the EIC-based estimator for the population mean always coincides with the sample average, while for the linear regression coefficient it only reduces to the ordinary least squares estimator when data-adaptive estimation of the nuisance parameters is not used. This implies that it may suffice for the applied researcher to 1) shift the synthetic data and apply the traditional estimators, and 2) to multiply the SE of the estimator with √ 1 + m/n to obtain valid inference from a single synthetic dataset. However, the EIC-based estimators are recommended since they are robust against model misspecification by allowing for more flexibility in the estimation of the nuisance parameters.

4 Simulation study

Our proposed debiasing strategy is empirically validated by a simulation study that covers both estimators 3.1 (sample mean) and 3.2 (linear regression coefficient). Having full control over the data generating process allows us to calculate the bias, SE and convergence rate of both estimators in synthetic data, with and without our debiasing strategy. The data generating process consists of the following four variables: age (normally distributed), atherosclerosis stage (ordinal with four categories), therapy (binary), and blood pressure (normally distributed). The Directed Acyclic Graph (DAG) in Figure 1 represents the dependency structure and we refer to Appendix A.7.1 for more details. This setting allows us to simultaneously target the population mean of age and the population effect of therapy ( A ) on blood pressure ( Y ) adjusted for stage ( X ).

4.1 Set-up

We conduct a Monte Carlo simulation study, where n independent records are sampled from the data generating process, forming the observed original dataset O 1 , ..., O n . This process is repeated 250 times, with the sample size n varying log-uniformly between 50 and 5000 (i.e., n ∈ { 50 , 160 , 500 , 1600 , 5000 } ). Per original dataset, following DGMs are trained: CTGAN and TVAE (Xu et al., 2019), of which a detailed explanation can be found in Appendix A.7.2. From these DGMs m synthetic data records are sampled that constitute the default synthetic dataset S 1 , ..., S m . We set m = n to retain the dimensionality of the original data. Subsequently, each default synthetic dataset is debiased with respect to both estimands using the steps provided in Section 3, leading to a debiased synthetic dataset . Finally, two estimators are calculated in each of three datasets: the sample mean of age and the linear regression coefficient of therapy on blood pressure adjusted for

50

500

5000

0%

25%

50%

75%

100%

Mean age

(MLE-based)

50

500

5000

Effect therapy

(MLE-based)

n (log scale)

Coverage

Generator

Original

CTGAN

TVAE

Debiased

No

Yes

stage . We always report the maximum likelihood estimation (MLE)-based estimators, as used in traditional statistical analysis, of which the standard errors (SEs) are inflated with the correction factor √ 1 + m/n to acknowledge the sampling variability of synthetic data. These estimators will deliver similar estimates as the EIC-based estimators since no data-adaptive estimation is used (see Section 3.3 and Appendix A.7.5).

4.2 Results

We now present the results of our simulation study. The DGMs were trained using the default hyperparameters as suggested by the package Synthcity (Qian et al., 2023). We also show results obtained for other hyperparameters (the default in the package SDV (Patki et al., 2016)) in Appendix A.7.4. In Section 4.2.1 we evaluate the empirical coverage of the 95 % confidence interval (CI) for the population parameters. Our debiasing strategy should enhance the coverage, preferably to the nominal level, allowing for (more) honest inference, which is the main contribution of our strategy. Then, we analyse step by step the various components that may influence the coverage by investigating the bias and SE of the estimators in Section 4.2.2, and their convergence rates in Section 4.2.3. Additional results are presented in Appendix A.7.4, including the convergence rates of the nuisance parameters estimated by the DGMs. Our code is available on Github: https: //github.com/syndara-lab/debiased-generation .

4.2.1 Coverage

By definition, 95 % ( empirical ) of the 95 % CIs ( nominal ) should cover the population parameter. Figure 2 depicts the empirical coverage of the 95 %CIs obtained from both original and synthetic samples for the population mean of age and the population effect of therapy on blood pressure adjusted for stage . The results indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage based on the default synthetic datasets decreases with increasing n due to slower shrinkage of the SE than the typical 1 over rootn rate, as calculated in Section 4.2.3 and previously elaborated in Decruyenaere et al. (2024). For the population regression coefficient, debiasing delivers coverage at the nominal level for TVAE across all sample sizes, but not for CTGAN , although it clearly provides more honest inferences than based on the default synthetic datasets. The residual loss of coverage likely results from not using (efficient) sample splitting (see Appendix A.3).

4.2.2 Bias and Standard Error

Figures 3a and 3b depict the estimates and their SE, respectively, for the sample mean of age obtained in the default and debiased synthetic datasets. Figures A5 and A6 in the appendix show these for the linear regression coefficient of therapy on blood pressure adjusted for stage . In Figure 3a, each dot is an estimate per Monte Carlo run and the true population parameters are represented by the horizontal dashed line. This figure allows a qualitative assessment of two key properties of estimators: empirical bias (i.e., the average difference between the estimates and the population parameter, as represented by the solid line) and empirical SE (i.e., the standard deviation of the estimates, as indicated by the vertical spread of the estimates). Ideally, both converge to zero as the sample size grows larger. The convergence rate conveys the rate at which this happens. The funnel represents the default behaviour of an unbiased estimator based on original data of which the SE diminishes at a rate of 1 over rootn .

As shown in Figure 3a, the sample mean of age is unbiased in the default synthetic datasets, but exhibits a large empirical SE that shrinks slowly with sample size due to the data-adaptive nature of

30

40

50

60

CTGAN

TVAE

Debiased:

no

50

500

5000

30

40

50

60

50

500

5000

Debiased:

yes

n (log scale)

Mean age (MLE-based)

(a)

0

4

CTGAN

TVAE

Debiased:

no

50

500

5000

0

4

50

500

5000

Debiased:

yes

n (log scale)

SE of mean age (MLE-based)

Standard error

Empirical

MLE-based

(b)

DGMs. It is exactly this variability that is, on average, underestimated by the MLE-based SE in Figure 3b. Debiasing reduces the empirical variability and accelerates its shrinkage, such that the average MLE-based SE approximates the empirical SE. For the linear regression coefficient of therapy on blood pressure adjusted for stage , debiasing reduces finite-sample bias and also improves shrinkage of the empirical SE. Albeit less pronounced, the average MLE-based SE still underestimates the empirical SE with CTGAN despite debiasing (see Figures A5 and A6 in the appendix).

4.2.3 Convergence Rate of Standard Error

Assuming a power law n -a in convergence rate for the empirical SE, we estimate the exponent a from five logarithmically spaced sample sizes n between 50 and 5000, shown in Table 1 and Figure A7 in the appendix. Standard statistical analysis assumes that the bias converges faster than the SE with the latter diminishing at a rate of 1 / √ n . However, the SEs produced by DGMs converge much slower (i.e., a SE < 0 . 5 ), leading to a progressively increasing underestimation of the empirical SE by the MLE-based SE (which assumes √ n -convergence) as the sample size grows larger. In turn, this results in too narrow CIs and poor coverage, as observed in Section 4.2.1. By contrast, our debiasing strategy renders estimators of which the SE converges at approximately rootn rates (i.e., a SE = 0 . 5 ), explaining the improvement in coverage of the CIs as a result of debiasing.

5 Case studies

To illustrate our findings and highlight their implications for the applied researcher, we conduct two case studies. First, Section 5.1 transfers the framework from our simulation study to the International Stroke Trial (IST) dataset (Sandercock et al., 2011). Second, Section 5.2 describes whether analysis results from the Adult Census Income dataset (Becker and Kohavi, 1996) are similar to those obtained from the real data, when the sample size m of the generated synthetic data is very large. In both case studies, estimated SEs in the default synthetic and debiased synthetic datasets are corrected by multiplying with factor √ 1 + m/n to acknowledge the sampling variability of synthetic data.

5.1 International Stroke Trial

We adapt the framework discussed in Section 4.1 to the IST dataset, one of the biggest randomised trials in acute stroke research (Sandercock et al., 2011). The dataset with 19285 complete cases now

constitutes our population . We mimic different hypothetical settings where an institution only has access to a limited sample of observations, with the sample size n varying between 50 and 5000 . In order to easily share the data with other researchers, the institution generates a synthetic dataset with sample size m , where m = n . We repeated this process 100 times per sample size n to be able to calculate the empirical coverage levels. For illustration purposes, we focus on the effect of aspirin on the outcome at 6 months and report the proportion of deaths for the two treatment arms (aspirin and no aspirin), and its corresponding risk difference. For each value of n , two default synthetic datasets were generated using both CTGAN and TVAE . Next, for each of these, we first split the default synthetic dataset by treatment, debias the data with relation to the population mean within each treatment arm, and then combine them back into one debiased synthetic dataset. We noticed that using the same hyperparameters as in the simulation study resulted in biased estimates, as can be seen in Figure A12 in the appendix. For this reason, we highlight the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al., 2016) instead.

One of the original research questions in Sandercock et al. (2011) was whether or not there is a difference in risk of death between the treatment arms. Suppose a researcher can repeatedly collect information on 500 subjects and uses original, default synthetic and debiased synthetic data to make an inferential statement about this risk difference. Figure 4 depicts the confidence intervals for the first 15 repetitions, with the vertical dashed lines representing the true risk difference of -0 . 009 as calculated based on our population (the full dataset). Should the researcher use the default synthetic data, they would falsely conclude (in 7 out of these 15 repetitions) that the risk is significantly different from -0 . 009 , while using the debiased synthetic dataset basically eliminates this high number of false-positives (with all these 15 intervals containing the population parameter), as is the case in the original data as well. More results can be found in Appendix A.8.1.

0.2

0

0.2

1

5

10

15

Original data

0.2

0

0.2

Default CTGAN

0.2

0

0.2

Debiased CTGAN

Risk difference death

Run

5.2 Adult Census Income Dataset

We also perform a case study on the Adult Census Income dataset, which comprises 45222 complete cases and 14 unique variables (Becker and Kohavi, 1996). We assume the researcher's interest lies in inferring the population mean of hours worked per week (estimated via the sample mean) and the average sex -adjusted difference in age between persons with an income of > $50 K a year vs. not (estimated via a linear regression model age ( Y ) ∼ income ( A ) + sex ( X ) ). Our goal in this case study is to confirm whether inferential results obtained using the debiased synthetic dataset are asymptotically equivalent (i.e. with m>>n in our debiasing strategy) to those obtained using the original data. To test this across different sample sizes, the original data constitute five different samples of the Adult Census Income dataset with sizes n varying log-uniformly between 50 and 45222 . For each original dataset, a default synthetic dataset of size m = 10 6 was generated by TVAE . Subsequently, this dataset was debiased following the steps described in Section 3.1 (sample mean) and Section 3.2 (linear regression coefficient).

Figure 5 depicts the 95 %CIs for both estimators, the five original sample sizes and the three versions of datasets. This indeed confirms that analysis on the debiased synthetic dataset leads to results of similar quality and CIs of similar length compared to the original dataset. By contrast, the analysis on the default synthetic dataset may yield results of inferior quality and even incorrect conclusions.

35

37.5

40

42.5

50

274

1503

8246

45222

Mean hours/week

0

5

10

15

50

274

1503

8246

45222

Effect income

Estimate (95% CI)

n (log-scale)

Original

Default synthetic

Debiased synthetic

6 Discussion

In this paper, we propose a new debiasing strategy that targets synthetic data created by DGMs towards the downstream task of statistical inference from the resulting synthetic data. We establish our theory for two estimators by applying insights from debiased or targeted machine learning literature (van der Laan and Rose, 2011; Chernozhukov et al., 2018) to the current setting where machine learning is not necessarily used in the analysis, but rather in the generation of synthetic data. We obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting DGMs and thereby improve the inferential utility of the synthetic data.

We illustrated the impact of our proposal through a simulation study on toy data and two case studies on real-world data. Our debiasing strategy results in rootn consistent estimators based on the synthetic data and thereby better coverage of the confidence intervals, allowing for more honest inference. While coverage was clearly improved, it was not guaranteed to be at the nominal level. Indeed, it may remain anti-conservative for some estimators and DGMs, due to slow convergence inherent to these models and/or due to residual overfitting bias that could not be removed since sample splitting was not performed. Future work should focus on efficient sample splitting, where the resulting bias reduction outweighs the increase in finite-sample bias that arises from training on smaller sample sizes. Alternatively, findings from Ghalebikesabi et al. (2022) on importance weighting could be incorporated, with the weights being targeted to eliminate the impact of the data-adaptive estimation of the weights. This may potentially relax the fast baseline convergence assumption, and enable the same debiased synthetic data to be used for multiple downstream analyses.

A key advantage of our debiasing strategy is that it may deliver synthetic data created by DGMs of which the analysis is equivalent to the original data analysis, provided that the synthetic sample size is chosen to be much larger than the original sample size. Although the risk of disclosure may increase with the size of non-DP synthetic data (Reiter and Drechsler, 2010), this trade-off is beyond the scope of our paper. More interestingly, in the case of DP synthetic data, our debiasing strategy may exploit their post-processing immunity that allows for data transformations without compromising privacy guarantees (Dwork and Roth, 2014). However, our strategy needs to be extended to incorporate the DP-constraints when studying the difference between θ ( ˜ P m ) and θ ( P ) in DP synthetic data.

Limitations of our proposal include the low-dimensional setting of our simulation and case studies, for which DGMs might be less well suited. The positive results found for two widely used estimators in this simple setting highlight the utility of a debiasing approach and are encouraging in terms of future larger-scale applications. However, before addressing these, it is important to first understand low-dimensional settings, where valid inference is already challenging to attain. While our debiasing strategy boils downs to a post-processing step, one could argue that the lack of change to the DGM's training strategy itself is actually a strength, since it renders our strategy generator-agnostic.

Another limitation concerns the fact that our debiasing strategy for a regression coefficient requires sampling of synthetic data conditional on a covariate, which is not available in all DGMs. However, this issue is partially mitigated in the case of conditioning on categorical variables, since one can always generate a synthetic dataset unconditionally and then only select samples that fit the condition - though this approach also has its limits, especially when conditioning on multiple covariates at once. Zhou et al. (2023) propose a deep generative approach to sample from a conditional distribution, even when working with high-dimensional data. Future work could explore this strategy.

Finally, our proposal requires that the person generating synthetic data is aware of the analyses that will be run on those data, and has access to the corresponding EICs needed for debiasing (which in particular rules out the possibility for debiasing w.r.t. non-pathwise differentiable parameters, such as conditional means or predictions). For each parameter of interest, the data generated by the DGM will then need to be debiased (simultaneously) w.r.t. that parameter's EIC, which is left for future research. In the case of original data, several debiased estimation strategies that do not require exact knowledge about the EIC already exist. These methods include a) approximating the EIC through finite-differencing (Carone et al., 2019; Jordan et al., 2022) or stochastic approximations via Monte Carlo (Agrawal et al., 2024), and b) automatically estimating the EIC from the data through auto-DML (Chernozhukov et al., 2022). Alternatively, kernel debiased plug-in estimation methods (Cho et al., 2023) enable simultaneous debiasing of all pathwise differentiable target parameters that meet certain regularity conditions, without requiring any knowledge about the EIC. Integrating their insights could further strengthen the foundations of our current work on the interplay between synthetic data, deep generative modelling, and debiased machine learning.

Acknowledgments and Disclosure of Funding

Paloma Rabaey's research is funded by the Research Foundation Flanders (FWO-Vlaanderen) with grant number 1170124N. This research also received funding from the Flemish government under the 'Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen' programme.

References

Agrawal, R., Witty, S., Zane, A., and Bingham, E. (2024). Automated efficient estimation using monte carlo efficient influence functions. arXiv preprint arXiv:2403.00158 .

Awan, J. and Cai, Z. (2020). One step to efficient synthetic data. arXiv preprint arXiv:2006.02397 .

Becker, B. and Kohavi, R. (1996). Adult. UCI Machine Learning Repository.

Bickel, P., Klaassen, C., Ritov, Y., and Wellner, J. (1993). Efficient and Adaptive Estimation for Semiparametric Models , volume 4. Baltimore Johns Hopkins University Press.

Carone, M., Luedtke, A. R., and van der Laan, M. J. (2019). Toward computerized efficient estimation in infinite-dimensional models. Journal of the American Statistical Association .

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1-C68.

Chernozhukov, V., Newey, W. K., and Singh, R. (2022). Automatic debiased machine learning of causal and structural effects. Econometrica , 90(3):967-1027.

Cho, B., Gan, K., Malenica, I., and Mukhin, Y. (2023). Kernel Debiased Plug-in Estimation. arXiv preprint arXiv:2306.08598 .

Decruyenaere, A., Dehaene, H., Rabaey, P., Polet, C., Decruyenaere, J., Vansteelandt, S., and Demeester, T. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In Kiyavash, N. and Mooij, J. M., editors, Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , volume 244 of Proceedings of Machine Learning Research , pages 966-996. PMLR.

Drechsler, J. (2011). Synthetic datasets for statistical disclosure control: theory and implementation , volume 201. Springer Science & Business Media.

Dwork, C. and Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science , 9(3-4):211-407.

Endres, M., Mannarapotta Venugopal, A., and Tran, T. S. (2022). Synthetic data generation: a comparative study. In Proceedings of the 26th International Database Engineered Applications Symposium , pages 94-102.

Fisher, A. and Kennedy, E. H. (2021). Visually communicating and teaching intuition for influence functions. The American Statistician , 75(2):162-172.

Ghalebikesabi, S., Wilde, H., Jewson, J., Doucet, A., Vollmer, S., and Holmes, C. (2022). Mitigating statistical bias within differentially private synthetic data. In Cussens, J. and Zhang, K., editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , volume 180 of Proceedings of Machine Learning Research , pages 696-705. PMLR.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems , 27.

Hernandez, M., Epelde, G., Alberdi, A., Cilla, R., and Rankin, D. (2022). Synthetic data generation for tabular health records: A systematic review. Neurocomputing , 493:28-45.

Hines, O., Dukes, O., Diaz-Ordaz, K., and Vansteelandt, S. (2022). Demystifying statistical learning based on efficient influence functions. American Statistician , 76(3):292-304.

Jordan, M., Wang, Y., and Zhou, A. (2022). Empirical gateaux derivatives for causal inference. Advances in Neural Information Processing Systems , 35:8512-8525.

Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint .

Nowok, B., Raab, G. M., and Dibben, C. (2016). Synthpop: Bespoke creation of synthetic data in R. Journal of Statistical Software , 74(11):1-26.

Patki, N., Wedge, R., and Veeramachaneni, K. (2016). The synthetic data vault. In IEEE International Conference on Data Science and Advanced Analytics (DSAA) , pages 399-410.

Qian, Z., Cebere, B.-C., and van der Schaar, M. (2023). Synthcity: facilitating innovative use cases of synthetic data in different data modalities. arXiv preprint .

Raab, G. M., Nowok, B., and Dibben, C. (2016). Practical data synthesis for large samples. Journal of Privacy and Confidentiality , 7(3):67-97.

Raghunathan, T. E. (2021). Synthetic data. Annual review of statistics and its application , 8:129-140.

Raghunathan, T. E., Reiter, J. P., and Rubin, D. B. (2003). Multiple imputation for statistical disclosure limitation. Journal of official statistics , 19(1):1.

Räisä, O., Jälkö, J., Kaski, S., and Honkela, A. (2023). Noise-aware statistical inference with differentially private synthetic data. In International Conference on Artificial Intelligence and Statistics , pages 3620-3643. PMLR.

Reiter, J. P. (2005). Releasing multiply imputed, synthetic public use microdata: an illustration and empirical study. Journal of the Royal Statistical Society Series A: Statistics in Society , 168(1):185-205.

Reiter, J. P. and Drechsler, J. (2010). Releasing multiply-imputed synthetic data generated in two stages to protect confidentiality. Statistica Sinica , pages 405-421.

Rubin, D. B. (1993). Statistical disclosure limitation. Journal of official Statistics , 9(2):461-468.

Sandercock, P. A., Niewada, M., Członkowska, A., and Group, I. S. T. C. (2011). The international stroke trial database. Trials , 12(1):101.

van Breugel, B., Qian, Z., and van der Schaar, M. (2023). Synthetic data, real errors: How (not) to publish and use synthetic data. arXiv preprint .

van der Laan, M. J. and Rose, S. (2011). Targeted Learning . Springer Series in Statistics. Springer New York, New York, NY.

Vansteelandt, S. and Dukes, O. (2022). Assumption-lean inference for generalised linear model parameters. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(3):657685.

Wan, Z., Zhang, Y., and He, H. (2017). Variational autoencoder based synthetic data generation for imbalanced learning. In 2017 IEEE symposium series on computational intelligence (SSCI) , pages 1-7. IEEE.

Wilde, H., Jewson, J., Vollmer, S., and Holmes, C. (2021). Foundations of bayesian learning from synthetic data. In International Conference on Artificial Intelligence and Statistics , pages 541-549. PMLR.

Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. (2019). Modeling tabular data using conditional GAN. Advances in neural information processing systems , 32.

Yan, C., Yan, Y., Wan, Z., Zhang, Z., Omberg, L., Guinney, J., Mooney, S. D., and Malin, B. A. (2022). A multifaceted benchmarking of synthetic electronic health record generation models. Nature communications , 13(1):7609.

Zhou, X., Jiao, Y., Liu, J., and Huang, J. (2023). A deep generative approach to conditional sampling. Journal of the American Statistical Association , 118(543):1837-1848.

A Appendix

A.1 Elaboration estimators for synthetic data

Population mean. The population mean of the observed data o is defined as

which we will denote short as ∫ OdP and its efficient influence curve (EIC) is ϕ ( O,P ) = O -θ ( P ) . Choosing ˜ P m = ˜ P m , we will then study the large sample behaviour of the synthetic data estimator:

Linear regression. Suppose we are interested in a specific coefficient θ ≡ θ ( P ) of some exposure A in the linear model

where X is a possibly high-dimensional covariate and ω ( . ) is an unknown function. Our proposal allows ω ( X ) to correspond to a standard linear model, but is less restrictive. The parameter θ in this linear model can be nonparametrically defined as

in the sense that it reduces to θ when the above model holds. Its EIC is (Vansteelandt and Dukes, 2022)

Denote the obtained synthetic data samples as S = ( ˜ Y , ˜ A, ˜ X ) . An estimator θ ( ˜ P m ) is then obtained by substituting in the above expression for θ ( P ) , the first expectation in the numerator and denominator by a sample average, E P ( A | X ) and E P ( Y | X ) by data-adaptive predictions E ˜ P m ( A | ˜ X i ) and E ˜ P m ( Y | ˜ X i ) obtained based on the synthetic data. This delivers estimator

A.2 Derivation of the impact of uncertainty affecting deep generative models

We are interested in knowing how much θ ( ˜ P m ) differs from θ ( P ) . For this, we study the difference θ ( ˜ P m ) -θ ( P ) , for which we consider 2 von Mises expansions (i.e., functional Taylor expansions). Throughout the calculation below, we use that influence curves ϕ ( O,P ) have the property of being mean zero when averaging w.r.t. the distribution P . We moreover let R ( . ) be a remainder term, which can generally be shown to be small, but must be studied on a case-by-case basis.

Here,

equals 1 over rootn times a term that is asymptotically normal (as n goes to infinity) with mean zero and variance E { ϕ ( O,P ) 2 } . Further, conditional on ̂ P n ,

equals 1 over rootm times a term that converges to E { ϕ ( S, ̂ P n ) 2 | ̂ P n } 1 / 2 times a standard normal distribution (as m goes to infinity). This follows from the mean zero property of influence curves and the fact that the synthetic data are drawn from the distribution ̂ P n . This mean zero property is essential as it implies that the variation of ̂ P n across repeated (observed data) samples does not

induce excess variability. Next letting also the sample size n go to infinity, we obtain convergence of m -1 / 2 ∑ m i =1 ϕ ( S i , ̂ P n ) to E { ϕ ( O,P ) 2 } N (0 , 1) under the assumption that E { ϕ ( S, ̂ P n ) 2 | ̂ P n } converges in probability to E { ϕ ( O,P ) 2 } . This is a weak assumption because

is generally smooth (continuous) in the distribution of the data (as in the considered two examples). Moreover, the flexibility offered by deep generative models (DGMs) makes it reasonable to assume that ̂ P n converges to P (e.g., in L 2 ( P ) ); while this convergence may be slow, no requirements on the rate of convergence are needed for the above assumption to be satisfied.

A.3 Note on sample splitting

In order to let

converge to zero, we will need the DGM to be trained on a different part of the data than the one on which the debiasing step will be applied. Such sample splitting is needed to prevent overfitting bias that may otherwise result from the highly data-adaptive nature of DGMs. In addition, we need ̂ P n (e.g., the DGM) to consistently estimate P in the sense that the squared mean (under P ) of ϕ ( O, ̂ P n ) -ϕ ( O,P ) (at fixed ̂ P n ) converges to zero in probability.

To prevent efficiency loss with sample splitting, one may consider the use of k -fold cross-fitting. For this, we randomly split the data in k folds, each time train the DGM on k -1 folds to obtain an estimator of the observed data distribution ̂ P ( k -1) n/k and calculate the bias

based on the n/k data points O i in the remaining fold. The average of the k obtained bias estimates can next be used for debiasing (see the next section for specific examples).

However, it remains to be seen from future work if the resulting bias reduction outweighs the increase in finite-sample bias that may result from training the DGM on smaller sample sizes. Preliminary simulations with a simple implementation suggested that this was not the case, which is why our results in the main text are reported without the use of sample splitting. Furthermore, the remainder of the theory discards this nuance for now as well.

A.4 Graphical summary

①

~n

-

1 2

②

~n

-

1 4

③

~m

-

1 2

q(

P

)

q(

Pn

)

q(

Pm

)

Variable

Density

Data generating process

Ground truth

P

Generative model

Pn

Parameter of interest

q

(.)

Population mean

Sample mean

Empirical distribution

Original

ℙ

n

Synthetic

ℙ

m

Based on original data

Based on synthetic data

50

500

5000

50

500

5000

Sample size

Estimate for

q

(

P

)

④

Repeated sampling variability

Original data uncertainty

Minimal synthetic data uncertainty

Additional synthetic data uncertainty

A.5 Elaboration on debiased strategy for linear regression coefficient

For the linear regression coefficient case, this section shows how the samples generated by a given DGMneed to adapted to eliminate the bias b , given by

To compute this bias, we must generate, for each observed value X i , a very large sample of measurements of A and Y with the given level X i , based on the DGM. Then E ̂ P n ( Y | X i ) can be calculated as the sample average of those values for Y , and likewise E ̂ P n ( A | X i ) can be calculated as the sample average of those values for A . Further based on a large sample generated based on the DGM, we calculate θ ( ̂ P n ) as the sample average of { A -E ̂ P n ( A | X ) }{ Y -E ̂ P n ( Y | X ) } divided by the sample average of { A -E ̂ P n ( A | X ) } 2 .

Debiasing of the DGM can now be done by adding b { ˜ A i -E ̂ P n ( A | ˜ X i ) } to the synthetic outcome observations generated by the DGM. This change does not affect the predictions E ̂ P n ( Y | X i ) from the DGM (because ˜ A i -E ̂ P n ( A | X i ) averages to zero for each choice of X i ). Further, with this change, θ ( ̂ P n ) also increases with b units as follows:

where the sum runs over a large sample of synthetic observations. With this change in θ ( ̂ P n ) , the previously calculated bias becomes

which is zero, and as such the debiasing with respect to term (6) is complete. Based on a new sample of synthetic observations (independent of those generated to calculate the bias), the synthetic data estimator is

This ensures that bias term (5) equals zero. Please note that this estimator coincides with the standard linear regression estimator on the debiased synthetic data when least squares predictions for E ˜ P m ( A | ˜ X i ) and E ˜ P m ( Y | ˜ X i ) are used.

A.6 Derivation of variance of debiased estimator θ ( ˜ P m )

With the suggested debiasing, we thus expect that

Since the synthetic data are independently drawn from the observed data, any covariance between the 2 leading terms must originate from the fact that ̂ P n depends on the observed data. This covariance equals zero since

where in the second equality we use that ̂ P n is determined by O 1 , ..., O n , in the third equality we use that S i only depends on O 1 , ..., O n via ̂ P n , and in the final equality we use that ϕ ( S i , ̂ P n ) has mean zero when the synthetic data are sampled from ̂ P n . This renders these terms asymptotically independent and their sum, hence, asymptotically normal. Moreover, since the variance of ϕ ( S i , ̂ P n ) converges to E { ϕ ( O,P ) 2 } (see the previous section; Appendix A.2), the variance of θ ( ˜ P m ) may thus be approximated by

thereby generalising results known for parametric synthetic data generators (Raab et al., 2016; Decruyenaere et al., 2024).

A.7 Simulation study

A.7.1 Data generating process

Inspired by an applied medical setting, we create a hypothetical disease, defined by a low-dimensional tabular data generation process. The dependency structure depicted by the directed acyclic graph (DAG) in Figure 1 in the main text displays the presence of four variables, each of them chosen to obtain a mix of data types. In our hypothetical disease, it is assumed that a patient is observed at a given point in time. At this time, patient data about age , atherosclerosis stage , and the random assignment of therapy is gathered. The continuous outcome variable blood pressure is evaluated at a later time point, making this design a simplification, since we do not consider the data as longitudinal.

The exact routine to reconstruct this data generating process is presented in the pseudo-code in Algorithm 1. Age (continuous) follows a normal distribution with mean 50 and standard deviation 10 . Atherosclerosis stage (ordinal) was generated according to a proportional odds cumulative logit model where an increase in age causes an increase in the odds of having a stage higher than a given stage k ( ν age = -0 . 05 and intercepts ν stage = { 2 , 3 , 4 } for stage I-III). Therapy (binary) is considered to be 1:1 randomly assigned and is therefore sampled from a Bernouilli distribution with a probability of 0 . 50 . The last variable, blood pressure (continuous), is sampled from a normal distribution with standard deviation 10 and where the baseline average of 120 increases with higher atherosclerosis stage ( β stage = { 0 , 10 , 20 , 30 } for stage I-IV, respectively) and absence of therapy ( β therapy = -20 ).

A.7.2 Deep Generative Models

We elaborate on the DGMs used to create synthetic data in our simulation study and cases studies. All experiments were run on our institutional high performance computing cluster using a single GPU (NVIDIA Ampere A100; 80 GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across 5 sample sizes; Adult Census Income Dataset case study: less than 4 hours). We focus on two commonly used DGMs, namely Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and Variational Autoencoders (VAEs) (Kingma and Welling, 2013).

A GAN consists of two competing neural networks, a generator and discriminator, and aims to achieve an equilibrium between both (Hernandez et al., 2022). This translates to a mini-max game, since the generator aims to minimise the difference between the real and generated data, while the discriminator aims to maximise the possibility to distinguish the real and generated data (Goodfellow

et al., 2014). We use the CTGAN implementation that was designed specifically for tabular data, proposed by Xu et al. (2019).

A VAE is a deep latent variable model, consisting of an encoder and a decoder (Kingma and Welling, 2013). The encoder models the approximate posterior distribution of the latent variables given an input instance, whereby typically a standard normal prior is assumed for the latent variables. The decoder allows reconstructing an input instance, based on a sample from the predicted latent space distribution. Encoder and decoder can be jointly trained by maximising the evidence lower bound (ELBO), i.e. the marginal likelihood of the training instances. Maximising the ELBO corresponds to minimising the Kullback-Leibler (KL) divergence between the predicted latent variable distribution for a given input instance and the standard normal priors, and minimising the reconstruction error of the input instance at the decoder output. Once again, we use the tabular implementation of a V AE ( TVAE ) proposed by Xu et al. (2019).

The results presented in the rest of the Appendix are obtained by training both types of DGM with default hyperparameters as suggested by the packages Synthcity and SDV (for both CTGAN and TVAE ). A comparison of the default hyperparameters in both packages is provided in Tables A1 and A2. Note that both packages implement the CTGAN and TVAE modules as originally proposed by Xu et al. (2019), where a cluster-based normaliser is used to preprocess numerical features.

A.7.3 Quality of synthetic data

We performed some additional analyses to assess the quality of the synthetic data obtained in our simulation study.

Average IKLD. The inverse of the KL divergence (IKLD) between original and synthetic data, averaged over 250 Monte Carlo runs and standardised between 0 and 1 , is presented in Table A3, where the debiased synthetic datasets have slightly higher IKLD than their default versions for all DGMs considered.

Failed generators. CTGAN (Synthcity) could not be trained in 14 runs (runs 38 , 69 , 102 , 225 for n = 500 ; runs 19 , 23 , 98 , 107 , 117 , 128 , 129 , 140 , 148 , 169 for n = 5000 ) due to an internal error in the package. As such, it was not possible to generate default and debiased synthetic data with CTGAN (Synthcity) in these runs. This comprises 1 . 12% (14 / 1250) of all CTGAN (Synthcity) trained and 0 . 28% (14 / 5000) of all generators trained. The other generative models did not produce errors during training, so that default synthetic data could be generated in every run.

Failed debiasing. TVAE (SDV) could not be debiased in 82 runs (runs 1 , 3 , 4 , 5 , 6 , 8 , 15 , 23 , 24 , 28 , 30 , 42 , 46 , 50 , 51 , 58 , 64 , 72 , 78 , 92 , 96 , 104 , 106 , 111 , 112 , 117 , 118 , 119 , 122 , 126 , 127 , 130 , 133 , 135 , 140 , 143 , 146 , 149 , 154 , 157 , 159 , 160 , 166 , 167 , 168 , 170 , 178 , 180 , 187 , 191 , 193 , 196 , 197 , 200 , 205 , 210 , 214 , 216 , 218 , 220 , 225 , 230 , 231 , 235 , 236 , 245 , 247 , 248 for n = 50 ; runs 13 , 22 , 97 , 117 , 139 , 145 , 146 , 148 , 161 , 186 , 193 , 197 , 235 , 246 for n = 160 ) due to sparse data, especially for small sample sizes. In particular, some X i in the original dataset may not be present in the default synthetic dataset generated by TVAE (SDV) , leading to an error when estimating the nuisance parameters needed for the debiasing step. This comprises 6 . 56% (82 / 1250) of all TVAE (SDV) trained and 1 . 64% (82 / 5000) of all generators trained. The other generative models did not produce errors during debiasing, so that debiased synthetic data could be generated in every run.

Exact memorisation. A sanity check was conducted to ensure that no records of the original data were memorised by the generative model. CTGAN (Synthcity) made the following number of exact copies of the original data in the synthetic data: one record ( 2 . 00% ) for n = 50 in three runs (runs 63 , 158 , 192 ). The other generative models did not make exact copies.

Non-estimable estimators. Due to sparse data, especially for small sample sizes, the linear regression coefficient of therapy on blood pressure adjusted for stage could not be estimated in a small subset of the 11140 (original and synthetic) datasets, producing extremely small ( < 1e -10 ) or large ( > 1e2 ) standard errors. Overall, 0 . 09% ( 10 / 11140 ) regression coefficient estimates could not be obtained: in 4 runs for default synthetic dataset of size m = 50 generated by CTGAN (SDV) and in 1 run for default synthetic dataset of size m = 160 generated by CTGAN (Synthcity) , and for their corresponding debiased synthetic datasets. The sample mean of age was always estimable.

A.7.4 Additional results

While the main text only reports results using the Synthcity default hyperparameters, we additionally wanted to report a wider variety of trained models, without manually tuning anything, which is why results for SDV are presented here as well.

Coverage for all estimators and models. The results shown in Figure A2 indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage in the default synthetic datasets decreases with increasing n due to slower shrinkage of the standard error (SE) than the typical 1 over rootn rate, as calculated in Section 4.2.3 and previously addressed in Decruyenaere

et al. (2024). While our debiasing strategy clearly improves the empirical coverage for the population regression coefficient, it does, however, not guarantee this to be at the nominal level for all sample sizes and DGMs considered. In particular, debiasing only seems to achieve coverage at the nominal level for TVAE (Synthcity) across all sample sizes and for TVAE (SDV) at sufficiently large sample sizes. Our approach still falls short for CTGAN (Synthcity) and CTGAN (SDV) , although providing more honest inference than the default synthetic datasets.

50

500

5000

0%

25%

50%

75%

100%

Mean age

(MLE-based)

50

500

5000

Effect therapy

(MLE-based)

n (log scale)

Coverage

Generator

Original data

CTGAN (SDV)

CTGAN (Synthcity)

TVAE (SDV)

TVAE (Synthcity)

Debiased

No

Yes

Bias and SE for mean age . As shown in Figure A3, the sample mean of age is unbiased in the default synthetic datasets, but exhibits large variability that shrinks slowly with sample size due to the data-adaptive nature of DGMs. Debiasing reduces this variability and accelerates shrinkage. Figure A4 indicates that the empirical SE for the sample mean of age is indeed, on average, underestimated by the MLE-based SE in default synthetic datasets. After debiasing, the average MLE-based SE approximates the empirical SE, albeit with minor deviations at smaller sample sizes.

30

40

50

60

CTGAN

(SDV)

CTGAN

(Synthcity)

TVAE

(SDV)

TVAE

(Synthcity)

Debiased:

no

50

500

5000

30

40

50

60

50

500

5000 50

500

5000 50

500

5000

Debiased:

yes

n (log scale)

Mean age (MLE-based)

Bias and SE for effect therapy on blood pressure adjusted for stage . It can be seen from Figure A5 that the linear regression coefficient of therapy on blood pressure adjusted for stage in the default synthetic datasets has finite-sample bias towards the null that converges to zero as the sample size grows larger. Additionally, its variability seems to diminish slower than expected. Debiasing reduces the finite-sample bias and improves the shrinkage of the SE. However, Figure A6 shows that the average MLE-based SE in the debiased synthetic datasets still underestimates the empirical SE with CTGAN (SDV) and CTGAN (Synthcity) despite debiasing, albeit less pronounced.

0

4

8

CTGAN

(SDV)

CTGAN

(Synthcity)

TVAE

(SDV)

TVAE

(Synthcity)

Debiased:

no

50

500

5000

0

4

8

50

500

5000 50

500

5000 50

500

5000

Debiased:

yes

n (log scale)

SE of mean age (MLE-based)

Standard error

Empirical

MLE-based

50

0

50

100

CTGAN

(SDV)

CTGAN

(Synthcity)

TVAE

(SDV)

TVAE

(Synthcity)

Debiased:

no

50

500

5000

50

0

50

100

50

500

500050

500

500050

500

5000

Debiased:

yes

n (log scale)

Effect therapy (MLE-based)

0

10

20

CTGAN

(SDV)

CTGAN

(Synthcity)

TVAE

(SDV)

TVAE

(Synthcity)

Debiased:

no

50

500

5000

0

10

20

50

500

5000 50

500

5000 50

500

5000

Debiased:

yes

n (log scale)

SE of effect therapy (MLE-based)

Standard error

Empirical

MLE-based

Convergence of SE for all estimators and models. The convergence rate of the empirical SE for the MLE-based estimators is shown in Table A4 and represented by the slope in Figure A7. The dashed line indicates the behaviour of the SE of an unbiased and √ n -consistent estimator based on observed data, whereas the dotted line indicates the assumed behaviour of the SE of the same estimator based on synthetic data, following the correction proposed by Raab et al. (2016). Lines for the empirical SE that are parallel to these lines indicate that the estimator converges at 1 over rootn rate, whereas more horizontal or vertical lines imply slower or faster shrinkage, respectively. After debiasing, all lines are parallel for both estimators, suggesting √ n -consistency. Note that the vertical offset of some lines, which represents the log asymptotic variance, is still too large for some DGMs despite debiasing. This is the case for the linear regression coefficient for therapy on blood pressure adjusted for stage obtained in the debiased synthetic dataset generated by CTGAN (SDV) and CTGAN (SDV) .

2

1

0

1

2

Default

Debiased

Mean age:

MLE-based

50

500

5000

1

0

1

2

3

50

500

5000

Effect therapy:

MLE-based

n (log-scale)

log(SE)

Empirical SE

Original data

CTGAN (SDV)

CTGAN (Synthcity)

TVAE (SDV)

TVAE (Synthcity)

Theoretical SE

Original data

Synthetic data

Nuisance parameters. Our debiasing strategy assumes that the two remainder terms in the von Mises expansion presented in Section 3 are o p ( m -1 / 2 ) and o p ( n -1 / 2 ) . This requires the difference between ˜ P m (i.e., estimate of ̂ P n based on the sampled synthetic dataset of m records) and ̂ P n (i.e., estimate of P by the DGM trained on n original records) and the difference between ̂ P n and P (i.e., the data generating process), to converge to zero in L 2 ( ̂ P n ) and L 2 ( P ) , respectively, at faster than n -to-the-quarter convergence rates. While this holds true for the convergence of ˜ P m to ̂ P n (see Table A5), this seems not always strictly attained for the functionals of ̂ P n with respect to P that appear in the efficient influence curves for the regression coefficient (see Table A6), in particular for CTGAN

(Synthcity) and, to a lesser extent, for CTGAN (SDV) and TVAE (Synthcity) . Nevertheless, a rootn consistent estimator was still obtained after debiasing these DGMs (see Table A4).

Summary. Table A7 summarises the effect of our debiasing strategy on bias, SE and coverage in the simulation study. Our strategy results in uniformly valid coverage for the population mean, allowing for honest inference. For the regression coefficient, coverage was clearly improved but may remain anti-conservative for some DGMs. This may originate from residual overfitting bias inherent to these DGMs that could not be removed since (efficient) sample splitting was not performed (see Appendix A.3).

A.7.5 Influence curve based estimation

Here, two estimators are estimated in the original and synthetic datasets: a maximum likelihood estimation (MLE)-based one, as used in traditional statistical analysis, and an efficient influence curve (EIC)-based one, as proposed in this paper and obtained after 5-fold cross-fitting during estimation of the nuisance parameters. Note that sample splitting was not used during the debiasing step. For the former, SEs are calculated via the regular expressions which discard the uncertainty associated with data-adaptive prediction during estimation, while for the latter, SEs are based on the EIC which acknowledges this uncertainty. Throughout, all estimated (model- or EIC-based) SEs are corrected with √ 1 + m/n to acknowledge the sampling variability of synthetic data. This correction factor was initially proposed by Raab et al. (2016) for parametric synthetic data generators, but was found to be insufficient for synthetic data created by DGMs (Decruyenaere et al., 2024). In Section 3.3, we give the formula for the variance of our EIC-based estimator on the debiased synthetic data, which generalises this correction factor to the setting where synthetic data were generated by DGMs.

The results of our simulation study using the EIC-based estimators, as shown below in Figures A8-A11 and Table A8, remain unchanged as compared to using the MLE-based estimators, since no data-adaptive predictions (e.g., machine learning) were used during estimation of the nuisance parameters. If data-adaptive estimation were to be used, we expect the MLE-based estimators to be overly optimistic, while the EIC-based estimators could handle the additional uncertainty introduced by data-adaptive estimation.

0%

25%

50%

75%

100%

MLE-based

EIC-based

Mean age

50

500

5000

0%

25%

50%

75%

100%

50

500

5000

Effect therapy

n (log scale)

Coverage

Generator

Original data

CTGAN (SDV)

CTGAN (Synthcity)

TVAE (SDV)

TVAE (Synthcity)

Debiased

No

Yes

30

40

50

60

Original

data:

default

CTGAN

(SDV):

default

CTGAN

(SDV):

debiased

CTGAN

(Synthcity):

default

CTGAN

(Synthcity):

debiased

TVAE

(SDV):

default

TVAE

(SDV):

debiased

TVAE

(Synthcity):

default

TVAE

(Synthcity):

debiased

Mean age:

MLE-based

30

40

50

60

Mean age:

EIC-based

50

0

50

100

Effect therapy:

MLE-based

50

500 5000

100

50

0

50

50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000

Effect therapy:

EIC-based

n (log scale)

Estimate

0

2

4

6

8

Original

data:

default

CTGAN

(SDV):

default

CTGAN

(SDV):

debiased

CTGAN

(Synthcity):

default

CTGAN

(Synthcity):

debiased

TVAE

(SDV):

default

TVAE

(SDV):

debiased

TVAE

(Synthcity):

default

TVAE

(Synthcity):

debiased

Mean

age

50

500 5000

0

5

10

15

20

25

50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000 50

500 5000

Effect

therapy

n (log scale)

SE of estimate (EIC-based)

Standard error

Empirical

EIC-based

MLE-based

2

1

0

1

2

Default

Debiased

Mean age:

MLE-based

2

1

0

1

2

Mean age:

EIC-based

1

0

1

2

3

Effect therapy:

MLE-based

50

500

5000

1

0

1

2

3

50

500

5000

Effect therapy:

EIC-based

n (log-scale)

log(SE)

Empirical SE

Original data

CTGAN (SDV)

CTGAN (Synthcity)

TVAE (SDV)

TVAE (Synthcity)

Theoretical SE

Original data

Synthetic data

A.8 Case studies

A.8.1 International Stroke Trial

We adapt the framework discussed in Section 4.1 to the International Stroke Trial (IST), one of the biggest randomised trials in acute stroke research (Sandercock et al., 2011). The dataset with 19285 complete cases now constitutes our population . We mimic different hypothetical settings where an institution only has access to a limited sample of observations, with the sample size n varying between 50 and 5000 .

In order to easily share the data with other researchers, the institution generates a synthetic dataset with sample size m , where m = n . Similarly to the simulation study, we repeated this process 100 times per sample size n , to be able to calculate the empirical coverage levels. For illustration purposes, we focus on the effect of aspirin on the outcome at 6 months and report the proportion of deaths for the two treatment arms (aspirin and no aspirin), and its corresponding risk difference.

For each value of n , two default synthetic datasets were generated using both CTGAN and TVAE . Given the interest in the proportion of death in the group with and without aspirin, we use the debiasing strategy with respect to the population mean. This implies that the default synthetic dataset was first split by treatment and then debiased with relation to the population mean within each treatment arm. The two debiased subdatasets were then afterwards combined into one debiased synthetic dataset for each generative model. For both the default and debiased synthetic dataset, the sampling variability of synthetic data is acknowledged by inflating the standard errors (SEs) by the correction factor √ 1 + m/n .

The funnel plots for the proportion of deaths in both treatment arms and the risk difference are shown in Figure A12. We noticed that using the same hyperparameters as in the simulation study resulted in biased estimates, as can be seen in Figure A12. For this reason, we highlight the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al., 2016) instead. Analogously to the simulation study, our debiasing strategy decreases the variance of the mean estimator in both treatment arms, remedying the slower-than-√ n -convergence observed in the default synthetic datasets. The impact for the applied researcher can be better understood by looking at the empirical coverage levels of the 95% CI for the true proportion of deaths in the aspirin arm, for all sample sizes and DGMs considered. Figure A13a illustrates that in contrast to the default synthetic datasets, the coverage levels based on the debiased synthetic datasets are all positioned around the nominal level.

One of the original research questions in Sandercock et al. (2011) was whether or not there is a difference in risk of death between the treatment arms. Figure A13b depicts the empirical type 1 error rate for the risk difference in death between aspirin and no aspirin group based on original data, default and debiased synthetic data. For the aforementioned reason, we focus on the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al., 2016). Should the researcher use the default synthetic data, they would very often falsely conclude that the risk is significantly different from the true difference of -0 . 009 , as calculated based on our population (the full dataset), while using the debiased synthetic dataset basically eliminates this high number of false-positives, as is the case in the original data as well.

0

0.2

0.4

0.6

0.8

Original data

CTGAN

(SDV):

default

CTGAN

(SDV):

debiased

CTGAN

(Synthcity):

default

CTGAN

(Synthcity):

debiased

TVAE

(SDV):

default

TVAE

(SDV):

debiased

TVAE

(Synthcity):

default

TVAE

(Synthcity):

debiased

Proportion

Deaths

Aspirin

0

0.2

0.4

0.6

Proportion

Deaths

No Aspirin

50

500

5000

0.4

0

0.4

50

500

5000 50

500

5000 50

500

5000 50

500

5000 50

500

5000 50

500

5000 50

500

5000 50

500

5000

Risk

Difference

Deaths

n (log scale)

Estimate

50

160

500

1600

5000

0%

25%

50%

75%

100%

n (log scale)

Coverage

(a)

50

160

500

1600

5000

0%

25%

50%

75%

100%

n (log scale)

Type 1 error

Generator

Original data

TVAE (Synthcity)

TVAE (SDV)

CTGAN (Synthcity)

CTGAN (SDV)

Debiased

No

Yes

(b)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state why our research is relevant (bias in synthetic data generated by deep generative models, leading to imprecision in statistical analysis and wrong conclusions), and propose a new strategy to debias this synthetic data. While this methodology is general, we clearly state that we apply it on two Deep Generative Models (CTGAN and TVAE), in a simulation study and two case studies to convince the reader of its merits.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We list limitations that are related to several facets of this paper. We refer to the assumption that is needed for the formal derivation of our strategy but frame this in the context of well-known analyses. We also discuss the low-dimensional setting of our simulation and case studies, for which DGMs might be less suited. Nevertheless, the positive results for two widely used estimators in this simple setting highlights the utility of a debiased approach and is simultaneously encouraging in terms of future larger-scale applications. In contrast to other work, we are aware that we do not suggest a tuning strategy of the DGM but instead rely on the debiasing of the generated synthetic data in a post-processing step. Nonetheless, we perceive this as a strength, since it renders our strategy generator-agnostic. However, the debiasing method of the regression coefficient still requires sampling of synthetic data conditional on a covariate, which is not a given in all types of DGM. Finally, when multiple parameters are of interest, the data generated by the DGMwill need to be debiased to ensure that several such restrictions hold (simultaneously) w.r.t. several EICs ϕ ( . ) , which is left for future research. In the Discussion section, we offer various suggestions and ideas for future research aimed at addressing the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be

used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 2 we clearly present our set up and its assumptions. Section 3 contains a detailed theoretical proof, including reasoning steps along the way, of our debiasing strategy, complemented by additional notes in the supplementary material (referred to as 'Appendix' in the manuscript).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Besides fully describing our debiasing methodology, we also provide ample details on the simulation study and case studies. Moreover, we have made our code for the simulation and case studies available online through our GitHub page, with the link included in the main text.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed

instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The simulation study is built on a fictitious yet realistic data generating process, which is provided in full in Appendix A.7.1. As DGMs, we use the default implementations provided in the code libraries Synthcity and SDV , which are open-source. The case studies are based on two public datasets: the International Stroke Trial dataset (Sandercock et al., 2011) and the Adult Census Income dataset (Becker and Kohavi, 1996). While we aimed to provide all details necessary to reproduce our experiments in the paper itself, we additionally make all code available in a public Github repository (the link is included in the main text). These code enables reproduction of all results for the simulation study and case studies.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Since the focus is not on building novel generative models, but rather on proposing a general methodology for debiasing the data generated by a generic generator, we simply use default hyperparameters for the DGMs as suggested by two code libraries ( SynthCity and SDV ). In our simulation study, our training data is obtained by sampling from the data generating process. We do not use any test splits, since the evaluation is done in terms of utility of statistical analyses, where we compare the obtained estimands with their ground truth values. Similar considerations are valid for the case studies.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: While we do include experimental results, we do not compare multiple generative models or methods. Rather, we present a novel methodology for debiasing synthetic data, and use various experiments to verify that it works as expected, by comparing the statistical utility of synthetic data with and without our debiasing strategy.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were run on our institutional high performance computing cluster using a single GPU (NVIDIA Ampere A100; 80 GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across 5 sample sizes; Adult census Income Dataset case study: less than 4 hours). This is stated as such in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research did not involve human subjects or participants. Furthermore, there are no data-related concerns: the simple data generating process that is proposed in the simulation study is fictitious, and apart from that we use public datasets. There are potential harmful consequences related to synthetic data, such as privacy disclosure risk and incorrect statistical inference from tabular synthetic datasets. In fact, our work addresses the second risk by debiasing synthetic datasets towards improving their inferential utility.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Positive societal impacts stem from the increased reliability of statistical analyses on synthetic data, which is made possible by our debiasing method. We do not foresee major negative societal impacts of our method. General concerns regarding tabular synthetic data do apply, though we believe our method remedies part of them (see our answer to Question 9).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not believe our paper poses such risks, since we propose a generic debiasing strategy for synthetic data, without releasing any new models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the paper, we credit the creators of the deep generative models used in our work (i.e. CTGAN and TVAE), as well as the open source code libraries whose implementations we used ( Synthcity and SDV ). Furthermore, we cite the sources of the public datasets we used. The International Stroke Trial exists under an Open Database License, and the Adult Census Income dataset exists under a Creative Commons Attribution 4.0 International License. Both are free to use, but require attribution, which we provided.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for debiasing DGM-generated tabular data such that the population average of the EIC for the observed data distribution and its estimation becomes zero, resulting in elimination of the corresponding biasing term. This is done through augmenting the DGM output such that the population mean of the synthetic data matches the sample average of the observed data. As such, the method is generator-agnostic. The authors validate their approach through extensive experiments, demonstrating significant reductions in bias while maintaining data utility.\n\nSTRENGTHS:\n1. The paper is technically sound with in-depth theoretical analysis that are backed up by practical examples. We appreciate the additional detailed investigations presented in the Appendix, which shows the dedication of the authors to do a robust evaluation of their methodology. \n\n2. The paper tackles the critical issue of bias in synthetic data, a relatively underexplored area. The proposed method integrates fairness constraints directly into the data generation process, a novel approach, as opposed to post-hoc fairness adjustments. The proposed methodology specifically addresses the bias term concerning the observed data distribution, O, and its DGM estimation, P, as opposed to previous efforts that (are said to) have focused on the bias term describing the discrepancy between the sampled estimated (synthetic) distribution, S, and the approximation of P as P¨ (P-hat) through S.\n\n3. The paper is very well written and well organized (it was a pleasure to read). Figures and tables are well-presented and support the narrative effectively.\n\n4. The results have significant implications for improving fairness in machine learning, an area of growing importance. The proposed method for debiasing synthetic tabular data is straightforward and well described and hence easily implemented and adaptable by others\n\nWEAKNESSES:\n1. The paper would benefit from a more in-depth evaluation of potential limitations or edge cases where the method might not perform as well.\n\n2. It is difficult to assess the novelty in the proposed method as there is no presentation of related works nor comparison with alternative methods. A more detailed review of related work would help contextualize the novelty of the proposed method.\n\n3. It would be good to put the work in contrast with related works would improve the paper and elucidate the authors contribution to the field.\n\n4. The significance is still difficult to assess due to the aforementioned lack of comparison to alternative approaches. Additionally, potential impacts on different domains (e.g., healthcare, finance) could be explored to highlight the broader significance.\n\nQUESTIONS:\n1. Could the authors provide more details on how their approach scales with larger datasets and more complex models?\n2. How does the proposed method perform in scenarios with multiple sources of bias or more intricate data dependencies?\n3. Can the authors elaborate on any potential trade-offs between debiasing effectiveness and computational efficiency?\n4. How is \"a very large sample\" defined? L146 says 1,000,000 observations is \"large\", but the quality analysis in Section A.7.2 has max n = 5,000. \n5. Is the necessary synthetic data sample size generalizable or does it depend on the complexity of the parameter space?\n\nLIMITATIONS:\nThe authors properly address limitations of their work. However, they could further discuss the scalability of the proposed method to large datasets.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the significant biases introduced in synthetic data generated by deep generative models (DGMs) that compromise the inferential utility of such data. The authors propose a new debiasing strategy based on techniques adapted from debiased or targeted machine learning. Their approach aims to reduce biases and enhance convergence rates, thereby improving the reliability of statistical analyses performed on synthetic data. The effectiveness of the proposed strategy is demonstrated through a simulation study and two case studies using real-world data.\n\nSTRENGTHS:\n- **Originality:** The paper tackles the important problem of bias in synthetic data generated by DGMs, which has significant implications for privacy protection and data analysis. The authors propose a novel debiasing strategy that adapts techniques from targeted machine learning to the context of synthetic data generation. This represents a new direction in addressing the challenges of inferential utility in synthetic data. The approach is generator-agnostic, making it widely applicable to various types of DGMs.\n- **Quality:** The submission is technically sound, with clear explanations of the theoretical foundations and derivations of the debiasing strategy. The authors provide a simulation study and two case studies that demonstrate the effectiveness of their approach in improving the coverage of confidence intervals and enabling more reliable statistical inference from synthetic data. The paper is well-written and organized, making it easy to follow the authors' reasoning and understand their contributions.\n- **Significance:** The results presented in the paper are relevant for the field of synthetic data generation and privacy protection. The proposed debiasing strategy has the potential to improve the reliability and applicability of synthetic data in statistical inference, which can have broad implications for various domains where privacy is a concern.\n\nWEAKNESSES:\n- **Limited Scope:** The simulation study and case studies focus on low-dimensional settings and a limited number of estimators (population mean and linear regression coefficient). While the positive results are encouraging, further evaluation is needed to assess the effectiveness of the debiasing strategy in high-dimensional settings and for a wider range of statistical analyses.\n- **Dependence on EICs:** The proposed debiasing strategy requires access to the Efficient Influence Curves (EICs) of the target parameters of interest. This limits the applicability of the method to parameters that are pathwise differentiable and have known EICs.\n- **Conditional Sampling Limitation:** The debiasing procedure for the linear regression coefficient requires sampling synthetic data conditional on covariates, which may not be feasible for all types of DGMs.\n\nQUESTIONS:\n- **High-Dimensional Settings:** How well does the debiasing strategy perform in high-dimensional settings where DGMs are more commonly used? Are there any challenges or limitations in applying the method to high-dimensional data?\n- **Complex Estimators:** Can the debiasing strategy be extended to more complex estimators beyond the population mean and linear regression coefficient?\n- **Alternative Debiasing Approaches:** Are there alternative debiasing approaches that could be explored to address the limitations of the proposed method, such as its dependence on EICs and the conditional sampling requirement?\n- **Privacy Considerations:** While the paper mentions the potential increase in privacy disclosure risk with larger synthetic datasets, it would be beneficial to discuss this trade-off in more detail. How can the debiasing strategy be balanced with privacy concerns, especially in the context of differentially private synthetic data generation?\n- **Failure of Assumptions**: Are there specific scenarios where the assumptions required for the debiasing method might not hold, and how would this impact the results?\n\nLIMITATIONS:\nThe authors themselves are highly aware and forthcoming about the limitations of this study, which in my eyes is a big bonus.\n- The authors discuss the low-dimensional setting of their simulations and acknowledge that DGMs might be less suited for such settings.\n- They note the need for the person generating synthetic data to be aware of the analyses that will be run, which may limit the method's applicability.\n- The authors note the need to condition on data covariates to generate appropriate synthetic data.\n- The potential trade-off between synthetic data sample size and privacy risks is mentioned, though it is beyond the scope of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper tackle the problem of creating/imputing unbiased synthetic data, and analyse the potential bias brought by these imputation methods.

STRENGTHS:
The tackled problem is important and very timely, and linked to the general question of biais of generative models [1].


[1] Wyllie, Sierra, Ilia Shumailov, and Nicolas Papernot. "Fairness feedback loops: training on synthetic data amplifies bias." The 2024 ACM Conference on Fairness, Accountability, and Transparency. 2024.

WEAKNESSES:
I do not know if this comes from my lack of knowledge in the field, but I found the paper very hard to understand, no clear theoretical result is encapsulated, and it is not even very clear to me what is proposed (I guess this is what is after line 91).
I thus have a lot of questions:
- line 90, how exactly $S_i$ is sampled? Are you sampling first $A_i$ and $X_i$, and then $Y_i|A_i, X_i$?
- What is your main theoretical result? Is it Equation 2? or Section 3.3?
- What is $\phi$ (I am not sure it is defined in the main text)
- How is the deep generative model learned in the first place?

QUESTIONS:
see above

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of debiasing synthetic data which can have signficant issues when used for statistical analysis. In particular, they show examples of mean estimation of a variable and parameter estimate of a regression model and demonstrate the benefits of the post-processing step of their approach (model agnostic).\n\nOverall: The paper is clearly written and the problem is well-defined. The approach is for debiasing is simple but effective and experiments validate the approach. Unclear, how it can work for more real-world settings.\n\nSTRENGTHS:\n(i) The problem of bias in the synthetic data is well explained by considering the two examples of population mean and the coefficient estimator in linear regression. The debiased estimators are derived for these two settings. \n\n(ii) Experiments are first shown the synthetic datasets and the debiased estimators have very good coverage as opposed to without debiasing. The convergence rate with O(1/sqrt(n)) is al\nso shown when debiased. Two real-world datasets are also considered and the issue of obtaining false conclusions from the synthetic data vs the debiased setting is illustrated.\n\nWEAKNESSES:\n(a) As mentioned in the discussion settings, the datasets are low-dimensional and the estimators are very simple such as mean and parameter estimate for linear regression. \n\n(b) It is unclear how one can generalize these to settings where the estimators are much more complex/ubiquitous such as the covariance matrix or group fairness.\n\nQUESTIONS:\n(1) The references are a bit sparse on the synthetic data literature and would it be possible to enhance it with appropriate hooks? There are quite a few surveys available now.\n(2) While the results are interesting, it is unclear how much is innovative in this work as opposed to building on the results from Chernozhukov, Decruyenaere and other cited works. Can you elucidate that clearly? \n(3) Also, what would be needed to handle more interesting estimators for downstream applications?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}