Debiasing Synthetic Data Generated by Deep Generative Models

Authors: alexander  decruyenaere
heidelinde  dehaene
paloma  rabaey
christiaan  polet
johan  decruyenaere
thomas  demeester
stijn  vansteelandt
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:31:54

Abstract: While synthetic data hold great promise for privacy protection, their statistical analysis poses significant challenges that necessitate innovative solutions. The use of deep generative models (DGMs) for synthetic data generation is known to induce considerable bias and imprecision into synthetic data analyses, compromising their inferential utility as opposed to original data analyses. This bias and uncertainty can be substantial enough to impede statistical convergence rates, even in seemingly straightforward analyses like mean calculation. The standard errors of such estimators then exhibit slower shrinkage with sample size than the typical 1 over root-$n$ rate. This complicates fundamental calculations like p-values and confidence intervals, with no straightforward remedy currently available. In response to these challenges, we propose a new strategy that targets synthetic data created by DGMs for specific data analyses. Drawing insights from debiased and targeted machine learning, our approach accounts for biases, enhances convergence rates, and facilitates the calculation of estimators with easily approximated large sample variances. We exemplify our proposal through a simulation study on toy data and two case studies on real-world data, highlighting the importance of tailoring DGMs for targeted data analysis. This debiasing strategy contributes to advancing the reliability and applicability of synthetic data in statistical inference.

Main paper: 
 introduction: 
 The concept of generating synthetic data as a means of privacy protection was initially introduced by Rubin (1993) within the framework of multiple imputation, a widely used technique for managing the statistical analysis of incomplete data sets. Since its inception, a substantial body of literature on synthetic data has emerged (Raghunathan et al., 2003; Raghunathan, 2021; Drechsler, 2011; Raab et al., 2016; Reiter, 2005) , with a recent surge in interest propelled by advancements in deep generative modelling technology (Raghunathan, 2021; van Breugel et al., 2023; Wan et al., 2017; Yan et al., 2022; Nowok et al., 2016; Endres et al., 2022; Hernandez et al., 2022) . In this work, we focus on tabular synthetic data and its inferential utility, which captures whether synthetic data can be used to obtain valid estimates and inference for a population parameter (Decruyenaere et al., 2024) .The substantial privacy protection potential offered by synthetic data is marred by significant challenges that undermine their inferential utility (Raab et al., 2016) . Previous work by Decruyenaere et al. (2024) has shown that these challenges are much more severe when using Deep Generative Models (DGMs), rather than parametric statistical models, which is why we focus on the former. First, standard confidence intervals and p-values obtained on synthetic data may drastically underestimate the uncertainty in synthetic data as these ignore the size of the original data. Indeed, synthetic data obtained via generators trained on small datasets will unsurprisingly deliver much worse quality than synthetic data obtained via generators trained on large datasets. This uncertainty in the generator must therefore be translated into analysis results, such as confidence intervals and p-values. Standard confidence intervals and p-values ignore this, as they do not distinguish whether the data is synthetic or real. Second, it is well known from the literature on plug-in estimation that data-adaptive methods (such as DGMs) cannot succeed to estimate all features of the data-generating distribution well (Bickel et al., 1993; van der Laan and Rose, 2011; Chernozhukov et al., 2018; Hines et al., 2022) . These methods are designed to optimally balance bias and variance w.r.t. a chosen criterion, such as prediction error. However, they cannot guarantee that such an optimal trade-off is simultaneously made w.r.t. all possible discrepancy measures that exist, such as mean squared error in specific functionals (mean, variance, least squares projections...) of the observed data distribution. Such data-adaptive methods therefore leave non-negligible bias in estimators of such functionals, leading to excess variability, slow convergence, and confidence intervals that do not cover the truth at nominal level (and may even never contain the truth, even in large samples) (Decruyenaere et al., 2024) .Related work. While several approaches have been proposed to account for the uncertainty arising from synthetic data generation, we are not aware of strategies for generating and analysing DGMbased synthetic data that guarantee valid inference. Raghunathan et al. (2003) developed a framework inspired by the work on multiple imputation for missing data, by combining the results of multiple synthetic datasets, but this is not readily applicable to DGM-based synthetic data. Räisä et al. (2023) extended this work for differentially private (DP) synthetic data, acknowledging the additional DP noise, but continue to consider parametric (Bayesian) data generation strategies.Our work instead focuses on obtaining valid inference from a single synthetic dataset, which is arguably more attractive for use by practitioners. Raab et al. (2016) derived alternative combining rules that reduce to the correction factor 1 + m/n for the standard error (SE) of an estimator in the case of inference from a single (non-DP) synthetic dataset of size m generated from an original dataset of size n. The method suggested by Awan and Cai (2020) to preserve efficient estimators in a single (DP or non-DP) synthetic dataset relies on generating data conditional on the estimate in the original data. Both procedures are only applicable to parametric generative models and therefore suffer from the same limitation as the aforementioned approaches. To enable Bayesian inference from a single DP synthetic dataset, Wilde et al. (2021) proposed a corrected analysis that relies on the availability of additional public data, while Ghalebikesabi et al. (2022) investigated importance weighting methods to remove the noise-related bias, but they do not study the impact on inference.Contributions. Our work is the first to propose a generator-agnostic solution that mitigates the impact of the typical slower-than-√ n-convergence of estimators in synthetic data created by DGMs. As far as we are aware, our approach is thus the only one that provides some formal guarantees for (more) honest inference in this setting. In this paper, we show how the statistical bias in estimators can be removed, by adapting results on debiased or targeted machine learning (van der Laan and Rose, 2011; Chernozhukov et al., 2018) to the current setting where machine learning is not necessarily used in the analysis, but rather in the generation of synthetic data. Although we build upon ideas from existing work, our extension is non-trivial, since (1) those works did not consider synthetic data; and (2) we demonstrate, with significant generality, how to mitigate the estimation errors in the DGM that would otherwise propagate into the estimator calculated on synthetic data.In Section 2 and 3, we propose a generator-agnostic debiasing strategy, directed towards the downstream statistical analysis of the synthetic data. As such, we obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting the generators. We illustrate this with a simulation study in Section 4, showing that the coverage of both the mean and linear regression coefficient estimators indeed improves. In Section 5, we further cement the utility of our debiasing strategy in a practical setting through two case studies. While the proposed strategy is generator-agnostic, we focus our analyses on two DGMs for tabular data: CTGAN and TVAE (Xu et al., 2019) . Finally, Section 6 concludes with a discussion on our method and its limitations.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We state why our research is relevant (bias in synthetic data generated by deep generative models, leading to imprecision in statistical analysis and wrong conclusions), and propose a new strategy to debias this synthetic data. While this methodology is general, we clearly state that we apply it on two Deep Generative Models (CTGAN and TVAE), in a simulation study and two case studies to convince the reader of its merits.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
notation and set-up: 
 The aim of this paper is to use synthetic data in order to learn a specific functional θ(.) of the observed data distribution P . We formalise the problem of learning θ(P ) based on synthetic data as follows. Suppose that, based on n independent (possibly high-dimensional) samples O 1 , ..., O n from P , we estimate the observed data distribution as P n . Here, P n may be obtained by fitting parametric models to the distribution of the observed data, or alternatively by training DGMs. Based on m independent (synthetic) random samples S 1 , ..., S m from P n , we then estimate θ(P ). For this, the data analyst (to whom P n is unknown) uses the m synthetic samples S 1 , ..., S m to approximate P n as P m , and obtains an estimator of θ(P ) given by θ( P m ). We use P n to denote the empirical distribution of the observed data and P m to denote the empirical distribution of the synthetic data. We index expectations by the distribution under which they are taken; e.g., E P (Y ) denotes the population expectation of Y .Throughout, we assume that the parameter of interest is sufficiently smooth in the data-generating distribution so that root-n consistent estimators (i.e., with SEs shrinking at 1 over root-n rate) can be obtained. Formally, we assume the θ(P ) is pathwise differentiable with efficient influence curve (EIC) ϕ(., P ) under the nonparametric model, as is satisfied for (most) standard statistical analyses. The EIC is a functional derivative of θ(P ) w.r.t. the data-generating distribution P (in the sense of a Gateaux derivative), which has mean zero under P (Fisher and Kennedy, 2021; Hines et al., 2022) .In what follows, we illustrate our debiasing strategy via two examples. Here, we briefly outline some of their theoretical foundations we build upon. Appendix A.1 clarifies how these debiased estimators originate from their EIC. We further refer to them as debiased or EIC-based estimators.Population mean. Adapting the traditional formulation of the population mean to the context of synthetic data, we choose P m = P m . As a result, we will study the large sample behaviour of the synthetic data sample average:θ( P m ) = 1 m m i=1 S i .The EIC of this sample average is ϕ(S, P n ) = S -θ( P n ).Linear regression coefficient. Second, suppose we are interested in a specific coefficient θ ≡ θ(P ) of some exposure A for the outcome Y in the linear model E P (Y |A, X) = θA + ω(X), where X is a possibly high-dimensional vector of covariates and ω(.) is an unknown function. Our proposal allows ω(X) to correspond to a standard linear model in all covariates, but is more generally valid.Building upon the nonparametric definition of θ as given in Appendix A.1, we adjust it to obtain an estimator in the setting of synthetic data. Let Y , X and A be jointly or sequentially modelled by some generative model, from which a single synthetic datasetS i = ( Y i , A i , X i ) is sampled (i = 1, . . . , m).We then consider the estimated regression coefficient of exposure A given byθ( P m ) = 1 m m i=1 A i -E Pm (A| X i ) Y i -E Pm (Y | X i ) 1 m m i=1 A i -E Pm (A| X i ) 2 ,where the nuisance parameters E Pm (A| X i ) and E Pm (Y | X i ) are estimated based on synthetic data. This EIC-based estimator coincides with the Maximum Likelihood Estimator (MLE) when least squares predictions for these nuisance parameters are used. Its EIC is (Vansteelandt and Dukes, 2022 )ϕ(S, P n ) = A -E Pn (A| X) Y -E Pn (Y | X) -θ( P n ) A -E Pn (A| X) E Pn A -E Pn (A| X) 2 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We list limitations that are related to several facets of this paper. We refer to the assumption that is needed for the formal derivation of our strategy but frame this in the context of well-known analyses. We also discuss the low-dimensional setting of our simulation and case studies, for which DGMs might be less suited. Nevertheless, the positive results for two widely used estimators in this simple setting highlights the utility of a debiased approach and is simultaneously encouraging in terms of future larger-scale applications. In contrast to other work, we are aware that we do not suggest a tuning strategy of the DGM but instead rely on the debiasing of the generated synthetic data in a post-processing step. Nonetheless, we perceive this as a strength, since it renders our strategy generator-agnostic. However, the debiasing method of the regression coefficient still requires sampling of synthetic data conditional on a covariate, which is not a given in all types of DGM. Finally, when multiple parameters are of interest, the data generated by the DGM will need to be debiased to ensure that several such restrictions hold (simultaneously) w.r.t. several EICs ϕ(.), which is left for future research. In the Discussion section, we offer various suggestions and ideas for future research aimed at addressing the limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In a first step, we establish an understanding of how much the estimate based on synthetic data θ( P m ) differs from the population parameter of interest θ(P ). For this, we study the difference θ( P m ) -θ(P ), for which we consider 2 von Mises expansions (i.e., functional Taylor expansions).In Appendix A.2 we derive thatEQUATIONWe now examine the eight terms of this equation and discuss why some may be negligible while other may introduce bias. First, R(•) are remainder terms, which can generally be shown to be small, but must be studied on a case-by-case basis; see Hines et al. (2022) for worked out examples. In order for these remainder terms to be o p (m -1/2 ) and o p (n -1/2 ), we generally need that faster than n -1/4 convergence rates are obtained for the unknown functionals of P m and P n that appear in the EICs. It is unknown whether these convergence rates are attainable for DGMs; whether they are, will partly depend on the number of parameters in the DGM itself, the dimension of the data and the complexity of the observed data distribution. The simulation study in Section 4 will give further insight into this.Second, the two empirical process terms (1) and ( 2) in the von Mises expansion can be shown to be o p (m -1/2 ) and o p (n -1/2 ) by Markov's inequality, under weak conditions. For (1) to converge to zero, we will need the difference between P m and P n to converge to zero (in L 2 ( P n ) at any rate), and the estimator to be calculated on a different part of the data than the one on which P m was estimated (Chernozhukov et al., 2018) . For (2) to converge to zero, we will need P n (e.g., the DGM) to consistently estimate P (in L 2 (P ) at any rate). In addition, it can be argued that the DGM needs to be trained on a different part of the data than that on which the debiasing step (see later) will be applied. In Appendix A.3, we elaborate on the necessity of sample splitting.We thus foresee that the two remainder and two empirical process terms are negligible under certain conditions. Third, as elaborated in Appendix A.2, we show that the large sample behaviour of both1 n n i=1 ϕ(O i , P ) (3) and 1 m m i=1 ϕ(S i , P n ) (4)is standard, and well understood; in particular, these terms vary around zero with variance that can be estimated well. By contrast, the terms- 1 m m i=1 ϕ(S i , P m ) (5) and - 1 n n i=1 ϕ(O i , P n ) (6)need some further discussion. Term (5) generally fails to have mean zero because the synthetic data S i do not originate from the distribution P m . This term therefore induces a bias in θ( P m ) that results from using data-adaptive estimates P m on the synthetic data; a similar term would appear if we instead analysed the real data. Term (6) likewise fails to have mean zero because the observed data O i do not originate from the distribution P n . Also this term thus induces a bias, now resulting from the use of a generative model to obtain P n . It may be large relative to (3) when DGMs are used, because of slow convergence of P n . It is precisely this term that causes estimators based on synthetic data to converge slowly with increasing sample size, as observed in Decruyenaere et al. (2024) .After identifying the two problematic terms, we now propose in a second step a targeting or debiasing strategy to remove these bias terms (5) and ( 6). As in van der Laan and Rose (2011) and Chernozhukov et al. (2018) , we will remove bias term (5) by analysing the data with debiased estimators based on the EIC that ensure that this bias term then becomes zero. Novel to our proposal is that we will additionally shift the generated data to ensure that also bias term (6) becomes zero. This bias term depends on the EIC, which itself depends on the target parameter of interest. In the next two paragraphs, we discuss how this can be done for the two considered estimators. Note that the proposed strategy does not require any actual finetuning or retraining of the DGM. For a given parameter of interest, a mere post-processing of the synthetic samples, based on access to the DGM as well as the original data it was trained on, allows eliminating the bias for a given parameter of interest. A graphical summary of the problem setting and our debiasing strategy can be found in Appendix A.4.For the population mean, the debiasing step with respect to term ( 5) is implicit since the traditional estimator as given in Section 2 is a debiased estimator. Therefore, the proposal to debias a given DGM with respect to the population mean θ(P ) = odP (o) amounts to first training the DGM and then augmenting the output for the variable of interest to ensure that bias term ( 6) is zero, or hence1 n n i=1 ϕ(O i , P n ) = 1 n n i=1 O i -θ( P n ) = O -θ( P n ) = 0.In other words, the population mean of the synthetic data under the DGM should match the sample average of the real data. Here, θ( P n ) can be approximated by generating a very large sample k (e.g., one million observations) based on the DGM and calculating their sample mean Y . The generative model must then be corrected, to ensure that this sample mean equals O. To obtain a debiased synthetic sample, we shift all samples generated by the given DGM by adding O -Y to the considered variable. Note that the sample average of such a set of m corrected synthetic samples will generally differ from O.For linear regression coefficients, this section shows how the samples generated by a given DGM need to be adapted to eliminate bias term (6), written as follows (see Appendix A.5):EQUATIONTo compute this bias, we must generate, for each observed value X i , a very large sample of measurements of A and Y with the given level X i , based on the DGM. Then E Pn (Y |X i ) and E Pn (A|X i ) can be estimated as the sample average of those values for respectively Y and A. Further based on a very large DGM-generated sample k (e.g. one million observations), we calculate θ( P n ) as follows:EQUATION)Debiasing of the DGM can now be done by adding the product b A i -E Pn (A| X i ) to the synthetic outcome observations Y i generated by the DGM. An in-depth elaboration is provided in Appendix A.5. With the proposed shifting, we ensure that the debiasing with respect to term (6) is completed.We then proceed our analysis with these shifted synthetic observations and employ the EIC-based estimator of Section 2, which ensures that bias term (5) equals zero as well.To summarise, when the remainder terms R(.) and the empirical process terms (1) and ( 2) are all o p (m -1/2 ) and o p (n -1/2 ) and, furthermore, the suggested debiasing approach is used so as to remove terms ( 5) and ( 6), then we expect thatθ( P m ) -θ(P ) = 1 m m i=1 ϕ(S i , P n ) + 1 n n i=1 ϕ(O i , P ) + o p (n -1/2 ) + o p (m -1/2 ).In particular, the resulting estimator θ( P m ) may even converge at root-n rates (under standard conditions of pathwise differentiability (Bickel et al., 1993; Hines et al., 2022) ) and has an easy-tocalculate variance that acknowledges the uncertainty in the generation of synthetic data (provided that the statistical convergence of the generator is not too slow). In Appendix A.6, we show that the variance of θ( P m ) may be approximated by1 m + 1 n E ϕ(O, P ) 2 ,thereby generalising results known for parametric synthetic data generators (Raab et al., 2016; Decruyenaere et al., 2024) , where the correction factor 1 + m/n was proposed. Thus, when m → ∞, the debiased estimator θ( P m ) based on debiased synthetic data has the same distribution as when the real data were analysed. Therefore, the proposed debiasing strategy delivers analysis results that are asymptotically equivalent to those obtained from the same analysis on the real data, provided that n/m = o(1). This means that results of the same quality and confidence intervals of the same expected length are then obtained, as will be illustrated in the case study in Section 5.2.Since E ϕ(O, P ) 2 is unknown, it merely remains to estimate it as1 m m i=1 ϕ(S i , P m ) 2 .This is a consistent estimator when P m converges to P n as m goes to infinity, and moreover, P n converges to P as n goes to infinity. We note that this sample variance will be subject to bias that results from 'poor' tuning of the DGM. Removing this bias is not required, because this variance will be scaled by 1/m + 1/n so that any bias becomes negligible in large samples. While the use of debiased estimators based on the EIC of E ϕ(O, P ) 2 may potentially improve performance, this goes beyond the scope of this work.Practical implications. Sections 3.1 and 3.2 show how bias term ( 6) is eliminated by shifting the synthetic variable of interest. We describe how bias term ( 5) is also removed by using debiased estimators, which we referred to as EIC-based estimators (see Section 2). As mentioned earlier, the EIC-based estimator for the population mean always coincides with the sample average, while for the linear regression coefficient it only reduces to the ordinary least squares estimator when data-adaptive estimation of the nuisance parameters is not used. This implies that it may suffice for the applied researcher to 1) shift the synthetic data and apply the traditional estimators, and 2) to multiply the SE of the estimator with 1 + m/n to obtain valid inference from a single synthetic dataset. However, the EIC-based estimators are recommended since they are robust against model misspecification by allowing for more flexibility in the estimation of the nuisance parameters. stage. We always report the maximum likelihood estimation (MLE)-based estimators, as used in traditional statistical analysis, of which the standard errors (SEs) are inflated with the correction factor 1 + m/n to acknowledge the sampling variability of synthetic data. These estimators will deliver similar estimates as the EIC-based estimators since no data-adaptive estimation is used (see Section 3.3 and Appendix A.7.5).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: In Section 2 we clearly present our set up and its assumptions. Section 3 contains a detailed theoretical proof, including reasoning steps along the way, of our debiasing strategy, complemented by additional notes in the supplementary material (referred to as 'Appendix' in the manuscript).Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
results: 
 We now present the results of our simulation study. The DGMs were trained using the default hyperparameters as suggested by the package Synthcity (Qian et al., 2023) . We also show results obtained for other hyperparameters (the default in the package SDV (Patki et al., 2016) ) in Appendix A.7.4. In Section 4.2.1 we evaluate the empirical coverage of the 95% confidence interval (CI) for the population parameters. Our debiasing strategy should enhance the coverage, preferably to the nominal level, allowing for (more) honest inference, which is the main contribution of our strategy. Then, we analyse step by step the various components that may influence the coverage by investigating the bias and SE of the estimators in Section 4.2.2, and their convergence rates in Section 4.2.3. Additional results are presented in Appendix A.7.4, including the convergence rates of the nuisance parameters estimated by the DGMs. Our code is available on Github: https: //github.com/syndara-lab/debiased-generation.By definition, 95% (empirical) of the 95% CIs (nominal) should cover the population parameter.Figure 2 depicts the empirical coverage of the 95% CIs obtained from both original and synthetic samples for the population mean of age and the population effect of therapy on blood pressure adjusted for stage. The results indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage based on the default synthetic datasets decreases with increasing n due to slower shrinkage of the SE than the typical 1 over root-n rate, as calculated in Section 4.2.3 and previously elaborated in Decruyenaere et al. (2024) . For the population regression coefficient, debiasing delivers coverage at the nominal level for TVAE across all sample sizes, but not for CTGAN, although it clearly provides more honest inferences than based on the default synthetic datasets. The residual loss of coverage likely results from not using (efficient) sample splitting (see Appendix A.3).Figures 3a and 3b depict the estimates and their SE, respectively, for the sample mean of age obtained in the default and debiased synthetic datasets. Figures A5 and A6 in the appendix show these for the linear regression coefficient of therapy on blood pressure adjusted for stage. In Figure 3a , each dot is an estimate per Monte Carlo run and the true population parameters are represented by the horizontal dashed line. This figure allows a qualitative assessment of two key properties of estimators: empirical bias (i.e., the average difference between the estimates and the population parameter, as represented by the solid line) and empirical SE (i.e., the standard deviation of the estimates, as indicated by the vertical spread of the estimates). Ideally, both converge to zero as the sample size grows larger. The convergence rate conveys the rate at which this happens. The funnel represents the default behaviour of an unbiased estimator based on original data of which the SE diminishes at a rate of 1 over root-n.As shown in Figure 3a , the sample mean of age is unbiased in the default synthetic datasets, but exhibits a large empirical SE that shrinks slowly with sample size due to the data-adaptive nature of DGMs. It is exactly this variability that is, on average, underestimated by the MLE-based SE in Figure 3b . Debiasing reduces the empirical variability and accelerates its shrinkage, such that the average MLE-based SE approximates the empirical SE. For the linear regression coefficient of therapy on blood pressure adjusted for stage, debiasing reduces finite-sample bias and also improves shrinkage of the empirical SE. Albeit less pronounced, the average MLE-based SE still underestimates the empirical SE with CTGAN despite debiasing (see Figures A5 and A6 in the appendix).Assuming a power law n -a in convergence rate for the empirical SE, we estimated the exponent a from five logarithmically spaced sample sizes n between 50 and 5000, shown in Table 1 and Figure A7 in the appendix. Standard statistical analysis assumes that the bias converges faster than the SE with the latter diminishing at a rate of 1/ √ n. However, the SEs produced by DGMs converge much slower (i.e., a SE < 0.5), leading to a progressively increasing underestimation of the empirical SE by the MLE-based SE (which assumes √ n-convergence) as the sample size grows larger. In turn, this results in too narrow CIs and poor coverage, as observed in Section 4.2.1. By contrast, our debiasing strategy renders estimators of which the SE converges at approximately root-n rates (i.e., a SE = 0.5), explaining the improvement in coverage of the CIs as a result of debiasing. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Besides fully describing our debiasing methodology, we also provide ample details on the simulation study and case studies. Moreover, we have made our code for the simulation and case studies available online through our GitHub page, with the link included in the main text.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailedAnswer: [Yes] Justification: Since the focus is not on building novel generative models, but rather on proposing a general methodology for debiasing the data generated by a generic generator, we simply use default hyperparameters for the DGMs as suggested by two code libraries (SynthCity and SDV). In our simulation study, our training data is obtained by sampling from the data generating process. We do not use any test splits, since the evaluation is done in terms of utility of statistical analyses, where we compare the obtained estimands with their ground truth values. Similar considerations are valid for the case studies. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
case studies: 
 To illustrate our findings and highlight their implications for the applied researcher, we conduct two case studies. First, Section 5.1 transfers the framework from our simulation study to the International Stroke Trial (IST) dataset (Sandercock et al., 2011) . Second, Section 5.2 describes whether analysis results from the Adult Census Income dataset (Becker and Kohavi, 1996) are similar to those obtained from the real data, when the sample size m of the generated synthetic data is very large. In both case studies, estimated SEs in the default synthetic and debiased synthetic datasets are corrected by multiplying with factor 1 + m/n to acknowledge the sampling variability of synthetic data.We adapt the framework discussed in Section 4.1 to the IST dataset, one of the biggest randomised trials in acute stroke research (Sandercock et al., 2011) . The dataset with 19285 complete cases now constitutes our population. We mimic different hypothetical settings where an institution only has access to a limited sample of observations, with the sample size n varying between 50 and 5000. In order to easily share the data with other researchers, the institution generates a synthetic dataset with sample size m, where m = n. We repeated this process 100 times per sample size n to be able to calculate the empirical coverage levels. For illustration purposes, we focus on the effect of aspirin on the outcome at 6 months and report the proportion of deaths for the two treatment arms (aspirin and no aspirin), and its corresponding risk difference. For each value of n, two default synthetic datasets were generated using both CTGAN and TVAE. Next, for each of these, we first split the default synthetic dataset by treatment, debias the data with relation to the population mean within each treatment arm, and then combine them back into one debiased synthetic dataset. We noticed that using the same hyperparameters as in the simulation study resulted in biased estimates, as can be seen in Figure A12 in the appendix. For this reason, we highlight the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al., 2016) instead.One of the original research questions in Sandercock et al. (2011) was whether or not there is a difference in risk of death between the treatment arms. Suppose a researcher can repeatedly collect information on 500 subjects and uses original, default synthetic and debiased synthetic data to make an inferential statement about this risk difference. Figure 4 depicts the confidence intervals for the first 15 repetitions, with the vertical dashed lines representing the true risk difference of -0.009 as calculated based on our population (the full dataset). Should the researcher use the default synthetic data, they would falsely conclude (in 7 out of these 15 repetitions) that the risk is significantly different from -0.009, while using the debiased synthetic dataset basically eliminates this high number of false-positives (with all these 15 intervals containing the population parameter), as is the case in the original data as well. More results can be found in Appendix A.8.1. We also perform a case study on the Adult Census Income dataset, which comprises 45222 complete cases and 14 unique variables (Becker and Kohavi, 1996) . We assume the researcher's interest lies in inferring the population mean of hours worked per week (estimated via the sample mean) and the average sex-adjusted difference in age between persons with an income of > $50K a year vs. not (estimated via a linear regression model age (Y ) ∼ income (A) + sex (X)). Our goal in this case study is to confirm whether inferential results obtained using the debiased synthetic dataset are asymptotically equivalent (i.e. with m >> n in our debiasing strategy) to those obtained using the original data. To test this across different sample sizes, the original data constitute five different samples of the Adult Census Income dataset with sizes n varying log-uniformly between 50 and 45222. For each original dataset, a default synthetic dataset of size m = 10 6 was generated by TVAE. Subsequently, this dataset was debiased following the steps described in Section 3.1 (sample mean) and Section 3.2 (linear regression coefficient).Figure 5 depicts the 95% CIs for both estimators, the five original sample sizes and the three versions of datasets. This indeed confirms that analysis on the debiased synthetic dataset leads to results of similar quality and CIs of similar length compared to the original dataset. By contrast, the analysis on the default synthetic dataset may yield results of inferior quality and even incorrect conclusions. 
discussion: 
 In this paper, we propose a new debiasing strategy that targets synthetic data created by DGMs towards the downstream task of statistical inference from the resulting synthetic data. We establish our theory for two estimators by applying insights from debiased or targeted machine learning literature (van der Laan and Rose, 2011; Chernozhukov et al., 2018) to the current setting where machine learning is not necessarily used in the analysis, but rather in the generation of synthetic data. We obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting DGMs and thereby improve the inferential utility of the synthetic data.We illustrated the impact of our proposal through a simulation study on toy data and two case studies on real-world data. Our debiasing strategy results in root-n consistent estimators based on the synthetic data and thereby better coverage of the confidence intervals, allowing for more honest inference. While coverage was clearly improved, it was not guaranteed to be at the nominal level. Indeed, it may remain anti-conservative for some estimators and DGMs, due to slow convergence inherent to these models and/or due to residual overfitting bias that could not be removed since sample splitting was not performed. Future work should focus on efficient sample splitting, where the resulting bias reduction outweighs the increase in finite-sample bias that arises from training on smaller sample sizes. Alternatively, findings from Ghalebikesabi et al. (2022) on importance weighting could be incorporated, with the weights being targeted to eliminate the impact of the data-adaptive estimation of the weights. This may potentially relax the fast baseline convergence assumption, and enable the same debiased synthetic data to be used for multiple downstream analyses.A key advantage of our debiasing strategy is that it may deliver synthetic data created by DGMs of which the analysis is equivalent to the original data analysis, provided that the synthetic sample size is chosen to be much larger than the original sample size. Although the risk of disclosure may increase with the size of non-DP synthetic data (Reiter and Drechsler, 2010) , this trade-off is beyond the scope of our paper. More interestingly, in the case of DP synthetic data, our debiasing strategy exploits their post-processing immunity that allows for data transformations without compromising any privacy guarantees (Dwork and Roth, 2014) . However, our strategy needs to be extended to incorporate the DP-constraints when studying the difference between θ( P m ) and θ(P ) in DP synthetic data.Limitations of our proposal include the low-dimensional setting of our simulation and case studies, for which DGMs might be less well suited. The positive results found for two widely used estimators in this simple setting highlight the utility of a debiasing approach and are encouraging in terms of future larger-scale applications. However, before addressing these, it is important to first understand low-dimensional settings, where valid inference is already challenging to attain. While our debiasing strategy boils downs to a post-processing step, one could argue that the lack of change to the DGM's training strategy itself is actually a strength, since it renders our strategy generator-agnostic.Another limitation concerns the fact that our debiasing strategy for a regression coefficient requires sampling of synthetic data conditional on a covariate, which is not available in all DGMs. However, this issue is partially mitigated in the case of conditioning on categorical variables, since one can always generate a synthetic dataset unconditionally and then only select samples that fit the condition -though this approach also has its limits, especially when conditioning on multiple covariates at once. Zhou et al. (2023) propose a deep generative approach to sample from a conditional distribution, even when working with high-dimensional data. Future work could explore this strategy.Finally, our proposal requires that the person generating synthetic data is aware of the analyses that will be run on those data, and has access to the corresponding EICs needed for debiasing (which in particular rules out the possibility for debiasing w.r.t. non-pathwise differentiable parameters, such as conditional means or predictions). For each parameter of interest, the data generated by the DGM will then need to be debiased (simultaneously) w.r.t. that parameter's EIC, which is left for future research. In the case of original data, several debiased estimation strategies that do not require exact knowledge about the EIC already exist. These methods include a) approximating the EIC through finite-differencing (Carone et al., 2019; Jordan et al., 2022) or stochastic approximations via Monte Carlo (Agrawal et al., 2024) , and b) automatically estimating the EIC from the data through auto-DML (Chernozhukov et al., 2022) . Alternatively, kernel debiased plug-in estimation methods (Cho et al., 2023) enable simultaneous debiasing of all pathwise differentiable target parameters that meet certain regularity conditions, without requiring any knowledge about the EIC. Integrating their insights could further strengthen the foundations of our current work on the interplay between synthetic data, deep generative modelling, and debiased machine learning.A.1 Elaboration estimators for synthetic data Population mean. The population mean of the observed data o is defined asθ(P ) = o dP (o) do do = odP (o),which we will denote short as OdP and its efficient influence curve (EIC) is ϕ(O, P ) = O -θ(P ).Choosing P m = P m , we will then study the large sample behaviour of the synthetic data estimator:θ( P m ) = sd P m (s) = 1 m m i=1 S i .Linear regression. Suppose we are interested in a specific coefficient θ ≡ θ(P ) of some exposure A in the linear modelE P (Y |A, X) = θA + ω(X),where X is a possibly high-dimensional covariate and ω(.) is an unknown function. Our proposal allows ω(X) to correspond to a standard linear model, but is less restrictive. The parameter θ in this linear model can be nonparametrically defined asθ(P ) = E P [{A -E P (A|X)} {Y -E P (Y |X)}] E P {A -E P (A|X)} 2in the sense that it reduces to θ when the above model holds. Its EIC is (Vansteelandt and Dukes, 2022 )ϕ(O, P ) = {A -E P (A|X)} [Y -E P (Y |X) -θ(P ) {A -E P (A|X)}] E P {A -E P (A|X)} 2Denote the obtained synthetic data samples as S = ( Y , A, X). An estimator θ( P m ) is then obtained by substituting in the above expression for θ(P ), the first expectation in the numerator and denominator by a sample average, E P (A|X) and E P (Y |X) by data-adaptive predictions E Pm (A| X i ) and E Pm (Y | X i ) obtained based on the synthetic data. This delivers estimatorθ( P m ) = 1 m m i=1 A i -E Pm (A| X i ) Y i -E Pm (Y | X i ) 1 m m i=1 A i -E Pm (A| X i ) 2 .A.2 Derivation of the impact of uncertainty affecting deep generative modelsWe are interested in knowing how much θ( P m ) differs from θ(P ). For this, we study the difference θ( P m ) -θ(P ), for which we consider 2 von Mises expansions (i.e., functional Taylor expansions).Throughout the calculation below, we use that influence curves ϕ(O, P ) have the property of being mean zero when averaging w.r.t. the distribution P . We moreover let R(.) be a remainder term, which can generally be shown to be small, but must be studied on a case-by-case basis.θ( P m ) -θ(P ) = θ( P m ) -θ( P n ) + θ( P n ) -θ(P ) = ϕ(S, P m )d( P m -P n ) + R( P m , P n ) + ϕ(O, P n )d( P n -P ) + R( P n , P ) = -ϕ(S, P m )d P n + R( P m , P n ) -ϕ(O, P n )dP + R( P n , P ) = ϕ(S, P m )d( P m -P n ) -ϕ(S, P m )d P m + R( P m , P n ) + ϕ(O, P n )d(P n -P ) -ϕ(O, P n )dP n + R( P n , P ) = ϕ(S, P n )d( P m -P n ) -ϕ(S, P m )d P m + R( P m , P n ) + ϕ(S, P m ) -ϕ(S, P n ) d( P m -P n ) + ϕ(O, P )d(P n -P ) -ϕ(O, P n )dP n + R( P n , P ) + ϕ(O, P n ) -ϕ(O, P ) d(P n -P ) = 1 m m i=1 ϕ(S i , P n ) - 1 m m i=1 ϕ(S i , P m ) + R( P m , P n ) + ϕ(S, P m ) -ϕ(S, P n ) d( P m -P n ) + 1 n n i=1 ϕ(O i , P ) - 1 n n i=1 ϕ(O i , P n ) + R( P n , P ) + ϕ(O, P n ) -ϕ(O, P ) d(P n -P ).Here,1 n n i=1 ϕ(O i , P )equals 1 over root-n times a term that is asymptotically normal (as n goes to infinity) with mean zero and variance E ϕ(O, P ) 2 . Further, conditional on P n ,1 m m i=1 ϕ(S i , P n )equals 1 over root-m times a term that converges to E ϕ(S, P n )2 | P n 1/2times a standard normal distribution (as m goes to infinity). This follows from the mean zero property of influence curves and the fact that the synthetic data are drawn from the distribution P n . This mean zero property is essential as it implies that the variation of P n across repeated (observed data) samples does not induce excess variability. Next letting also the sample size n go to infinity, we obtain convergence of m -1/2 m i=1 ϕ(S i , P n ) to E ϕ(O, P ) 2 N (0, 1) under the assumption that E ϕ(S, P n ) 2 | P n converges in probability to E ϕ(O, P ) 2 . This is a weak assumption becauseE ϕ(S, P n ) 2 | P n = ϕ(o, P n ) 2 d P n (o)is generally smooth (continuous) in the distribution of the data (as in the considered two examples). Moreover, the flexibility offered by deep generative models (DGMs) makes it reasonable to assume that P n converges to P (e.g., in L 2 (P )); while this convergence may be slow, no requirements on the rate of convergence are needed for the above assumption to be satisfied.In order to let ϕ(O, P n ) -ϕ(O, P ) d(P n -P )converge to zero, we will need the DGM to be trained on a different part of the data than the one on which the debiasing step will be applied. Such sample splitting is needed to prevent overfitting bias that may otherwise result from the highly data-adaptive nature of DGMs. In addition, we need P n (e.g., the DGM) to consistently estimate P in the sense that the squared mean (under P ) of ϕ(O, P n ) -ϕ(O, P ) (at fixed P n ) converges to zero in probability.To prevent efficiency loss with sample splitting, one may consider the use of k-fold cross-fitting. For this, we randomly split the data in k folds, each time train the DGM on k -1 folds to obtain an estimator of the observed data distribution P (k-1)n/k and calculate the bias- 1 n/k i ϕ(O i , P (k-1)n/k ),based on the n/k data points O i in the remaining fold. The average of the k obtained bias estimates can next be used for debiasing (see the next section for specific examples).However, it remains to be seen from future work if the resulting bias reduction outweighs the increase in finite-sample bias that may result from training the DGM on smaller sample sizes. Preliminary simulations with a simple implementation suggested that this was not the case, which is why our results in the main text are reported without the use of sample splitting. Furthermore, the remainder of the theory discards this nuance for now as well.① ~n-1 2 ② ~n-1 4 ③~m-1 2 q(P ) q(P n ) q(P m ) (1) First, a random sample of original data of size n with empirical distribution Pn (orange histogram) is collected from this ground truth. Theory on asymptotic linearity prescribes that the sample mean (orange dashed vertical line) will deviate from the population mean by order of 1 over root-n (grey arrow).(2) Subsequently, a deep generative model is trained on these original data, yielding an estimated distribution Pn (blue solid curve). Its mean (blue solid vertical line) may in turn differ from the original sample mean by an order larger than 1 over root-n (red arrow), which is referred to as regularisation bias in Decruyenaere et al. (2024) .(3) Finally, synthetic data of size m (here m = n) are sampled from the estimated distribution Pn, forming the empirical distribution Pm = Pm (blue histogram with sample mean indicated by the blue dashed vertical line). The mean of both distributions Pn and Pm will again differ by order of 1 over root-m (green arrow). Ideally, the data analyst, who uses the synthetic data to estimate θ(P ) by θ( Pm), needs to take into account these three sources of random variability. (4) The large sample behaviour of the synthetic data estimator θ( Pm) is depicted by repeating the above procedure multiple times across increasing sample sizes of n = m and storing each estimate of the synthetic sample mean. Although the estimator remains unbiased for θ(P ) (dashed line), its empirical standard error becomes larger than in the original data due to the additional sources of variability. However, the correction factor 1 + m/n to the model-based standard error previously proposed by Raab et al. (2016) only captures the original data sampling variability (grey funnel) and a lower bound of the synthetic data sampling variability (green funnel), while the uncertainty associated with the regularisation bias (red funnel) remains unaccounted for. Since it cannot readily be expressed analytically, our debiasing strategy will eliminate the latter by shifting the mean of the distribution θ( Pn) estimated by the generative model towards the original sample mean (thereby removing the red arrow and funnel). Additionally, choosing synthetic sample sizes of m → ∞ will shrink the synthetic data sampling variability (ultimately removing the green arrow and funnel), such that the synthetic data estimator exhibits similar large sample behaviour as in original data.For the linear regression coefficient case, this section shows how the samples generated by a given DGM need to adapted to eliminate the bias b, given by b = n i=1 A i -E Pn (A|X i ) Y i -E Pn (Y |X i ) -θ( P n ) A i -E Pn (A|X i ) n i=1 A i -E Pn (A|X i ) 2 = n i=1 A i -E Pn (A|X i ) Y i -E Pn (Y |X i ) n i=1 A i -E Pn (A|X i ) 2 -θ( P n ) To computeN i=1 A i -E Pn (A| X i ) Y i + b A i -E Pn (A| X i ) -E Pn (Y | X i ) N i=1 A i -E Pn (A| X i ) 2 ,where the sum runs over a large sample of synthetic observations. With this change in θ( P n ), the previously calculated bias becomesn i=1 A i -E Pn (A|X i ) Y i -E Pn (Y |X i ) n i=1 A i -E Pn (A|X i ) 2 -θ( P n ) + bwhich is zero, and as such the debiasing with respect to term (6) is complete. Based on a new sample of synthetic observations (independent of those generated to calculate the bias), the synthetic data estimator isθ( P m ) = 1 m m i=1 A i -E Pm (A| X i ) Y i + b A i -E Pm (A| X i ) -E Pm (Y | X i ) 1 m m i=1 A i -E Pm (A| X i ) 2 = 1 m m i=1 A i -E Pm (A| X i ) Y i -E Pm (Y | X i ) 1 m m i=1 A i -E Pm (A| X i ) 2 + b.This ensures that bias term (5) equals zero. Please note that this estimator coincides with the standard linear regression estimator on the debiased synthetic data when least squares predictions for E Pm (A| X i ) and E Pm (Y | X i ) are used.With the suggested debiasing, we thus expect thatθ( P m ) -θ(P ) = 1 m m i=1 ϕ(S i , P n ) + 1 n n i=1 ϕ(O i , P ) + o p (n -1/2 ) + o p (m -1/2 ).Since the synthetic data are independently drawn from the observed data, any covariance between the 2 leading terms must originate from the fact that P n depends on the observed data. This covariance equals zero sinceE ϕ(S i , P n )ϕ(O j , P ) = E E ϕ(S i , P n )|O 1 , ..., O n ϕ(O j , P ) = E E ϕ(S i , P n )| P n , O 1 , ..., O n ϕ(O j , P ) = E E ϕ(S i , P n )| P n ϕ(O j , P ) = 0where in the second equality we use that P n is determined by O 1 , . Inspired by an applied medical setting, we create a hypothetical disease, defined by a low-dimensional tabular data generation process. The dependency structure depicted by the directed acyclic graph (DAG) in Figure 1 in the main text displays the presence of four variables, each of them chosen to obtain a mix of data types. In our hypothetical disease, it is assumed that a patient is observed at a given point in time. At this time, patient data about age, atherosclerosis stage, and the random assignment of therapy is gathered. The continuous outcome variable blood pressure is evaluated at a later time point, making this design a simplification, since we do not consider the data as longitudinal.The exact routine to reconstruct this data generating process is presented in the pseudo-code in Algorithm 1. Age (continuous) follows a normal distribution with mean 50 and standard deviation 10. Atherosclerosis stage (ordinal) was generated according to a proportional odds cumulative logit model where an increase in age causes an increase in the odds of having a stage higher than a given stage k (ν age = -0.05 and intercepts ν stage = {2, 3, 4} for stage I-III). Therapy (binary) is considered to be 1:1 randomly assigned and is therefore sampled from a Bernouilli distribution with a probability of 0.50. The last variable, blood pressure (continuous), is sampled from a normal distribution with standard deviation 10 and where the baseline average of 120 increases with higher atherosclerosis stage (β stage = {0, 10, 20, 30} for stage I-IV, respectively) and absence of therapy (β therapy = -20).We elaborate on the DGMs used to create synthetic data in our simulation study and cases studies. All experiments were run on our institutional high performance computing cluster using a single GPU (NVIDIA Ampere A100; 80GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across 5 sample sizes; Adult Census Income Dataset case study: less than 4 hours). We focus on two commonly used DGMs, namely Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and Variational Autoencoders (VAEs) (Kingma and Welling, 2013) .A GAN consists of two competing neural networks, a generator and discriminator, and aims to achieve an equilibrium between both (Hernandez et al., 2022) . This translates to a mini-max game, since the generator aims to minimise the difference between the real and generated data, while the discriminator aims to maximise the possibility to distinguish the real and generated data (Goodfellow et al., 2014) . We use the CTGAN implementation that was designed specifically for tabular data, proposed by Xu et al. (2019) .A VAE is a deep latent variable model, consisting of an encoder and a decoder (Kingma and Welling, 2013) . The encoder models the approximate posterior distribution of the latent variables given an input instance, whereby typically a standard normal prior is assumed for the latent variables. The decoder allows reconstructing an input instance, based on a sample from the predicted latent space distribution. Encoder and decoder can be jointly trained by maximising the evidence lower bound (ELBO), i.e. the marginal likelihood of the training instances. Maximising the ELBO corresponds to minimising the Kullback-Leibler (KL) divergence between the predicted latent variable distribution for a given input instance and the standard normal priors, and minimising the reconstruction error of the input instance at the decoder output. Once again, we use the tabular implementation of a VAE (TVAE) proposed by Xu et al. (2019) .The results presented in the rest of the Appendix are obtained by training both types of DGM with default hyperparameters as suggested by the packages Synthcity and SDV (for both CTGAN and TVAE). A comparison of the default hyperparameters in both packages is provided in Tables A1 and A2 . Note that both packages implement the CTGAN and TVAE modules as originally proposed by Xu et al. (2019) , where a cluster-based normaliser is used to preprocess numerical features. We performed some additional analyses to assess the quality of the synthetic data obtained in our simulation study.Average IKLD. The inverse of the KL divergence (IKLD) between original and synthetic data, averaged over 250 Monte Carlo runs and standardised between 0 and 1, is presented in Table A3 , where the debiased synthetic datasets have slightly higher IKLD than their default versions for all DGMs considered. 23, 98, 107, 117, 128, 129, 140, 148, 169 for n = 5000) due to an internal error in the package. As such, it was not possible to generate default and debiased synthetic data with CTGAN (Synthcity) in these runs. This comprises 1.12% (14/1250) of all CTGAN (Synthcity) trained and 0.28% (14/5000) of all generators trained. The other generative models did not produce errors during training, so that default synthetic data could be generated in every run.Failed debiasing. TVAE (SDV) could not be debiased in 82 runs (runs 1, 3, 4, 5, 6, 8, 15, 23, 24, 28, 30, 42, 46, 50, 51, 58, 64, 72, 78, 92, 96, 104, 106, 111, 112, 117, 118, 119, 122, 126, 127, 130, 133, 135, 140, 143, 146, 149, 154, 157, 159, 160, 166, 167, 168, 170, 178, 180, 187, 191, 193, 196, 197, 200, 205, 210, 214, 216, 218, 220, 225, 230, 231, 235, 236, 245, 247, 248 for n = 50; runs 13, 22, 97, 117, 139, 145, 146, 148, 161, 186, 193, 197, 235, 246 for n = 160) due to sparse data, especially for small sample sizes. In particular, some X i in the original dataset may not be present in the default synthetic dataset generated by TVAE (SDV), leading to an error when estimating the nuisance parameters needed for the debiasing step. This comprises 6.56% (82/1250) of all TVAE (SDV) trained and 1.64% (82/5000) of all generators trained. The other generative models did not produce errors during debiasing, so that debiased synthetic data could be generated in every run.Exact memorisation. A sanity check was conducted to ensure that no records of the original data were memorised by the generative model. CTGAN (Synthcity) made the following number of exact copies of the original data in the synthetic data: one record (2.00%) for n = 50 in three runs (runs 63, 158, 192) . The other generative models did not make exact copies.Non-estimable estimators. Due to sparse data, especially for small sample sizes, the linear regression coefficient of therapy on blood pressure adjusted for stage could not be estimated in a small subset of the 11140 (original and synthetic) datasets, producing extremely small (< 1e-10) or large (> 1e2) standard errors. Overall, 0.09% (10/11140) regression coefficient estimates could not be obtained: in 4 runs for default synthetic dataset of size m = 50 generated by CTGAN (SDV) and in 1 run for default synthetic dataset of size m = 160 generated by CTGAN (Synthcity), and for their corresponding debiased synthetic datasets. The sample mean of age was always estimable.While the main text only reports results using the Synthcity default hyperparameters, we additionally wanted to report a wider variety of trained models, without manually tuning anything, which is why results for SDV are presented here as well.Coverage for all estimators and models. The results shown in Figure A2 indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage in the default synthetic datasets decreases with increasing n due to slower shrinkage of the standard error (SE) than the typical 1 over root-n rate, as calculated in Section 4.2.3 and previously addressed in Decruyenaere et al. (2024) . While our debiasing strategy clearly improves the empirical coverage for the population regression coefficient, it does, however, not guarantee this to be at the nominal level for all sample sizes and DGMs considered. In particular, debiasing only seems to achieve coverage at the nominal level for TVAE (Synthcity) across all sample sizes and for TVAE (SDV) at sufficiently large sample sizes. Our approach still falls short for CTGAN (Synthcity) and CTGAN (SDV), although providing more honest inference than the default synthetic datasets. Bias and SE for mean age. As shown in Figure A3 , the sample mean of age is unbiased in the default synthetic datasets, but exhibits large variability that shrinks slowly with sample size due to the data-adaptive nature of DGMs. Debiasing reduces this variability and accelerates shrinkage. Figure A4 indicates that the empirical SE for the sample mean of age is indeed, on average, underestimated by the MLE-based SE in default synthetic datasets. After debiasing, the average MLE-based SE approximates the empirical SE, albeit with minor deviations at smaller sample sizes. Bias and SE for effect therapy on blood pressure adjusted for stage. It can be seen from Figure A5 that the linear regression coefficient of therapy on blood pressure adjusted for stage in the default synthetic datasets has finite-sample bias towards the null that converges to zero as the sample size grows larger. Additionally, its variability seems to diminish slower than expected. Debiasing reduces the finite-sample bias and improves the shrinkage of the SE. However, Figure A6 shows that the average MLE-based SE in the debiased synthetic datasets still underestimates the empirical SE with CTGAN (SDV) and CTGAN (Synthcity) despite debiasing, albeit less pronounced.Convergence of SE for all estimators and models. The convergence rate of the empirical SE for the MLE-based estimators is shown in Table A4 and represented by the slope in Figure A7 . The dashed line indicates the behaviour of the SE of an unbiased and √ n-consistent estimator based on observed data, whereas the dotted line indicates the assumed behaviour of the SE of the same estimator based on synthetic data, following the correction proposed by Raab et al. (2016) . Lines for the empirical SE that are parallel to these lines indicate that the estimator converges at 1 over root-n rate, whereas more horizontal or vertical lines imply slower or faster shrinkage, respectively. After debiasing, all lines are parallel for both estimators, suggesting √ n-consistency. Note that the vertical offset of some lines, which represents the log asymptotic variance, is still too large for some DGMs despite debiasing. This is the case for the linear regression coefficient for therapy on blood pressure adjusted for stage obtained in the debiased synthetic dataset generated by CTGAN (SDV) and CTGAN (SDV). Nuisance parameters. Our debiasing strategy assumes that the two remainder terms in the von Mises expansion presented in Section 3 are o p (m -1/2 ) and o p (n -1/2 ). This requires the difference between P m (i.e., estimate of P n based on the sampled synthetic dataset of m records) and P n (i.e., estimate of P by the DGM trained on n original records) and the difference between P n and P (i.e., the data generating process), to converge to zero in L 2 ( P n ) and L 2 (P ), respectively, at faster than n-to-the-quarter convergence rates. While this holds true for the convergence of P m to P n (see Table A5 ), this seems not always strictly attained for the functionals of P n with respect to P that appear in the efficient influence curves for the regression coefficient (see Table A6 ), in particular for CTGANHere, two estimators are estimated in the original and synthetic datasets: a maximum likelihood estimation (MLE)-based one, as used in traditional statistical analysis, and an efficient influence curve (EIC)-based one, as proposed in this paper and obtained after 5-fold cross-fitting during estimation of the nuisance parameters. Note that sample splitting was not used during the debiasing step. For the former, SEs are calculated via the regular expressions which discard the uncertainty associated with data-adaptive prediction during estimation, while for the latter, SEs are based on the EIC which acknowledges this uncertainty. Throughout, all estimated (model-or EIC-based) SEs are corrected with 1 + m/n to acknowledge the sampling variability of synthetic data. This correction factor was initially proposed by Raab et al. (2016) for parametric synthetic data generators, but was found to be insufficient for synthetic data created by DGMs (Decruyenaere et al., 2024) . In Section 3.3, we give the formula for the variance of our EIC-based estimator on the debiased synthetic data, which generalises this correction factor to the setting where synthetic data were generated by DGMs.The results of our simulation study using the EIC-based estimators, as shown below in Figures A8-A11 and Table A8 , remain unchanged as compared to using the MLE-based estimators, since no data-adaptive predictions (e.g., machine learning) were used during estimation of the nuisance parameters. If data-adaptive estimation were to be used, we expect the MLE-based estimators to be overly optimistic, while the EIC-based estimators could handle the additional uncertainty introduced by data-adaptive estimation. NeurIPS Paper Checklist 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: All experiments were run on our institutional high performance computing cluster using a single GPU (NVIDIA Ampere A100; 80GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across 5 sample sizes; Adult census Income Dataset case study: less than 4 hours). This is stated as such in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research did not involve human subjects or participants. Furthermore, there are no data-related concerns: the simple data generating process that is proposed in the simulation study is fictitious, and apart from that we use public datasets. There are potential harmful consequences related to synthetic data, such as privacy disclosure risk and incorrect statistical inference from tabular synthetic datasets. In fact, our work addresses the second risk by debiasing synthetic datasets towards improving their inferential utility.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Positive societal impacts stem from the increased reliability of statistical analyses on synthetic data, which is made possible by our debiasing method. We do not foresee major negative societal impacts of our method. General concerns regarding tabular synthetic data do apply, though we believe our method remedies part of them (see our answer to Question 9).Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.Answer: [NA] Justification: We do not release any new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: This paper proposes a debiasing strategy for synthetic data generated by deep generative models (DGMs), which aims to eliminate the bias that arises when estimating population parameters from synthetic data. The authors argue that existing methods for generating synthetic data can lead to biased estimates, and propose a new approach that involves post-processing the synthetic data to remove the bias. The method is generator-agnostic, meaning it can be applied to different types of DGMs, and is based on techniques adapted from debiased or targeted machine learning.

The reviewers praised the paper for its technical soundness, originality, and significance, noting that it addresses an important problem in the field of synthetic data generation. However, they also raised some concerns about the paper's readability and clarity, particularly for readers without a strong background in the field. The reviewers also asked for more information about the scalability of the proposed method to larger datasets and more complex models, and about the potential trade-offs between debiasing effectiveness and computational efficiency.

The authors responded to these concerns in their rebuttal, providing additional explanations and clarifications, and acknowledging the limitations of their study. They also suggested avenues for future research, including investigating the performance of the method in scenarios with multiple sources of bias or more intricate data dependencies.

After the rebuttal and discussion, three reviewers lean towards acceptance, while one leans towards rejection. However, the latter reviewer reports only a confidence of 1 and has not acknowledged the authors' rebuttal. Overall, I would thus recommend to accept the submission. However, I would further recommend that the authors seriously take the reviewer feedback into account for the camera-ready version, especially when it comes to the clarity of the presentation of the method and theoretical results to a wider ML audience and the positioning of this work within the related literature.

Reviews: review: paper_summary: paper_summary: The paper considers the problem of debiasing synthetic data which can have signficant issues when used for statistical analysis. In particular, they show examples of mean estimation of a variable and parameter estimate of a regression model and demonstrate the benefits of the post-processing step of their approach (model agnostic).

Overall: The paper is clearly written and the problem is well-defined. The approach is for debiasing is simple but effective and experiments validate the approach. Unclear, how it can work for more real-world settings.
strength_weakness: strength_weakness: (a) As mentioned in the discussion settings, the datasets are low-dimensional and the estimators are very simple such as mean and parameter estimate for linear regression. 

(b) It is unclear how one can generalize these to settings where the estimators are much more complex/ubiquitous such as the covariance matrix or group fairness.
questions: questions: (1) The references are a bit sparse on the synthetic data literature and would it be possible to enhance it with appropriate hooks? There are quite a few surveys available now.
(2) While the results are interesting, it is unclear how much is innovative in this work as opposed to building on the results from Chernozhukov, Decruyenaere and other cited works. Can you elucidate that clearly? 
(3) Also, what would be needed to handle more interesting estimators for downstream applications?
limitations: limitations: NA
paper_summary: paper_summary: This paper tackle the problem of creating/imputing unbiased synthetic data, and analyse the potential bias brought by these imputation methods.
strength_weakness: strength_weakness: I do not know if this comes from my lack of knowledge in the field, but I found the paper very hard to understand, no clear theoretical result is encapsulated, and it is not even very clear to me what is proposed (I guess this is what is after line 91).
I thus have a lot of questions:
- line 90, how exactly $S_i$ is sampled? Are you sampling first $A_i$ and $X_i$, and then $Y_i|A_i, X_i$?
- What is your main theoretical result? Is it Equation 2? or Section 3.3?
- What is $\phi$ (I am not sure it is defined in the main text)
- How is the deep generative model learned in the first place?
questions: questions: see above
limitations: limitations: yes
paper_summary: paper_summary: This paper addresses the significant biases introduced in synthetic data generated by deep generative models (DGMs) that compromise the inferential utility of such data. The authors propose a new debiasing strategy based on techniques adapted from debiased or targeted machine learning. Their approach aims to reduce biases and enhance convergence rates, thereby improving the reliability of statistical analyses performed on synthetic data. The effectiveness of the proposed strategy is demonstrated through a simulation study and two case studies using real-world data.
strength_weakness: strength_weakness: - **Limited Scope:** The simulation study and case studies focus on low-dimensional settings and a limited number of estimators (population mean and linear regression coefficient). While the positive results are encouraging, further evaluation is needed to assess the effectiveness of the debiasing strategy in high-dimensional settings and for a wider range of statistical analyses.
- **Dependence on EICs:** The proposed debiasing strategy requires access to the Efficient Influence Curves (EICs) of the target parameters of interest. This limits the applicability of the method to parameters that are pathwise differentiable and have known EICs.
- **Conditional Sampling Limitation:** The debiasing procedure for the linear regression coefficient requires sampling synthetic data conditional on covariates, which may not be feasible for all types of DGMs.
questions: questions: - **High-Dimensional Settings:** How well does the debiasing strategy perform in high-dimensional settings where DGMs are more commonly used? Are there any challenges or limitations in applying the method to high-dimensional data?
- **Complex Estimators:** Can the debiasing strategy be extended to more complex estimators beyond the population mean and linear regression coefficient?
- **Alternative Debiasing Approaches:** Are there alternative debiasing approaches that could be explored to address the limitations of the proposed method, such as its dependence on EICs and the conditional sampling requirement?
- **Privacy Considerations:** While the paper mentions the potential increase in privacy disclosure risk with larger synthetic datasets, it would be beneficial to discuss this trade-off in more detail. How can the debiasing strategy be balanced with privacy concerns, especially in the context of differentially private synthetic data generation?
- **Failure of Assumptions**: Are there specific scenarios where the assumptions required for the debiasing method might not hold, and how would this impact the results?
limitations: limitations: The authors themselves are highly aware and forthcoming about the limitations of this study, which in my eyes is a big bonus.
- The authors discuss the low-dimensional setting of their simulations and acknowledge that DGMs might be less suited for such settings.
- They note the need for the person generating synthetic data to be aware of the analyses that will be run, which may limit the method's applicability.
- The authors note the need to condition on data covariates to generate appropriate synthetic data.
- The potential trade-off between synthetic data sample size and privacy risks is mentioned, though it is beyond the scope of this paper.
paper_summary: paper_summary: The authors propose a method for debiasing DGM-generated tabular data such that the population average of the EIC for the observed data distribution and its estimation becomes zero, resulting in elimination of the corresponding biasing term. This is done through augmenting the DGM output such that the population mean of the synthetic data matches the sample average of the observed data. As such, the method is generator-agnostic. The authors validate their approach through extensive experiments, demonstrating significant reductions in bias while maintaining data utility.
strength_weakness: strength_weakness: 1. The paper would benefit from a more in-depth evaluation of potential limitations or edge cases where the method might not perform as well.

2. It is difficult to assess the novelty in the proposed method as there is no presentation of related works nor comparison with alternative methods. A more detailed review of related work would help contextualize the novelty of the proposed method.

3. It would be good to put the work in contrast with related works would improve the paper and elucidate the authors contribution to the field.

4. The significance is still difficult to assess due to the aforementioned lack of comparison to alternative approaches. Additionally, potential impacts on different domains (e.g., healthcare, finance) could be explored to highlight the broader significance.
questions: questions: 1. Could the authors provide more details on how their approach scales with larger datasets and more complex models?
2. How does the proposed method perform in scenarios with multiple sources of bias or more intricate data dependencies?
3. Can the authors elaborate on any potential trade-offs between debiasing effectiveness and computational efficiency?
4. How is "a very large sample" defined? L146 says 1,000,000 observations is "large", but the quality analysis in Section A.7.2 has max n = 5,000. 
5. Is the necessary synthetic data sample size generalizable or does it depend on the complexity of the parameter space?
limitations: limitations: The authors properly address limitations of their work. However, they could further discuss the scalability of the proposed method to large datasets.

Comments: title: Response
comment: Thanks to the authors for the detailed rebuttal. I have updated my score to 5 based on all the comments and I think it needs some work to place it appropriately in the literature.
I thank the authors for their detailed and thorough response. I will keep my original score.
Thanks for these clarifications. Given this, I will retain my original score.