Debiasing Synthetic Data Generated by Deep Generative Models
Authors: Alexander Decruyenaere
Heidelinde Dehaene
Paloma Rabaey
Johan Decruyenaere
Christiaan Polet
Thomas Demeester
Stijn Vansteelandt
Venue: NeurIPS 2024 poster (2024)
Keywords: deep generative model
efficient influence function
inferential utility
synthetic data
Primary area: generative_models

Abstract: While synthetic data hold great promise for privacy protection, their statistical analysis poses significant challenges that necessitate innovative solutions. The use of deep generative models (DGMs) for synthetic data generation is known to induce considerable bias and imprecision into synthetic data analyses, compromising their inferential utility as opposed to original data analyses. This bias and uncertainty can be substantial enough to impede statistical convergence rates, even in seemingly straightforward analyses like mean calculation. The standard errors of such estimators then exhibit slower shrinkage with sample size than the typical 1 over root-$n$ rate. This complicates fundamental calculations like p-values and confidence intervals, with no straightforward remedy currently available. In response to these challenges, we propose a new strategy that targets synthetic data created by DGMs for specific data analyses. Drawing insights from debiased and targeted machine learning, our approach accounts for biases, enhances convergence rates, and facilitates the calculation of estimators with easily approximated large sample variances. We exemplify our proposal through a simulation study on toy data and two case studies on real-world data, highlighting the importance of tailoring DGMs for targeted data analysis. This debiasing strategy contributes to advancing the reliability and applicability of synthetic data in statistical inference.

Debiasing Synthetic Data Generated by Deep Generative Models
Alexander Decruyenacre Ghent University Hospital SYNDARA
Heidelinde Dehaene
Ghent University Hospital SYNDARA
Paloma Rabaey
Christiaan Polet Ghent University Hospital SYNDARA
Ghent University imec
Johan Decruyenaere Ghent University Hospital SYNDARA
Thomas Demeester
Ghent University imec
Stijn Vansteelandt
Ghent University Department of Applied Mathematics, Computer Science and Statistics
Joint first authors and corresponding authors (firstname.. laattname Juugent.
Abstract
While synthetic data hold great promise for privacy protection, their statistical analysis poses significant challenges that necessitate innovative solutions. The use deep generative models (DGMs) for synthetic data generation is known to induce considerable bias and imprecision into synthetic data analyses, compro- mising their inferential utility as opposed to original data analyses. This bias and uncertainty can be substantial enough to impede statistical convergence rates, in seemingly straightforward analyses like mean calculation. The standard errors of such estimators then exhibit slower shrinkage with sample size than the typical over root-n rate. This complicates fundamental calculations like p-values and con- fidence intervals, with straightforward remedy currently available. In response to these challenges, we propose a new strategy that targets synthetic data created by DGMs for specific data analyses. Drawing insights from debiased and targeted machine learning, our approach accounts for biases, enhances convergence rates, and facilitates the calculation of estimators with easily approximated large sample variances. We exemplify our proposal through simulation study on toy data and two case studies on real-world data, highlighting the importance of tailoring DGMs for targeted data analysis. This debiasing strategy contributes to advancing the reliability and applicability of synthetic data in statistical inference.
1 Introduction
The concept of generating synthetic data as a means of privacy protection was initially introduced by Rubin (1993) within the framework of multiple imputation, widely used technique for managing the statistical analysis of incomplete data sets. Since its inception, a substantial body of literature on synthetic data has emerged (Raghunathan et al. 2003 Raghunathan 2021 Drechsler 2011
Raab et al., 2016 Reiter 2005). with recent surge in interest propelled by advancements in deep generative modelling technology (Raghunathan 2021 van Breugel et al.) (0033 Wan et al. [017]] Yan et al. 2022] Nowok et al.) 2016 [Endres et al. 2022 Hernandez et al.) (2022) In this work, we focus on tabular synthetic data their inferential utility, which captures whether synthetic data can be used to obtain valid estimates and inference for population parameter (Decruyenaere et al. 2024..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The substantial privacy protection potential offered by synthetic data is marred by significant chal- lenges that undermine their inferential utility (Raab et al., 2016). Previous work by Deeruyenaere et al. (224) has shown that these challenges are much more severe when using deep generative models (DGMs), rather than parametrie statistical models, which is why we focus on the former. First, standard confidence intervals and p-values obtained on synthetic data may drastically underestimate the uncertainty in synthetic data as these ignore the size of the original data. Indeed, synthetic data obtained via generators trained on small datasets will unsurprisingly deliver much worse quality than synthetic data obtained via generators trained on large datasets. This uncertainty in the generator must therefore be translated into analysis results, such as confidence intervals and p-values. Standard confidence intervals and p-values ignore this, as they do not distinguish whether the data are synthetic or real. Second, it is well known from the literature on plug-in estimation that data-adaptive methods (such as DGMs) cannot succeed to estimate all features of the data- lata-generating distribution well (Bickel et al. 1993 van der Laan and Rose 2011; CCernozhukov et al., 2018: [Hines al., (022). These methods are designed to optimally balance bias and variance w.r.t. a chosen criterion, such as prediction error. However, they cannot guarantee that such an optimal trade-off is simultaneously made w.r.t. all possible discrepancy measures that exist, such as mean squared error in specific functionals (mean, variance, least squares projections...) of the observed data distribution. Such data-adaptive methods therefore leave non-negligible bias in estimators of such functionals, leading to excess variability, slow convergence, and confidence intervals that do not cover the truth at nominal level (and may even never contain the truth, even in large samples) (Decruyenaere et al. (024)).
Related work. While several approaches have been proposed to account for the uncertainty arising from synthetic data generation, we are not aware of strategies for generating and analysing DGM- based synthetic data that guarantee valid inference. Raahhnathan et al. (2003) developed a framework inspired by the work on multiple imputation for missing data, by combining the results of multiple synthetic datasets, but this is not readily applicable to DGM-based synthetic data. Räisä et al. (2023) extended this work for differentially private (DP) synthetic data, acknowledging the additional DP noise, but continue to consider parametric (Bayesian) data generation strategies.
Our work instead focuses on obtaining valid inference from single synthetic dataset, which is arguably more attractive for use by practitioners, |Rabb et al.) (2016) derived alternative combining rules that reduce to the correction factor / + m/nn for the standard error (SE) of an estimator in the case of inference from a single (non-DP) synthetic dataset of size m generated from an original dataset of size n. The method suggested by/Awan and Cail (2020) to preserve efficient estimators in a single (DP or non-DP) synthetic dataset relies on generating data conditional on the estimate in the original data. Both procedures are only applicable to parametric generative models and therefore suffer from the same limitation as the aforementioned approaches. To enable Bayesian inference from a single DP synthetic dataset, Wilde et al. (202I) proposed a corrected analysis that relies on the availability of additional public data, while Ghalebikesabi et al. (2022) investigated importance weighting methods to remove the neise-related bias, but they do not study the impact on inference.
Contributions. Our work is the first to propose generator-agnostic solution that mitigates the impact of the typical slower-thann-/ie foaunnn/nncooveggenee of estimators in synthetic data created by DGMs. As far are aware, our approach is thus the only one that provides some formal guarantees for (more) honest inference in this setting. In this paper, we show how the statistical bias in estimators can be removed, by adapting results on debiased or targeted machine learning (van der Laan and Rose 2011 Chernozhukov et al. 2018) to the current setting where machine learning is not necessarily used in the analysis, but rather in the generation of synthetic data. Although we build upon ideas from existing work. our extension is non-trivial, since (1) previous work did not consider synthetic data; and (2) we demonstrate, with significant generality, how to mitigate the estimation errors in the DGM that would otherwise propagate into the estimator calculated on synthetic data.
In Section 2 and 3 we propose generator-agnostc debiasing strategy, directed towards the down- stream statistical analysis of the synthetic data. As such, we obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting the generators. We illustrate this with simulation study in Section 4 showing that the coverage of both the mean and linear regression coefficient estimators indeed improves. In Section we further cement the utility of our debiasing strategy in a practical setting through two case studies. While the proposed strategy is generator-agnostic. we focus our analyses on two DGMs for tabular data: CTGAN and TVAE (Xu et al. 2019). Finally, Section&] concludes with a discussion on our method and its limitations.
2

---

2 Notation and Set-Up
The aim of this paper is to use synthetic data in order to learn a specific functional ((.) of the observed data distribution P. We formalise the problem of learning ((P) based on synthetic data as follows. Suppose that, based on n independent (possibly high-dimensional) samples Oi,... On from P. we estimate the observed data distribution as P.. Here, Pn may be obtained by fitting parametric models to the distribution of the observed data, or alternatively by training DGMs. Based on m independent (synthetic) random samples Si..., Sm from Pn. we then estimate (PP). For this, the data analyst (to whom P., is unknown) uses the m synthetic samples Si...,, Sm. to approximate Pr as Pm. and obtains an estimator of (P)) given by (Pmm). We use P., to denote the empirical distribution of the observed data and P,,, to denote the empirical distribution of the synthetic data. We index expectations by the distribution under which they are taken; e.g., EY denotes the population expectation of Y.
Throughout, assume that the parameter of interest is sufficiently smooth in the data data-generating distribution so that root-n consistent estimators (i.e., with SEs shrinking at over root-n rate) can be obtained. Formally, (EIC) d(., P) under the nonparametric model, as is satisfied for (most) standard statistical analyses. The EIC is functional derivative of 0 ((P) w.r.t. the data-generating distribution P (in the sense of Gateaux derivative), which has mean zero under P (Fisher and Kennedy 2021 Hines et al. 2022).
assume the 0 (P) is pathwise differentiable with efficient influence curve
In what follows, illustrate our debiasing strategy via two examples. Here, briefly outline some of their theoretical foundations we build upon. Appendix (A. A.....Jl clarifies how these debiased estimators originate from their EIC. We further refer to them debiased or EIC-based estimators.
Population mean. synthetic data, we choose Pm. = Pm. As result, we will study the large sample behaviour of the synthetic data sample average:
Adapting the traditional formulation of the population mean to the context of
m ((Pm) = _ >> S
1
m
The EIC of this sample average is P 5
Linear regression coefficient. Second, suppose we are interested in specific coefficient 0= O(P) of some exposure for the outcome Y in the linear model EY(Y X) 0A w(X), where X is a possibly high-dimensional vector of covariates and w-)) is an unknown function. Our proposal allows (XX) to correspond to a standard linear model all covariates. but is more generally valid. Building upon the nonparametric definition of 0 as given in Appendix [ we adjust it to obtain an estimator in the setting of synthetic data. Let Y, X and be jointly or sequentially modelled by some generative model, from which a single synthetic dataset S: (Y,, A., X,) is sampled (i m.....m. We then consider the estimated regression coefficient of exposure A given by
E
A Epp (A(X.) Y. Ep.. (M1
((Pm.)
Em [A, -AEE, (A(X.))
where the nuisance parameters Ep_ (A|X, X2) and Ep. (Y( are estimated based on synthetic data.
This ElC-based estimator coincides with the Maximum Likelihood Estimator (MLE) when least squares predictions for these nuisance parameters used. Its EIC is (Vansteelandt and Dukes 2022
{ F.. (A[x)} Y Ep. (11 ((P.) Ā Ep, (4|x))
$15, P..)
A Ep, (413)) }"
Ep,
3 Methodology
In a first step, we establish an understanding of how much the estimate based on synthetic data 0 ((Pm) differs from the population parameter of interest O(P). For this, we study the difference (Pm.) 0(P), for which we consider 2 von Mises expansions (i.c., functional Taylor expansions).
3

---

In Appendix A.2 we derive that
((m))-((P) =
( Pr) - Σ
P( + +H((RPPp R((mm:P Pr)
1=1
+
0(s Pm)) - ((S) d(Pm - -P,)
(1)
Σ (0,,, P) 1 Σ O0O,, Pn) + R(PP,
i=1
+ / 000, Pn) p)) d(Pr, -
(2)
We now examine the eight terms of this equation and discuss why some may be negligible while other may introduce bias. First, R(-) are remainder terms, which can generally be shown to be small, but must be studied on a case-by-case basis; see Hines et al.] (2022) for worked out examples. In order for these remainder terms to be op(m -1/2) and On 112), we generally need that faster than n n-144 1/4 convergence rates are obtained for the unknown functionals of Pm and P,, that appear in the EICs. is unknown whether these convergence rates attainable for DGMs; whether they are, will partly depend on the number of parameters in the DGM itself, the dimension of the data and the complexity of the observed data distribution. The simulation study in Section 4 will give further insight into this.
Second, the two empirical process terms and in the von Mises expansion can be shown to be op(m 1//2) and Op(n (m1) by Markov's inequality, under weak conditions. For (1) to converge to zero, we will need the difference between Pm and Pn, to converge to zero (in L (Px) at any rate), and the estimator to be calculated on different part of the data than the one on which Pm was estimated (aaeeeddCChhmmeen (Chernozhukov et al., 2018). For (2) to converge zero, will need P,, (e.g.. the DGM) to consistently estimate P (in (P)) at any rate). In addition, it can be argued that the DGM needs to be trained on different part of the data than that on which the debiasing step (see later) will be applied. In Appendix A.3 we elaborate on the necessity of sample splitting.
We thus foresee that the two remainder and two empirical process terms negligible under certain conditions. Third, as elaborated in Appendix A.2 we show that the large sample behaviour of both
m
o(O,,
and
(S Pr)
is standard, and well understood; in particular, these terms vary around zero with variance that can be estimated well. By contrast, the terms
1 Σ
(SS
(5) and
Σ 6(O,, P.))
(6)
need some further discussion. Term 5 generally fails to have mean zero because the synthetic data S; do not originate from the distribution P.. This term therefore induces a bias in 0 (Pm) that results from using data-adaptive estimates Pm on the synthetic data; similar term would appear if we instead analysed the real data. Term (6) likewise fails to have mean zero because the observed data O, do not originate from the distribution P,, Also this term thus induces bias, now resulting from the use of a generative model to obtain P,. It may be large relative to (3) when DGMs are used, because of slow convergence of P,.. It is precisely this term that causes estimators based on synthetic data to converge slowly with increasing sample size, as observed in Decruyenaere et al. (2024).
After identifying the two problematic terms, we now propose in a second step targeting or debiasing strategy to remove these bias terms (5) and (6). As in van der Laan and Rose (0IT) and Chernozhukoy et al. (2018), we will remove bias term (5) by analysing the data with debiased estimators based on the EIC that ensure that this bias term then becomes zero. Novel to our proposal is that we will additionally shift the generated data to ensure that also bias term (6) becomes zero. This bias term depends on the EIC, which itself depends on the target parameter of interest. In the next two paragraphs, we discuss how this can be done for the two considered estimators. Note that the proposed strategy does not require any actual finetuning or retraining of the DGM. For given parameter of interest, mere post-procsssing of the synthetic samples, based on access to the DGM as well as

---

the original data it was trained on, allows eliminating the bias for given parameter of interest. A graphical summary of the problem setting and our debiasing strategy can be found in Appendix A..4
3.1 Population mean
For the population mean, the debiasing step with respect term (5) is implicit since the traditional estimator as given Section i a debiased estimator. Therefore, the proposal to debias given DGM with respect to the population mean ((P) = odP(o) amounts to first training the DGM and then augmenting the output for the variable of interest to ensure that bias term (6) is zero, or hence
T2 Σ o00 P.) = n2 0. - ((P.) = 0.
In other words, the population mean of the synthetic data under the DGM should match the sample average of the real data. Here, 0 ((Pr) can be approximated by generating a very large sample & (e.g.. one million samples) based the DGM and calculating their sample mean Y. The generative model must then be corrected, to ensure that this sample mean equals O. To obtain debiased synthetic sample, we shift all samples generated by the given DGM by adding 7 - Y to the considered variable. Note that the sample average of such a set of m corrected synthetic samples will generally differ from O.
3.2 Linear regression coefficient
For linear regression coefficients, this section shows how the samples generated by a given DGM need be adapted eliminate bias term (6). written as follows (see Appendix(A.5
ii A Ep, (AIX,) (v, Ep. (x ((P)).
E A, E.p, (A|x:)
To compute this bias, we must generate, for each observed value X₁, a very large sample of measure- ments of A and Y with the given level X,, based on the DGM. Then Ep, (Y (X,) and Ep, (A(X,) can be estimated as the sample average of those values for respectively Y and A. Further based very large DGM--Genrenin sample ls (e.g. million samples), calculate 0 (P..) follows:
Σ A Ep. (A|x,) Y, - Ep. (M1,))
(8)
Σ k A Ep, (AIx,))
Debiasing the DGM can now be done by adding the product bf Ep, (A(x.)) to the synthetic
outcome observations Y; generated by the DGM. An in-depth elaboration is provided in Appendix
AA.5)
With the proposed shifting, we ensure that the debiasing with respect to term (6) is completed.
We then proceed our analysis with these shifted synthetic observations and employ the EIC-based estimator of Section 2] which ensures that bias term (5) equals zero as well.
3.3 Properties
To summarise, when the remainder terms R(.) and the empirical process terms 1 and ( are all op (m 1/2) and on and, furthermore, the suggested debiasing approach used so as to remove terms (5) and (6), then we expect that
-
Σ (A.
Σ (0., o0 1122) + op(m 11/3).
In particular, the resulting estimator O Pm may even converge at root-n rates (under standard
conditions of pathwise differentiability (Bickel et al. 1993] Hines et al. (2022)) and has an easy-to- calculate variance that acknowledges the uncertainty the generation synthetic data (provided that the statistical convergence of the generator is not too slow). In Appendix A.6, we show that the variance of 0 (PP..) may be approximated by
1 ++
1 n
E P1O
5

---

thereby generalising results known for parametric synthetic data generators (Raab et al. 2016 Decruyenaere et al. 2024 where the correction factor v m/n was proposed. Thus, when m - oo, the debiased estimator ((Pm) based on debiased synthetic data has the same distribution as when the real data were analysed. Therefore, the proposed debiasing strategy delivers analysis results that are asymptotically equivalent to those obtained from the same analysis on the real data, provided that n m = o(1). This means that results of the same quality and confidence intervals of the same expected length are then obtained, as will be illustrated in the case study Section 5.2
Since E {6(O, is unknown, it merely remains to estimate it as
(Si, P.
This is a consistent estimator when Pm converges to Pn. as m goes to infinity, and moreover, Pr. converges to P as goes to infinity. We note that this sample variance will be subject to bias that results from 'poor' tuning of the DGM. Removing this bias is not required, because this variance will be scaled by 1/m + 1/n so that any bias becomes negligible in large samples. While the use of debiased estimators based on the EIC of E {Ø(O, }) may potentially improve performance, this goes beyond the scope of this work.
Practical implications. Sections 3.1 and 3.2 show how bias term (6) is eliminated by shifting the synthetic variable of interest. We describe how bias term (5) is also removed by using debiased estimators, which we referred to as ElC-based estimators (see Section 2 As mentioned carlier, the ElC-based estimator for the population mcan always coincides with the sample average, while for the linear regression coefficient it only reduces to the ordinary least squares estimator when data-adaptive estimation of the nuisance parameters is not used. This implies that it may suffice for the applied researcher to 1) shift the synthetic data and apply the traditional estimators, and 2) multiply the SE of the estimator with vI + m/n to obtain valid inference from single synthetic dataset. However, the EIC-based estimators are recommended since they are robust against model misspecification by allowing for more flexibility in the estimation of the nuisance parameters.
Simulation study
Our proposed debiasing strategy is empirically validated by a simulation study that covers both estimators 3.1
(sample mean) and 3.22 (linear regression coefficient). Having full control over the data generating process allows to calculate the bias, SE and convergence rate of both estimators in synthetic data, with and without our debiasing strategy. The data generating process consists of the following four variables: age (normally distributed), atherosclerosis stage (ordinal with four categories), therapy (binary), and blood pressure (normally distributed). The Directed Acyclic Graph (DAG) Figure represents the dependency structure and we refer to Appendix [A.7I]] for more details. This setting allows us to simultancously target the population mean of age and the population effect of therapy (A) on blood pressure (Y) adjusted for stage (X).
atherosclerosis stage
blood pressure
age
therapy
Figure 1: DAG for the variables in the simulation study.
4.1 Set-up
We conduct Monte Carlo simulation study, where n independent records are sampled from the data generating process, forming the observed original dataset OI,..., On. This process repeated 250 times, with the sample size varying log-uniformly between 50 and 5000 (i.e., 12 E [50, 160, 500, 1600, 5000)). Per original dataset, following DGMs are trained: CTGAN and From these
TVAE (Xu et al., 2019). of which detailed explanation can be found in Appendix DGMs m synthetic data records are sampled that constitute the default synthetic dataset S1, Sm.
We set m = n to retain the dimensionality of the original data. Subsequently, each default synthetic dataset is debiased with respect to both estimands using the steps provided in Section 3 leading to a debiased synthetic dataset. Finally, two estimators are calculated in each of three datasets: the sample mean of age and the linear regression cocfficient of therapy blood pressure adjusted for

---

Mean age (MLE-based)
Effect therapy (MLE-based)
Generator Original
100%
Coverage
75%
CTGAN
50%
TVAE
25%
Debiased
0%
No
50
500
500050
500
5000
Yes
(log scale)
Figure 2: Empirical coverage of the 95% confidence interval for the population mean of age and the population effect of therapy on blood pressure adjusted for stage.
stage. We always report the maximum likelihood estimation (MLE)-based estimators, as used in traditional statistical analysis, of which the standard errors (SEs) are inflated with the correction factor / 1 + m/n to acknowledge the sampling variability of synthetic data. These estimators will deliver similar estimates as the EIC-based estimators since no data- adaptive estimation used (see Section 3.3
and and Appendix A.7.5).
4.2 Results
We now present the results of our simulation study. The DGMs were trained using the default hyperparameters as suggested by the package Synthcity (Qian et al. 2023 We also show results obtained for other hyperparameters (the default in the package SDV Patki et al. 2016)) in Appendix A.7.4 In Section I.2.1 we evaluate the empirical coverage of the 95% conndence interval (CI) for the population parameters. Our debiasing strategy should enhance the coverage, preferably to the nominal level, allowing for (more) honest inference, which is the main contribution of our strategy. Then, we analyse step by step the various components that may influence the coverage by investigating the bias and SE of the estimators in Section 4.2.2 and their convergence rates in Section 4.2.3 Additional results are presented in Appendix A.7.4 including the convergence rates of the nuisance parameters estimated by the DGMs. Our code is available Github: https -enneeeteeneeeriionn
//gihhub. com/ conn/syndra-a..
4.2.1 Coverage
By definition, 95% (empirical) of the 95% CIs (nominal) should cover the population parameter. Figure 2 depicts the empirical coverage of the 95% CIs obtained from both original and synthetic samples for the population mean of age and the population effect of therapy on blood pressure adjusted for stage. The results indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage based the default synthetic datasets decreases with increasing due to slower shrinkage of the SE than the typical 1 over root-n rate, as calculated in Section [4.2.3] and previously elaborated in Decruyenaere al. (2024). For the population regression coefficient, debiasing delivers coverage at the nominal level for TVAE across all sample sizes, but not for CTGAN, although it clearly provides more honest inferences than based the default synthetic datasets. The residual loss of coverage likely results from not using (efficient) sample splitting (see Appendix A.3)
4.2.2 Bias and Standard Error
Figures [3a aad 3hjdepict [ the estimates and their SE, respectively, for the sample mean of age obtained in the default and debiased synthetic datasets. Figures and an A6 the appendix show these for the linear regression cocfficient of therapy on blood pressure adjusted for stage. In Figure Ba] each dot is an estimate per Monte Carlo run and the true population parameters are represented by the horizontal dashed line. This figure allows a qualitative assessment of two key properties of estimators: empirical bias (i.c., the average difference between the estimates and the population parameter, represented by the solid line) and empirical SE (i.e., the standard deviation of the estimates, as indicated by the vertical spread of the estimates). Ideally, both converge zero as the sample size grows larger. The convergence rate conveys the rate at which this happens. The funnel represents the default behaviour of unbiased estimator based on original data of which the SE diminishes at rate of 1 over root-n.
As shown in Figure 3a the sample mean of age is unbiased in the default synthetic datasets, but exhibits a large empirical SE that shrinks slowly with sample size due to the data-adaptive nature of
7

---

CTGAN
TVAE
CTGAN
TVAE
SE of mean age (MLE-based)
Mean age (ML MLE-based]
Debiased:
Debiased:
50
no
40
30
Standard error -Epirical
60
Debiased:
Debiased:
MLE-based
50
Yes
40
30
50
500
500050
500
5000
50
500
5000 50
500
5000
n (log scale)
n (log scale)
(a)
(b)
Figure 3: Each dot in Figure (a) is an estimate for the population mean of age per Monte Carlo run. The funnel indicates the behaviour of unbiased and Vn--consistent estimator based observed data. Figure (6) depicts the empirical and average MI.E-based SE for the sample mean of age.
DGMs. It is exactly this variability that is, on average, underestimated by the MLE-based SE in Figure Debiasing reduces the empirical variability and accelerates its shrinkage, such that the average MLE-based SE approximates the empirical SE. For the linear regression coefficient of therapy on blood pressure adjusted for stage, debiasing reduces finite-sample bias and also improves shrinkage of the empirical SE. Albeit less pronounced, the average MLE-based SE still underestimates the empirical SE with CTGAN despite debiasing (see Figures A5 and A6 in the appendix).
3b
4.2.3 Convergence Rate of Standard Error
Assuming power law n= in convergence rate for the empirical SE, we estimate the exponent from five logarithmically spaced sample sizes n between 50 and 5000, shown in Table | and Figure in the appendix. Standard statistical analysis assumes that the bias converges faster than the SE with the latter diminishing at a rate 1/ /n. However, the SEs produced by DGMs converge much slower (i.e., asE < 0.5), leading to progressively increasing underestimation of the empirical SE by the MLE-based SE (which assumes Vii-convergence) as the sample size grows larger. In turn, this results in too narrow Cls and poor coverage, as observed in Section 4.2.1 By contrast, debiasing strategy renders estimators of which the SE converges at approximately root-n rates (i.c., asg = 0.5), explaining the improvement in coverage of the CIs as result of debiasing.
Table 1: Estimated exponent [95% CI] for the convergence rate n for empirical SE.
Estimator
CTGAN
TVAE
CTGAN
TVAE
Default synthetic datasets
Debiased synthetic datasets
Mean age
0.01 [-0.16; 0.18)
0.21 [0.04; 0.39]
0.50 [0.46; 0.54]
0.47 [0.44; 0.50]
Effect therapy
0.31 (0.22; 0.40] 0.09 [-0.13; 0.31]
0.43 [0.31; 0.55] 0.46 [0.38; 0.55]
5 Case studies
To illustrate our findings and highlight their implications for the applied researcher, we conduct two case studies. First, Section [. transfers the framework from our simulation study to the International Stroke Trial (IST) dataset (Sandereock et al.) (01T). Second, Section [5.2 describes whether analysis results from the Adult Census Income dataset (Becker and Kohavi] 1996) are similar to those obtained from the real data, when the sample size m of the generated synthetic data very large. In both case studies, estimated SEs in the default synthetic and debiased synthetic datasets are corrected by multiplying with factor / + m/n to acknowledge the sampling variability of synthetic data.
5.1 International Stroke Trial
We adapt the framework discussed Section 4. to the IST dataset, one of the biggest randomised trials in acute stroke research (Sandercock et al. 2011). The dataset with 19285 complete cases now
8

---

constitutes our population. We mimic different hypothetical settings where an institution only has access to limited sample of observations, with the sample size n varying between 50 and 5000. In order to easily share the data with other researchers, the institution generates synthetic dataset with sample size m, where m = n. We repeated this process 100 times per sample size n to be able to calculate the empirical coverage levels. For illustration purposes, we focus on the effect of aspirin on the outcome at 6 months and report the proportion of deaths for the two treatment arms (aspirin and no aspirin), and its corresponding risk difference. For each value of two default synthetic datasets were generated using both CTGAN and TVAE. Next, for each of these, we first split the default synthetic dataset by treatment, debias the data with relation to the population mean within each treatment arm, and then combine them back into one debiased synthetic dataset. We noticed that using the same hyperparameters in the simulation study resulted in biased estimates, can be seen in Figure A12 in the appendix. For this reason, we highlight the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al. 2016) instead.
One of the original research questions in Sandercock et al. (201T) was whether or not there is a difference in risk of death between the treatment arms. Suppose researcher can repeatedly collect information on 500 subjects and uses original, default synthetic and debiased synthetic data to make an inferential statement about this risk difference. Figure depicts the confidence intervals for the first 15 repetitions, with the vertical dashed lines representing the true risk difference of -0.009 as calculated based on population (the full dataset). Should the researcher data, they would falsely conclude (in 7 out of these 15 repetitions) that the risk is significantly different from -0.009, while using the debiased synthetic dataset basically climinates this high number of false-positives (with all these 15 intervals containing the population parameter), as is the case in the original data as well. More results can be found in Appendix A.8.T
the default synthetic
Original data
Default CTGAN
Deblased CTGAN
Mean hours/week
15
45222
Original
8246
1503.
Default synthetic
274
Debiased synthetic
10
(log-scale)
50
Run
3'5
37.5
40
42.5
Effect income
5
45222
8246
1503
274
50
-0.2
0
0.2
-0.2
0.2
-02
5
10
15
Risk difference death
Estimate (95% CI)
Figure 4: Empirical coverage of 95% CIs for the risk difference for death in each of the three datasets (m n = 500) for the IST case study.
Figure 5: 95% CIs for the population mean and regression coefficient for m 10 and different sample sizes n for the Adult Income case study.
5.2 Adult Census Income Dataset
We also perform a case study on the Adult Census Income dataset, which comprises 45222 complete cases and 14 unique variables (Becker and Kohavi, 1996). We assume the researcher's 's interest lies in inferring the population mean of hours worked per week (estimated via the sample mean) and the average sex-adjusted difference in age between persons with an income of > $50K a year vs. not (estimated via a linear regression model age (Y) ~ income (A) + sex (X)). Our goal in this case study is to confirm whether inferential results obtained using the debiased synthetic dataset are asymptotically equivalent (i.e. with m >> n in our debiasing strategy) to those obtained using the original data. To test this across different sample sizes, the original data constitute five different samples of the Adult Census Income dataset with sizes n varying log-uniformly between 50 and 45222. For each original dataset, default synthetic dataset of size 10 was generated by TVAE. Subsequently, this dataset was debiased following the steps described in Section 3.1 (sample mean) and Section 3.2 (linear regression coefficient).
Figure degures depicts the 95% CIs for both estimators, the five original sample sizes and the three versions of datasets. This indeed confirms that analysis the debiased synthetic dataset leads to results of similar quality and Cls of similar length compared to the original dataset. By contrast, the analysis on the default synthetic dataset may yield results of inferior quality and even incorrect conclusions.
9

---

6 Discussion
In this paper, we propose a new debiasing strategy that targets synthetic data created by DGMs towards the downstream task of statistical inference from the resulting synthetic data. We establish our theory for two estimators by applying insights from debiased or targeted machine learning literature (van der Laan and Rose 2011 Chernozhukoy al., 2018) to the current setting where machine learning is not necessarily y used in the analysis, but rather in the generation of synthetic data. We obtain estimators that are less sensitive to the typical slow (statistical) convergence affecting DGMs and thereby improve the inferential utility of the synthetic data.
We illustrated the impact of our proposal through simulation study on toy data and two case studies on real-world data. Our debiasing strategy results in root-n consistent estimators based on the synthetic data and thereby better coverage of the confidence intervals, allowing for more honest inference. While coverage was clearly improved, it was not guaranteed to be at the nominal level. Indeed, it may remain anti-conservative for some estimators and DGMs, due to slow convergence inherent to these models and/or due to residual overfitting bias that could not be removed since sample splitting was not performed. Future work should focus on efficient sample splitting, where the resulting bias reduction outweighs the increase in finite- maite-sample bias that arises from training on smaller sample sizes. Alternatively, findings from Ghalebikesabi et al. (2022) on importance weighting could be incorporated, with the weights being targeted to eliminate the impact of the data-adaptive estimation of the weights. This may potentially relax the fast baseline convergence assumption, and enable the same debiased synthetic data to be used for multiple downstream analyses.
A key advantage of our debiasing strategy is that it may deliver synthetic data created by DGMs of which the analysis is equivalent to the original data analysis, provided that the synthetic sample size is chosen to be much larger than the original sample size. Although the risk of disclosure may increase with the size of non-DP synthetic data (Reiter and Drechsler 2010), this trade-off is beyond the scope of our paper. More interestingly, in the case of DP synthetic data, our debiasing strategy may exploit their proccesinn immunity that allows for data transformations without compromising privacy guarantees (Dwork and Roth 2014). However, our strategy needs to be extended to incorporate the DP-constraints when studying the difference between (P...) and ((P) in DP synthetic data.
Limitations of our proposal include the low-dimensional setting of our simulation and case studies, for which DGMs might be less well suited. The positive results found for two widely used estimators in this simple setting highlight the utility of a debiasing approach and are encouraging in terms of future larger-scale applications. However, before addressing these, it is important first understand low-dimensional settings, where valid inference is already challenging to attain. While our debiasing strategy boils downs post-processing step, one could argue that the lack of change to the DGM's training strategy itself is actually a strength, since it renders our strategy
Another limitation concerns the fact that our debiasing strategy for a regression coefficient requires sampling of synthetic data conditional on a covariate, which is not available in all DGMs. However, this issue is partially mitigated in the case of conditioning categorical variables, since one can always generate a synthetic dataset unconditionally and then only select samples that fit the condition though this approach also has its limits, especially when conditioning on multiple covariates once. Zhou et al. (2023) propose a deep generative approach to sample from a conditional distribution, even when working with high-dimensional data. Future work could explore this strategy.
Finally, our proposal requires that the person generating synthetic data is aware of the analyses that will be run on those data, and has access to the corresponding EICs needed for debiasing (which in particular rules out the possibility for debiasing w.r.t. non-pathwise differentiable parameters, such conditional means predictions). For each parameter of interest, the data generated by the DGM will then need to be debiased (simultaneously) w.n.t that parameter's s EIC, which is left for future research. In the case of original data, several debiased estimation strategies that do not require exact knowledge about the EIC already exist. These methods include a) approximating the EIC through finiteddifferencing (Carone et al. 2019 Jordan al., 2022) stochastic approximations via Monte Carlo (Agrawal et al. 2004)), and b) automatically estimating the EIC from the data through auto-DML. (Chernozhukov al. 2022). Alternatively, kernel debiased plug-in estimation methods (Cho et al. 2023) enable simultancous debiasing of all pathwise differentiable target parameters that meet certain regularity conditions, without requiring any knowledge about the EIC. Integrating their insights could further strengthen the foundations of our current work on the interplay between synthetic data, deep generative modelling, and debiased machine learning.
10

---

Acknowledgments and Disclosure of Funding
Paloma Rabaey's research is funded by the Research Foundation Flanders (FWO-Vlaanderen) with grant number I 177122NN. This research also received funding from the Flemish government under the "Onderzoeekproorramaa Artificiële Intelligentie (AI) Vlaanderen" programme.
References
Agrawal, R., Witty, S., Zane, A., and Bingham, E. (2024). Automated efficient estimation using monte carlo efficient influence functions. arXiv preprint arXiv:2403.000
Awan, J. and Cai, Z. (2020). One step to efficient synthetic data, arXiv preprint arXiv:2006.02397.
Becker, B. and Kohavi, R. (1996). Adult. UCI Machine Learning Repository.
Bickel, P., Klaassen, C., Ritov, Y., and Wellner, (1993). Efficient and Adaptive Estimation for Semiparametric Models, volume Baltimore Johns Hopkins University Press.
Carone, M., Luedtke, A. R., and van der Laan, M. J. (2019). Toward computerized efficient estimation in aninite-dimensiinall models. Journal of the American Statistical Association.
Chernozhukov, V., Chetverikov, D., Demirer, M., Dufto, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 21(1):C1-C68.
Chemoozuuovv V., Newey, W. K., and Singh, R. (2022). Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3):967-1027.
Cho, B., Gan, K., Malenica, I., and Mukhin, Y. (2023). Kernel Debiased Plug-in Estimation, arXiv preprint ar Xxvv:2306.08599
Decruyenaere, A., Dehaene, H., Rabaey, P., Polet, C., Decruyenaere, J., Vansteelandt, S., and Demeester, T. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In Kiyavash, N. and Mooij, J. M., editors, Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pages 966-996. PMLR.
Drechsler, J. (2011). 1). Synthetic datasets for statistical disclosure control: theory and implementation, volume 201. Springer Science & Business Media.
Dwork, C. and Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 (0044:2211441 1-407.
Endres, M. Mannarapotta Venugopal, A., and Tran, T. S. (2022). Synthetic data generation: comparative study. In Proceedings of the 26th International Database Engineered Applications Symposium, pages 942
Fisher, A. and Kennedy, E. H. (2021). Visually communicating and teaching intuition for influence functions. The American Staristician, 75(2):162-172.
Ghalebikesabi, S., Wilde, H., Jewson, J., Doucet, A., Vollmer, S., and Holmes, C. (2022). Mitigating statistical bias within differentially private synthetic data. In Cussens, J. and Zhang, K., editors, Proceedings of the Thirty-Eight Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 696-705. PMI.R
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances neural information processing systems, 27.
Hernandez, M., Epelde, G., Alberdi, A., Cilla, R., and Rankin, D. (2022). Synthetic data generation for tabular health records: A systematic review. Neurocomputing, 493:28-45.
Hines, O., Dukes, O., Diaz-Ordaz, K., and Vansteelandt, S. (2022). Demystifying statistical learning based efficient influence functions. American Statistician, 76(3):292-304,

---

Jordan, M., Wang, Y., and Zhou. A. (2022). Empirical gateaux derivatives for causal inference. Advances in Neural Information Processing Systems, 35:8512-8525.
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint.
Nowok, B., Raab, G. M., and Dibben, C. (2016). Synthpop: Bespoke creation of synthetic data in R. Journal of Statistical Software, 74(11):1-26
Patki, N., Wedge, R., and Veeramachaneni, K. (2016). The synthetic data vault. In IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 399-410.
Qian, Z., Cebere, B.-C., and van der Schaar, M. (2023). Synthcity: facilitating innovative use cases of synthetic data in different data modalities. arXiv preprint.
Raab, G. M., Nowok, B., and Dibben, C. (2016). Practical data synthesis for large samples. Journal of Privacy and Confidentiality, 7(3):67-97.
Raghunathan, T. E. (2021). Synthetic data. Annual review of statistics and its application, 8:129-140.
Raghunathan, E., Reiter, J. P., and Rubin, D. B. (2003). Multiple imputation for statistical disclosure limitation. Journal of official statistics, 19(1):1.
Räisä, O., Jälkö, J., Kaski, S., and Honkela, A. (2023). Noise-aware statistical inference with differentially private synthetic data. In International Conference Artificial Intelligence and Statistics, pages 3620-3643. PMLR.
Reiter, J. P. (2005). Releasing multiply imputed, synthetic public use microdata: an illustration and empirical study, Journal of the Royal Statistical Society Series A: Statistics in Society, 168(1):185-205.
Reiter, J. P. and Drechsler, J. (2010). Releasing multiply-imputed synthetic data generated in two stages protect confidentiality. Statistica Sinica, pages 405-421.
Rubin, (1993). Statistical disclosure limitation. Journal of official Statistics, 9(2):461-455
Sandercock, P. A., Niewada, M., Czlonkowska, A., and Group, S. T. C. (2011). The international stroke trial database. Trials, 12(1):101.
van Breugel, B., Qian, Z., and der Schaar, M. (2023). Synthetic data, real errors: How (not) to publish and use synthetic data. arXiv preprint.
van der Laan, M. J. and Rose, S. (2011). Targeted Learning. Springer Series in Statistics. Springer New York, New York, NY.
Vansteclandt, S. and Dukes, O. (2022). Assumption- inference for generalised linear model parameters. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):657- 685.
Wan, Z., Zhang, Y., and He, H. (2017). Variational autoencoder based synthetic data generation for imbalanced learning. In 2017 IEEE symposium series on computational intelligence (SSCI), pages 1-7. IEEE
Wilde, H., Jewson, J., Vollmer, S., and Holmes, C. (2021). Foundations of bayesian learning from synthetic data. In International Conference Artificial Intelligence and Statistics, pages 541-549. PMLR.
Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. (2019). Modeling tabular data using conditional GAN, Advances in neural information processing systems, 32.
Yan, C., Yan, Y.. Wan, Z., Zhang, Z., Omberg, L., Guinney, J., Mooney, S. D., and Malin, B. A. (2022). A multifaceted benchmarking of synthetic electronic health record generation models. Nature communications, 13(1):7609.
Zhou, X., Jiao, Y., Liu, J., and Huang, J. (2023). deep generative approach conditional sampling. Journal of the American Statistical Association, 118754))
12

---

A
Appendix
A.1 Elaboration estimators for synthetic data
Population mean.
The population mean of the observed data is defined
((P) f dP(o) do - fodP(o. odP(o).
f
do
which we will denote short as f OdP and its efficient influence curve (EIC) is 6(0,P) = - --0(())
Choosing P.. P,,, will then study the large sample behaviour of the synthetic data estimator:
0(Pm) ʃ sdlP sddPm(s)
S.
Linear regression. A in the linear model
Suppose we are interested in specific coefficient = O(P) of some exposure
Ep = 0. + w(X),
where is a possibly nigh-dimensional covariate and w(.) is an unknown function. Our proposal allows (XX) to correspond to standard linear model, but is less restrictive. The parameter 0 in this linear model can be nonparametrically defined as
O(P) Ep I[A Ep(A (Y Ep({
Ep
{A Ep(A
in the sense that reduces to when the above model holds. Its EIC is (Vansteelandt and Dukes 2022
{A -A-EPP E Ep(Y - (A Ep(A[x))
()
Ep (A- (A-Ex(A)X)]
Denote the obtained synthetic data samples as 5= (Ý, A, 8). An estimator 8 (P...) ) is then obtained by substituting the above expression for ((P), the first expectation in the numerator and denomi- nator by sample average, Er(A|X and (x)(XX by data-adaptive predictions Ep.,. (A) X,) and Ep, (Y (Y)),) obtained based on the synthetic data. This delivers estimator
Σm
m
A.
B (AIX.) { Ei (11X,))
((P..)
E A, Ep.. (A1
13

---

A.2 Derivation of the impact of uncertainty affecting deep generative models
We are interested in knowing how much ((Pm) differs from (P). For this, we study the difference ((Pm) - O(P). for which consider 2 von Mises expansions (i.e., functional Taylor expansions). Throughout the calculation below, we use that influence curves (OO, P) have the property of being mean zero when averaging w.r.t the distribution P. We moreover let R(-) be a remainder term, which can generally be shown to be small, but must be studied on a case-by-case basis.
((Pm) ((pm)-((P) = o(Pm)) -
/ (m). P R(P Pn)
(a)(PP - P) + R(Pp. P)
/ O(S Pm)d + [((Pmm Pn) - (10,Pa) R(P P)
= / ((SPPm) -P,) - -f 0(S Pm) + R(P P.))
/ (.)(PP))(P,.- -P) (,),P,)dP,, + R(P,, P) = / O(S P.)(PP - -P,)
+ R(P Pa)
/ (m() (1S, P.)) d(F - -Pa)
+ / ((,P))((P, - P) - - / (0,0, (.),P.))PP. + R(Pnn P)
/
- -(0))) p)) d(IP - -P)
Σ P( 1 Σ (S Pmm) + R2( ms Pa)
m
/ P()) (15
d( d(p -Pr)
1 Σed 2) Σ n
(0,,, R(P,.P)
/ (1) c(0 pi} d(P,.. P).
Here,
n
(0,, P)
equals 1 over root-n times a term that is asymptotically normal (as n goes to infinity) with mean zero and variance E {Ø(0, p)2]. Further, conditional on P...
(S, P.))
equals over root-m times term that converges E P... 1/2
times standard normal distribution (as m goes to infinity). This follows from the mean zero property of influence curves and the fact that the synthetic data are drawn from the distribution P,.. This mean zero property is essential as it implies that the variation of P,, across repeated (observed data) samples does not
14

---

induce excess variability, Next letting also the sample size go to infinity, we obtain convergence of m 1/2 E oS,, P.)) to E ())) ((, 1) under the assumption that E ( P.)
converges in probability to E {0(O,, This is a weak assumption because
E F60 - 6(o, P.)
generally smooth (continuous) in the distribution of the data (as in the considered two examples). Moreover, the flexibility offered by deep generative models (DGMs) makes it reasonable to assume that Pn, converges to P (e.g.. in L2 (P)); while this convergence may be slow, no requirements on the rate convergence are needed for the above assumption to be satisfied.
15

---

A.3
Note on sample splitting
In order to let
((. d(P
converge to zero, we will need the DGM to be trained on different part of the data than the one which the debiasing step will be applied. Such sample splitting is needed to prevent overfitting bias that may otherwise result from the highly data-adaptive nature of DGMs. In addition, we need Pa (c.g., the DGM) to consistently estimate P in the sense that the squared mean (under P) of s00, P.) (), P) (at fixed P.) converges to zero in probability.
To prevent efficiency loss with sample splitting, one may consider the use of k-fold cross-fitting. For this, we randomly split the data in & folds, each time train the DGM on - 1 folds to obtain an estimator of the observed data distribution P(K-1)m/ 1lm-1)//k and calculate the bias
n/kk Σ M0 M P(-D))/k)))
1
based on the n/k data points O; in the remaining fold. The average of the k obtained bias estimates can next used for debiasing (see the next section for specific examples).
However, it remains to be seen from future work if the resulting bias reduction outweighs the increase in sinite-sample bias that may result from training the DGM on smaller sample sizes. Preliminary simulations with a simple implementation suggested that this was not the case, which is why our results in the main text are reported without the use of sample splitting. Furthermore, the remainder of the theory discards this nuance for now as well.
16

---

A.4
Graphical summary
Data generating process
(P))
0 m
e
Ground truth P
Generative model P,
Parameter of interest (()
1/4
Population mean
Density
Sample
m
Empirical distribution
Original P..
Synthetic F.
Variable
Based original data
Based synthetic data
Estimate for ((P)
Repeated sampling variability
Original data uncertainty
Minimal synthetic data uncertainty
Additional synthetic data uncertainty
50
500
5000 50
500
5000
Sample size
Figure A1: Suppose that interest lies in inferring the mean ((.) (orange solid vertical line) of the ground truth distribution P (orange solid curve) from synthetic data. (1) First, random sample of original data of size with empirical distribution P., (orange histogram) is collected from this ground truth. Thcory on asymptotic linearity prescribes that sample mean (orange dashed vertical line) will deviate from the population mean by order of I over root-r (grey arrow). (2) Subsequently. a deep generative model is trained on these original data, yielding an estimated distribution P., (blue solid curve). Its mean (blue solid vertical line) may turn differ from the original sample mean by an order larger than over root-n (red arrow), which is referned to regularisation bias in Decruyenaere et al. 2024 (3) Finally, synthetic data of size m (here = n)) are sampled from the estimated distribution P.. forming the empirical distribution P. = P.. (blue histogram with sample mean indicated by the blue dashed vertical line). mean of both distributions P. and P., will again differ by order of over root-m (green arrow). Ideally, the data analyst, who uses the synthetic data to estimate ((P) by (P,.) needs take into account these three sources random variability.
(4) The large sample behaviour of the synthetic data estimator # P..) is depicted by repeating the above procedure multiple times across increasing sample sizes = m and storing cach estimate of the synthetic sample mean. Although the estimator remains unbiased for (PP) (dashed linc), its empirical standard eтrоr becomes larger than in the original data to the additional sources of variability. However, the correction factor /1 + m/n to the model-bascd standard eтrоr previously proposed by Raab et al. (201)) only captures the original data sampling variability (grey funnel) and lower bound of the synthetic data sampling variability (green funnel), while the uncertainty associated with the regularisation bias (red funnel) remains unaccounted for. Since cannot readily expressed analytically, our debiasing strategy eliminate latter by shifting mean of the distribution O(P))) estimated by the generative model towards the original sample and funnel). Additionally, choosing synthetic sample sizes -> oo shrink the synthetic data sampling variability (ultimately removing the green arrow and funnel), such that the synthetic data estimator exhibits similar large sample behaviour as in original data.
(thereby removing the red arrow
17

---

A.5
Elaboration on debiased strategy for linear regression coefficient
For the linear regression coefficient case, this section shows how the samples generated by given DGM need to adapted to eliminate the bias b, given by
Σ i= A. Ep, (A(x,) } Y, -pp, (MXX) - ((P)) { - Ep, (AIX))
b
A Ep, (AIX.)) Li=1 A; Ep. (AIX,)
Σ Si=1 {
(x1xa) ((Pr)
E A. - Ep, (A|x.)
To compute this bias, we must generate, for each observed value X,, a very large sample of measure- ments of A and Y with the given level X,, based on the DGM. Then E'p, (Y (X+) can be calculated be calculated as the sample average of those values for A. Further based large sample generated based the DGM, we calculate (Pa)) as the sample average of {A Ep, (A(X) { - EE, (Y (x1x) divided by the sample average {A- Ep, (A
as the sample average of those values for Y, and likewise Ep, (A[[;)
Debiasing of the DGM can now be done by adding b( Ep. (A[X;)) to the synthetic outcome observations generated by the DGM. This change does not affect the predictions Ep, (Y|X,) from the DGM (because A; pp, (A(X;) averages to zero for each choice of X,). Further, with this change, (Px)) also increases with b units as follows:
N A Ep, (A(X))
Y, + b A. - Ep, (A(X,) Ep, (Y1
E A, F.. (A
where the sum runs over large sample of synthetic observations. With this change in 0 O(P,), the previously calculated bias becomes
Σ E. A. {
Ep. (A(x))
} { (r, Ep, (M]x))
(o(P)) + +b
E A, E,, (A(X,)
which is zero, and as such the debiasing with respect to term (6) is complete. Based on a new sample of synthetic observations (independent of those generated to calculate the bias), the synthetic data estimator is
1 m
Em -=
A
Ep, (A|x.)
Y, + b {A, -Ej,, (A(x.) Ej. (Y13))
((P...
m - ( -FP (AI }
À. - Epp (A)x,) } {Y, -ypppp (x1))
2i=1
+
1 EE A; E (A|
This ensures that bias term (5) equals zero. Please note that this estimator coincides with the standard linear regression estimator on the debiased synthetic data when least squares predictions for Ep (A(X;) and Ep., (Y] are used.
18

---

A.6 Derivation of variance of debiased estimator 0 Pm))
With the suggested debiasing, thus expect that
8(Pm) ((P) 1 Σ Pa) Σ o + o (m(m)
Since the synthetic data are independently drawn from the observed data, any covariance between the 2 leading terms must originate from the fact that P, depends on the observed data. This covariance equals zero since
(o p)}
E E BBB M(S 10.00,,...,00066
| 0( Pn) O On 610,
E [Bfols. P.). 60 =0
where the second equality we use that Pr. determined by OI,..., Ons in the third equality we use that S; only depends on O....., On via Pa. and the final equality we use that ((S,, P.) has mean zero when the synthetic data are sampled from Px. This renders these terms asymptotically independent and their sum, hence, asymptotically normal. Moreover, since the variance of db (t,, P.) converges E (o(O,P)) }) (see the previous section; Appendix A.2 the variance of 0 Pm)) may thus be approximated by
( n
E ()(
thereby generalising results known for parametric synthetic data generators (Raab Decruyenaere et al. 2024).
2016
19

---

A.7 Simulation study
A.7.1 Data generating process
Algorithm 1: Data generating process for hypothetical disease.
input Requeesed number of data records n.
Dataut:aa D with records, each made up of attributes: age, stage, therapy, bp.
D + Empty dataframe. for +- 1 to do
age < Normal(mean 50, std = 10)
Mage + 0.05
vt. VI1. 6 2,3,4
"PI,epII,cpE
Sigmoid(vi Vage x age), Sigmoid(v11 Mage (uge)),igmmii(uni Vage X age) stage Categoricallcat
(epı, op11 - cpr, epın1 cprr, 1prr:l))
therapy + (ateegorcaaleal = [False, True]. = (..5.
Biherapy -- 20
B1, B11, B111, 3Iv t- 0, 10, 20, 30
pbp + 120 + Sstage + Seheropy x therapy
bp Normal = pbp; std 10)
D. - {age, stage, theropy, bp}
end
Inspired by an applied medical setting, we create a hypothetical disease, defined by a low-dimensional tabular data generation process. The dependency structure depicted by the directed acyclic graph (DAG) in Figure T in the main text displays the presence four variables, each them chosen to obtain mix of data types. In our hypothetical disease, it is assumed that patient is observed at a given point in time. At this time, patient data about age, atherosclerosis stage, and the random assignment of therapy is gathered. The continuous outcome variable blood pressure is evaluated at later time point, making this design simplification, since we do not consider the data as longitudinal.
The exact routine to reconstruct this data generating process is presented in the pseudo-code in Algorithm T Age (continuous) follows a normal distribution with mean and standard deviation 10. Atherosclerosis stage (ordinal) generated according proportional odds cumulative logit model where an increase in age causes an increase the odds of having a stage higher than a given stage l (Vage is considered to be 1:1 randomly assigned and is therefore sampled from a Bernouilli distribution with probability of 0.50. The last variable, blood pressure (continuous). is sampled from normal distribution with standard deviation 10 and where the baseline average of 120 increases with higher atherosclerosis stage (Bstage (0, 10, 20, 30} for stage I-IV, respectively) and absence of therapy (Btherapy = 20)).
-0.05 and intercepts Vstage {2, 3, 4} for stage I-III). Therapy (binary)
A.7.2 Deep Generative Models
We elaborate on the DGMs used to create synthetic data our simulation study and cases studies. All experiments were run on our institutional high performance computing cluster using single GPU (NVIDIA Ampere A A100; S0GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across sample sizes; Adult Census Income Dataset case study: less than 4 hours). We focus on two commonly used DGMs, namely Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) and Variational Autoencoders (VAEs) (Kingma and Welling 2013).
GAN consists of two competing neural networks, a generator and discriminator, and aims to achieve equilibrium between both (Hernandez et al. 2022). This translates to a mini-max game, since the generator aims to minimise the difference between the real and generated data, while the discriminator aims to maximise the possibility to distinguish the real and generated data (Goodfellow
20

---

et al., 2014). We use the CTGAN implementation that was designed specifically for tabular data, proposed by Xu al. (2019).
VAE is a deep latent variable model, consisting of an encoder and a decoder (Kingma and Welling 2013). The encoder models the approximate posterior distribution of the latent variables given input instance, whereby typically a standard normal prior is assumed for the latent variables. The decoder allows reconstructing an input instance, based sample from the predicted latent space distribution. Encoder and decoder can be jointly trained by maximising the evidence lower bound (ELBO), i.. the marginal likelihood of the training instances. Maximising the ELBO corresponds to minimising the Sullback-Leible (KL) divergence between the predicted latent variable distribution for given input instance and the standard normal priors, and minimising the reconstruction error of the input instance at the decoder output. Once again, we use the tabular implementation of VAE (TVAE) proposed by Xu al. (2019).
The results presented in the rest of the Appendix are obtained by training both types of DGM with default hyperparameters as suggested by the packages Synthcity and SDV (for both CTGAN and TVAE). A comparison of the default hyperparameters in both packages is provided in Tables A] and
A2
Note that both packages implement the CTGAN and TVAE modules originally proposed by Xu
et al.
2019), where a cluster-based normaliser is used to preprocess numerical features.
Table A1: Comparison of CTGAN default hyperparameters between SDV and Synthcity.
Hyperparameter
SDV
Synthcity
generator_dee _diim discriminator dim
(256, 256)
(500, 500)
(256, 256)
(500, 500)
generator senerator_dropout
0.0
0.1
discriminator drrpout
0.5
0.1
generator Ir generator _ecca
2 x 10
1 x 10-3
1 10
x 10
discriminator_Ii Ir discriminator_dee _ecattorrrecyyyeeeee batch_size
x 10
1 10-3
1 10
x 10-3
500
200
discriminator_steg
1
epochs
300
2000
Table A2: Comparison of TVAE default hyperparameters between SDV and Syntheity.
Hyperparameter
SDV
Synthcity
embedding dim
128
500
encoder dim
(128, 128)
(500, 500, 500)
decoder_dit dim
(128, 128)
(500, 500, 500)
pacoder_droponn
0.0
0.1
decoder droopot
0.0
0.1
Ir
x 10
1 10-3 3
decay loss factor
x 10
1 x 10-5 1 200
batch_ epochs
500
300
1000
A.7.3 Quality of synthetic data
We performed some additional analyses to assess the quality of the synthetic data obtained in our simulation study.
Average IKLD. The inverse of the KL divergence (IKLD) between original and synthetic data, averaged over 250 Monte Carlo runs and standardised between ( and 1, is presented in Table A3 where the debiased synthetic datasets have slightly higher IKLDD than their default versions for all DGMs considered.
21

---

Table A3: The IKLD between original and synthetic data, averaged over 250 Monte Carlo runs and standardised between 0 and 1. Higher values indicate similar datasets in terms of underlying distribution.
Generator
Debiased
Y1
50
160
Y
500
"
1600
n
5000
CTGAN (SDV)
no 0.849
0.924
0.939
0.963
0.980
CTGAN (SDV)
yes 0.860
0.935
0.957
0,966
0.981
CTGAN (Syntheity)
no 0.838
0.894
0.867
0,890
0,929
CTGAN (Synthcity)
yes 0.849
0.906
0.885
0,897
0.933
TVAE (SDV)
no 0.704
0.819
0.908
0,985
0,980
TVAE (SDV)
yes 0.734
0.832
0.913
0,987
0,983
TVAE (Synthcity)
no 0.791
0.830
0.887
0.929
0.977
TVAE (SDV)
yes
0.813
0.860
0.913
0.943
0.980
Failed generators. CTGAN (Syntheity) could not be trained 14 runs (runs 38, 69, 102, 225 for n = 500; runs 19, 23, 98, 107, 117, 128, 129, 140, 148, 169 for n 5000) due to an internal error in the package. As such, it was not possible to generate default and debiased synthetic data with CTGAN (Synthcity) in these runs. This comprises 1.12% (14/1250) of all CTGAN (Synthcity) trained and 0.28% (14/5000) of all generators trained. The other generative models did not produce errors during training, so that default synthetic data could be generated in every run.
Failed debiasing. 15, 23, 24, 28, 30, 42, 46, 50, 51, 58, 64, 72, 78, 92, 96, 104, 106, 112, 117, 118, 119, 122, 126, 127, 130, 133, 135, 140, 143, 146, 149, 154, 157, 159, 160, 166, 167, 168, 170, 178, 180, 187, 191, 193, 196, 197, 200, 205, 210, 214, 216. 218, 220. 225, 230, 231, 235, 236, 245, 247,247,2 248 247,248 for n 50; runs 13, 22, 97, 117, 139, 145, 146, 148, 161, 186, 193, 197, 235, 246 for n = 160) due to sparse data, especially for small sample sizes. In particular, some X; in the original dataset may not be present the default synthetic dataset generated by TVAE (SDV), leading to an error when estimating the nuisance parameters needed for the debiasing step. This comprises 6.56% (82 (82/1250) of all TVAE (SDV) trained and 1.64% (82/5000) /5000) of all generators trained. The other generative models did not produce errоrs during debiasing, that debiased synthetic data could be generated in every run.
TVAE (SDV) could not be debiased 82 runs (runs 1,3, 4,,4,55,0 6,
Exact memorisation. A sanity check was conducted to ensure that records of the original data were memorised by the generative model. CTGAN (Synthcity) made the following number of exact copies of the original data in the synthetic data: record (2.00%) for = 50 in three runs (runs 63, 158, 192). The other generative models did not make exact copics.
Non-estimable estimators. Due to sparse data, especially for small sample sizes, the linear regres- sion coefficient of therapy on blood pressure adjusted for stage could not be estimated in small subset of the 11140 (original and synthetic) datasets, producing extremely small (< 1e-10 or large (> 1e2) standard errors. Overall, 0.09% (10/11140) (011) regression coefficient estimates could not be obtained: in rums for default synthetic dataset of size 50 generated by CTGAN (SDV) and in 1 run for default synthetic dataset of size m = 160 generated by CTGAN (Synthcity). and for their corresponding debiased synthetic datasets. The sample mean of age was always estimable.
A.7.4 Additional results
While the main text only reports results using the Syntheity default hyperparameters, we addition- ally wanted to report wider variety of trained models, without manually tuning anything, which is why results for SDV are presented here as well.
Coverage for all estimators and models. The results shown in Figure A2 indicate that our debiasing strategy delivers empirical coverage levels for the population mean that approximate the nominal level for all sample sizes and DGMs considered. By contrast, the coverage in the default synthetic datasets decreases with increasing n due to slower shrinkage of the standard error (SE) than the typical I over root-n rate, calculated in Section 4.2.3 and previously addressed in Decruyenaere
22

---

et al. 2024). While our debiasing strategy clearly improves the empirical coverage for the population regression coefficient, it does, however, not guarantee this to be the nominal level for all sample sizes and DGMs considered. In particular, debiasing only seems to achieve coverage at the nominal level for TVAE (Synthcity) across all sample sizes and for TVAE (SDV) at sufficiently large sample sizes. Our approach still falls short for CTGAN (Synthcity) and CTGAN (SDV), although providing more honest inference than the default synthetic datasets.
Mean age (MLE-based)
Effect therapy (MLE-based)
100%
Generator
Coverage
75%
Original data
(SDV)
CTGAN (SDV)
TVAE (Synthcity)
50%
CTGAN (Synthcity)
Debiased
25%
Yes
0%
50
500
5000 50
500
5000
(log scale)
Figure A2: Empirical coverage of the 95% confidence interval for the MLE-based estimators.
Bias and SE for mean age. As shown in Figure A3 the sample mean of age is unbiased in the default synthetic datasets, but exhibits large variability that shrinks slowly with sample size due the data-adaptive nature of DGMs. Debiasing reduces this variability and accelerates shrinkage. Figure A4jindicates that the empirical SE for the sample mean of age indeed, on average, underestimated by the MLE-based SE in default synthetic datasets. After debiasing, the average MLE-based SE approximates the empirical SE, albeit with minor deviations at smaller sample sizes.
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Synthcity)
Debiased:
Mean age (MLEE (LLE-based)
50-
40-
30
60
Deblased:
50-
y Fe5
40-
30
50
500
5000
50
500
5000 50
500
5000
50
500
5000
(log scale)
Figure A3: The horizontal dashed line represents the population mean of age and each dot is MLE-based estimate per Monte Carlo run (250 dots in total per value of n). The dashed funnel indicates the behaviour of an unbiased and /n-consistent estimator based on observed data.
Bias and SE for effect therapy on blood pressure adjusted for stage. It can be seen from Figure thtt the linear regression coefficient therapy on blood pressure adjusted for stage in the default synthetic datasets has finite-sample bias towards the null that converges to zero as the sample size grows larger. Additionally, its variability seems to diminish slower than expected. Debiasing reduces the finite-sample bias and improves the shrinkage of the SE. However, Figure A6 shows that the average MLEEbased SE in the debiased synthetic datasets still underestimates the empirical SE with CTGAN (SDV) and CTGAN (Synthcity) despite debiasing, albeit less pronounced.
23

---

CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Synthcity)
SE of mean MLE-based)
8
Debiased: no
4-
Standard error Empirical MLE-based
Debiased:
yes
50
500
5000
50
500
5000
50
500
5000
50
500
5000
(log scale)
Figure A4: The empirical and MI.E-based standard error for the sample mean of age.
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Synthcity)
Effect therapy (ILE-based)
100-
Debiased:
50
0-
Debiased:
0-
500
50
500
5000
50
500
5000 50
500
5000
50
500
5000
(log scale)
The horizontal dashed line represents the population effect of therapy on blood pressure adjusted for stage and each dot a MLE-based estimate per Monte Carlo run (250 dots total per value of n)). The dashed funnel indicates the behaviour of an unbiased and /n-consistent estimator based on observed data.
Figure A5:
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Synthcity)
SE of effect therapy (ML E--baaed)
20
Debiased: no
Standard error Emppriial
20
Debiased:
MLE-based
yes
10
50
500
5000
50
500
5000
50
500
5000
50
500
5000
(log scale)
Figure A6: The empirical and MLE-based standard error for the (MLE-based) effect of therapy on blood pressure adjusted for stage.

---

Convergence of SE for all estimators and models. The convergence rate of the empirical SE for the MLE-based estimators is shown in Table A4 and represented by the slope in Figure A7 The dashed line indicates the behaviour of the SE of an unbiased and /n-connistent estimator based on observed data, whereas the dotted line indicates the assumed behaviour of the SE of the same estimator based on synthetic data, following the correction proposed by |Rab et al. (2016). Lines for the empirical SE that are parallel to these lines indicate that the estimator converges at over root-n rate, whereas more horizontal or vertical lines imply slower or faster shrinkage, respectively. After debiasing, all lines are parallel for both estimators, suggesting /n=ccnnisttnccy Note that the vertical offset of some lines, which represents the log asymptotic variance, is still too large for some DGMs despite debiasing. This is the case for the linear regression coefficient for therapy on blood pressure adjusted for stage obtained in the debiased synthetic dataset generated by CTGAN (SDV) and CTGAN (SDV).
Table A4: Estimated exponent a [95% CI] for the power law convergence rate n - for empirical SE.
Estimator
CTGAN (SDV)
CTGAN (Syntheity)
TVAE (SDV)
TVAE (Synthcity)
Default synthetic datasets
Mean age Effect therapy
0.33 [0.04: 0.62]
0.01 [-0.16; 0.18)
0.10 [-0.13: 0.32]
0.21 [0.04: 0.39] 0.09 [-0.13; 0.31]
0.23 [-0.09; 0.56]
0.31 [0.22: 0.40]
0.28 (0. 10: 0.46]
Debiased synthetic datasets
Mean age Effect therapy
0.42 [0.36: 0.48]
0.50 [0.46; 0.54]
0.45 [0.44; 0.47]
0.47 [0.44: 0.50]
0.58 [0.43; 0.73]
0.43 [0.31: 0.55]
0.61 [0.43: 0.79]
0.46 [0.38; 0.55]
Default
Debiased
2
Mean age:
Empirical SE
-Original data
-TGAN (SDV)
log(SE)
CTGAN (Synthcity)
TVAE (SDV)
Effect therapy:
TVAE (Synthcity)
ML E- babed
Theoretical SE
Original data
Synthetic data
50
500
5000
50
500
5000
(log-scale)
Figure A7: Convergence rate of the empirical standard error (SE) for the MI.E-based estimators. If the SE is of the form SE cn -a where is a constant, then log ESE) log(c) + (-a) log (n). Therefore slope a represents the convergence rate and the vertical offset log(c) indicates the log asymptotic variance. The dashed line indicates the behaviour of the SE of an unbiased and yn- consistent estimator based on observed data, whereas the dotted line indicates the assumed behaviour of the SE of the same estimator based on synthetic data, following the correction proposed by Raab et al. (2016).
Nuisance parameters. Our debiasing strategy assumes that the two remainder terms the von Mises expansion presented in Section 3 are op (mm2) and (n This requires the difference between P... (i.e., estimate of P,, based on the sampled synthetic dataset of m records) and P,, (i.e., estimate of P by the DGM trained on n original records) and the difference between P,, and P (i.e., the data generating process), to converge to zero in L2 (P.)) and L2(P), respectively, faster than n-to-the-quarter convergence rates. While this holds true for the convergence of P., to P,, (see Table A5). this seems not always strictly attained for the functionals of P,, with respect to P that appear in the efficient influence curves for the regression coefficient (see Table A6). in particular for CTGAN
25

---

(Synthcity) and, to lesser extent, for CTGAN (SDV) and TVAE (Syntheity). Nevertheless, root-n consistent estimator was still obtained after debiasing these DGMs (see Table A4p.
Table A5: Estimated exponent a [95% CIJ for the power law convergence rate n for the difference between functionals of P... and P, in L2 (Pa).
Functional
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Syntheity)
X
0.47 [0.41; 0.52]
0.50 [0.44; 0.56]
0,45 [0.39; 0.50]
0.54 [0.44; 0.63)
A 17
0.46 [0.40; 0.52]
0.46 [0.37; 0.55]
0.58 [0.39; 0.78]
0.50 [0.45; 0.54]
E X III
0.46 [0.35; 0.57]
0.49 [0.37; 0.61]
0.49 [0.21; 0.77]
0.55 [0.52; 0.58)
E X = IV]
0.43 [0.37; 0.50]
0.48 [0.42; 0.54]
0.42 [0.32; 0.53]
0.49 [0.44; 0.55]
Y X = Z]
0.50 [0.34; 0.66]
0.53 [0,48; 0,58]
0,43 [0.35; 0.50]
0.47 [0.36; 0.57]
B X 17]
0.50 [0.35; 0.64]
0.48 [0.36; 0.60]
0.60 [0.30; 0.89]
0.44 (0.40; 0.48]
BY X 111
0.52 [0.41: 0.63]
0.51 [0.35; 0.67]
0.59 [0.24; 0.95]
0.50 [0.45; 0.56]
IV]
0.51 [0.42; 0.61]
0.50 [0.44; 0.55]
0.56 [0.49; 0.63]
0.42 [0.35; 0.50)
Table A6: Estimated exponent a [95% CI] for the power law convergence rate n for the difference
between functionals of P.. and P in L2 (P).
Functional
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Syntheity)
E X /
0.24 [0.12: 0.36]
0.16 [0.02: 0.30]
0.46 [0.08; 0.84]
0.30 [0.25; 0.35]
E 17)
0.19 [-0.01; 0.38]
0.22 [0.08: 0.36]
0.64 [0.25; 1.04]
0.32 (0.24; 0.40]
EAX III]
0.18 [-0.03; 0.40]
0.22 [0.08; 0.36]
0.51 [0.09; 0.941
0.27 [0.15; 0.39]
EAX [V]
0.19 (0.01; 0.38]
0.23 [0.08; 0,38]
0.44 [0.07; 0.82]
0.21 [0.02; 0.40)
EY = 1]
0.37 [0.08; 0.66]
0.21 [0.06; 0.35]
0.31 10:16; 0.46]
0.27 [0.15; 0.39]
Y X 17]
0.30 [0.08; 0.53]
0.14 [0.01; 0.27]
0.49 [0.03; 0.95]
0.24 [0.10; 0.38)
E X- II7
0.27 [0.04; 0.51]
0.17 [0.03; 0.32]
0.31 [-0.01; 0.63]
0.18 [0.07; 0.30]
EY X =I]
0.26 (0.02: 0.49]
0.21 10.10: 0.33]
0.17 [0.02: 0.33]
0.19 [0.04; 0.34]
((P.)
0.33 [0.04; 0.61]
0.35 [0.28; 0.43]
0.37 [0.12; 0.62]
0.29 [0.07; 0.51]
Summary, Table A7 summarises the effect our debiasing strategy on bias, SE and coverage in the simulation study. Our strategy results in uniformly valid coverage for the population mean, allowing for honest inference. For regression coefficient, coverage was clearly improved but may remain anti-conservative for some DGMs. This may originate from residual overfitting bias inherent to these DGMs that could not be removed since (efficient) sample splitting was not performed (sce Appendix A.3)
Table A7: Behaviour of estimators in debiased synthetic data in the simulation study.
Mean age
Effect therapy
Model
Bias
SE
Coverage
Bias
SE
Coverage
root-n & unbiased root-m & unbiased
nominal
root-n & small underestimated root-m & underestimated
hias at
anti-conser-
CTGAN (SDV)
unbiased
all
vative at all w anti-conser- vative at all n nominal only at large nominal at all n
nominal
CTGAN (Syntheity)
unbiased
unbiased bias root-n & small unbiased root-n unbiased
all
& unbiased root-n unbiased
nominal
TVAE (SDV)
unbiased
nominal at all n
TVAE (Synthcity)
unbiased
unbiased
26

---

A.7.5 Influence curve based estimation
Here, two estimators are estimated in the original and synthetic datasets: maximum likelihood estimation (MLE))based one, as used in traditional statistical analysis, and an efficient influence curve (EIC)-based one, proposed in this paper and obtained after 5-fold pross-fitting during estimation of the nuisance parameters. Note that sample splitting was not used during the debiasing step. For the former, SEs are calculated via the regular expressions which discard the uncertainty associated with data-adaptive prediction during estimation, while for the latter, SEs are based on the EIC which acknowledges this uncertainty. Throughout, all estimated (model- or EIC-based) SEs corrected with /1 m/n to acknowledge the sampling variability of synthetic data. This correction factor was initially proposed by Raab et al. (2016) for parametric synthetic data generators, but was found to be insufficient for synthetic data created by DGMs (Decruyenaere al. 2024). In Section [.3 give the formula for the variance of our ElC-based estimator on the debiased synthetic data, which generalises this correction factor the setting where synthetic data were generated by DGMs.
The results of our simulation study using the EIC-based estimators, as shown below in Figures A8 A Table A8 remain unchanged as compared to using the MI.E-based estimators, since no data-adaptive predictions (c.g., machine learning) were used during estimation of the nuisance parameters. If data-adaptive estimation were to be used, expect the MLE-based estimators to be overly optimistic, while the ElC-based estimators could handle the additional uncertainty introduced by data-adaptive adaptive estimation.
MLE-based
EIC-based
100%
Mean age
50%
Generator
25%
Original data
TVAE (SDV)
Coverage
CTGAN (SDV)
TVAE (Synthcity)
0%
CTGAN (Synthcity)
100%
Debiased
Effect therapy
No
Yes
50%
25%
0%
50
500
5000 50
500
5000
(log scale)
Figure A8: Empirical coverage of the 95% confidence interval for the maximum likelihood estimation (MLF) bbassed and efficient influence curve (EIC)-based estimators.
27

---

Orignnal
CESAN (SpVE
CISAN (SPM)
CIGAN tsyttheyt:
CEGAN (Synticryl
TVAE Geblased
TWAE (Symtherys
TIME (Symincity
Eata
ISPVI
MLE.E. Baled
60
Ex pmp
E
40
Estinate
1.00
1n 4mb
-50
50
1100
dlog scale)
Figure A9: The horizontal dashed line represents the population parameter and each dot is a maximum likelihood estimation (MLE)- based or efficient influence curve (EIC) estimate per Monte Carlo run (250 dots in total per value of n). The dashed funnel indicates the behaviour an unbiased and /n-consistent estimator based observed data.
Oriiial defbult
EIGAN debuit
CTGAN ocb-ased
CTGAN dcsuut
CTGAN ecpased
TWAE defoult
TVAE deblosed
TVAE detbult
TWWE Sebiaseo
St of estimaate LUC-bbedd!
Moun
Standard error
Emarical
MLE based
50 5005200
500
5000
500
20 600 5000
5000
50 5003050 5050
50
5000
50
5050
5050
(log scale)
Figure A10: The empirical standard error of the efficient influence curve (EIC)-based estimators shown. Standard errors are estimated via the maximum likelihood estimation (ML.E)-based or EIC-based expressions.
Table A8: Estimated exponent a [95% CI] for the power law convergence rate n n for empirical SE the maximum likelihood estimation (MLE)-based and efficient influence curve (EIC)-based estimators.
Estimator
CTGAN (SDV)
CTGAN (Syntheity)
TVAE (SDV)
TVAE (Syntheity)
Default synthetic datasets
Mean age (MLE)
0.33 [0.04; 0.62]
0.01 [-0.16; 0.18)
0.10 [-0.13; 0.32]
0.21 [0.04; 0.39]
Mean age (HIC)
0.33 (0.04; 0.62]
0.01 16; 0.18]
0.10 1-0,13; 0.32]
0.21 [0.04; 0.39
Effect therapy (MLE)
0.23 [-0.09; 0.56]
0.31 (0.22: 0.40]
0.28 10: 0.46]
0.09 [-0.13; 0.31]
Effect therapy (EIC)
0.33 [-0.11; 0.78]
0.31 [0.24; 0.38]
0.30 [0:13; 0.48]
0.10 [-0.13; 0.32]
Debiased synthetic datasets
Mean age (MLE)
0.42 [0.36; 0.48]
0.50 (0.46; 0.54]
0.45 [0.44; 0.47]
0.47 [0.44; 0.50]
Mean age (EIC)
0.42 [0.36; 0.48]
0.50 (0.46; 0.54]
0.45 [0.44; 0.47]
0.47 [0.44; 0.50]
Effect therapy (MIF)
0.58 [0.43; 0.73]
0.43 (0.31: 0.55]
0.61 [0.43; 0.79]
0.46 [0.38; 0.55
Effect therapy (EIC)
0.63 [0.50: 0.75]
0.43 [0.32: 0.54]
0.64 [0.48; 0.79]
0.46 [0.38; 0.55]
28

---

Default
Debiased
Mean age: MLE- MLE-based
Mean age: EIG-
Empirical SE
-Original data
-CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Synthcity)
Effect therapy:
MLE based
Theoretical SE
OOOignnia data
Synthetic data
Effect therapy. ElC-bacs ElC-based
50
500
5000
50
500
5000
(log-scale)
Figure All: Convergence rate of the empirical standard enror (SE) for the maximum likelihood estimation (MI.E)-based and efficient influence curve (EIC)-based estimators. If the SE of the form SE cn where is constant, then log (SF) log(c) (-a) log (n). Therefore slope represents the convergence rate and the vertical offset log(c) indicates the log asymptotic variance. The dashed line indicates the behaviour of the SE of unbiased and /Ti-consistent estimator based on observed data, whereas the dotted line indicates the assumed behaviour of the SE the same estimator based on synthetic data, following the correction proposed by Raab et al. 2016).
29

---

A.8
Case studies
A.8.1 International Stroke Trial
We adapt the framework discussed in Section 4.1 to the International Stroke Trial (IST), one of the biggest randomised trials in acute stroke research Sandercock et al. 2011). The dataset with 19285 complete cases now constitutes our population. We mimic different hypothetical settings where an institution only has access to a limited sample of observations, with the sample size n varying between 50 and 5000.
In order to easily share the data with other researchers, the institution generates a synthetic dataset with sample size m, where m n. Similarly the simulation study, we repeated this process 100 times per sample size to be able to calculate the empirical coverage levels. For illustration purposes, we focus on the effect of aspirin on the outcome at months and report the proportion of deaths for the two treatment arms (aspirin and no aspirin), and its corresponding risk difference.
For each value of n, two default synthetic datasets were generated using both CTGAN and TVAE. Given the interest in the proportion of death in the group with and without aspirin, we use the debiasing strategy with respect to the population mean. This implies that the default synthetic dataset was first split by treatment and then debiased with relation to the population mean within each treatment arm. The two debiased subdatasets were then afterwards combined into one debiased synthetic dataset for each generative model. For both the default and debiased synthetic dataset, the sampling variability of synthetic data is acknowledged by inflating the standard errors (SEs) by the correction factor VI+ + m/n.
The funnel plots for the proportion of deaths in both treatment arms and the risk difference are shown in Figure A12 We noticed that using the same hyperparameters as in the simulation study resulted in biased estimates, as can be seen in Figure [A12 For this reason, we highlight the results obtained by training with the default hyperparameters suggested by the package SDV (Parki et al. 2016) instead. Analogously to the simulation study, our debiasing strategy decreases the variance of the mean estimator in both treatment arms, remedying the slower-than--nn-oone /n-convergeccce observed in the default synthetic datasets. The impact for the applied researcher can be better understood by looking at the empirical coverage levels of the 95% CI for the true proportion of deaths in the aspirin arm, for all sample sizes and DGMs considered. Figure 3a illustrates that in contrast to the default synthetic datasets, the coverage levels based on the debiased synthetic datasets are all positioned around the nominal level.
One of the original research questions in Sandercock et al. (201T) was whether not there is difference in risk of death between the treatment arms. Figure Ab depicts the empirical type 1 error rate for the risk difference in death between aspirin and aspirin group based original data, default and debiased synthetic data. For the aforementioned reason, we focus on the results obtained by training with the default hyperparameters suggested by the package SDV (Patki et al.) 2016). Should the researcher use the default synthetic data, they would very often falsely conclude that the risk is significantly different from the true difference of -0.009, as calculated based on our population (the full dataset), while using the debiased synthetic dataset basically eliminates this high number of false- posittvess as is the case in the original data as well.
Table A9: Estimated exponent a [95% CI] for the power law convergence rate n - for empirical SE. Note that a convergence rate could not be estimated for the default synthetic data when hyperpa- rameters suggested by the package Synthcity. This occurred because there was no variance in the
estimates for sample sizes of 1600 and 5000.
Estimator
Original
CTGAN (SDV)
CTGAN (Synthcity)
TVAE (SDV)
TVAE (Syntheity)
Default synthetic datasets
Proportion death aspirin group
0.54 [0.50: 0.59]
0.02 [-0.06; 0.11]
NaN [NaN; NaN]
0.23 [-0.17, 0.62)
NaN [NaN; NaN]
Proportion aspirin group Risk differeace death
0.54 (0.52: 0.57]
0.02 [-D. 15; 0818
(NaN;; NaN]
0.23 [-0.14 0.61]
(NaN NaN]
0.54 [0.51; 0.56]
0.01[-0.24: 0.26]
NaN [NaN: NaN]
0.46 (0.14: 0.78]
NaN [NaN NaN]
Debiused synthelic dutusets
Proportion death aspirin group Proportion death aspirin group Risk difference death
0.54 [0.48: 0.61]
0.59 [0.52. 0.67]
0.53 [050, 0.56]
0.60 [0.51: 0.70]
0.52 [0.46: 0.59)
0.58 [0.55; 0.62]
0.53 [0.49; 0.58]
0.58 [0.49; 0.67]
0.55 J0.48: 0.62
0.57 [0.53; 0.60]
0.53 (0.50, 0.57]
0.57 [0.53; 0.61
30

---

Original data
(SDV):
(SDM)
Dsynthaily:
(Syrincity:
(SDV:
(SDV)
(Synincity byll
Esynibaity
Proportinn
0.2
Estimate
0.6
Proonioa 1000
0.4-
0.4
0.
-0.4
50
500
5000
50
500
5000
500
5000
50
5000
50
500
sobo
500 5000
50
5000
30
300
5000
550
5000
flog Scale)
Figure A12: Estimates for the proportion of death in both treatment arms and their corresponding risk difference estimates. We show results for original data, and both default synthetic data (left) and debiased synthetic data (right) for all four generators. The horizontal dashed line represents the population proportion of death in each group and the corresponding risk difference, and each dot an estimate per Monte Carlo run (100 dots in total value of n). The dashed funnel indicates the behaviour an unbiased and /n-consistent estimator based on observed data.
100%
Generator
75%
75%
Driginal dats
CTGAN (Syntheit)
Coverage
Type ctror
TWAI (Syntheity)
CTGAN (SDV)
50%
50%
TVAE
Deblased
No
0%
50
160
500
1600
5000
16000
5000
(log scale)
(log scale)
(a)
(b)
Figure A13: Figure (a) shows the empirical coverage of the 95% CI for the true proportion of death in the aspirin treatment arm. In Figure (b). one can find the empirical type 1 error rate for the risk difference in death between aspirin and no aspirin group based on original data, default and targeted synthetic data. The null hypothesis states that the risk difference is equal to -0.009, the risk difference as observed in the population (i.e. the original IST data). Tests were conducted significance level, where the black horizontal line on the figure depicts this nominal level.
5%
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state why our research is relevant (bias in synthetic data generated by deep generative models, leading to imprecision in statistical analysis and wrong conclusions), and propose a new strategy to debias this synthetic data. While this methodology is general, we clearly state that apply it two Deep Generative Models (CTGAN and TVAE), in simulation study and two case studies to convince the reader of its merits.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We list limitations that are related several facets of this paper. We refer to the assumption that is needed for the formal derivation of our strategy but frame this in the context of well-known analyses. We also discuss the diwddimesiional setting of our simulation and case studies, for which DGMs might be less suited. Nevertheless, the positive results for two widely used estimators in this simple setting highlights the utility of debiased approach and is simultaneously encouraging in terms of future larger-scale applications. In contrast to other work, we are aware that we do not suggest a tuning strategy of the DGM but instead rely the debiasing of the generated synthetic data in a post-processing step. Nonetheless, we perceive this as a strength, since renders our strategy generator-agnostic. However, the debiasing method of the regression coefficient still requires sampling of synthetic data conditional on a covariate, which not a given in all types of DGM. Finally, when multiple parameters are of interest, the data generated by the DGM will need to debiased to ensure that several such restrictions hold (simultaneously) w.r.t. several EICs ((.), which is left for future research. In the Discussion section, offer various suggestions and ideas for future research aimed at addressing the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be
32

---

used reliably to provide closed captions for online lectures because it fails to handle technical Jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In Section 2 we clearly present our set up and assumptions. Section 3 contains detailed theoretical proof, including reasoning steps along the way, of our debiasing strategy, complemented by additional notes the supplementary material (referred to 'Appendix' in the manuscript).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Besides fully describing our debiasing methodology, we also provide ample details on the simulation study and case studies. Moreover, we have made our code for the simulation and case studies available online through our GitHub page, with the link included in the main text.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
33

---

instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The simulation study is built on a fictitious yet realistic data generating process, which provided full in Appendix [A.7.]] As DGMs, we use the default implementations provided in the code libraries Synthcity and SDV, which are open- source. 'he case studies are based on two public datasets: the International Stroke Trial dataset (Sandercock et al.. 2011) and the Adult Census Income dataset (Becker and Kohavi 1996). While we aimed to provide all details necessary to reproduce our experiments in the paper itself, we additionally make all code available in public Github repository (the link is included in the main text). These code enables reproduction all results for the simulation study and case studies.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc/puut http:/cccccggguueeee//ee/////e// http:///uddess/Cooo Caaeeee/oooSSSmmiieeeee
www.issssoppcccccooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
34

---

Answer: [Yes]
Justification: Since the focus is not on building novel generative models, but rather on proposing general methodology for debiasing the data generated by generic generator, we simply use default hyperparameters for the DGMs as suggested by two code libraries (SynthCity and SDV). In our simulation study, training data is obtained by sampling from the data generating process. We do not use any test splits, since the evaluation is done in terms of utility of statistical analyses, where compare the obtained estimands with their ground truth values. Similar considerations are valid for the case studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: While we do include experimental results, do not compare multiple generative models or methods. Rather, we present a novel methodology for debiasing synthetic data, and use various experiments verify that it works expected, by comparing the statistical utility of synthetic data with and without our debiasing strategy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were run on our institutional high performance computing cluster using a single GPU (NVIDIA Ampere A100; S0GB GPU memory) and single CPU (AMD EPYC 7413), taking less than 24 hours to complete (simulation study: less than 15 minutes per individual run across 5 sample sizes; International Stroke Trial case study: less than 75 minutes per individual run across 5 sample sizes; Adult census Income Dataset case study: less than 4 hours). This is stated such in the appendix.
35

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httph::////// http://:///uurps.ccooo
Answer: [Yes]
Justification: This research did not involve human subjects or participants. Furthermore, there are no data-related concerns: the simple data generating process that is proposed the simulation study is fictitious, and apart from that use public datasets. There are potential harmful consequences related to synthetic data, such as privacy disclosure risk and incorrect statistical inference from tabular synthetic datasets. In fact, our work addresses the second risk by debiasing synthetic datasets towards improving their inferential utility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive societal impacts stem from the increased reliability of statistical analyses on synthetic data, which is made possible by our debiasing method. We do foresee major negative societal impacts of our method. General concerns regarding tabular synthetic data do apply, though we believe our method remedies part of them (see our answer to Question 9).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
36

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not believe paper poses such risks, since we propose a generic debiasing strategy for synthetic data, without releasing any new models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the paper, credit the creators of the deep generative models used in our work (i.e. CTGAN and TVAE), as well as the open source code libraries whose used (Synthcity and SDV). Furthermore, we cite the sources of the public datasets we used. The International Stroke Trial exists under an Open Database License, and the Adult Census Income dataset exists under Creative Commons Attribution 4.0 International License. Both are free to use, but require attribution, which we provided.
implementations
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
37

---

Answer:
[NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "afzAM9YTJe", "reviewer": "Reviewer_eYfd", "summary": "The paper considers the problem of debiasing synthetic data which can have signficant issues when used for statistical analysis. In particular, they show examples of mean estimation of a variable and parameter estimate of a regression model and demonstrate the benefits of the post-processing step of their approach (model agnostic).\n\nOverall: The paper is clearly written and the problem is well-defined. The approach is for debiasing is simple but effective and experiments validate the approach. Unclear, how it can work for more real-world settings.", "questions": "(1) The references are a bit sparse on the synthetic data literature and would it be possible to enhance it with appropriate hooks? There are quite a few surveys available now.\n(2) While the results are interesting, it is unclear how much is innovative in this work as opposed to building on the results from Chernozhukov, Decruyenaere and other cited works. Can you elucidate that clearly? \n(3) Also, what would be needed to handle more interesting estimators for downstream applications?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(i) The problem of bias in the synthetic data is well explained by considering the two examples of population mean and the coefficient estimator in linear regression. The debiased estimators are derived for these two settings. \n\n(ii) Experiments are first shown the synthetic datasets and the debiased estimators have very good coverage as opposed to without debiasing. The convergence rate with O(1/sqrt(n)) is al\nso shown when debiased. Two real-world datasets are also considered and the issue of obtaining false conclusions from the synthetic data vs the debiased setting is illustrated.", "weaknesses": "(a) As mentioned in the discussion settings, the datasets are low-dimensional and the estimators are very simple such as mean and parameter estimate for linear regression. \n\n(b) It is unclear how one can generalize these to settings where the estimators are much more complex/ubiquitous such as the covariance matrix or group fairness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer eYfd for their careful reading of the manuscript and constructive review.\n\n**What would be needed to handle more interesting estimators for downstream applications? It is unclear how one can generalize these to settings where the estimators are much more complex/ubiquitous such as the covariance matrix or group fairness.**\n\nGiven the importance of this concern, we expand on this topic in the global response to all reviewers. We share the concern that theory should be aligned with practical real-world settings and we advocate that future research should focus on extending our results to broader settings. However, Decruyenaere et al. (2024) have shown that inferential utility is compromised when statistical analysis is performed on synthetic data created by deep generative models (DGMs), even in low-dimensional settings. \nWe see addressing these low-dimensional settings as a necessary first step, before tackling more complex settings where valid inference is already challenging in observed data, let alone synthetic data.\n\n**The references are a bit sparse on the synthetic data literature and would it be possible to enhance it with appropriate hooks? There are quite a few surveys available now.**\n\nIndeed, it would be useful to contrast our work more extensively with additional recent research on synthetic data in the context of inferential utility. \nOther reviewers raised the same question and therefore we elaborate on this in the global rebuttal. There, we provide a comparison of our work with related literature, which we will add to the revised manuscript. \nWe are convinced that this addition will improve our manuscript and will better capture how our work stands out w.r.t. other contributions.\n\n**While the results are interesting, it is unclear how much is innovative in this work as opposed to building on the results from Chernozhukov, Decruyenaere and other cited works. Can you elucidate that clearly?**\n\nFor a global overview of related work and our specific contribution, we would like to refer to the global rebuttal. In addition, we will address the specific question related to the work of Chernozhukov et al. (2018) and Decruyenaere et al. (2024). \n\nOur debiasing strategy originates from the observation in Decruyenaere et al. (2024) that inferential utility is compromised when naive statistical analyses are performed on synthetic data created by deep generative models (DGMs), even when applying a previously proposed correction factor to the standard error.\nThey claim that this can be attributed in great extent to the additional layer of uncertainty resulting from the synthetic data generation process and the deviating convergence rate of parameters when estimated in synthetic data, leading to unreliable confidence intervals and overly confident conclusions made from these synthetic data. In our work, we first aim to establish an understanding of this deviation when the parameter of interest is estimated based on a synthetic versus observed sample (i.e. $\\theta(\\widetilde{P}_m)-\\theta(P)$). By doing so, we are able to identify two bias terms which can be targeted in order to decrease this deviation between synthetic and original data.\n\nFor the first bias term (referred to as bias term (5) in the manuscript), we rely on theory proposed by van der Laan and Rose (2011) and Chernozhukov et al. (2018), allowing us to analyze the data with debiased estimators.\nOur contribution lies in the way we handle the second bias term (i.e. bias term (6) in the manuscript). As stated in the manuscript, finding a solution for this bias term relies on the efficient influence curve and therefore on the target parameter of interest. In the first two subsections of the Methodology section, we specifically work out a strategy for the population mean and the linear regression coefficient, showing how our debiasing strategy combines a solution for both bias term (5) and (6).\nWe finalize the Methodology section with stating the properties of the resulting estimator (i.e. the debiased estimator based on the synthetic data). \n\nTo summarize, we show how the problems concerning inferential utility can be removed, by targeting synthetic data generated by DGMs to perform optimally for the specific data analysis that is envisaged. This builds on ideas from debiased or targeted machine learning (van der Laan and Rose (2011), Chernozhukov et al. (2018)), but is nonetheless a non-trivial extension as (a) those works do not consider synthetic data; and (b) it required us to work out how the estimation errors in the DGM propagate into the estimator calculated on synthetic data, and based on this, how those data must be adapted to dampen propagation of those errors. \nAs far as we are aware, our approach is the only one that provides formal guarantees for (more) reliable inference from synthetic data created by deep generative models.\n\n\nChernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. _The Econometrics Journal_, 21(1):C1–C68. \n\nDecruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In _The 40th Conference on Uncertainty in Artificial Intelligence._ \n\nvan der Laan, M. J. and Rose, S. (2011). _Targeted Learning._ Springer Series in Statistics. Springer New York, New York, NY.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgIqjhrz4W", "reviewer": "Reviewer_LYaP", "summary": "This paper tackle the problem of creating/imputing unbiased synthetic data, and analyse the potential bias brought by these imputation methods.", "questions": "see above", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The tackled problem is important and very timely, and linked to the general question of biais of generative models [1].\n\n\n[1] Wyllie, Sierra, Ilia Shumailov, and Nicolas Papernot. \"Fairness feedback loops: training on synthetic data amplifies bias.\" The 2024 ACM Conference on Fairness, Accountability, and Transparency. 2024.", "weaknesses": "I do not know if this comes from my lack of knowledge in the field, but I found the paper very hard to understand, no clear theoretical result is encapsulated, and it is not even very clear to me what is proposed (I guess this is what is after line 91).\nI thus have a lot of questions:\n- line 90, how exactly $S_i$ is sampled? Are you sampling first $A_i$ and $X_i$, and then $Y_i|A_i, X_i$?\n- What is your main theoretical result? Is it Equation 2? or Section 3.3?\n- What is $\\phi$ (I am not sure it is defined in the main text)\n- How is the deep generative model learned in the first place?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer LYaP for their careful reading of the manuscript and constructive review.\n\n**I do not know if this comes from my lack of knowledge in the field, but I found the paper very hard to understand.**\n\nWe are aware that the proposed strategy is not trivial and combines different concepts in order to eliminate the deviating behaviour of estimators seen in synthetic data (Decruyenaere et al., 2024). However, we hope that the proposed guidance below for Section 3 of a \nrevised manuscript improves readability.\nWe will also rewrite the introduction to stress the difference with other related work and emphasize our contribution in the field. Therefore, we have included an extended comparison with related work in the global rebuttal. We summarize our contributions as follows.\nInferential utility of synthetic data is compromised when data are generated with deep generative models (DGMs), even when using earlier proposed correction factors (Decruyenaere et al., 2024). In other related work, procedures were developed relying on multiple synthetic datasets or focussing on only statistical generators (and hence implicitly rely on root-_n_ consistent estimators). In this manuscript, we propose a debiasing strategy that allows to create synthetic data via DGMs while maintaining valid inference for a population parameter.\n\n**Line 90, how exactly $S_i$ is sampled? Are you sampling first $A_i$ and $X_i$, and then $Y_i | A_i, X_i$?**\n\nIn this work, we focus on DGMs but within this setup, the proposed strategy is generator-agnostic. Therefore, the specific sampling procedure depends on the generator that is used in specific cases. In our simulation and case studies, we apply TVAE and CTGAN (Xu et al., 2019) to generate synthetic samples. In these algorithms, the trained decoder (i.e. TVAE) or generator (i.e. CTGAN) are used to generate synthetic samples where all features are sampled jointly as $Y_i, A_i, X_i$ (and thus not conditional as $Y_i | A_i, X_i$).\n\nIn our proposed method, we perform a post-processing of the obtained synthetic samples to eliminate the bias that would otherwise occur when the parameter of interest is estimated based on synthetic samples. The exact procedure for this post-processing is explained in Section 3.1 and 3.2. In Section 3.2 it can be seen that we need to estimate e.g. $E_{\\widehat{P}_n}(Y|X_i)$ and this requires the generation of synthetic values of $Y$ conditional on a given observed level $X_i$. This can be circumvented by generating, based on the DGM, a very large synthetic sample, taking the subset where $X$ equals $X_i$, and estimating the mean within this subset. If $X$ were continuous, then the DGM would necessitate a built-in conditional sampling module, which is not a given in any type of DGM, and lies beyond the scope of our simulation study.\n\n**What is your main theoretical result? Is it Equation 2?**\n\nOur main theoretical result can be found in Sections 3.1 and 3.2, where we propose a debiasing strategy for two estimators: the population mean and the linear regression coefficient. In short, we propose to shift the DGM-based synthetic data.\nThe remainder of Section 3 provides the theoretical background for why this debiasing approach works. We understand the confusion given the detailed yet dense explanation within the page limit. In order to prevent future confusion, we will slightly rewrite this part to ensure that the setup of Section 3 is clear. In the remainder, we provide an outline of how we would guide the reader through Section 3.\n\nWe first aim at establishing an understanding of the deviation when the parameter of interest is estimated on synthetic versus observed data (i.e. $\\theta(\\widetilde{P}_m)-\\theta(P)$). By doing so, the equation provides us with eight elements that we then further explore. We start with the two remainder terms and the two empirical process terms and elaborate on why we foresee that these terms are negligible under certain conditions. In the next paragraph, we state that we foresee no problems with terms (3) and (4), but that terms (5) and (6) however will cause bias if not taken into account.\nWe then proceed by suggesting a new approach to tackle those two latter terms. For term (5) we rely on theory proposed by van der Laan and Rose (2011) and Chernozhukov et al. (2018) by analyzing the data with debiased estimators.\n\nOur contribution lies in the way we handle term (6). Finding a solution for this bias term relies on the efficient influence curve and therefore on the parameter of interest. In the next two subsections, we specifically worked out the proposed strategy for the population mean and a linear regression coefficient and we pinpoint how bias term (5) and (6) can be resolved. We finalize the Methodology section with stating the properties of the resulting estimator (i.e. the debiased estimator based on synthetic data).\n\n**What is $\\phi$?**\n\n$\\phi(\\cdot)$ is defined as the efficient influence curve on line 75, but it could enhance readability to repeat this on line 98-99.\n\n**How is the deep generative model learned in the first place?**\n\nWe are not sure that we fully understand this question as we rely on commonly used generators such as TVAE and CTGAN (Xu et al., 2019) to generate synthetic data. They are trained on original data and our proposed strategy does not alter this standard training phase.\n\nChernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. _The Econometrics Journal_, 21(1):C1–C68.\n\nDecruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In _The 40th Conference on UAI._\n\nvan der Laan, M. J. and Rose, S. (2011). _Targeted Learning._ Springer Series in Statistics. Springer New York, New York, NY. \n\nXu, L., Skoularidou, M. et al. (2019). Modeling tabular data using conditional GAN. _Advances in neural information processing systems_, 32.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Y1GtEDbI8", "reviewer": "Reviewer_4cJP", "summary": "This paper addresses the significant biases introduced in synthetic data generated by deep generative models (DGMs) that compromise the inferential utility of such data. The authors propose a new debiasing strategy based on techniques adapted from debiased or targeted machine learning. Their approach aims to reduce biases and enhance convergence rates, thereby improving the reliability of statistical analyses performed on synthetic data. The effectiveness of the proposed strategy is demonstrated through a simulation study and two case studies using real-world data.", "questions": "- **High-Dimensional Settings:** How well does the debiasing strategy perform in high-dimensional settings where DGMs are more commonly used? Are there any challenges or limitations in applying the method to high-dimensional data?\n- **Complex Estimators:** Can the debiasing strategy be extended to more complex estimators beyond the population mean and linear regression coefficient?\n- **Alternative Debiasing Approaches:** Are there alternative debiasing approaches that could be explored to address the limitations of the proposed method, such as its dependence on EICs and the conditional sampling requirement?\n- **Privacy Considerations:** While the paper mentions the potential increase in privacy disclosure risk with larger synthetic datasets, it would be beneficial to discuss this trade-off in more detail. How can the debiasing strategy be balanced with privacy concerns, especially in the context of differentially private synthetic data generation?\n- **Failure of Assumptions**: Are there specific scenarios where the assumptions required for the debiasing method might not hold, and how would this impact the results?", "limitations": "The authors themselves are highly aware and forthcoming about the limitations of this study, which in my eyes is a big bonus.\n- The authors discuss the low-dimensional setting of their simulations and acknowledge that DGMs might be less suited for such settings.\n- They note the need for the person generating synthetic data to be aware of the analyses that will be run, which may limit the method's applicability.\n- The authors note the need to condition on data covariates to generate appropriate synthetic data.\n- The potential trade-off between synthetic data sample size and privacy risks is mentioned, though it is beyond the scope of this paper.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** The paper tackles the important problem of bias in synthetic data generated by DGMs, which has significant implications for privacy protection and data analysis. The authors propose a novel debiasing strategy that adapts techniques from targeted machine learning to the context of synthetic data generation. This represents a new direction in addressing the challenges of inferential utility in synthetic data. The approach is generator-agnostic, making it widely applicable to various types of DGMs.\n- **Quality:** The submission is technically sound, with clear explanations of the theoretical foundations and derivations of the debiasing strategy. The authors provide a simulation study and two case studies that demonstrate the effectiveness of their approach in improving the coverage of confidence intervals and enabling more reliable statistical inference from synthetic data. The paper is well-written and organized, making it easy to follow the authors' reasoning and understand their contributions.\n- **Significance:** The results presented in the paper are relevant for the field of synthetic data generation and privacy protection. The proposed debiasing strategy has the potential to improve the reliability and applicability of synthetic data in statistical inference, which can have broad implications for various domains where privacy is a concern.", "weaknesses": "- **Limited Scope:** The simulation study and case studies focus on low-dimensional settings and a limited number of estimators (population mean and linear regression coefficient). While the positive results are encouraging, further evaluation is needed to assess the effectiveness of the debiasing strategy in high-dimensional settings and for a wider range of statistical analyses.\n- **Dependence on EICs:** The proposed debiasing strategy requires access to the Efficient Influence Curves (EICs) of the target parameters of interest. This limits the applicability of the method to parameters that are pathwise differentiable and have known EICs.\n- **Conditional Sampling Limitation:** The debiasing procedure for the linear regression coefficient requires sampling synthetic data conditional on covariates, which may not be feasible for all types of DGMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer 4cJP for their careful reading of the manuscript and constructive review.\n\n**How well does the debiasing strategy perform in high-dimensional settings <...>?**\n\nWe elaborate on this in our global response as well, but agree that future work should additionally investigate high-dimensional settings. Complex joint distributions of the observed data may impede the baseline convergence of the DGM, which in turn could diminish the utility gain provided by our debiasing approach. This is a limited weakness of our proposal as it may be the unavoidable consequence of working with high-dimensional data (e.g., debiased machine learning strategies likewise provide no guarantees in high-dimensional complex settings). Nevertheless, we expect that our approach will still improve inference, as is likewise observed for debiased estimators based on machine learning predictions.\n\n**Can the debiasing strategy be extended to more complex estimators <...>?**\n\nWe elaborate on this in our global response, but note that the proposed debiasing strategy is already applicable to all pathwise differentiable parameters. This includes essentially all finite-dimensional parameters used in routine practice, such as variances, regression coefficients,... An empirical study for other parameters than the population mean and linear regression coefficient is of interest, but beyond the scope of this work, whose aim is to show, for the first time, that valid inference on DGM-based synthetic data is feasible, but requires proper debiasing.\n\n**Are there alternative debiasing approaches that could be explored to address the limitations of the proposed method <...>?**\n\nWe refer the reviewer to our global response on related work and alternative debiasing approaches. Regarding their specific questions on the dependence on EICs and conditional sampling, future work could:\n- integrate insights from the literature on automatic debiased machine learning (Chernozhukov et al. (2022)) that allows to estimate the EIC automatically from data.\n- combine the findings from our work and work of Ghalebikesabi et al. (2022) on importance weighting, where the weights could be targeted to eliminate the impact of the data-adaptive estimation of the weights. This could relax the fast baseline convergence assumption, and enable the same ‘debiased’ synthetic data to be used for multiple analyses. The latter might also be possible by undersmoothing the DGM, by extending the recent work on kernel debiased plug-in estimation to the synthetic data context.\n- incorporate strategies for conditional sampling from DGMs (e.g. the work of Zhou et al. (2023)). Note that in case of a limited number of discrete covariates, conditional sampling can be approximated by (unconditionally) generating a large synthetic sample and taking a conditional subset, as done in our simulation study and second case study.\n\n**How can the debiasing strategy be balanced with privacy concerns <...>?**\n\nThe privacy-utility trade-off is well-known in synthetic data literature and relates to the sample size of synthetic datasets: as m → +∞, the synthetic dataset has higher probability to contain synthetic records that are close to the original, resulting in higher disclosure risk (Reiter, J. and Drechsler, J. (2010)), while the inferential utility improves in terms of more precise estimates (as can be seen from the expressions for the standard error). Although formal privacy assessment is beyond the scope of our paper, this trade-off is relevant to our work. Default synthetic data created by differentially private (DP) generative models will by definition not have worse privacy with increasing sample size, but will only have better inferential utility when the estimators remain roughly √n-consistent. With DP DGMs, this is not the case (see Decruyenaere et al. (2024)), but it could possibly be attained by extending our debiasing approach, without compromising any privacy guarantees due to the post-processing immunity of DP. This extension should incorporate the additional DP-constraints into the derivation of the difference between $\\theta(\\tilde{P}_m)$ and $\\theta(P)$. It is not clear yet what this would look like, given that different DP generators add noise in different ways (e.g. DPGAN adds noise to the gradients, while PATE-GAN adds noise to the majority vote in the training procedure). We will discuss this nuance more clearly in our revised manuscript.\n\n**Are there specific scenarios where the assumptions required for the debiasing method might not hold <...>?**\n\nOur approach relies on fast baseline convergence of the DGM: faster than n-to-the-quarter convergence rates for the unknown functionals of $\\tilde{P}_m$ and $\\hat{P}_n$ that appear in the EIC are required. Their attainability depends on the number of parameters in the DGM itself, the dimension of the data and the complexity of the observed data distribution. If these rates are not attained, then we expect that our approach will still improve coverage, but that no root-n consistent estimators will be obtained based on the synthetic data, resulting in increasingly anti-conservative coverage with larger sample size n.\n\nChernozhukov, V., Newey, W., and Singh, R. (2022). Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3), 967-1027.\n\nDecruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on UAI.\n\nGhalebikesabi, S., Wilde, H. et al. (2022). Mitigating statistical bias within differentially private synthetic data. In The 38th Conference on UAI.\n\nReiter, J. and Drechsler, J. (2010). Releasing multiply-imputed synthetic data generated in two stages to protect confidentiality. Statistica Sinica, 20, 405–421.\n\nZhou, X., Jiao, Y. et al. (2023). A deep generative approach to conditional sampling. Journal of the American Statistical Association, 118(543), 1837-1848.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTqThkJ8W1", "reviewer": "Reviewer_ZgH6", "summary": "The authors propose a method for debiasing DGM-generated tabular data such that the population average of the EIC for the observed data distribution and its estimation becomes zero, resulting in elimination of the corresponding biasing term. This is done through augmenting the DGM output such that the population mean of the synthetic data matches the sample average of the observed data. As such, the method is generator-agnostic. The authors validate their approach through extensive experiments, demonstrating significant reductions in bias while maintaining data utility.", "questions": "1. Could the authors provide more details on how their approach scales with larger datasets and more complex models?\n2. How does the proposed method perform in scenarios with multiple sources of bias or more intricate data dependencies?\n3. Can the authors elaborate on any potential trade-offs between debiasing effectiveness and computational efficiency?\n4. How is \"a very large sample\" defined? L146 says 1,000,000 observations is \"large\", but the quality analysis in Section A.7.2 has max n = 5,000. \n5. Is the necessary synthetic data sample size generalizable or does it depend on the complexity of the parameter space?", "limitations": "The authors properly address limitations of their work. However, they could further discuss the scalability of the proposed method to large datasets.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is technically sound with in-depth theoretical analysis that are backed up by practical examples. We appreciate the additional detailed investigations presented in the Appendix, which shows the dedication of the authors to do a robust evaluation of their methodology. \n\n2. The paper tackles the critical issue of bias in synthetic data, a relatively underexplored area. The proposed method integrates fairness constraints directly into the data generation process, a novel approach, as opposed to post-hoc fairness adjustments. The proposed methodology specifically addresses the bias term concerning the observed data distribution, O, and its DGM estimation, P, as opposed to previous efforts that (are said to) have focused on the bias term describing the discrepancy between the sampled estimated (synthetic) distribution, S, and the approximation of P as P¨ (P-hat) through S.\n\n3. The paper is very well written and well organized (it was a pleasure to read). Figures and tables are well-presented and support the narrative effectively.\n\n4. The results have significant implications for improving fairness in machine learning, an area of growing importance. The proposed method for debiasing synthetic tabular data is straightforward and well described and hence easily implemented and adaptable by others", "weaknesses": "1. The paper would benefit from a more in-depth evaluation of potential limitations or edge cases where the method might not perform as well.\n\n2. It is difficult to assess the novelty in the proposed method as there is no presentation of related works nor comparison with alternative methods. A more detailed review of related work would help contextualize the novelty of the proposed method.\n\n3. It would be good to put the work in contrast with related works would improve the paper and elucidate the authors contribution to the field.\n\n4. The significance is still difficult to assess due to the aforementioned lack of comparison to alternative approaches. Additionally, potential impacts on different domains (e.g., healthcare, finance) could be explored to highlight the broader significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer ZgH6 for their careful reading of the manuscript and constructive review.\n\n**It would be good to put the work in contrast with related works <...>.**\n\nWe noted that other reviewers raised similar questions and have therefore included an extended comparison with related work in the global rebuttal, which will also be integrated into the manuscript.\n\n**Could the authors provide more details on how their approach scales with larger datasets and more complex models?**\n\nWe elaborate on this in our global response as well, but agree that future work should additionally investigate high-dimensional settings. Complex joint distributions of the observed data may impede the baseline convergence of the DGM, which in turn could diminish the utility gain provided by our debiasing approach. This is a limited weakness of our proposal as it may be the unavoidable consequence of working with high-dimensional data (e.g., debiased machine learning strategies likewise provide no guarantees in high-dimensional complex settings). Nevertheless, we expect that our approach will still improve inference, as is likewise observed for debiased estimators based on machine learning predictions.\n\nConcerning an extension to more complex estimators, we would like to highlight that the proposed debiasing strategy is already applicable to all pathwise differentiable parameters. This includes essentially all finite-dimensional parameters used in routine practice, such as variances, regression coefficients,... An empirical study for other parameters than the population mean and linear regression coefficient is of interest, but beyond the scope of this work, whose aim is to show, for the first time, that valid inference on DGM-based synthetic data is feasible, but requires proper debiasing. Lastly, we would like to stress that our proposed strategy is generator-agnostic and can therefore be extended to more complex DGMs.\n\n**How does the proposed method perform in scenarios with multiple sources of bias or more intricate data dependencies?**\n\nRegarding the first part of the question concerning the multiple sources of bias, we wonder whether there is a misunderstanding regarding bias in the context of fairness. In view of this, we clarified the meaning of bias in our manuscript in the global rebuttal. In particular, we focus on the estimation bias that arises in estimators based on synthetic data generated by DGMs. In our paper, we\ntarget each estimator separately (e.g. the mean of age vs. the effect of therapy on blood pressure), but future work should focus on targeting all these sources of bias at once. This entails that the person generating the synthetic data is aware of all the analyses that will be run on those data, and has access to the corresponding EICs. Whether targeting multiple estimators at once affects the efficacy of the debiasing approach remains to be studied. The effect of more intricate data dependencies on the performance of our debiased strategy is an interesting yet open research question at this point. Our approach relies on fast baseline convergence of the DGM, which is influenced by the dimension of the data and the complexity of the true data-generating model.\n\n**Can the authors elaborate on any potential trade-offs between debiasing effectiveness and computational efficiency?**\n\nThere are indeed potential trade-offs that should be taken into consideration. As alluded to in the paper, the current implementation of our strategy does not use sample splitting to estimate the bias term and therefore does not require the generative model to be retrained, making our strategy computationally efficient. Future extensions that necessitate a sample splitting procedure during which the generative model is retrained on different subsets of the data (and as such, computation time will depend on the dimensionality of these subsets) should focus on both computational and statistical efficiency. Our theory currently discards sample splitting due to preliminary results suggesting that the bias reduction did not outweigh the increase in finite-sample bias that may result from training the DGM on smaller sample sizes. This suggests a trade-off between debiasing effectiveness and statistical properties such as finite-sample bias and increased variability when using sample splitting.\n\n**How is “a very large sample” defined? <...> Is the necessary synthetic data sample size generalizable <...>?**\n\nIn the set-up of our simulation study (which Section A.7.2 relates to), we consider different sample sizes of observed and synthetic data: $n \\in$ {50, 160, 500, 1600, 5000}. These sample sizes were chosen to reflect a range that is common in real medical settings. Next, in the theory of our debiasing strategy for a linear regression coefficient we need the terms $E_{\\hat{P_n}}(A|X_i)$ and $E_{\\hat{P_n}}(Y|X_i)$. Given that these reflect unknown conditional population means, we need a way to approximate these in practice. In this case, one often creates an artificial “population” by generating a very large sample of e.g. one million observations, to then approximate the population mean by calculating the sample mean of this Monte Carlo sample. The purpose of this very large sample is merely to recreate a population of samples generated by the trained DGM and is thus not comparable to the magnitude of sample sizes of the observed data. We believe one million observations will suffice to capture the distribution of samples generated by the DGM, regardless of the specific type of DGM.\n\nChernozhukov, V., Chetverikov, D. et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68.\n\nDecruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In The 40th Conference on UAI.\n\nvan der Laan, M. J. and Rose, S. (2011). Targeted Learning. Springer Series in Statistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their careful reading of the manuscript and the constructive reviews. We were pleased to read that the presentation of our work was positively received by **Reviewers eYfd, 4cJP** and **ZgH6**, though we agree with **Reviewer LYaP** that the methodology section could benefit from further clarification.
We noticed that there were three recurring topics, which we have chosen to address in this global response, while elaborating further in reviewer-specific rebuttals where needed.

**Reviewer eYfd, 4cJP** and **ZgH6** raised concerns about the generalizability of our results to high-dimensional settings and more complex estimators. We agree with the reviewers' point. However, we would like to highlight that our paper is, to the best of our knowledge, the first to demonstrate, with significant generality, how valid statistical inferences can be derived from synthetic data. This addresses a critical yet largely overlooked issue with synthetic data: standard analyses of synthetic data based on deep generative models (DGMs) often result in significantly biased inferences, leading to confidence intervals that rarely contain the true parameter values.
Decruyenaere et al. (2024) have shown that this problem persists even for basic statistics such as means and regression coefficients, which are central to most routine analyses. This is why our current paper initially focuses on proposing a solution for these fundamental quantities. Moreover, our methodology is applicable to all pathwise differentiable parameters, encompassing essentially all finite-dimensional parameters, including variances, correlations, average treatment effects, and regression coefficients in common statistical models.

While empirical investigation of our proposed solution across these various contexts is undoubtedly important, it falls beyond the scope of the present work.

We noted that **Reviewer LYaP** and **ZgH6** linked our work to the field of fairness in machine learning. We want to make clear that when we talk about bias, we mean statistical bias, being the average difference between the estimate obtained in the data and the true underlying parameter. We do not mean to remove bias towards some particular subpopulation, which is how this term is often used in fairness literature. We apologize if this caused confusion. We do agree that fairness is an important concept, and future work could therefore focus on extending our debiasing approach to metrics like group fairness, as part of the extensions discussed in our previous point. 

Finally, most reviewers suggest to clarify whether alternative approaches exist to obtain valid inference from synthetic data and to contrast these with our approach.
While several approaches have been proposed to account for the uncertainty arising from synthetic data generation, we are not aware of strategies for generating and/or analyzing DGM-based synthetic data that guarantee valid inference.
Raghunathan et al. (2003) developed a framework inspired on the work of multiple imputation for missing data, by combining the results of multiple synthetic datasets, but this is not readily applicable to DGM-based synthetic data.
Räisä et al. (2023a) extended this work for differentially private (DP) synthetic data, acknowledging the additional DP noise during synthetic data generation, but continue to consider parametric (Bayesian) data-generation strategies.

Our work instead focuses on obtaining valid inference from a single synthetic dataset, which is more attractive for use by practitioners.
The method suggested by Awan and Cai (2020) to preserve efficient estimators in a single (DP or non-DP) synthetic dataset relies on generating synthetic data conditional on the estimate in the original data and is only applicable to parametric generative models and therefore suffers the same limitation as the aforementioned approaches.
To allow for Bayesian inference from a single DP synthetic dataset, Wilde et al. (2021) proposed a corrected analysis that relies on the availability of additional public data, while Ghalebikesabi et al. (2022) investigated importance weighting methods to remove the noise-related bias, but they do not study the impact on inference. 
Relative to the aforementioned approaches, our work is the first to consider the impact of the typical slower-than-$\sqrt{n}$-convergence of estimators in (DP or non-DP) synthetic data created by deep generative models. As far as we are aware, our approach is thus the only one that provides some formal guarantees for (more) honest inference in this setting. We will add this paragraph on related work to our revised manuscript.

Awan, J., and Cai, Z (2020). One Step to Efficient Synthetic Data. _arXiv preprint arXiv_:2006.0239.

Decruyenaere, A., Dehaene, H. et al. (2024). The real deal behind the artificial appeal: Inferential utility of tabular synthetic data. In _The 40th Conference on Uncertainty in Artificial Intelligence._ 

Ghalebikesabi, S., Wilde, H. et al. (2022). Mitigating statistical bias within differentially private synthetic data. _Proceedings of the Thirty-Eighth Conference on UAI_, in _Proceedings of Machine Learning Research_ 180:696-705.

Raghunathan, T. E., Reiter, J. P., and Rubin, D. B. (2003). Multiple imputation for statistical disclosure limitation. _Journal of official statistics_, 19(1), 1.

Räisä, O., Jälkö, J. et al. (2023a). Noise-aware statistical inference with differentially private synthetic data. _Proceedings of The 26th International Conference on AISTATS_, in _Proceedings of Machine Learning Research_, 206, 3620–3643.

Räisä, O., Jälkö, J., and Honkela, A. (2023b). On Consistent Bayesian Inference from Synthetic Data. _arXiv preprint arXiv_:2305.16795.

Wilde, H., Jewson, J. et al. (2021). Foundations of Bayesian learning from synthetic data. _Proceedings of The 24th International Conference on AISTATS_, in _Proceedings of Machine Learning Research_, 130, 541–549.

Meta-review: This paper proposes a debiasing strategy for synthetic data generated by deep generative models (DGMs), which aims to eliminate the bias that arises when estimating population parameters from synthetic data. The authors argue that existing methods for generating synthetic data can lead to biased estimates, and propose a new approach that involves post-processing the synthetic data to remove the bias. The method is generator-agnostic, meaning it can be applied to different types of DGMs, and is based on techniques adapted from debiased or targeted machine learning.

The reviewers praised the paper for its technical soundness, originality, and significance, noting that it addresses an important problem in the field of synthetic data generation. However, they also raised some concerns about the paper's readability and clarity, particularly for readers without a strong background in the field. The reviewers also asked for more information about the scalability of the proposed method to larger datasets and more complex models, and about the potential trade-offs between debiasing effectiveness and computational efficiency.

The authors responded to these concerns in their rebuttal, providing additional explanations and clarifications, and acknowledging the limitations of their study. They also suggested avenues for future research, including investigating the performance of the method in scenarios with multiple sources of bias or more intricate data dependencies.

After the rebuttal and discussion, three reviewers lean towards acceptance, while one leans towards rejection. However, the latter reviewer reports only a confidence of 1 and has not acknowledged the authors' rebuttal. Overall, I would thus recommend to accept the submission. However, I would further recommend that the authors seriously take the reviewer feedback into account for the camera-ready version, especially when it comes to the clarity of the presentation of the method and theoretical results to a wider ML audience and the positioning of this work within the related literature.
Decision: Accept (poster)