Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference
Authors: Harry Amad
Zhaozhi Qian
Dennis Frauen
Julianna Piskorz
Stefan Feuerriegel
Mihaela van der Schaar
Venue: NeurIPS 2025 poster ()
Keywords: synthetic data
treatment effect estimation
medical data
generative models
evalution metrics
Primary area: machine_learning_for_sciences

Abstract: Causal inference is essential for developing and evaluating medical interventions, yet real-world medical datasets are often difficult to access due to regulatory barriers. This makes synthetic data a potentially valuable asset that enables these medical analyses, along with the development of new inference methods themselves. Generative models can produce synthetic data that closely approximate real data distributions, yet existing methods do not consider the unique challenges that downstream causal inference tasks, and specifically those focused on treatments, pose. We establish a set of desiderata that synthetic data containing treatments should satisfy to maximise downstream utility: preservation of (i) the covariate distribution, (ii) the treatment assignment mechanism, and (iii) the outcome generation mechanism. Based on these desiderata, we propose a set of evaluation metrics to assess such synthetic data. Finally, we present STEAM: a novel method for generating Synthetic data for Treatment Effect Analysis in Medicine that mimics the data-generating process of data containing treatments and optimises for our desiderata. We empirically demonstrate that STEAM achieves state-of-the-art performance across our metrics as compared to existing generative models, particularly as the complexity of the true data-generating process increases.

Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference
Harry Amad DAMTP University of Cambridge hmka30cam ac. uk
Zhaozhi Qian" Elm Europe zqianGelm.
Dennis Frauen
LMU Munich
Munich Center for Machine Learning
Julianna Piskorz DAMTP University of Cambridge
Stefan Feuerriegel
Mihaela van der Schaar
LMU Munich
DAMTP University of Cambridge
Munich Center for Machine Learning
Abstract
Causal inference is essential for developing and evaluating medical interventions, yet real-world medical datasets are often difficult to access due to regulatory bar- riers. This makes synthetic data potentially valuable asset that enables these medical analyses, along with the development of new inference methods them- selves. Generative models can produce synthetic data that closely approximate real data distributions, yet existing methods do not consider the unique challenges that downstream causal inference tasks, and specifically those focused trealments, pose. We establish a set of desiderata that synthetic data containing treatments should satisfy maximise downstream utility: preservation of the covariate distribution, (ii) the treatment assignment mechanism, and (iii) the outcome genera- tion mechanism. Based on these desiderata, we propose set of evaluation metrics to assess such synthetic data. Finally, we present STEAM: novel method for generating Synthetic data for Treatment Effect Analysis Medicine that mimics the data-generating process of data containing treatments and optimises for our desiderata. We empirically demonstrate that STEAM achieves state-of-the-arr per- formance across our metrics as compared to existing generative models, particularly as the complexity of the true data- getta-genrating process increases.
Introduction
Access medical data crucial for advancing healthcare research, enabling novel medical discoveries and providing a testbed for developing new analytical approaches, such as machine-learning -baee causal inference methods
6 94 288 However, regulations restrict access to patient data for research purposes 3 [92). Synthetic data, which is gaining significant recognition in medical literature [48), offers way to increase data availability, and recent initiatives are aiming to generate synthetic data for broad 'health data research' [20 (2). such as testing of learning algorithms (33) and replication of clinical trial results [17]. Importantly, the promise of synthetic data hinges on its ability preserve information critical to relevant downstream tasks. The majority of recent literature [ focuses on downstream predictive tasks, shaping standard evaluation and generation practices this setting.
Medical data typically contain treatment assignment variables which invite unique approaches to downstream analysis, beyond standard prediction (28). Data containing treatments are typically anal- ysed via causal inference (e.g. treatment effect estimation methods) to examine causal relationships
Work completed while at University of Cambridge.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

between covariates, treatments, and outcomes. Despite this, many works in synthetic data which use medical data for motivation and validation [3
98 10 employ standard. -oriicttoonoieeeeedd gen- [8 A for more details). Such failure to acknowledge the
eration and evaluation techniques (see Appendix likely downstream uses of synthetic data containing treatments leads to low-quality data generation, and evaluation via misaligned metrics.
Evaluation. Standard synthetic data evaluation involves holistic statistical comparisons of synthetic and real data (Table T In this paradigm, causal inference tasks are overlooked, as treatments are handled like any other variable, limiting the relevance of such assessment in medical settings. To illustrate this, consider the following key questions that an analyst working with a synthetic medical dataset, D,, may ask: Q1 How representative are the patient covariates in Dz?: Q2 How accurate are the treatment assignment decisions in D,?; and Q3 How much error might be introduced treatment effect estimates derived from DD?? These questions require differentiation between covariates, treatments, and outcomes, and they cannot be answered with current evaluation protocols.
Generation. Generic generative models (Table are typically designed to minimise the divergence between real and synthetic joint distributions. In doing so, these models overlook the specific causal relationships between covariates, treatments, and outcomes that are essential for downstream medical applications. Existing causal generative models, on the other hand, generally assume access to a causal graph that describes all causal relationships between variables, which is often unrealistic in complex domains like medicine, where true causal structure is rarely known (27).
In this work address these limitations by proposing novel approaches to evaluation and generation of synthetic data containing treatments, operating under reasonable assumptions and explicitly considering the downstream use of such data. In doing so, we make the following contributions:
1. Desiderata: By examining typical medical analyses conducted on data containing treatments, we establish a set of desiderata that synthetic data should satisfy in this context (44)
2. Evaluation: We show that existing synthetic data evaluation metrics are inadequate in this setting, as they do not measure adherence these desiderata. As remedy, we propose principled set of metrics derived from our desiderata, allowing meaningful evaluation of synthetic data containing treatments (55)
3. Generation: We propose STEAM, method for generating synthetic data that augments generic generative models to encode inductive biases to optimise for our desiderata (16). Empirical analysis:
We demonstrate that STEAM generates state-of-the-arr synthetic data con- taining treatments, particularly as the real data-generating process (DGP) grows in complexity, and in high-dimensional scenarios 7
2 Problem formulation
Setup. We consider a data owner with access to observational or experimental real data D. {(x!)
(i) w!(") y( sampled from population P. where x( X' is vector of d binary or continuous covariates, w:(*) E {0, }} is a binary treatment assignment, and (,()) E V binary or continuous outcome. We refer to the set of all variables in D, as V (Xi,.., Xa, W.
We denote the propensity score with π (x) Pwix = x-
Objective. We wish to enable the release of synthetic data to downstream users with various analysis goals, such as estimation of propensity scores, average treatment effects (ATEs), and conditional average treatment effeett (CATEs). To do so, we aim generate synthetic data D. ((x!0, W(()) y x())) from a distribution Q and evaluate how well D. captures the information in D, that is relevant to likely downstream tasks with a set of metrics M(D,,D.)
3 Related work
Evaluation. Evaluation of synthetic tabular data, the modality we focus on here, has two common fo- cuses: resemblance and predictive utility 773 While predictive utility assesses model accuracy when
'Our code is available at https:://gttubb.cmm/aaaaaa vanderschearLaa/STEAA
and https://github... com/

---

Table 1: Synthetic data evaluation methods. d and p distance and correlation functions, Sp is the support of distribution P. 'Q?": which the questions from { does the method answer?
Method
Formula
Differentiates between X, W, Y Q?
Marginal
Eie
X
Existing
Correlation
MI] Eijev d(p(P, Pa), (Q: Q7))
1 d(Pv. Qv)
Joint
X
Prec., Rec.
[Sp n S / [Sp n Sol/|Sr)
Pa.x, R3.x
Equations 3 and
01
Ours
JSD,
Equation
Q2
UPELE
Equation
Q3
Table 2: Synthetic data generation methods, d is a distance function. PAg (V4) refers to the set of parents of node V; in the causal graph G.
Example methods
Distributional target
Assumptions D,
D, application
Generie gen. models
NFlow 80
CTGAN
TVAE 97
min d(Qv, Pv)
None
Prediction
TabDDPM 58
ARF 9
min d(Q PusIPA((M)))
CGMs
DCM 12
Interventional/ counterfactual
VACA
83
g
min di (Qyyn P
queries on G
Ours
min d(Q,, Px) min di d(Qii Pwix) min d(Qriw.x. PrIwx)
Caual inference with treatments
STEAM
Valid DGP
trained on D,, it is unsuitable for causal tasks lacking observable ground-truths 44] On the other hand, resemblance metrics compare statistical properties of DD and D, (marginals, correlations, joint distributions, supports; Table IT. Existing methods handle all variables similarly. not differentiating between X, W, and Y, and they therefore cannot answer Q1-3. Our metries ($5) remedy this.
Generation. We consider two related approaches to generative modeling (Table 2). Firstly, generic generative models make minimal assumptions D., minimising the difference between real and synthetic joint distributions. In doing so, these methods forego any treatment-related inductive biases, unlike our STEAM method. Causal generative models (CGMs), on the other hand, require full knowledge of the causal graph, G. to model each conditional distribution as dictated by G. STEAM's assumptions (66) that the underlying DGP of D, is of the form X Px. W Pwix: Y PYJw.x. are much less restrictive, not requiring knowledge of individual causal relationships between variables, and they will hold for wide array datasets containing treatments. For further elaboration on specific evaluation and generation methods, see Appendix B
4 Desiderata for synthetic data containing treatments
We cover three essential distributions for treatment data: (i) the covariate distribution Px, (ii) the treatment assignment mechanism Pwix, and (iii) the outcome generation mechanism Pr|wx- While their importance is clear in the causal inference community, hence Q1-3, this has not been adequately recognised by the synthetic data community, and methods that target them are missing. To bridge this gap, we establish desiderata for synthetic data containing treatments based on these distributions.
3

---

(i) The covariate distribution Px describes the population of interest and, in medicine, it is standard to report its characteristics 1951 as it determines to whom analysis is relevant.
Why is its preservation important? Inadequate covariate coverage in Dj can result exclusion from downstream analysis of members of the population whose covariates are not well explored, as making reliable inferences can become infeasible (78) 81]. On the other hand, generating out-of-distriutioo covariates in D3 can cause groundless extrapolation by synthefically-rrained models, leading to potential misuse.
(ii) The treatment assignment mechanism Pwix is nuisance parameter in many treatment effect models [3), and it can be a target for analysis itself when examining treatment protocols.
Why is its preservation important? Since Pwix is used as a nuisance parameter, errors in its modelling propagate to treatment effect estimates derived from D,.. Furthermore, Pw]x x can guide the difficult task of CATE model selection [47], so poor preservation can lead to inconsistency in model selection between D, and D., which is unideal [37]. Misrepresenting Pwix also risks misreporting treatment protocols. Given that extreme propensities of (x) s 0 (or xxx 1) are common in data such as electronic health records [63], often because of safety, inaccurate Qwix could encourage exploration of treatments in patient subgroups for which they highly unsafe.
(iii) The outcome generation mechanism PyJw,x is the distribution through which treatment effects can be estimated, by comparing the statistical functionals of Py w=1.X and Py P
Why is its preservation important? Priwxx must be preserved, so that D, can permit accurate estimation of treatment effects. Q is inaccurate, then even a perfect model could not estimate correct treatment effects from Dz. and the worse this relationship is preserved, the less useful D3 becomes.
Preserving (ii- is necessary and sufficient for Q to be a high-quality approximation of P. Mod- elling each distribution well is evidently necessary given the above reasons, and it is also sufficient, which is clear from the following decomposition of P:
P(X, = Px P PwIx (W(X) (Y
The components (ii- offer a complete factorisation of the joint distribution, and therefore Q matching P in each component is sufficient for Q to match P entirely. As such, accurate modelling of (i)-(ii) forms our desiderata for synthetic data containing treatments. Generation methods should seek to maximise adherence to these desiderata, and evaluation metrics should assess how successful D, is in this regard (and therefore answer Q1-3).
On causal assumptions. Even if these desiderata are satisfied, D. may not permit useful analysis via causal inference. Assumptions, such as typical identifiability assumptions? must still be critically examined by analysts, since any violations in D, will almost surely be violated in a faithful D, as well. Accounting for violated assumptions is task orthogonal to synthetic data generation, with existing literature [52 29), and we do not consider it necessary for Q to improve upon any biases in P. allowing post-generation methods to rectify them, if necessary, instead.
How to evaluate synthetic data containing treatments
With our desiderata established, we now investigate how to evaluate the adherence of D,.
5.1 Inadequacy of existing metrics
Existing evaluation metrics do not measure how well D, satisfies our desiderata. These metrics do not differentiate between X, W, and Y, and they therefore cannot directly assess any of Qx. Qwix. or Qriw.x. voint-distribution-leee metrics, such Kullback-Leible (KL) divergence [59], are the most common approach, offering a holistic assessment of how well Q models P. However, these are
Consistencyy ( Y (W) overlap: 0 < π(() < 1, and unconfoundednes: (0), ( TL W

---

only loosely related to our desiderata, and they do not allow user to disentangle how each of (i)-(ii) preserved, limiting the depth of information offered on D.. Furthermore, we argue that they tend to be dominated by X as it grows in dimensionality. and they will lose sensitivity to the treatment assignment and outcome generation mechanisms. By this, we mean that these metrics tend to fail to notice differences in the modelling of Pwix or w by two proposal distributions.
For an illustrative example of this phenomenon with KL divergence, consider a simple / which can be factorized as = II= Px; Pwix PrIw.x. Let there be two learnable distributions Q%, and n22 with the same form Qh = II Qx Quix Q Y and which only differ in either Ow, or By (i..., they either model Pwix or Py|w;X differently). In this setting, the following holds:
Theorem 1. Let P. Qa, Q² be of the above form, and M be KL divergence. we assume that Qhe and Q have sufficient capacity to have bounded error on each component, iee. Vi, < M(Px,, O%2) Ex, and 0 < M(Pwix. then:
&y &w and 0 < MMPYiI Qrlwx)) EY
M(P,Q')) +1, as 000
(2)
M(P.Q%)
Proof. See Appendix C
Theorem I shows that KL divergence loses sensitivity to W and Y|W, X grows, suggesting that it will struggle to select between Q% and Q²= despite any difference in their modelling of Pwix or Pylwxx. For an empirical example this phenomenon arising across an extended array of joint-distributioo--leel metrics, see Appendix D
5.2 Metrics for synthetic data containing treatments
These findings motivate us design our own metries for synthetic data containing treatments which directly measure adherence to desiderata (i)-iii) and offer answers to Q1-3.
5.2.1 The covariate distribution
Evaluating Qx requires comparing the generally high-dimensional covariate distributions of DD and D,. which is non-trivial. Nevertheless, this is a standard synthetic data evaluation task, as does not contain treatments. We see precision/recall analysis the most useful approach. There is typically a trade-off between these two qualities, which generative models balance differently [8] (2). and
by measuring them both, a data holder can guide generation towards their preferences for covariate realism and diversity. Without a strong preference, balancing the two is recommended (50).
We propose the use of the integrated Pa and Rg scores, introduced in (2]. which compare the supports of D, and D., for E [0, 1] Intuitively, Pa captures how well D, falls within the support of D, and R3 reflects how well D, is covered by the support of D,. We denote the covariate precision and recall with Pa, x and R3,x respectively, which are calculated by applying integrated Pa and R3 to the covariates of D, and D, only, as in Eq. (3) and Eq. 4
Pa. = 1 - /'' PIX,, E E; al do
(3)
Ra,x (D, D.) = -2 / |P(X, 8.) 3 d3
(4)
where X, and S. are the embedding X. = =x(X.) and --support as defined (2). respectively.
We have 0 Pa.x, R3. 1, and scores near I indicate a realistic and diverse Qx. Together, these metrics can be used to answer Q1.
An a-support is the minimum volume subset of the domain that contains probability mass a.
5

---

5.2.2 The treatment assignment mechanism
While in general we do not have access to Pwix and Qwix. we know that, for each X = they are Bernoulli distributions, since W a binary variable. The success probabilities can be estimated from D, and D. with a probabilistic classifier, which can be used to form approximations of Pwix and Qwix. There is then an array of valid options to compare these approximations. We propose the use of Jensen-Shannon distance given its desirable properties of symmetry, smoothness, and boundedness (we discuss alternatives Appendix B For given probabilistic classifier ft, we define Pwix=x Ber (x))(x((x) and Qw x= Bern m. (x)) where and म. trained on D, and D. respectively, and we measure the preservation of Pwix as in Eq. (
(D, Epx
(KL(P |I M) + KL(QWIX-XX x-x (1)
(5)
where M = =1(PWIX=> + @wix=x) and KT. KL divergence using log2-
JSD, can be used to answer Q2. We have < JSD, < matches Pwix well. The validity of ,sD, will depend on the accuracy of म., conducting Ā model selection is an important pre-evaluation step, although, amongst reasonable model choices, we find the information offered by JSD,, does not significantly differ.
1, with scores near 1 indicating that Qwis
5.2.3 The outcome generation mechanism
To evaluate the preservation of PrJw.x, we consider a treatment effect analogue of predictive utility. We address the unavailability of ground-truths by seeking parity in performance between D, and D,, rather than quantifying error from an oracle value. Such evaluation is inherently task dependent, yet the specific quantity D3 may be used estimate is unclear. Assessment should therefore centre on complex task, in which comparable performance will likely imply the same for simpler tasks. In this case, we consider the most difficult treatment effect task likely to arise in the medical field CATE estimation as similarity in this between DD and DD will tend to imply similarity in simpler tasks, such as ATE estimation definition, it will also yield an accurate ATE, but the reverse is not true, i.e., D, can reproduce the correct ATE and yet contain arbitrarily incorrect CATEs. Therefore, we evaluate how well QrIw.x preserves Py W..x by calculating the PEHE between synthetic- and real-trained CATE learners (see Appendix E for alternatives). Given family F of CATE learners é, where f, and f, are trained on Dr and D, respectively, assess the preservation of PyJw,x in Eq. 6
0 If DD yields accurate CATES across the full patient population then, by
UPenE(Dr, D.) = F
Σ REF
E (X) f. (X))
(6)
UPeHE can answer Q3. We average over JF since CATE model validation is difficult [16]. so i cannot be set as the best performing model in a similar fashion as is done for JSD,, (we discuss choices for F in Appendix F). As such, UPERE rewards synthetic data which permit proximity in CATES across an array of potential learners, where lower UPENE indicates better preservation of PrJwxx
Generating synthetic data containing treatments
To illustrate the standard DGP of data containing treatments, shown in the middle of Figure 4 consider a simple hospital dataset. Patient covariates X, such as height, weight, etc., are drawn from an underlying covariate distribution Px, which is dictated by the local population. Treatments are then assigned by a domain expert, such as a doctor, conditioned on X, i.e., W ~ Pwix. Finally, patients' outcomes dictated by the dynamics of their ailments, conditional upon W and X, i... ~ Py We now propose STEAM. novel aggoss-g framework for generating Synthetic data for Treatment Effect Analysis in Medicine which mimics this DGP.
IISD(P Q) = / [KL(P]) M) + 1 KL(Q || M), where M = (P Q).
'Dentting the potential outcomes Y(O) and Y(I), ATE is defined as ATE E - Y (0)) and CATE rxx)= = [Y (1) (0) X = x].

---

6.1 STEAM
STEAM, shown on the right of Figure 4 conducts a three-step generation process, mimicking the real DGP to push Q closer towards P in structure and directly target each distribution from our desiderata. STEAM involves three components:
1. Qx. X is generated from generic generative model trained to match Px.
2. QwIx- Treatments are assigned according to propensity function trained on D,. If DD is experimental data with known PwIx. then Qwix be directly set as the distribution, negating the need for any optimisation at this step.
QYIwx Potential outcome (PO) estimators are trained on D, to match Py|W=o,X and Py|w=1.X. and the relevant outcome is generated for each instance based on their assigned treatment.
Each component can be defined from diverse array of potential models. Qx can be any generic generative model, Qwix can be any classifier, and Q can use any regressors. Generation via the STEAM framework can therefore be framed augmenting any generic base generative model to improve its generation of medical data for use in downstream causal inference tasks, allowing it to easily fit within existing synthetic data generation pipelines.
7 Empirical analysis
In 7.1.1 and 7.1.2 we compare generic generative models and CGMs, respectively, with STEAM models medical data. Then, in 47.2, examine performance in number of targeted settings to better understand where the STEAM framework is particularly successful.
logistic regression classifier, and Q the PO estimators from S-Iearner [60]. We use the open source synthcity [79] for all generic generative models, and we indicate what for Qx in STEAM with subscript, i.e., STEAM, uses generative model o for Qx We detail data and experimental set-ups in Appendix
In STEAM, consistently set Qwix
7.1 Generation medical data containing treatments
To compare STEAM with existing generation frameworks, we consider performance across three medical datasets:
1. AIDS Clinical Trial Group (ACTG) study 175. A trial on subjects with HIV-1 33
Infant Health and Development Program (IHDP). semi-synthetic medical dataset, with real covariates and simulated outcomes, using data from an experiment evaluating the effect of specialist childcare on the cognitive scores premature infants [1]]
3. Atlantic Causal Inference Competition 2016 (ACIC). A semi-synthetic medical dataset, with real covariates and simulated outcomes, containing data from the Collaborative Perinatal Project (74).
7.1.1
Comparison with generic generative models
We compare state-of-the-an generic tabular data generators with their STEAM counterparts across all three datasets. We choose basclines across the major families of tabular data generators: CTGAN 197,, TVAE [97], ARF NFlow [80]. and TabDDPM (58). We display comprehensive results for each dataset and model combination in Table
Takeaway. the vast majority of cases, generation via STEAM leads to better performance across our metrics. Improvements in terms of JSD, and UPENE are most notable, indicating that STEAM significantly increases the preservation of Pwix and Py w.X- There relatively little difference in Pa.x and R,,x between generic and STEAM models, which is expected. While STEAM does isolate the modelling of the covariates in Qx giving that component model an ostensibly easier task than its joint-level alternative, modelling the complete Px, w,y. is nearly of equivalent difficulty to modelling Px when the number of covariates is high, since they dominate the dimensionality. Table 3 also demonstrates that STEAM's performance is sensitive to the

---

Table Medical data generation with generic and STEAM models, averaged over 20 runs with 95% Cls. Coloured numbers are the relative differences between each STEAM and generic model.
Dutaset
Model
Pu,x (1)
Ra.x
JSD
UPERE
ACTG
TVAE
0.926 0.013
0.483 = 0.010
0.946 + 0.004
0.564 0.017
STEAM IYAE
0,929 0.008 (+0.003)
0.186 = 0. 0.009 (+0.003)
0.958 + 0.001 (+0.012)
0.492 + 0.011 (-0.072)
ARF
0.818 D. 0.012
0.453 = 0.00T
0.960 + 0.004
0.577 0.015
STEAM
0.836 0.008 (+0.018)
0.464 0.00T (+0.01))
0.962 1 0.004 (+0.00))
0.423 0.016 (:0.54)
CTGAN
0.880
0.020
0.444 = 0.014
0.934 + 0.008
0.586 + 0.017
CTGAN
0.892
(+0.003)
0.437 =
(-0.007)
(+0.025)
0.436
(:0.150)
NEWow
0.817
0.032
0.418 = 0.008
0.913 1 0.016
0.643 + 0.026
STEAM NFlww
0.837
0. 040 (+0.020)
0,417 = 0.015 (-0.001)
0.962 + 0.005 (+0.049)
0.445 0. 0.020 (-0.198)
TabDDPM
0.067
0.060
0.036 0.035
0.812 + 0.029
1.761 + 0.230
STEAM TAODPM
0.609
D. 0.106 (+0.542)
0.310 = 0.033 (+0.274)
0.952 1 0.009 (+0.140)
0.468 1: 0.013 (- (-1.293)
IHDP
CTGAN
0.663
0.018
0.419 = 0.013
0.888 + 0.010
2.521 + 0.161
STEAM CIGAN
0.671
0.014 (+0.011)
0.124
(+0.055)
+ 0.928
(+0.040)
1.709 0.052 (-0.112)
1abDDPM
0.477 + 0.036
0.340 = 0.022
0.862 + 0.011
2.706 + 0.138
TADDPM
0.553
0.029 (+0.076)
0.396 :
(+0.056)
0.918 1
(+0.056)
2.346 0.088 (:0.660)
ARF
0.528 0.009
0.381 = 0.010
0.921 + 0.009
3.019 + 0.117
STEAM AXP
0.565 0. 0.0114 (+0.037)
0.394 = 0.010 (+0.013)
0.921 + 0.009 (+0.000)
1.629 0. 0.056 (-1.390)
TVAE
0,622 0.014
0.410 0,010
0.880 + 0.014
3.198 0.172
STEAM TVAE
0.629 + 0.015 (+0.007)
0.112 = 0 0.011 (+0.002)
0.927 + 0.007 (+0.047)
2.100 + 0.075 (-1.098)
NHlow
0.406 028
0,309
+
3.835
STEAM
0.435
0.034 (+0.029)
0.333
0.020 (+0.024)
0.921 1 0.007 (+0.039)
2.177 1. 0.118 (-1.658)
ACIC
TVAE
0.901 1 0.014
0.513 : 0.004
0.929 1 0.005
4.223 1: 0.138
STEAM TWAE
0.900 O. 0.0111 (-0.00))
0,514 = 0.004 (+0.001)
0.972 + 0.002 (+0.043)
2.422 0.118 (-1.801)
CTGAN
0,880
0.016
0.421 0.013
0,942 1 0.005
4.518 0.186
STEAM CTGAN
0.873
0.014 (-0.007)
0.424 : 0.014 (+0.003)
0.972 1 0.002 (+0.030)
2.268 1: 0.154 (-2.250)
ARF
0.828 0.003
0.430 0.002
0.945 + 0.002
4.633 0.146
STEAM
0.835 + 0.00-1 (+0.007)
0.130 = 0. 0011 (+0.000)
0.977 + 0.002 (+0.032)
2.449 + 0.149 (-2.184)
NHlow
0.748
0,333
5.068 0.282
STEAM
0.744
0.021 (-0.004)
0.333 1 0.010 (+0.000)
0.971 1. 0.002 (+0.133)
2.938 1 0.149 (:2.130)
TabDDPM
0.124 D. 0.028
0,002 = 0.001
0.813 + 0.023
9.281 1.033
STEAM TAIDIM
0.141 0.035 (+0.017)
0.002 0.000 (+0.000)
0.955 1 (+0.44))
4.497 0.501 (44784)
choice of Qx. as performance significantly differs between different STEAM configurations the same dataset.
7.1.2 Comparison with causal generative models
To fairly compare with CGMs, we must first address their more restrictive assumptions, as discussed in 3 For the ACTG, IHDP, and ACIC data, we do not know the true causal graphs; we simply know which features are the treatment and outcome. To construct reasonable causal graphs using this knowledge, we consider three methods:
1. Construction of a naive graph, Gnaive in which each covariate X causes W and Y, W causes Y, and every pair of covariates has a causal relationship between them;
2. Using the constraint- based PC causal discovery algorithm
89 to discover a graph, Gdisooveed
Pruning Gdiscovered by removing any edges which contradict the DGP we assume, i.e., edges from Y to W or X, or from W to X are removed, to form Gpruned-
In Table we compare the best STEAM models from Table 3 with two CGMs (using the best of the above graph methods): the additive noise model (ANM) [46 implementation from DoWhy-GCM (9]
and a diffusion- based causal model (DCM) from 12 (full results in Appendix MM.
Takeaway. For each dataset, we see that STEAM outperforms the CGMs in almost every metric, only being outperformed in P. x on the ACIC dataset. These results validate that, when the true causal graph is unknown, our less restrictive assumptions enable better generation of synthetic data containing treatments than CGMs.
7.2 Comparisons simulated data
To investigate the performance delta between STEAM and generic generation, we use simulated data with tunable experimental knobs, similar to [4] These knobs include covariate dimensionality
8

---

Table Medical data generation with CGMs and STEAM models, averaged over runs with 95% Cls. Coloured numbers are the relative differences between each CGM and STEAM model.
Dataset
Model
Pa,x (1)
Rg.x
JSD
UPERK (1)
ACTG
STEAM TYAE
0.929 0.008
0.486 0.009
0.004
0.492 +
DCM Granat
0.758 = 0.013 (-0.171)
0.358 0.007 (-0.128)
0.957 + 0.003 (-0.001)
0.596 + 0.017 (+0.104)
ANM Sincosand
0.836
U.007 (-0.093)
0.419
0.007 (-0.67)
0.932 0.004 (-0006)
0.578 + 0.019 (+0.086)
THDP
STEAM CITIAN
0.674 = 0.014
0.424 + 0.011
0.928 + 0.009
1.709 + 0.052
DCM Gransi
0.658 0.011 (-0.016)
0.360
0.007 (-0.64)
0.893 0.008 (-0.035)
2.059 = 0.140 (+0.350)
ANM Spreed
0.589 = 0.012 (-0.085)
0.359 + 0.009 (-0.065)
0.892 + 0.008 (-0.036)
1.865 + 0.059 (+0.156)
ACIC
STEAM
0.900 = 0.014
0.514 + 0.004
U.972 + 0.002
2.422 + 0.118
DCM Gisoweeed
0.942 0.004 (+0.042)
0.422
0.003 (-0.092)
0.957 0. 0.003 (-0.015)
4.249 + 0.132 (+1.827)
ANM Siscoorrd
0.929 0.003 (+0.029)
0.404 0.003 (-0.110)
0.872 0.002 (-0.100)
4.193 + 0.127 (+1.771)
d, propensity function π xd) + [0, 1), and prognostic and predictive functions Mprog. Mpred. (d) T Simulated sample i is generated by drawing X (0, Ia), W Bern ( (X (x)., and ( ([ (X(1) W(+), 'Preed (X 1). With this DGP,
performance
datasets tailored to specific situations. Across these experiments, we consistently compare between
TabDDPM and STEAM and the default settings for each experimental knob are: d 10,
(X) = (1+ -
Mprog (X) X12 + X22,, Mpred. (X) X3² + X²².
7.2.1
Increasing covariate dimensionality
To investigate performance as D, increases in dimensionality, we vary d E (5, 10, 20, 50).
UpShs
Takeaway. The performance delta between STEAM and generic generation grows with the dimensionality of X. This follows the intuition that, d grows, Px will dominate the joint dis- tribution, and the comparatively small Pwix and Prjw. will be overlooked by generic models. The top of Figure 1 shows that, as d increases, both STEAMTANDDPM and TabDDPM pre- serve Py w. worse, but STEAMTNNDPMM is less affected. The bottom of Figure iis similar, showing that TabDDPM degrades more than STEAMMaaDDPM. in preserving Pwix as d grows. Direct modelling of these small, but important, distributions by STEAM results in better performance high dimensions.
5 10
20
50
Standard
0.7
50
Figure 1: UPERE and JSD, as d increases. Averaged over 10 runs, with 95% Cls.
7.2.2 Increasing treatment assignment complexity
To investigate performance Pwix increases in complexity, vary the number of covariates upon which it depends. We set (XX)
1.00
0.95
(1 + 1/K Et_, x2)-1 for E (1, 2,,,4,55)
0.90
Takeaway. STEAM increasingly outperforms generic generation in preserving more complex Pwix- Figure 2 shows that, K increases, STEAMTANNPPM maintains a good estimate of Pwix, with JSD, consistently near 1. TabDDPM degrades with K, widening the performance gap. Direct modelling by STEAM allows more complex Pwix to be preserved.
Standard
STEAM
0.80
Figure 2: JSD K in- creases. Averaged over 10 runs, with 95% Cls.
'Prognostic variables affect an outcome regardless of treatment, while predictive variables only affect treated outcomes. Prognostic/predietive functions dictate the effects of the relevant variables the outcome.

---

7.2.3 Outcome heterogeneity
Standard
To investigate performance outcomes become increasingly heteroge- neous, we vary the number of covariates upon which Py Wy depends. We set Mpred. (X) > X2 for K E (3,4, 6, 6)
UNEVE w
Takeaway. As Py becomes increasingly heterogeneous, its preservation by STEAMTaNDDPM degrades only slightly, and much more dramatically for TabDDPM, as shown in Figure 3 Again, direct modelling of Q by STEAM better preserves complex distributions.
1
Figure 3: UPERE as K in- creases. Averaged over 10 runs, with 95% Cls.
The performance delta between STEAM and generic generation grows in complex settings. When generating synthetic copies of real data with high-dimensionality, or complex dependencies Pwix or Pyjw,x. STEAM increasingly outperforms. These situations are likely to emerge in real-world data, which is often highly complex, heightening the relevance of the STEAM framework.
Discussion
In this paper, we tackle problem impeding progress in the medical
synthetic data. Existing methods produce data that are poor for causal inference tasks, which are evaluated with misaligned metries. Our evaluation metrics and generation framework, grounded in our desiderata which stem from the needs of analysts, remedy this.
We allow meaningful evaluation with our metrics, proposed in [ that answer the key questions Q1-3 of downstream analysts from { STEAM generates synthetic data of substantially higher quality than existing methods, demonstrated across a range of experiments in 7 well in ablation study (Appendix]) ! and hyperparameter stability study (Appendix K. While we focus on medical data, our methods are also applicable to other fields where data contain treatments, or interventions, such education, marketing, and public policy. In Appendix [ we demonstrate performance on the Jobs dataset [6]]. showing STEAM has similar benefits in non medeicaal settings.
8.1
Future work
There are many future research directions in this setting. particular, generating synthetic data that respects formal definitions of privacy is an important step. In Appendix N we prove the immediate compatibility of STEAM with existing differentially private methods, and we conduct initial experiments showing promising results. Developing sophisticated mechanisms for assigning privacy budget across the component models in STEAM is a potential area future work, as we discuss in Appendix N.4
Beyond privacy, extending STEAM to operate in more complicated settings presents several opportu- nities. One extension, for instance, in settings with valid instrumental variable Z, could be to adapt the STEAM generation process to follow the corresponding causal structure:
~~ Pz, X PxIz, W Pwixz Y Py|wxx
This would produce synthetic data suitable for treatment effect analysis using instrumental variable methods 38 30,, which do not require the unconfoundedness assumption.
Other extensions could be designed for settings with multiple, or continuous, treatment options. STEAM can easily extend to accommodate multiple treatments by using multi-class classifier QwIx, and PO regressors compatible with 2 treatment arms for Qy More significant architectural changes would be required for continous treatments, Qwix would have to be replaced with regressor, and Qy W.X would require PO regressors designed for continuous treatments, such those based on the generalised propensity score [42.
10

---

Acknowledgements
Harry Amad's studentship is funded by Canon Medical Systems Corporation.
References
[1] Haleh Akrami, Sergul Aydore, Richard M. Leahy, and Anand A. Joshi. Robust variational autoencoder for tabular data with beta divergence, 2020.
[2] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela Van Der Schaar. How faithful your synthetic data? Sample-level metries for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290-306. PMLR, 2022.
[3] George Annas. HIPAA regulations: new era of medical-record privacy? New England Journal of Medicine, 348: 1486-1490, 2003.
[4] Peter C Austin. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3):399-424, 2011.
[5] Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. Differentially private query release through adaptive projection. In International Conference on Machine Learning, pages 457-467. PMLR, 2021.
[6] Howard Bauchner, Robert M. Golub, and Phil B. Fontanarosa. Data sharing: An ethical and scientific imperative. JAMA, 315(12):1238-1240) 2016.
[7] André Bauer, Simon Trapp, Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive exploration of synthetic data generation: A survey, 2024.
[8] Reza Bayat. study on sample diversity in generative models: GANs vs. diffusion models. In URL http:://ppmmeeeeeeeee//mmmmmte/
2023.
[9] Patrick Blöhaum, Peter Götz, Kailash Budhathoki, Atalanti Mastakouri, and Dominik Janzing, Dowhy-gem: An extension of dowhy for causal inference in graphical causal models. Journal of Machine Learning Research, 25(147):1-15 2024.
[10] Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. International Conference on Learning Representations, 2023.
[1I] Jeanne Brooks- Gunnn, Fong-ruey Liaw, and Pamela Kato Klebanov. Effects of early inter- vention on cognitive function of low birth weight preterm infants. The Journal of Pediatrics, 120(3):350-355 1992.
[12] Patrick Chao, Patrick Blöbaum, Sapan Patel, and Shiva Prasad Kasiviswanathan. Modeling causal mechanisms with diffusion models for interventional and counterfactual queries. In Transactions on Machine Learning Research, 2024.
[13] Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F, Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks. In Machine learning for Healthcare Conference, 2017.
[14] Jonathan Crabbé, Alicia Curth, Ioana Bica, and Mihaela van der Schaar. Benchmarking heterogeneous treatment effect models through the lens of interpretability. In Advances in Neural Information Processing Systems, volume 35, pages 12295-12309, 2022.
[15] Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. International Conference on Artificial Intelligence and Statistics, pages 1810-1818 PMLR, 2021.
11

---

[16] Alicia Curth and Mihaela Van Der Schaar. In search of insights, not magic bullets: Towards demystification of the model selection dilemma heterogeneous treatment effect estimation. In International Conference Machine Learning, pages 6623-6642, PMLR, 2023.
[17] Saverio D' D'amico, Daniele Oaio' Claudia Sala, Lorenzo Oall'Olio, Elisabetta Sauta, Matteo Zampini, Gianluca Asti, Luca Lanino, Giulia Maggioni, Alessia Campagna, et al. Synthetic data generation by artificial intelligence accelerate research and precision medicine in hematology. JCO Clinical Cancer Informatics, 7:e2300021, 2023.
[18] Rajeev H Dehejia and Sadek Wahba. Causal effects in non-cxperimental studies: Re-evaluating the evaluation of training programs. Working Paper 6586, National Bureau of Economic Research, June 1998.
[19] Rajeev H. Dehejia and Sadek Wahba. Propensity score- matremmathhing methods for nonexperimental causal studies. The Review of Economics and Statistics, 84(1):151-161, 02 2002.
[20] Digital Health Cooperative Research Centre (DHCRC). Synthetic data community of practice (SynD).
[21] Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition, 2018.
[22] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pages 265-284. Springer, 2006.
[23] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Founda- tions and Trends in Theoretical Computer Science, 9(3 4):211 19:211-407, 2014.
[24] Khaled El Emam, Lucy Mosquera, and Chaoyi Zheng. Optimizing the synthesis of clinical trial data using sequential trees. Journal of the American Medical Informatics Association, 28(1):3-13, 2021.
[25] Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional GANs, 2017.
[26] Fiona Katharina Ewald, Ludwig Bothmann, Marvin N. Wright, Bernd Bischl, Giuseppe Casalicchio, and Gunnar König, A Guide to Feature Importance Methods for Scientific Inference, page 440-464. Springer, Cham, Switzerland, 2024.
[27] Simon Ferreira and Charles K Assaad. Identifying macro causal effects in C-DMGs. arXiv preprint arXiv:2504.0/55/ 2025.
[28] Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958-9))) -68, Apr 2024.
[29] Dennis Frauen and Stefan Feuerriegel. Estimating individual treatment effects under unob- served confounding using binary instruments. arXiv preprint arXiv:2208.088: 2022.
[30] Dennis Frauen and Stefan Feuerriegel. Estimating individual treatment effects under un- observed confounding using binary instruments. In International Conference on Learning Representations, 2023.
[31] Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth. Jamie Hayes, Soham De, Samuel I. Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:2302. 13861, 2023.
[32] Andre Goncalves, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle, and Ana Paula Sales. Generation and evaluation of synthetic patient data, BMC Medical Research Methodol- ogy, 20:108, 2020.
12

---

[33] Aldren Gonzales, Guruprabha Guruswamy, and Scott R Smith. Synthetic data in health care: A narrative review, PLOS Digital Health, 2(1):c0000022 2023.
[34] lan Goodfellow, Jean Pouget-Abadigg Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of (he ACM, 63(11):139-144, 2020.
[35] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Reseurch, 13(1):723-773 2012.
[36] Scott M. Hammer, David A. Katzenstein, Michael D. Hughes, Holly Gundacker, Robert Schooley, Richard H. Haubrich, W. Keith Henry, Michael M. Lederman, John P. Phair, Manette Niu. Martin Hirsch, and Thomas C. Merigan. A trial comparing nucleoside monotherapy with combination therapy in hiv-infected adults with cd4 cell counts from 200 to 500 per cubic millimeter. New England Journal of Medicine,
1996.
[37] Lasse Hansen, Nabeel Seedat, Mihaela van der Schaar, and Andrija Petrovic. Reimagining synthetic tabular data generation through data-centric AI: A comprehensive benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2023.
[38] Jason Hartford, Greg Lewis, Kevin Seyton-Brown and Matt Taddy. Deep iv: A flexible approach for counterfactual prediction. In International Conference on Machine Learning, pages 1423. PMLR, 2017.
[39] Tobias Hatt, Jeroen Berrevoets, Alicia Curth, Stefan Feuerriegel, and Mihaela van der Schaar. Combining observational and randomized data for estimating heterogeneous treatment effects, 2022.
[40] Erik Hermansson and David Svensson. On discovering teeatment-efee modifiers using virtual twins and causal forest ML in the presence of prognostic biomarkers, pages 624-640. Springer, Cham, Switzerland, 09 2021.
[41] Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Compula- tional and Graphical Statistics, 20(1):217-240, 2011.
[42] Keisuke Hirano and Guido W. Imbens. The Propensity Score with Continuous Treatments, chapter pages 73 84. John Wiley & Sons, Ltd, 2004.
[43] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-668 6851, 2020.
[44] Paul W Holland. Statisties and causal inference. Journal of the American statistical Association,
81 81(36))9945 1986.
[45] Naoise Holohan, Stefano Braghin, P61 Mac Aonghusa, and Killian Levacher. Diffprivlib: the IBM Differential Privacy Library. arXiv: /907002444 2019.
[46] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlin- ear causal discovery with additive noise models. Advances in Neural Information Processing Systems, 21, 2008.
[47] Alihan Hüyük, Qiyao Wei, Alicia Curth, and Mihaela van der Schaar. Defining expertise: Applications to treatment effect estimation. In International Conference on Learning Repre- sentations, 2024.
[48] Aryan Jadon and Shashank Kumar. Leveraging generative Al models for synthetic data generation in healthcare: Balancing research and privacy. In 2023 International Conference on Smart Applications, Communications and Networking (SmartNets). IEFF, 2023.
[49] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC- III: freely accessible critical care database. Scientific Data, 3: 3:160035, 2016.
13

---

[50] James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data-what, why and how? WwXiv=2205.032555 2022.
[51] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. PATE-GAN: Generating syn- thetic data with differential privacy guarantees. In International Conference on Learning Representationss 2018.
[52] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics, pages 2281-2290. PMLR, 2019.
[53] L. V. Kantorovich. Mathematical methods of organizing and planning production. Management Science, 6(4):366 422, 1960.
[54] Dhamanpreet Kaur, Matthew Sobiesk, Shubham Patil, Jin Liu, Puran Bhagat, Amar Gupta, and Natasha Markuzon. Application of Bayesian networks to generate synthetic health data. Journal of the American Medical Informatics Association, 28(4):801-811, 2020.
[55] Edward H Kennedy et al. Optimal doubly robust estimation of heterogeneous causal effects. arXiv:2004. 1tte2000044449,,, 2020.
[56] Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, 2022.
[57] Aneesh Komanduri, Xintao Wu, Yongkai Wu, and Feng Chen. From identifiable causal representations controllable counterfactual generation: survey on causal generative modeling, 2024.
[58] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling tabular data with diffusion models. In International Conference on Machine Learning, 2023.
[59] S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
[60] Sören R Künzel, Jasjeet S Sekhon. Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 1 116(10):41664166 2019.
[61] Robert J LaLonde. Evaluating the econometric evaluations of training programs with experi- mental data. American Economic Review, 76:604-62 620, 1986.
[62] Dongha Lee, Hwanjo Yu, Xiaoqian Jiang, Deevakar Rogith, Meghana Gudala, Mubeen Tejani, Qiuchen Zhang, and Li Xiong. Generating sequential electronic health records using dual adversarial autoencoder. Journal of the American Medical Informatics Association, 27(9): ((99)1411 1419, 2020.
[63] Fan Li, Laine Thomas, and Fan Li. Addressing extreme propensity scores via the overlap weights. American Journal of Epidemiology,
(1::20025277 09 2018.
[64] Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37511,14 1991.
[65] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference Learning Representations, 2023.
[66] Terrance Liu. Giuseppe Vietri, and Steven Z Wu. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems, 34:690-702, 2021.
[67] Christos Louizos, Uri Shalit, Joris Mooij. David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Advances Neural Information Processing Systems, 2017.
14

---

[68] Yuchen Ma, Valentyn Melnychuk, Jonas Schweisthal, and Stefan Feuerriegel. DiffPO: A causal diffusion model for learning distributions of potential outcomes. Advances in Neural Information Processing Systems, 2024.
[69] Frank J Massey Jr. The Bolmogorov-Smmmon test for goodness of fit. Journal of the American Statistcal Association, 46(253)):8878,, 1951.
[70] Ryan McKenna, Gerome Miklau, and Daniel Sheldon. Winning the nist contest: A scalable and general approach to differentially private synthetic data, arXiv preprint Siti::2008.0977. 2021.
[71] Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. AIM]: An adaptive and iterative mechanism for differentially private synthetic data. Proceedings of the VLDB Endowment, 1):2599 2022.
[72] Microsoft Startups. South Australian Health: Synthetic data enables safe EHR data sharing, synthetic-data
2023, http:://www.mmccoooff.coommm safe-chr-datt--saaring/
[73] Hajra Murtaza, Musharif Ahmed, Naurin Farooq Khan, Ghulam Murtaza, Saad Zafar, and Ambreen Bano. Synthetic data generation: State of the art health care domain. Computer Science Review, 48:100646, 2023.
[74] Kenneth R Niswander. The collaborative perinatal study of the national institute of neurological diseases and stroke. The Woman and Their Pregnancies, 1972.
[75] Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11(10):1011-1083 June 2018.
[76] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009.
[77] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, seikít-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[78] Maya L Petersen, Kristin E Porter, Susan Gruber, Yue Wang, and Mark van der Laan. Diagnosing and responding to violations in the positivity assumption. Statistical Methods in Medical Research, 21(1))31-54 October 2010.
[79] Zhaozhi Qian, Bogdan- Constantin Cebere, and Mihaela van der Schaar. Synthcity: facilitating innovative use cases of synthetic data in different data modalities, 2023.
[80] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
[81] Kara E Rudolph, Catherine Gimbrone, Ellicott C. Matthay, Iván Díaz, Corey S Davis, Katherine Keyes, and Magdalena Cerdá. When effects cannot be estimated: Redefining estimands to understand the effects of naloxone access laws. Epidemiology, 33(5):689-698, 2022.
[82] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, 2018
[83] Pablo Sánchez-Martin, Miriam Rateike, and Isabel Valera. VACA: Designing variational graph autoencoders for causal queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8159-8168, 2022.
[84] Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085, 2017.
15

---

[85] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in Neural Information Processing Systems, 32, 2019.
[86] Tao Shi and Steve Horvath. Unsupervised learning with random forest predictors. Journal of Computational and Graphical Stativlics, 15(1):118-138, 2006.
[87] Yishai Shimoni, Ehud Karavani, Sivan Ravid, Peter Bak, Tan Hung Ng. Sharon Hensley Alford, Denise Meade, and Yaara Goldschmidt. An evaluation toolkit to guide model selection and cohort definition in causal inference, 2019.
[88] Yang Song, Jascha Sohh-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[89] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT Press, Cambridge, MA, 2001.
[90] Amirsina Torfi, Edward A Fox, and Chandan K Reddy. Differentially private synthetic medical data generation using convolutional gans. Information Sciences, 586:485-500, 2022.
[91] Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. npj Digital Medicine, 3:147, 2020.
[92] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing. 10:3152676:10-5555 5555, 2017.
[93] David S. Watson, Kristin Blesch, Jan Kapar, and Marvin N. Wright. Adversarial random forests for density estimation and generative modeling. 2023.
[94] Felix Nikolaus Wirth, Thierry Mcurers, Marco Johns, and Fabian Prasser. Privacy-rreseriing data sharing infrastructures for medical research: systematization and comparison. BMC Medical Informatics and Decision Making, 21:1-13, 2021.
[95] Robert F Wolff, Karel M Moons, Richard D Riley, Penny F Whiting, Marie Westwood, Gary S Collins, Johannes B Reitsma, Jos Kleijnen, Sue Malletr, and PROBAST Group. PRÓBAST: A tool to assess the risk of bias and applicability of prediction model studies. Annals of Internal Medine, 170(1)551-58 2019.
[96] Liyang Xie, Kaixiang Lin. Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network, 2018.
[97] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Advances in Neural Information Processing Systems, 2019.
[98] Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D Mooney, and Bradley A Malin. A multifaceted benchmarking of synthetic electronic health record generation models. Nature Communications, 13(1):7609, 2022.
[99] Jinsung Yoon, Lydia N Drumright, and Mihacla van der Schaar. Anonymization through data synthesis using generative adversarial networks and Health Informatics, 24(8):2378-22 2288, 2020.
IEEE Journal of Biomedical
[100] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthík Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cor- mode, and Ilya Mironov. Opacus: User-friendly differential privacy library PyTorch. arXiv: 12298, 2021.
16

---

A Example synthetic data papers using medical data containing treatments
Here, we detail selected works in the synthetic data literature that do not consider the downstream task of causal inference. Note that we do not claim that this list is exhaustive, as this is pervasive problem in the synthetic data literature, and we mean only to provide few examples here demonstrate this problem and provide motivation for our paper.
A.1 MedGAN
In the paper 'Generating Multi-label Discrete Patient Records using Generative Adversarial Networks' [13], the authors propose a GAN-based approach to generate 'realistic synthetic patient records'. In doing so, they experiment on multiple datasets containing treatments, including one from Sutter Palo Alto Medical Foundation (PAMF), which consists of longitudinal medical records of 258,000 patients, as well as the MIMIC-III dataset 49, which includes 46,000 intensive care unit patient records. Both datasets include treatments administered to patients, and they therefore invite downstream analysts to conduct causal inference tasks, such as treatment effect estimation.
Nevertheless, in this paper, standard generation and evaluation practices are followed, not differen- tiating between covariates, treatments, and outcomes. In particular, the evaluation protocol relies on marginal comparisons and predictive utility assessment, which offers limited information as to how well the generation method, medGAN, produces useful data for causal inference. Furthermore, medGAN itself draws samples directly from the joint distribution, not mimicking the DGP of treatment data, or optimising for the distributions most important for causal inference.
A.2 TabDDPM
The paper "TabDDPM: Modelling Tabular Data with Diffussion Models' (58] proposes a diffusion- based tabular data generation method. While not explicitly geared towards medical data, this paper does use medical data with variables that could be seen as treatments in its experiments. It, therefore, least implicitly, positions itself work on data containing treatments, and invites users conduct generation with TabDDPM on such data. Specifically, the cardiovascular disease dataset from https disease dataset is used, and
//ww.kaggle com/ htt///daassts/eeeee /sslllllaaa waniiaaaaa////a http//aaddiviuuuuucc this data could be analysed via causal inference by setting 'physical activity' as treatment, to estimate its effect on cardiovascular disease.
However, in this paper, only standard evaluation and generation methods are used, and the needs of downstream analysts pursuing causal inference tasks predictive utility measures, and the TabDDPM method generates all variables in a sample simultane- ously, not optimising for the distributions most important for causal inference.
acknowledged. Evaluation involves only
A.3 GReaT
The paper 'Language Models are Realistic Tabular Data Generators' [10] proposes an LLM-based generator. Similar the TabDDPM paper, this paper is not explicitly geared towards medical data, but it demonstrates on medical data containing treatments, thereby implicitly condoning its use on this type of data. Specifically, the dataset sick from https: //www//wwww//oo openml. org/ sttrcc?ttttte Srtabsort=runskide taw.usss.--tttttttsssaaaaaaeee
demonstrated on, which could be analysed via causal inference assess the effect of "thyroxine' or 'antithyroid' treatments. Nevertheless, once again, this paper does not consider downstream analysis involving causal inference, only evaluating its GReaT method with predictive utility metries and a discriminator score.
A.4 Benchmarking process for synthetic electronic health records
Finally, the paper 'A Multifaceted benchmarking of synthetic electronic health record generation models' proposes a benchmarking framework for on synthetic electronic health record (EHR) data [98]. Naturally, EHRs will include treatments administered to patients, and they will likely be analysed with treatment effect estimation in mind. In the proposed benchmarking framework. the evaluation procedures-includiing marginal comparison, correlation comparison, and predictive do not differentiate between covariates, treatments, and outcomes, or acknowledge the needs of downstream analysts conducting causal inference.
17

---

B Extended literature review
To provide useful context for readers, extend literature review here.
Evaluation. We extend the synthetic data evaluation practices summarised in Table Thre..
Marginal comparison. Assessing the distributional distance between synthetic and real marginals is often used to offer quantitative assessment of how well individual variables modelled [98 9 32]
The distance function d to conduct this can be set from a variety of choices, including including KL divergence [59], -ansennShannon distance [64]. Wasserstein distance [53], Catmoogrrv--mminnne
score 1691, MMD 35 and many more.
Correlation matrix comparison. Correlation- baasation--baed assessment can offer a sense of how well inter- dependencies between variables are modelled synthetic data (73. This commonly involves calculating synthetic and real 2-way correlation matrices, and assessing their difference, by setting d as distance such as Frobenius norm [32 and absolute error (54).
Joint distribution comparison. Metries based on notions of statistical divergence can offer a means of quantifying how different the entire joint distributions of real and synthetic data are [99 9 [0]. The distance function d can be set to largely the same family of functions as in the marginal comparison
case.
Precision and recall analysis. Precision and recall, originally proposed for generative model as- sessment in [22 measure if generated samples covered by real samples, and vice versa. Alpha precision and beta recall [2] are refined versions of the original metrics, which account for the densities of the real and generative distributions, rather than just comparing supports.
Discriminator performance. Discriminator performance is a slightly unique evaluation practice, involving 'discriminator', which predicts whether instances are synthetic real, where poor performance of the discriminator indicates realism the synthetic data [54 62 10
Predictive utility. Predictive utility metrics offer a practical evaluation of synthetic data by quantifying the performance of synthetically-train -triinne predictive model. The "train on synthetic, test on real" (TSTR) paradigm [25] is the common approach to such assessment, measuring the accuracy of synthetically-traiee trainedd model in predicting target label on a real test set.
Generic generative models. We expand on the generic generative modelling paradigm outlined in Table 2 by describing specific existing generative models which adhere to it. These models approximate the real joint distribution using diverse range of techniques.
GAN-based models. GANs (34 consist of a generator and discriminator network, which are trained adversarially generate and identify synthetic data, respectively, until the samples are realistic. Originally proposed for image generation, GANs have been adapted to tabular generation, and there are many methods that adopt this popular architecture (c.g., CTGAN (977, TableGAN (75)), including those specifically designed for medical data (e.g., MedGAN (3).
VAE-based models. VAEs [56] are another common architecture, which learn to encode data into lower-dimensional latent space and then decode it back to reconstruct the original data. They generate new data samples by sampling from the latent distribution and decoding these samples, and their application to tabular data involve techniques to handle mixed data types (e.g.. TVAE (97)), and regularisation for improved robustness (e.g., RTVAE
Diffuusion cittusioo-based models. Diffusion models [43 88 learn the gradient of the data distribution, and generate data via progressive denoising, beginning with noisy sample and using neural network to predict and remove noise over a number of timesteps, including for tabular data (e.g. TabDDPM (58)). Extensions for direct treatment effect estimation exist [68], but not for synthetic data generation for this purpose.
Forest-based models. Random forests can estimate the density of probability distribution, as leaf nodes partition the data space into distinct hyper-rectangles with estimated densities of the proportion of samples that fall into them. Samples can then be drawn from this estimated density. Random forests can easily handle heterogeneous data types, so their application to tabular data synthesis natural. They particularly fast to train and generate from [86] 93].
18

---

Normalizing flow-based models. Normalizing flows estimate the target density by transforming tractable density (e.g., Gaussian) into the target through a series of invertible transformations, called 'flows'. Probabilities from the target distribution can then be found using the change of variables formula (e.g. [80). Recent work has shown their theoretical similarity to diffusion models [65].
Causal generative models. Causal generative models [57 9 12 are class of generative models, distinct from generic tabular data generators, that approximate the underlying structural causal model (SCM) (76) of dataset. While such models are related to our work with STEAM and are likely to better preserve causal relationships than generic generators in settings where they can be used, their assumptions can restrict their practical use cases. In comparison to STEAM. they generally differ in terms of their (1) assumptions, (2) motivation, and (3) ficxibility, which we detail here.
(I): Importantly, causal generative models typically assume that the data holder has knowledge of the entire causal graph of the real data, which is more restrictive assumption than we make in this work. We assume that our specification of the underlying DGP X ~ Px, W ~ PwIx, Y ~ Py correct for datasets containing treatments, but do not require knowledge of the causal graph, we do not need to know the causal links between individual variables. We do not assume knowledge of causal relationships amongst the covariates, nor knowledge of which covariates cause treatment assignment or patient outcomes. As such, we make less restrictive assumptions than works that require knowledge of causal graph, and we suggest that our approach more realistic in complex, real-world scenarios, such those that arise in medicine, where the true causal graph is unlikely to be available.
w;x is
(2): The motivation for causal generative models is typically to allow the generation of data to answer graph- specific interventional and counterfactual queries that require knowledge of the full causal graph. With STEAM, however, we seek to generate useful synthetic data only from the observational distribution, for use by analysts with goals such as treatment effect estimation (e.g.. CATE).
(3): STEAM's design can, in principle, incorporate any generative model for Qx essentially acting wrapper around Qx to improve its generation quality for causal inference tasks. This allows STEAM to very easily empower many existing generative modelling frameworks, without having to incorporate bespoke generators. Also, allows STEAM models to continuously improve along with the base generative model. Existing causal generative models do not generally allow such flexibility, and therefore cannot as easily benefit from future improvements in underlying generic generative models.
Despite these differences, conduct empirical comparisons between STEAM and some baseline causal generative models in Appendix M
Privacy. Despite the popularity of some of the above generators, memorisation of training samples a phenomenon observed in generative models (31) Therefore, provably private generation is often desired to limit the amount of information leaked. Differential privacy 22 is the most common standard adopted, and there are multiple generators which guarantee this, including GAN-based methods (e.g., PATE-GAN [5]), DP-GAN (66) and query-based methods (e.g., GEM (66. MST
(70] RAP 5
AIM 71
19

---

C Theorem 1 proof
Theorem 1. Let P. Qh, Q² be of the form described in $5.1 and M be KL divergence. If assume that Q and Q have sufficient capacity to have bounded error each component, i.e. Vi,
0 < <MMxX Oxx,) Ex, and O MMPwixx @wix ew, and M(Py (MMYy w, Q7 Q
Ey,k. then:
M(P.Q")) M(P, Q=2)
1, as + 00
Proof. From the factorizations of P and Q, KL divergence decomposes:
( = Σ (xx, +Exx (Ra(Pwixll@WW +Bpx.w [DKL(Pywwxxl
(8)
As such, the following holds for when KL divergence is set as M. Define the ratio:
R(d) M(P.QO)
M(P.Q%=)
Substituting the decompositions, we have:
R(d) E M(Px Ox + Epx
[M(Wwxx Ow.1
+ +Epx,
M(Priwxx. . Opiix))
M(Px Ox) Epx M(Pwix Q (w]x)
+ -Exx.w
M(Priwx (1(
As the dimensionality d increases, the marginal summations Σ M(Px 1xxx) grow linearly with d, since each M(/x, || is, by assumption, non-negative, and they therefore dominate the bounded conditional contributions:
Erx [MPWIxX OWI
EW.ks
E x
PY. M(PriwxX QY(.XX)
< <YY.A.
M(Px, 2x)
Thus, M(Px...r. Oxw,Y) Eim M(Px, 22) and R(d)
1.
Ei M(Px Rxx "x)
Therefore:
R(d) 1, as d +
20

---

D Empirical demonstrations of current metric failure
D.1 Failure to identify changes to the outcome generation mechanism
We demonstrate this with a simple experiment investigating how four Ds of size n = 1000, which only differ in their outcome generation mechanisms, assessed by an array of current metrics. We simulate D, from a simple DGP with 10 covariates with f'x = =N(00 1), Pw x = (0.5,.
Priwx= = N(W .x², 1). We generate four Dj with the same Q% d Px, O - Pwix, Vi E (1,2,3,4) 4). We vary each y ~ N(W +,(X, (1 W) $, (X, 0), 1) where $ represents a potential outcome (PO) estimator with the architecture from either an S-Learner, T-Learner [60], DragonNet [85], or TARNet (84. These four architectures will model Vv differently, inducing the only point of variation amongst the D.--
Since we simulate D,, we know the ground-truth treatment effects, and oracle metric can be established to determine the true quality each Dj We define this as the precision estimating heterogencous effects (PEHE) 4 of estimates from CATE learner trained on D and the ground- truth CATEs. In Table we report the scores of Pa. Rg [2], inverse KL divergence [59], Kolmogorov- distance (JSD) [64] on each D:, All report very similar scores across the D with most offering no statistically significant best option, suggesting that their quality is the same. The oracle metric, however, determines that D using a T-Learnr for & is a clear best, and Di with an S-Learner as t; is more than twice as bad at preserving the true treatment effects. Clearly, even in moderately sized dataset, these metrics cannot reliably identify changes in Q3 w despite the large effect that this distribution has on downstream performance.
Smirnov (KS) score [69]. Wasserstein distance (WD) [53], and
Table 5: Joint-distribttion-leeel metrics on Dj which differ in Q architecture only. Averaged
over 10 runs, with 95% Cls.
v Pa (1)
Rg (1)
Inv. KL (t)
KS (7)
WD (L)
JSD (L)
Oracle (4)
T-Leamer
0,927 + 0.001
0.584 + 0.006
0,947
0.000
0.979
0.000
0.002
0.000
0.002 + 0.000
0.525 = 0.012
TARNet
0,919 1: 0,002
0.573 1 0.005
0,950 1 0.006
0.985
0.001
0.002 : 0.000
0,002 : 0.000
0.616 = 0.015
DragonNet
0,921 + 0.001
0.574 0.004
0,947 0.000
0.984
+ 0.001
0.002 = 0.000
0.002
+ 0.000
0.618 = 0.007
S-Leamer
0,926 + 0.002
0.579
0.007
0,957 0.009
0.990
0.000
0.002 = 0.000
0.001 + 0.000
1.279 0.015
Comparison to UPENE- Since we only alter Qyjwx between each Di, all have the same xa.x, R3.x, and JSD,, In Table/owe report UPERE each dataset, and we see that it fully reproduces the oracle ranking, and correctly identifies the best dataset to a statistically significant level, which no existing metric could do.
Table 6: UPEnE on DÍ with varied Q Averaged over 10 runs, with 95% Cls.
Qriwx UrERE (L)
Oracle (1)
T-Learner
0.693 1 0.013
0.525 1 0.012
TARNet 0.731 0.016
0.616 0.015
DragonNet 0.754 0.019
0.618 0.007
S-Learner 0.906 0.019
1,279 = 0.015
D.2 Failure to identify changes to the treatment assignment mechanism
We conduct a similar experiment varying Qiv x across three Dí. We simulate D, ~ Px, from DGP with 5 covariates, all of which contribute to the propensity score. We set Px = \o 1). Pwix= = Bem(((()), x(X) (1 + e 1/5 E:, x.)-1. Priwxx NO. 1). We generate three
Dj- ~ Qx. which vary only in the degree to which they correctly model π T(X) by setting Q* d
Px. 2 Q Pr|wxx Vi E (,) and @ = Bern (X)) where m (X) (1 -X1-1
T2 (X) = + -1/3 X,y-1, and Ta (X) (1 + - In this way, know that,
21

---

Table correct var.: The number of correctly identified variables in the propensity score. Pa: a precision. Rg: B recall. Inv. KL: Inverse KL divergence. KS: Kolmogorov- score. WD: Wasserstein distance. JSD: SensennShannon distance. JSD,: Ours. Averaged over 10 runs, with 95%
Cls.
correct
Pa (t)
(t)
Iuv. KL
KS (t)
WD
JSD
JSD,
0.863
0.456 4: 0.017
+ 0.002
0,961
3
0.868 1. 0.022
0.457 1: 0.022
0,981 1 0.006
0.966
1 0.003
0.018
0.001
0.004
0.000
0.942 1 0.006
0.866 + 0.021
0.433 +: 0.013
0.985 1 0.003
0.965 : 0.003
0.018 0.001
0.004 + 0.000
0.908 + 0.013
in truth, Qx w.y is a better model than Qx w.y. W,,,yy which in turn is better than Qx,w.r- and we can now assess how well existing metrics, and our JSD, metric, recover this ranking.
We display the scores of Pa. R3. inverse KI., Kolmogorov-Smirov score, Wasserstein distance, Jensen-Shannon distance, and our metric JSD, on each D in Table 7 We see that the existing metrics report very similar scores actoss the three datasets, and none offer statistically significant best option. This contrasts with our JSD, metric, which correctly orders the three models and selects Qx.w,y the best option statistically significant level.
To further elucidate the differences rankings between existing and our metrics, both in this experiment and the outcome generation comparison in ! we list each ranking and their Spearman's 's rank correlation coefficient with the oracle ranking in Tables and | Assessment via our metrics is the only protocol that reproduces the oracle ranking across both experiments.
Table 8: Treatment assignment experiment: rankings by different metrics, sorted by Spearman's 's rank correlation coefficient (r,)) with oracle ranking. Numbering indicates the oracle order of n, (X)..
Metric
Ranking
Ta (1)
Pa
2,3,1
-0.5
KS
2,1.3
0.5
Inv. KL
1,3.2
0.5
R
2,1.3
0.5
WD
1,2,3
JSD
1,2,3
Table 9: Outcome generation experiment: Rankings by different metrics, sorted by Spearman's 's rank correlation coefficient (r.)) with oracle ranking. Qr are numbers by oracle ranking, T-Learner,
2: TARNet, DragonNet, 4: S-Learner.
Metric
Ranking
T (1)
KS 4,2,3,1
-0.80
Inv. KI.
4,2,1,3
-0.40
Pa 1,4.3,2
0.20
R3
1,4,3,2
0,20
WD
2,3,1,4
0.40
UPENE
1,2,3,4
D.3 Existing metric failure: extreme example
As 'proof by contradiction' that current metrics can offer good level of information on the preser- vation (ii-, we present some extreme examples. We show that joint-distributionleeee metrics do not have enough resolution to identify how well (i)-(ii) are preserved, even if Q comprehensively fails in modelling any one of the component distributions Px, Pwix. or Pr|wxx
We perform series of experiments where we evaluate adversarial synthetic versions of simulated dataset, with each synthetic version failing one of the above components, and we show that standard metrics do not identify these failure modes. We simulate real data using the DGP in CATENets
22

---

from [13 and we create three D, that perfectly model two component distributions of Dr but poorly approximate the remaining one. For poorly modelled Px. we set X 0; for poorly modelled Pwix, we assign all instances with W O; and for poorly modelled Py we draw Y from normal distribution with mean regardless treatment. All such D, are useless for treatment effect estimation.
Table 10: Scores on adversarially created D, which poorly perform on desiderata (1), (i), or (ii)..
Inv. KL (f) Pa (1)
Rg (t)
MMD (1)
Poor (i) 0.681
0,902
0.368
0.085
Poor (ii) 0.685
0.501
0.333
0.074
Poor (iii)
0.844
0,905
0.430
0.008
We report the inverse of KL divergence, Pa. Rg, and MMD, which all have range [0, 1]. for these synthetic datasets in Table I0 We see that these conventional evaluation metrics do not accurately reflect the invalidity of each D, for treatment effect estimation. None report significantly low scores, despite the failure of each Dz. R3 reflects these failures best, although its scores still do not adequately reflect how these datasets render correct treatment effect analysis impossible, and it does not allow granular enough analysis disentangle which component distribution poorly modelled.
In further detail, for the experiments that examine poor modelling of Px and Pwix, we simulate D, of size 1000 with d covariate as follows:
X ~ NO
W ~ (emoulli(0.5)
(10)
Y(O), (()) = 0
(11)
Y = (1 (1)))))(() + +Y(1+ 1)
(12)
We manufacture D, that exhibits poor modelling of Px by generating W and Y from the true distributions as above, but set X = O. For D; that exhibits poor modelling of Pwix, we generate X and Y from their true distributions, but set all
To demonstrate assessment under poor modelling of Pylwxx we set the covariate D, to be predictive, such that affects the value of the potential outcome (()), but not Y (0). The distributions remain the same the above, although now the potential outcomes
Y(0) Y()= X
(13)
(14)
We manufacture D, that poorly models of Pyjw.x by generating X and W from their true distribu- tions, but we set (()), (1) = O.
23

---

E Discussion on alternative metrics
While we propose set of metries M for evaluation of Ds. there are many possible alternatives to each choice we make. Our choices enable evaluation of how well D, adheres to our desiderata, but, like any metrics, they may be sub-optimal for certain data holders with specific preferences. Here, we list some alternative definitions, and we detail when they may be preferable. We would like to emphasise that conducting any reasonable assessment of the preservation of (i)-(ii) beneficial compared to standard evaluation practices.
E.1 Alternative covariate distribution assessment
As we state in the main paper, comparison of Px and Qxx is essentially a standard synthetic data evaluation problem, and therefore any standard protocol can be applied.
For example, if the dimensionality of X is small, manual evaluation via visualisation may be preferable to the precision/recall analysis we suggest, this can provide more granular and interpretable assessment. On the other hand, a single all-encompassing score is desired, rather than the two-dimensional metric (Pa.x. R3.x ). then statistical divergence metrics can offer this. These onc-dimensional metrics can lead to more straightforward model selection than (Pa.x. Rg,x), as ordering based on a two-dimensional metric can be ambiguous.
E.2 Alternative treatment assignment mechanism assessment
Similarly, there is a vast array of metrics which could be substituted into Eq. (5) over Jensen-Shannon distance, which could measure the difference between Pwix and Qwix. These include metrics such as KL divergence and Wasserstein distance, which are also very common in machine learning literature. For example, data holder may prefer KL divergence if they want Qwix for failing to place density where Pwix is probable, encouraging mode-covering behaviour. On the other hand, if data holder wants more harshly punish Qw x for spreading from the modes, Wasserstein distance may be preferable, leading to mode-seeking behaviour. JSD achieves balance between these focuses, but if data holder has strong preference for one or the other, these alternate choices would be preferable. Nevertheless, we suggest that, apart from extreme scenarios, most reasonable methods to assess the preservation of Pwix will lead similar analysis.
harshly punish
away
E.3 Alternative outcome generation mechanism metrics
Raw similarity of PEHE in CATE estimation between D, and D3 may not be the most important quantity of interest for certain data holders. This can be particularly true in medical practice, raw performance not the only important aspect of downstream model. We propose some alternatives that may be more applicable in the following situations:
1. Correct estimation of the sign of the CATE may be of heightened importance if the CATE learner is assisting with policy decisions. The wrong CATE sign will lead incorrect policy administration, whereas the magnitude of the effect may not be as important for decision-making.
2. Discovering the correct drivers of effect heterogeneity may be important, as how a learner arrives at its final estimation is particularly important to consider in applications such as in drug discovery or clinical practice [40
E.3.1 Policy assignment
If policy guidance is of interest, then quantification of how well the sign of CATE estimates is preserved between D, and D, may be desired, which can be done as follows:
Upolicy Drr D.) =
Σ PEF
Exx (Fsymthh (X) x Freat() > 0))
(15)
F
where / is the indicator function.
24

---

E.3.2 Feature importance
If assessing how well D, permits the discovery of the correct drivers of effect heterogeneity is important, this can be quantified through the use of feature importance methods. Given CATE leamer 7, feature importance methods offer a means to measure the sensitivity of the model to each covariate by assigning an importance score o; (f, x) to each feature I; that reflects its importance in the prediction of the CATE F Fxx). There are many different instantiations of feature importance methods with different strengths [26], and the metric we propose here is method-agnostie. We
quantify how well py is modelled according to feature importance similarity between D, and Dr as follows:
Um (D, D.)
Σ PEF
So (peaal, Asymh,+))
(16)
F
where Se is cosine similarity, and Areal and Asyath, are d-dimensional vectors with ith entries
A Exx lai X)], {real, synth)
(17)
Unt [-1, 1). where Unt indicates total agreement in the feature importances of Dr and D.. while Unn = 0 indicates that the feature importances are uncorrelated, suggesting that Qrjw. does allow discovery of the correct drivers of heterogeneity. Finally, Unt 1 indicates antithetical feature importances, suggesting a drastic failure of Qrjwx in estimating Py Py]w:x.
25

---

Defining F for UPEHE
In CATE estimation, model validation is difficult task [6] As such, is reasonable expect that set of downstream analysts conducting CATE estimation on D, will use different learners. Therefore, we want UPENE reflect the expected difference downstream performance between D, and D, across diverse array of potential learners, such that it is representative for the entire population of analysts, and has limited bias towards any particular learner class. achieve this, we propose averaging UPEhE across family of CATE learners F, and we suggest that larger [F] and diverse selection of the learners within preferable.
Of course, there is trade-off between the size of F, and therefore the stability of Upehe, and the computational cost of repeated CATE estimation. With this in mind, to limit the computation involved in calculating UPEhE, we suggest that users should be selective of the learners included in J to maximize learner diversity, and minimise FFI For example, our experiments, we set | and we chose learners from both of the high-level CATE learning strategies described in [5] (i.e., one-step plug-in learners, and two-step learners). Specifically, for the one-step learners we use S- and T-learners [60]. and for the two-step learners we use RA- and DR-learners [55]. All four of these learners conduct CATE estimation differently, and encode different inductive biases in their approaches, and thus they form good diverse base for F.
For our experiments, on each of the real datasets, the runtime for calculating UPERE for one run is shown in Table 11 Note that these are much less than the typical generation times for each dataset, so this step is unlikely be large time burden for the data holder. Also note that these calculations can be parallelised across the learner classes, which we did not do, and this can improve the computational feasibility of using a larger [Fl
Table 11: Runtime to calculate UpEhE
Dataset UPERE runtime (s)
ACTG
26
IHDP
60
ACIC
191
26

---

G STEAM diagram
See Figure
D for flowcharts of the generic DGP for synthetic data generation, real datasets containing treatments, and STEAM. STEAM is designed to closely mimic the real DGP.
Standard Syuthetic DGP
STEAM DGP
Coudition
Real DGP
X
W
X W
Outgut data.
X
X
W
X
WY
Training data.
Real dala
X.I.
Qx
Quix
Syuthetic datal
Px
Pwr
DY
Distribation
X
W
y
W
Generative model
Figure 4: DGPs for generic generative models (left), real datasets (middle), and STEAM (right).
27

---

H Main experimental details
Here we add any additional details to the experiment set-ups from 7 All experiments were run on an Azure VM with 48-Core AMD EPYC Milan CPU and A100 GPU. We report typical runtimes where relevant. An estimated total compute time for all experimental runs is ~72 hours. This does not include the compute required for preliminary experimentation.
For all generative models, we use the open source library synthcity 79 (Apache-2.0 License), and we do not change the default hyperparameters. We set the treatment and outcome generators of STEAM using a logistic regression function from scikit-learn 777 and S-Learner from CATENets
15 respectively.
We release code here btstp:///ggttab..omm/arrrrr///// vanderschaar waw.errscchaaaabSSTAAAA aab/STEAM
and here https: com/
H.1 Generation of medical data containing treatments
To assess sequential generation in number of real-world scenarios, evaluate performance
ACTG [36] and on the popular treatment effect estimation datasets IHDP [41] and ACIC [21]. We also report further results in Table 12 non- non-medical dataset, Jobs [61, which is also popular amongst the treatment effect estimation community, to show that STEAM can be applied beyond the medical context, to any dataset containing treatments. More in-depth descriptions of the datasets used are here:
1. AIDS Clinical Trial Group (ACTG) study 175. A clinical trial on subjects with HIV-1 [36]. Preprocessed as in [39] to compare CD4 counts at the beginning of the study and after 20 = weeks across treatment arms using zidovudine (ZDV) and zalcitabine (ZAL) vs. ZDV only. The ACTG dataset contains n = 1056 instances with d = 12 covariates and a continuous outcome, and we use the publicly available version from /tttps//ggiiin http://..o/ooootottoooee http:/bbbhtttcoccce
com
Infant Health and Development Program (IHDP). semi-synthetic medical dataset, with real covariates and simulated outcomes, using data from randomised experiment designed to evaluate the effect of specialist childcare on the cognitive test scores of premature infants [1. Confounding and treatment imbalance were introduced in [41] to mimic an observational dataset. The IHDP dataset consists of n = 747 instances with d 25 covariates and continuous outcome. We use the publicly available version from https: /e::::/itttte g/gggtttuu..... com/
AMLab- Amsterdam Ceteraamm///EEEE.e CEVAE 67 with the first batch of simulated outcomes.
3. Atlantic Causal Inference Competition 2016 (ACIC). A semi-synthetic medical dataset, with real covariates and simulated outcomes, containing data from the Collaborative Perinatal Project (74). The data was modified in (21 to simulate an observational study examining the impact of birth weight in twins on IQ. The ACIC dataset consists of n = 4802 instances with d = 58 covariates and continuous outcome. We use the publicly available version from the causallib package [87 (Apache-200 License) available here https: :ttt:p://githul com/ using the first simulated set of treatments and potential outcomes.
$iomedSciAA/causaall
Jobs. Jobs contains experimental data from male sub- sampae from the National Supported Work Demonstration from [6]] to evaluate the effect of job training on income. The Jobs dataset consists of 722 instances with d = 7 covariates and a continuous outcome. We use the
publicly available version used in 18 19 from https: //upers./ neeerrrss.bb.........ee.. org/ ree//dddeeeaa
We report extended results for all models tested, and further results on the Jobs dataset, in Table 12 For each model on each dataset, conduct 20 runs. typical run for given real dataset and generative model took 15 minutes.
H.2 Simulated experiments
For our simulated insight experiments, we compare the performance of a standard TabDDPM with STEAMTa0DPPN. and we report average results over 10 runs. A typical run took 5 minutes. For
simulation of D,, we use the DGP from CATENets [5]]
28

---

H.3 Differentially private generation
For our experiment, which showcases the performance STEAM when satisfying DP (Appendix N we compare the generative performance of baseline methods AIM [71] GEM [66], MST [70), RAP [5] with their STEAM counterparts. We use the code provided by 71 their GitHub https //github. coa.bbanmm/rynnn1 12358/ pttt:p-vvittppmmmmmm for the AIM and MST implementations, and we use for the GEM and RAP implementations. We use the default hyperparameter settings of these implemen- tations, with the workload set as 3-way marginals. For the STEAM models, we use the relevant base model for Qx. DP random forest from the diffprivlib library 45 (MIT License) for Qw x. and a custom implementation of T-Learner [60] based on [5] which guarantees DP by training with DP stochastic gradient descent, implemented with the Opacus library [T00] (Apache-2.0 License). We report comparative results on varied c, averaged over runs.
the code provided in the GitHub httptb//ttsssstttieeee com/ http://eraaceelddppeeeeeeeeeee
29

---

I
Extended results
We report the full set of results for each model and dataset from 7.1 in Table 12 including on the Jobs dataset. We pair each standard model with its STEAM analogue, and report the relative difference between them for each metrie, where (green) indicates better performance by STEAM, and (red) indicates better performance by standard modelling. We see that STEAM clearly outperforms. Almost all STEAM models perform better in each metric than all standard models.
Table 12: Pa.x R ,SD,, and UPEhE values for STEAM and standard models. Averaged over 20 runs, with 95% confidence intervals. Each STEAM model is placed after its corresponding standard model. Coloured numbers in brackets indicate relative difference between standard and STEAM model, where (green) indicates better performance by STEAM, and (red) indicates better performance by standard modelling.
Dataset
Model
Pa,x (t)
R3,x (1)
JSD, (t)
Uran (4)
ACTG
0.926 +
0.946 1:
0.564 1: 0.017
STEAM TWAE
0.929 + 0.008 (+0.003)
0.486 + 0.009 (+0.003)
0.958 1: 0.004 (+0.012)
0.492 + 0.011 (.0.072)
ARF
0.818 + 0.012
0.453 = 0.007
0.960 + 0.004
0.577 + 0.015
STEAM
0.836 1 0.008 (+0.018)
0.464 1 0.007 (+0.011)
0.962 1 0.004 (+0.002)
0.423 1 0.016 (:0.15))
CTGAN
0.889 + 0.020
0.444 + 0.014
0.934 + 0.008
0.586 + 0.017
STEAM CTGAN
0.892 0.017 (+0.003)
0.137 = 0.012 (-0.007)
0.959 +
(+0.025)
0.436 + 0.012 (-0.150)
NElow
0.817 + 0.032
0.418 = O. 008
0,913 + 0.016
0.643 + 0.026
STEAM NFWS
1 (+0.020)
0.417 : 0.015 (:0.001)
0.962 1
(+0.049)
1
(.0.198)
TahDDPM
0.067 + 0.060
0.036 + 0.035
0.812 + 0.029
1.761 + 0.230
STEAM TRUENPM
0.009 - 106 (+0.542)
0.310 = 055 (+0.274)
0.952 + 0.009 (+0.140)
0.468 + (-1.299)
IIDP
CTGAN
0.663 1 0.018
0.419 1: 0.013
0.888 1: 0.010
2.521 1. 0.161
STEAM CTGAN
0.674 + 0.014 (+0.011)
0.424 + 0.011 (+0.005)
0.928 + 0.009 (+0.040)
1.709 : 0.052 (-0.812)
TabDDPM
0.477 = O. 036
0.340 = 0.022
0.862 + 0.011
2.706 + 0.18
STEAM THDIPM
0.553 0.029 (+0.076)
0.015 (+0.056)
+
(+0.056)
+
(-0.360)
ARF
0.528 + 0.009
0.381 + 0.010
0.921 1 0.009
3.019 1 0.117
STEAM ARE
0.565 = 0. 0.011 (+0.037)
0.394 = 0.010 (+0.013)
0,921 +
(+0.000)
1.629 + 0.056 (-1.390)
TVAE
0.622 + 0.014
0.410 + 0.010
0,880 + 0.014
3.198 + 0.172
STEAM TYXE
0.629 1 0.015 (+0.007)
0.412 1 0.011 (+0.002)
0.927 1 0.007 (+0.047)
2.100 1. 0.075 (.1998
NFlow
0.406 0.028
0.309 0.012
0.882 + 0.012
3.835 + 0.345
STEAM
0.135 = 0.034 (+0.029)
0.333 = 0. 020 (+0.024)
0.921 + 0.007 (+0.039)
2.177 + 0.118
ACIC
TVAE
0.901 1 0.014
0.513 1: 0.004
0.929 1: 0.005
4,223 1: 0.138
STEAM TWAE
0.900 : 0.014 (-0.001)
0.514 + 0.004 (+0.001)
0.972 + 0.002 (+0.043)
2.422 + 0.1 (-1.801)
CTGAN
0.880 = 0.016
0.421 = 0. 010
0.942 + 0.005
4.518 + 0,186
STEAM CIGAN
0,873 = 0.014 (-0.007)
0.424 = 0.014 (+0.003)
0.972 + 002 (+0.030)
2,268 + 0.154 (-2.250)
ARF
0.828 1 0.002 0.003
0.430 1 0.002
0.945 1 0.
4.633 1 0.146
STEAM ARE
0.835 : 0.004 (+0.007)
0.430 + 0.004 (+0.000)
0.977 1 0. (+0.032)
2.449 1 0.1 (-2.184)
NFlow
0.748 0.019
0.333 0.014
0,838 + 0.035
5.068 + 282
STEAM
0.744 1 0.021 (-0.004)
0.333 : 0.010 (+0.000)
0.971 1: 0. (+0.133)
2.938 1: 0.1 (-2.130)
TabDDPM
0.124 + 0.028
0.002 - 1
0.813 + 0.023
9.281 + 1.033
STEAM TEUEMM
0.141 = 0.035 (+0.017)
0.002 = 000 (+0.000)
0.955 + 0.019 (+0.142)
1.497 + 0.501 (-4.784)
Jobs
TadDDMM
0.890 + 0.014
0.177 = 0.011
0.9-19 + 001
3.335 + 0.516
STEAM TantrM
0.929 : 0.009 (+0.039)
0.493 + 0. 0.008 (+0.016)
0.954 + O. 0.003 (+0.005)
1.446 + 0.052 (-1.889)
ARF
0.832 0.010
0.431 0.019
0.961 + 0.004
3.173 + 0.691
STEAM
0.863 = 0.011 (+0.091)
0.481 + 0.016 (+1.050)
0.953 + DUL (-0.011)
2,280 1 0.381 (-0.893)
1
1
1
1 0.336
STEAM
0.887 1 0.014 (+0.001)
0.300 + 0.012 (+0.012)
0.949 4 O. 000 (+0.005)
1.540 + 167 (-2.931)
CTGAN
0.830 = 0.0:99
0.339 -
0.925 + 0.033
4.608 + 0.792
STEAM
0.778 = U. 077 (-0.052)
0.298 = 0. 030 (-0.041)
0.939 4 007 (+0.014)
1.846 + 0.270 (-2.762)
NFlow
0.716 = O 058
0.374 - 0.017
0.920 + 0.018
5.445 + 0.883
STEAM NFOA
(+0.084)
0.375 = (+0.001)
0.952 4 (+0.032)
2.666 + (-2.779)
30

---

J
Ablation study
To add to the evidence of STEAM's efficacy, we conduct an ablative study by assessing how jointl modelling Px.w affects performance. On the medical datasets used in the main body of the paper, we compare performance of the best standard models with their relevant ablation STEAM j.int X.W. which models Px, w with the generative model and Py] with PO estimator, and regular STEAM. We report the results in Table 13
We see that the ablative model, while often improving upon standard generation, is not as effective as STEAM. Directly modelling Pwix. as STEAM does, better preserves the treatment assignment and outcome generation mechanisms, and both JSD,, and UPERE are significantly improved by STEAM in most cases. Using the full inductive bias of directly modelling each distribution of our desiderata, and following the true DGP of data containing treatments is the best approach to generation.
Table 13: X, Rg.x, JSD,,, and UPERE values standard, ablation, and STEAM models. Ablation results averaged over 5 runs, with 95% Cls, other results use 20 runs.
Dataset
Model
Pa,x (1)
Rg.x (1)
JSD (t)
UpERE (4)
ACTG
TVAE
0.926 0013
0.483 +: 0.010
0,946 0.004
0.564 +: 0.017
STEAM TVAE, X.W (abiation)
0.918 0.021
0.473 + 0.012
0,939 = 0.010
0.475 + 0.012
TVAP
+ 0.009
0.492 - 0.011
0.419
0.010
2.521 +: 0.161
STEAM jii (ablation)
0.639 0.021
0,428 + 0.009
0,908 = 0.019
2.140 + 0,134
STEAM CIGAN
0.674 0.014
0.424 + 0.011
0.009 0,928
1.709 + 0.052
ACIC
TVAE
0.901 +: 0.014
0.513 1: 0.004
0.929
0.005
4.223 :: 0.138
STEAM JWAR (abiation)
0.873 0.022
0.512 + 0.010
0.930
0.019
2.447
= 0.249
STEAM
0,900 + 0.011
0.511 + 0.001
0.972 0.002
2.422 = 0.118
J.1
Qwix ablation
We conduct a further ablation study to identify how setting different classifiers as Qwix x within STEAM models affects performance. On the IHDP dataset, we compare setting Qwix with logistic regression classifier, as is done in the main experimental section, against random forest classier. We report the respective JSD,, scores for each model in Ta] We can see that there is no significant difference between how each classifier preserves the treatment assignment mechanism in IHDP.
Model selection for QwIx is a much smaller priority than selecting Qx which can significantly affect all metrics, as shown in Table 3
Table 14: JSD, values for STEAM models with different classifiers for Qwix-
Dataset
Model
JSD, (t)
IHDP
STEAM CTGAN, log.reg.
0.928 + 0.009
STEAM CIGAN,
0.919 + 0.020
STEAM Ta6DDPM, log.reg.
0.918 + 0.011
STEAM TabDDPM, rf.
0.931 0.015
STEAM ARF. log.
0,921 = 0.009
STEAM ARF, rf.
0.922 + 0.016
STEAM TYAE, log.rog.
0.927 + 0.007
STEAM TVAE,
0.915 + 0.020
STEAM NElow, log.reg.
0.921 + 0.007
STEAM NFlow, if.
0.924 + 0.016
31

---

K Hyperparameter stability
The performance of generative models is typically sensitive hyperparameters. To assess the stability of STEAM's performance across hyperparameters, on IHDP, we compare CTGAN with STEAMcTeAN with multiple hyperparameter configurations. We report results by changing three hy- perparameters: number of hidden units within the generator layers (generator_ n_ mamis_hiddees ( number of hidden layers within the generator (generator _aattorr___larrr__ae ____ayers__hidee ble [6). and activation functions used in the generator (generator_nonle (Table 17 keeping all other hyperparameters default.
hidden) (Ta-
The performance gap between STEAMCTGAN and CTGAN is relatively stable across these configura- tions. STEAMCTGAN outperforms CTGAN in each metric almost all hyperparameter levels. The most statistically significant differences are consistently noted in the JSD, and Upehe metrics, which compatible with the results displayed in the main paper.
Table 15:
Comparison of STEAM with standard generation on IHDP at different caatttorr__uutt_sse hidden levels. Averaged over 5 runs, with 95% Cls.
generator n_ unita bidden biddddeeb
Model
Pa,x (1)
Ra.x (t)
JSD, (t)
UPERE (U)
CTGAN
0.517 = 0.026
0,396 +: 0,015
0.863 +: 0.033
2,914 +: 0.047 2.194 0.265
STEAM CTGAN
0.565 0.011
0.405 + 0.011
0.941 + 0.000
50
CTGAN
0.622
+: 0.043
0.916 1:
0.141
STEAM CTGAN
0.664
0.020
0.444 + 0.017
0.905 + 0.041
1,960 4: 0.174
100
CTGAN
0.607
0.038
0.418 + 0.032
0.894 + 0.010
2.560 = 0.289
STEAM CTGAN
0.682
0.016
0.439 + 0.018
0.912 1 0.004
2,097 1 0.095
300
CTGAN
0.619
0.030
0.434 + 0.030
0.908 1: 0.023
2.426 1 0.289
STEAM CTGAN
0.699
0.018
0.458 1: 0.015
0.928 1 0.016
2.028 0.163
CTGAN
0.663
0.018
0.419 + 0.013
0.888 + 0.010
2.521 0.161
STEAM CTTAN
0.674 = 0.014
0.424 + 0.011
0,928 + 0.009
1.709 1 0.052
Table 16:
Comparison of STEAM with standard generation on IHDP at different generator_ laarornnnaaay______eea hidden levels. Averaged over 5 runs, with 95% Cls.
generator. D_ l__.aayrr__hhhee hidden
Model
Pa.x (t)
Ra,x (t)
JSD (t)
UrEnE ())
2
CTGAN
0,663 1 0.018
0.419 1: 0.013
0.888 1: 0.010
2,521 4: 0.161
STEAM CIGAN
0.674 + 0.014
0.424
0.011
0.928
0.009
1.709
0.052
CTGAN
0.595 + 0.067
0.395
0.066
0.868
0.064
2.982 0.647 0.143
STEAM CTGAN
0.693 1 0.075
0.441
0.043
0.924
0.018
2.028
CTGAN
0,583 + 0.049
0.259
0.074
0.807
0.054
3.278 0.191
STEAM (TIGAN
0.596 + 0.220
0,301 +: 0.084
0.886 : 0.014
2.690 + U.836
5
CTGAN
0.490 + 0.092
0,313 1: 0.069
0.770
0.127
2.871 + 0.599
STEAM CTGAN
0.691 + 0.071
0.386
0.041
0.915 0.010
2.498 0.536
Table 17: Comparison of STEAM with standard generation on IHDP at different generator_nonl.. settings. Averaged over 5 runs, with 95% Cls.
generator, nonlinn Model
Pa,x (1)
Rs.x
JSD (t)
UVenE (L)
ReLU
CTGAN
0.663 0018
0.419 + 0.013
0.888 + 0.010
2.521 + 0.161 1.709 1: 0.052
STEAM CTGAN
0.64 0.014
0.424 1 0011
0.928 1 0.009
SELU
CTGAN
0.604
0.020
0.499
0.15
0.855 + 0.023
2,509 + 0.160
STEAM
0.699 -: 0.017
0.445 = 0.025
0.929 + 0.014
2,043 = 0.130
Leaky ReLU
CTGAN
0.648 + 0.045
0.415 + 0.015
0,889 + 0.016
2.482 + 0.210
STEAM CTGAN
0.457
0.916 0.011
2.036 1 0.135
32

---

L Congeniality bias
Congeniality bias []6] is a phenomenon that may arise from generation with STEAM. In this scenario, it refers to the fact that downstream models which are structurally similar to the outcome generator, QyIwxx may be advantaged in their performance on D,. For example, the potential outcomes from an S-learner are used for Qr the outcome generation mechanism in D, may be modelled such way that allows downstream S-learners to better estimate CATEs than other learners. While we acknowledge this phenomenon may disadvantage certain downstream models, we note that our outcome error metric, UPEhE, averages across a number of downstream learner types, such that conducting generative model selection with UPERE should lead to good performance across wide variety of downstream learners, not just those similar to Qy w helping to reduce this congeniality bias.
33

---

M Causal generative model comparison
For the baseline models in 7.1.2 use the code provided by [2] in their GitHub https
7/gi //github..r http://ptrickkcchaahaac Di ffafffsiionn wow.sssaaaasaaaaaaaalllllll and we use the same hyperpa- rameter settings for both ANM and DCM as that work.
In Table 18 we report the full set of results for ANM and DCM with each of these graph discovery methods. We that the differences between the graph discovery methods relatively small, except on the IHDP dataset, where Gdscooeree is significantly better than Gmive for the DCM model.
Table 18: Pa,x. Rt.x.. JSD and UPEHE values for CGMs with different graph discovery methods. Averaged over runs, with 95% confidence intervals.
Dataset Model
Pa,x (1)
R3,x (1)
JSD (1)
Upehe (L)
ACTG
DCM Gnaive
0.773 + 0.013
0.369 = 0.006
0.937 0.006
0.665 0.034
DCM
discovered
0.756 + 0,011
0.350 = 0.007
0.956 + 0.005
0.605 + 0.023
DCM pruned
0.758 + 0.013
0.358 0.007
0.957
0.003
0.596 + 0.017
ANM naive
0.787 + 0.007
0.389 0.008
0
0.580 + 0.017
ANM
discovered
0.836 = 0.007
0.419 = 0.007
0.004
0.578 = 0.019
ANM
prunnd
0.839 = 0.008
0.412 = 0.005
0.005
0.582 = 0.014
IHDP*
DCM
Guaive
0.557 = 0,010
0.340 = 0.009
0.883 0.016
4.878 + 0.395
DCM discoveeed
0.658 = 0.011
0.360 = 0.007
0
2.059 = 0.140
ANM
9
naive
0.597 = 0.029
0.379 = 0.011
0.900
1.868 = 0.147
ANM
Gdscovvrre
0.589 + 0.012
0.359 + 0.009
0.892 0.008
1.865 + 0.059
ACICI
DCM
Gissoovered
0.942 + 0.004
0.422 = 0.003
0.957 0.003
4.249 + 0.132
DCM Goruned
0.939 + 0.004
0.420 0.004
0.959 + 0.002
4.340 + 0.159
ANM Giscoverred
0.929 + 0.003
0.404 + 0.003
0.872 + 0.002
4.193 + 0.127
ANM Grrune
0.930 + 0.004 0.404
0.003
0.880
0.002
4.481 + 0.174
pruned was the same as Gdsscoered for IHDP
Excessive runtime caused the exclusion of Goaive ACIC results
34

---

N Differential privacy with STEAM
Theoretical guarantees of the privacy of synthetic data are often required in high-stakes scenarios, such medicine. STEAM can permit this, satisfying DP when its three component models do, application of the post-processing and composition theorems of DP [23].
Proposition 1. Q Qwix, and Qy w satisfy (cx. 6x)- (ew 6w and (ey, dy )-iffeerrnnti privacy respectively, STEAM satisfies (Esoatt -iiferrrfreet privacy, where Evoaal Ex + EW cy, Owall = ox + Sw + 5y.
Proof. See below.
There are a number existing DP generative models, classifiers, and regressors which can be set as Qx, Qwix, and Qyjw;x respectively enable this.
N.1 STEAM differential privacy proof
The theoretical guarantee of STEAM's differential privacy (DP) when using individual DP compo- nents is grounded in the post-processing and composition theorems of DP [3], as we state in the main body of the paper. We make this derivation clear here by first outlining the post-processing and composition theorems in full.
Theorem Post-Processing Theorem). Let M N + R be randomised algorithm that is (e. 6)- differentially private. Let f: R R' be an arbitrary randomised mapping. Then the composition f o M : N - R' is (e, --differrenltlll private.
Theorem (Composition Theorem). Let M; NI - R. be an (Ei, diiifferrree private algo- rithm for E [k]. Define Maa N + I R₁ as:
(x)(F) = (M)( Mz(x), Ma(x)),
then Ma is (2im Eis Ei 8.) differentially private.
Given these theorems, we have our guarantee of DP generation with STEAM, as stated in Proposition 1. Specifically:
Proof. Qx generates X, and satisfies (cx, -xfffefrrrntt privacy by assumption.
By the post-processing theorem, inputting X as the condition to Qw x does not affect its privacy. Qwix generates W, and satisfies (ew, ,6 -differential privacy by assumption.
By the post-processing theorem, inputting W and X as the conditions to Qriw,x does not affect their privacy. Qy] generates Y, and satisfies (cy, )diffferrnttt privacy by assumption.
STEAM generates (X, W, Y,,,, and the composition of Qx. Qwix. and Qr i.e. STEAM (Qx, Qwix, Qy]w;))
Therefore, by the composition theorem STEAM satisfies (Erotal d -differential privacy, where
Gtoaa EX + Ew + Ey, deoall 8x dw + dy.
N.2 Differential privacy experiments
We now examine STEAM's performance in (c, 6)-DP generation. For comparison we use a state- of-the-art DP generative model, AIM [7I), set at privacy level (e, 8), and, for STEAM, Qx as (e/3, 8/3) AIM,, Qwix as an 4/3,6 8/3)-DP random forest, and Qr as an (e/3, 8/3)-T- Learner, such that STEAM is also (c, DDP. We compare performance on the ACTG dataset across E {0.25, 0.5, 1, 2, 3, 10, 15) with 8 = 10 in Figure 5
set
Takeaway. STEAMAIM models PyJw.x better all tested values of e, as Upehe is significantly lower than for standard AIM. Pwix is better modelled by STEAMAM at small e, with equivalent performance between the methods less conservative budgets. Px, on the other hand, is better preserved by standard AIM, scoring higher on Pa.x and R3 most This is likely because
35

---

1.0
1.0
1.0
Standard STEAM
0.4
0.4
0.7
0.2
0.2
0.6
107
10
10' 10
Figure 5: Pa.x (1), R3.x (1), ISD, (1), and UPEn (4) evaluating STEAMAIM and standard AIM privacy budgets. Averaged over runs, shaded area represents 95% Cls.
assigning Qx one third the budget the standard AIM model and having model largely the same distribution, high-dimensionality of X. As such, with uniform distribution (c. across each component, there is trade-off between ATEAMAIM and standard AIM, where STEAMAIM better preserves Pwix and Pyjw:x. while standard AIM preserves Px better. Distributing (c, 6) differently amongst Qx. Qwix. and Qrlw. could address this trade-off, discuss Appendix N.4
for the removed W and Y. is prohibitively restrictive given the
N.3 Extended differential privacy results
We now report (c, 8DDD generation results on the ACTG across a wider set of baseline models. In Figure 6 we compare GEM (66] with STEAMGEM. in Figure [ we compare MST [70] with STEAMMST. and in Figure we compare RAP [S] with STEAMRAP. While there are some nuances each baseline comparison, the general takeaway remains similar: STEAM models preserve Pwix and Pr better, while standard models preserve Px better.
1.0
1.
1.0
Standard
1.50
0.0
1.25
11.00
0.4
0.75
0.2
0.2
0.6
0.0
10
0.0
10
10:
1.5
10
10
Figure 6: Pa,x (1), R3.x JSD, n (1), and UpehE (4) evaluating STEAMGEM and standard GEM privacy budgets. Averaged over runs, shaded area represents 95% Cls.
1.0
1.0
1.0
Standard
0.8
0.9
1100
0.4
0.75
0.2
0.2
0.6
0.0 10a
10:
0.0 10
10
0.5
10L
Figure 7: Pa.x (1), Rex (1), JSD (1), and UPEhE (L) evaluating STEAMMST and standard MST across privacy budgets. Averaged over runs, shaded area represents 95% Cls.
1.0
1.
1.0
144
Standard
0.8
12
0.4
0.2-
0.2
0.6
0.6
0.0
10:
0.0 10
10
0.5
10
Figure 8: Pa.x (t), Ra. (1), ,SD, (1), and UPERE (L) evaluating STEAMRAP and standard RAP across privacy budgets. Averaged over runs, shaded area represents 95% Cls.
worth noting, however, that when baseline models perform poorly in modelling Px. as the case for GEM and RAP, then the relevant STEAM model exhibits similar performance in this regard.
36

---

N.4 Allocation of the privacy budget in STEAM
In STEAM, uniform distribution the privacy budget e amongst the three component models ensures (c, 8).DP. However, such allocation is uninformed on the difficulty of modelling of Px. Pwix, and Pr|wx, and their relative importance downstream analysts.
In relation to the importance of each distribution, one immediate improvement can be to distribute according some preference function f (0, 20) A which takes input of the budget e and weights w for the relative importance of good modelling in Qx. Qwix. and Qy w.X. and outputs a corresponding distribution. For example, simple preference function definition would be f(c, prior knowledge of the importance level of each component distribution to downstream analysts. Another approach, if is not necessary specify the desired distribution priori, is treat it hyperparameter, to be tuned over a series of runs to optimize some metric, such as a combination of Pa,x, Rg,x, JSD,, and UPHIE.
A² (where A is the 2-simplex)
where could be defined by data holder with some
Incorporating knowledge of the complexity of modelling Pxx Pwix, and Py w. is more difficult. While some proxy measures could be established, such as the number of covariates in X indicating the complexity Fwix, establishing robust understanding how the complexity these distributions relate and compare, is highly non-trivial, and such, we leave this for future work.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide theoretical and experimental results show misalignment of popular metrics with treatment effect use cases in 5 We provide experimental evidence of our generation methods' superior performance in
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer: [Yes]
Justification: Proof for our theorem $ is in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose hyperparameter settings in Appendix H and provide code here
and here https: /ttpps://githb.... com/
wanderschhaarlab/STE..
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
39

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code here btrphtttt////tthb.../aaaaaa//// and here https: https:://gitub.... cot///aadeeessccaaaaaa///1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://mips.cc.
coblic//guide//Coeeetuiiiis cotpssiinoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtp:/cccpbllcggiuiieeeee witeeee//ooSSSeSSS...ee.a www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose hyperparameter settings in Appendix H and provide code here http://gtthub.coom/aarraaaa3
and here https: // thub. com/
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include 95% CIs with all reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
40

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe compute resources and experiment run times in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read and conform with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts of our work in
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
41

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and name licenses for all assets used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
42

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide documented code bttp://ps/////thhhoo//oaaaaaaa// and here
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
43

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Core method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtdddard components.
Please refer to our LLM policy (https: for what should or should not be described.
44


Reviews:
[{"review_id": "f7SEcWQdG8", "reviewer": "Reviewer_vwba", "summary": "This paper focuses on the availability of medical data for use by the causal evaluation community.  Medical domains are a highly-valued use-case for causal inference methods, but due to patient privacy restrictions, actual medical data is often unavailable.  Synthetic data generation methods exist that approximate the distributions of empirical data.  However, these methods generally treat all the variables in the same way, which is not ideal for causal inference applications, since treatment and outcome have a much greater importance and are used in different ways than the covariates.  The authors propose STEAM, a framework for generating synthetic data that separately generates covariate, treatment, and outcome distributions to match the source data's distributions.  They authors compare STEAM, using different generic synthetic data generation methods for the covariate distribution, against those synthetic data generation methods and find that, in most cases, STEAM improves performance.", "questions": "On line 94, you when talking about generic generative models and causal generative models, state that \"neither of these existing approaches are tailored to data containing treatment, and they do not target our desiderata\".  I agree with respect to generic generative models, but it seems odd to me to claim that \"causal generative models\" aren't tailored to answering causal questions.  The main issue from my understanding with causal generative models is that they assume you have access to the full causal structure, which is a strong assumption in practice.  In what way are they \"not tailed to data containing treatment\"?\n\nIn addition to the data type assumptions that STEAM is able to make use of (i.e., assuming treatment is binary, so using a Bernoulli for treatment and a logistic regression for outcome), it seems like a major benefit of STEAM is that the treatment and outcome are not having their importance overshadowed by the sheer number of covariates.  Do any of the generic generation methods allow you to adjust the importance of variables, or do any sort of term weighting?  If so, I'd be curious to see if that improves their performance at all.", "limitations": "I think the limitations are adequately discussed", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I think this is a strong paper overall.  Given the general difficulty of evaluation in the causal inference space, guidance for how to produce more realistic synthetic data is a welcome contribution.  The authors justify the importance of treatment outcome and treatment separately from covariates, and the flow of the paper overall is clear.  While the idea behind STEAM isn't overly complex (mostly amounting to \"model W, X, and Y separately rather than jointly\"), it's still a valuable idea to explore, and I think the authors make the case for its practical utility well.  The set of experiments performed against generic generation methods are great and demonstrate the utility of STEAM well.\n\nThere's a bit too much repetition in the opening few pages that could be trimmed to make room for, for example, Figure 4 or causal generative model results.  Looking at the first 4 pages of the paper, the authors make some statement that amounts to \"generic distribution methods don't treat W (and/or X/Y) separately, which is bad\" 9 times from my count (on lines 39, 47, 58, 83, 87, 95, 100, 115, 125).  Obviously, as this is the main thesis of the paper, it's perfectly fine to repeat it.  However, the amount that it's repeated started to become tedious by the end of page 4, since it just felt like the you were repeating yourselves over and over again.  Even removing just a few of these would both save you some space and reduce the repetitive feeling significantly.\n\nOn line 197 and 205, the authors mention \"Figure 4\", which doesn't actually seem to be in the paper.  On further investigation, there is a Figure 4 in Section N of the appendix, but that's not mentioned either time the figure is referenced in the main body of the paper.  It also feels weird to put a diagram showing the core flow of the proposed framework in the appendix.\n\nWhile the evaluation that is in the paper is good (I like the set up of using each generic generation method for the covariate distribution as a comparison), there are additional evaluations that feel missing.  While I understand that STEAM has the advantage of making fewer assumptions than causal generation methods, I would still be interested in seeing a comparison with them.  The authors are working in a setting where some level of ground truth structure is assumed (covariates are pre-treatment, and treatment is pre-outcome), so from my understanding, the main missing piece for causal generation methods would be not knowing the causal structure among the covariates.  However, I'd still like to see some additional rows in Table 3 for a few different causal generation methods.  It seems like there are a few ways you could do this, such as, treating all covariates as independent (just having them all have arrows into both W and Y), applying a causal structure learning algorithm to the covariates and generating based on that, treating the covariates as fully connected...", "was_revised": true, "final_justification": "With the authors agreeing to make the listed changes (cleaning up the narrative and adding the CGM results to the main paper), the paper will only improve. I will maintain my vote for acceptance.", "rebuttal": "Thank you for your thoughtful comments and suggestions. We give answers to each of the following in turn:\n\n- (A) Comparison with causal generative models\n- (B) Repetition in writing\n- (C) Adjusting variable importance in generic generative models\n\n---\n\n**(A) Comparison with causal generative models**\n\nThank you for highlighting the family of causal generative models (CGMs). We agree that a direct comparison with CGMs is crucial for positioning our work, even though they operate under stronger assumptions than STEAM.\n\nIndeed, we have already conducted a comparison in Appendix L of the best performing STEAM models from Table 3 with two performant CGMs: an additive noise model (ANM) [1] implementation from the _DoWhy-GCM_ python package [2], and a diffusion-based causal model [3]. To fairly apply these methods, we first must reconcile their requirement of a complete causal graph. We investigate three distinct methods for defining the causal graph structure that assume no more knowledge than is encoded in STEAM:\n\n1) A naive graph where covariates are fully connected, all covariates cause W and Y, and W causes Y ($\\mathcal{G}_\\text{naive}$);\n2) A graph learned using the PC causal discovery algorithm [4] ($\\mathcal{G}_\\text{discovered}$);\n3) A pruned version of the discovered graph that removes any edges violating the known data generating process, i.e. any backwards edges from Y to W or X, or from W to X are removed ($\\mathcal{G}_\\text{pruned}$).\n\nWe repeat these results from Appendix L below:\n\n| Dataset | Model | $\\boldsymbol{P_{\\alpha, \\mathbf{X}}}$ ($\\uparrow$) | $\\boldsymbol{R_{\\beta, \\mathbf{X}}}$ ($\\uparrow$) | $\\boldsymbol{\\textbf{JSD}_\\pi}$ ($\\uparrow$) | $\\boldsymbol{U_\\textbf{PEHE}}$ ($\\downarrow$) |\n|---|---|---|---|---|---|\n| ACTG | $STEAM_\\text{TVAE}$ |**0.929 $\\pm$ 0.008**| **0.486 $\\pm$ 0.009**| 0.958 $\\pm$ 0.004   | **0.492 $\\pm$ 0.011**   |\n|  | DCM $\\mathcal{G}_\\text{naive}$ | 0.773 $\\pm$ 0.013 | 0.369 $\\pm$ 0.006 | 0.937 $\\pm$ 0.006 | 0.665 $\\pm$ 0.034 |\n|  | DCM $\\mathcal{G}_\\text{discovered}$ | 0.756 $\\pm$ 0.011 | 0.350 $\\pm$ 0.007 | 0.956 $\\pm$ 0.005 | 0.605 $\\pm$ 0.023 |\n|  | DCM $\\mathcal{G}_\\text{pruned}$ | 0.758 $\\pm$ 0.013 | 0.358 $\\pm$ 0.007 | 0.957 $\\pm$ 0.003 | 0.596 $\\pm$ 0.017 |\n|  | ANM $\\mathcal{G}_\\text{naive}$ | 0.787 $\\pm$ 0.007 | 0.389 $\\pm$ 0.008 | 0.954 $\\pm$ 0.005 | 0.580 $\\pm$ 0.017 |\n|  | ANM $\\mathcal{G}_\\text{discovered}$ | 0.836 $\\pm$ 0.007 | 0.419 $\\pm$ 0.007 | 0.952 $\\pm$ 0.004 | 0.578 $\\pm$ 0.019 |\n|  | ANM $\\mathcal{G}_\\text{pruned}$ | 0.839 $\\pm$ 0.008 | 0.412 $\\pm$ 0.005 | 0.952 $\\pm$ 0.005 | 0.582 $\\pm$ 0.014 |\n|---|---|---|---|---|---|\n| IHDP | $STEAM_\\text{CTGAN}$ | 0.674 $\\pm$ 0.014  | **0.424 $\\pm$ 0.011**| **0.928 $\\pm$ 0.009**| **1.709 $\\pm$ 0.052** |\n|  | DCM $\\mathcal{G}_\\text{naive}$ | 0.557 $\\pm$ 0.010 | 0.340 $\\pm$ 0.009 | 0.883 $\\pm$ 0.016 | 4.878 $\\pm$ 0.395 |\n|  | DCM $\\mathcal{G}_\\text{discovered}$ | 0.658 $\\pm$ 0.011 | 0.360 $\\pm$ 0.007 | 0.893 $\\pm$ 0.008 | 2.059 $\\pm$ 0.140 |\n|  | DCM ${\\mathcal{G}_\\text{pruned}}^*$ | 0.658 $\\pm$ 0.011 | 0.360 $\\pm$ 0.007 | 0.893 $\\pm$ 0.008 | 2.059 $\\pm$ 0.140 |\n|  | ANM $\\mathcal{G}_\\text{naive}$ | 0.597 $\\pm$ 0.029 | 0.379 $\\pm$ 0.011 | 0.900 $\\pm$ 0.005 | 1.868 $\\pm$ 0.147 |\n|  | ANM $\\mathcal{G}_\\text{discovered}$ | 0.589 $\\pm$ 0.012 | 0.359 $\\pm$ 0.009 | 0.892 $\\pm$ 0.008 | 1.865 $\\pm$ 0.059 |\n|  | ANM ${\\mathcal{G}_\\text{pruned}}^*$ | 0.589 $\\pm$ 0.012 | 0.359 $\\pm$ 0.009 | 0.892 $\\pm$ 0.008 | 1.865 $\\pm$ 0.059 |\n|---|---|---|---|---|---|\n| $\\text{ACIC}^\\dagger$ | $STEAM_\\text{ARF}$ | 0.939 $\\pm$ 0.004| 0.393 $\\pm$ 0.004 | **0.977 $\\pm$ 0.002** | **2.176 $\\pm$ 0.141**|\n|  | DCM $\\mathcal{G}_\\text{discovered}$ | 0.942 $\\pm$ 0.004 | 0.422 $\\pm$ 0.003 | 0.957 $\\pm$ 0.003 | 4.249 $\\pm$ 0.132 |\n|  | DCM $\\mathcal{G}_\\text{pruned}$ | 0.939 $\\pm$ 0.004 | 0.420 $\\pm$ 0.004 | 0.959 $\\pm$ 0.002 | 4.340 $\\pm$ 0.159 |\n|  | ANM $\\mathcal{G}_\\text{discovered}$ | 0.929 $\\pm$ 0.003 | 0.404 $\\pm$ 0.003 | 0.872 $\\pm$ 0.002 | 4.193 $\\pm$ 0.127 |\n|  | ANM $\\mathcal{G}_\\text{pruned}$ | 0.930 $\\pm$ 0.004 | 0.404 $\\pm$ 0.003 | 0.880 $\\pm$ 0.002 | 4.481 $\\pm$ 0.174 |\n\n\\* $\\mathcal{G}$ pruned is the same as $\\mathcal{G}$ discovered for IHDP\n\n$\\dagger$ Excessive runtime caused the exclusion of $\\mathcal{G}_\\text{naive}$ ACIC results\n\nThe STEAM models outperform these CGM baselines on nearly all metrics and datasets, often to a statistically significant degree, regardless of the underlying causal graph used. STEAM is only outperformed in the ${P_{\\alpha, \\mathbf{X}}}$ and ${R_{\\beta, \\mathbf{X}}}$ metrics on the ACIC dataset by the DCM generative model. These results validate our core thesis: when dealing with treatment data, and where the complete causal graph is unknown, STEAM’s encoding of only the high-level, known causal structure leads to more useful and robust synthetic data generation.\n\nFurther, to your point on our phrasing in line 94 that CGMs are not \"tailored to data containing treatments\", we agree that this should be made more precise. Our intention was to highlight that, while CGMs are of course designed with causality in mind and to answer broad sets of causal queries (that require a full causal graph), STEAM is specifically tailored for treatment data. In this setting, the high-level causal structure is known, but finer-grained relationships are typically not. STEAM is designed to leverage precisely this level of knowledge without requiring more restrictive assumptions. We have revised this section in the manuscript to articulate this important distinction more clearly.\n\n**Update:** We will **move the causal generative modelling results** from Appendix L to **Section 7.1**, and **revise line 94** to clarify the differences in assumptions and typical use cases between STEAM and CGMs.\n\n---\n\n**(B) Repetition in writing**\n\nWe appreciate your help in improving the quality of our writing. We agree that we are overly repetitive of our main contention in the first few sections, and we have removed some of the repetitions identified. We use the resulting space to move Figure 4 into the main body of the text.\n\n**Update:** We will **reduce repetitive phrasing** and **move Figure 4 into the main body** of the text.\n\n---\n\n**(C) Adjusting variable importance in generic generative models**\n\nThank you for bringing up this interesting potential alternative approach. To our knowledge, there is not substantial existing literature on biasing tabular generative models towards better preserving particular subsets of variables. Nevertheless, it seems like simple modifications could be made to the loss functions of many typical generative models to introduce a weighting vector $w$. For example, a diffusion model could be trained with a weighted noise prediction loss to bias the model to better preserve the treatment and outcome distributions\n\n$\\mathcal{L} = \\mathbb{E}_{t, x_0, \\epsilon} [\\sum_d w_d (\\epsilon_d - NN(x_t, t)_d)^2]$\n\nwhere $w_d$ is high for the $W$ and $Y$ variables. Such approaches would require manually tuning these weights, however, and it is not clear what the optimal weights to give to the treatment/outcome variables would be, which is a consideration that is not required in STEAM.\n\nMore fundamentally, this weighting scheme would target the _marginal_ distributions of $W$ and $Y$, while we are most interested in their _conditional_ distributions, conditioned on $X$, for downstream tasks like examnination of treatment assignment protocols, and treatment effect estimation. STEAM, on the other hand, specifically targets these conditionals of interest.\n\n---\n\nThank you once again. We hope that we have addressed all your comments, and we greatly appreciate your feedback.\n\n---\n\n[1] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.\n\n[2] Patrick Blöbaum, Peter Götz, Kailash Budhathoki, Atalanti A Mastakouri, and Dominik Janzing. Dowhy-gcm: An extension of dowhy for causal inference in graphical causal models. Journal of Machine Learning Research, 25(147):1–7, 2024.\n\n[3] Patrick Chao, Patrick Blöbaum, Sapan Patel, and Shiva Prasad Kasiviswanathan. Modeling causal mechanisms with diffusion models for interventional and counterfactual queries, 2024.\n\n[4] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001."}, {"review_id": "bqMnYLG0j8", "reviewer": "Reviewer_2zJC", "summary": "This paper addresses the problem of generating and evaluating synthetic data for medical causal inference tasks. The authors identify three core aspects critical for downstream causal analysis: (i) the covariate distribution, (ii) the treatment assignment mechanism, and (iii) the outcome generation mechanism. They propose STEAM, a novel framework that generates synthetic data by explicitly targeting and minimizing errors in each of these aspects. Through experiments on three medical datasets, the authors demonstrate that STEAM consistently outperforms existing generative models, achieving lower error across all three evaluation criteria.", "questions": "- It seems to me that the proposed methods can be broadly applicable to causal inference tasks in various domains. However, the paper is positioned specifically within the medical setting. Could the authors clarify what aspects of STEAM are uniquely tailored to medical data, and whether there are domain-specific considerations that guided the design?\n- The proposed approach relies on the unconfoundedness assumption, which may not always hold in practice, especially in real-world medical data. Could the authors discuss how STEAM could be adapted or extended to handle settings where this assumption is violated?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is exceptionally well-written and easy to follow. The key questions are clearly articulated and the proposed solution is well motivated and explained. I found the paper a pleasure to read.\n- The experimental evaluation is comprehensive and demonstrates the effectiveness of the proposed approach across multiple datasets.\n- The distinction between STEAM and existing generative methods is clearly stated and well justified.\n\nWeaknesses:\n- My main concern is that the proposed approach assumes unconfoundedness, which may not always be realistic in medical settings. It would strengthen the paper to discuss how STEAM might be generalized or adapted when this assumption does not hold.", "was_revised": true, "final_justification": "I have read the authors' responses and would like to maintain my initial assessment of this work; I think this work can benefit from more revisions based on the discussions and hope my comments are helpful in their future work.", "rebuttal": "Thank you for your thoughtful comments and suggestions. We give answers to each of the following in turn:\n\n- (A) Confounding\n- (B) Applications beyond medicine\n\n---\n\n**(A) Confounding**\n\nThank you for raising this important point. We agree that unconfoundedness is a strong assumption that may not hold in many real-world datasets. We do note, however, that this is a standard assumption in a large body of treatment effect estimation literature [1,2,3,4,5,6,7,8].\n\nOur goal in generating synthetic data, as discussed in Section 4 (lines 116-121), is to create a high-fidelity replica of a real dataset, preserving its characteristics, including any biases and potential confounding. We aim for any synthetic data to be an accurate reflection of real data, rather than seeking improvement and reduction of these biases. This is because we see the problem of detecting, correcting for, and operating under unobserved confounding as orthogonal to the task of synthetic data generation, and indeed there already exist methods for analysis in this setting [9,10]. Such methods can equally be applied when conducting analysis on STEAM-generated data, just like real data.\n\nIn fact, one hindrance to improving causal inference methods for complex settings (e.g. those with unobserved confounding) is the lack of freely available datasets that can act as sandboxes for algorithmic development, due to privacy concerns. By preserving real data distributions as accurately as possible, we hope that STEAM-generated data can be used as valuable and safe tools for researchers to develop and validate new methods for analysis under confounding.\n\nThat said, we agree that discussing extensions to STEAM for settings where we do not assume unconfounding would strengthen the paper. One extension, for instance, could be valid if an instrumental variable ($Z$) were available, as the STEAM generation process could be adapted to follow the corresponding causal structure:\n\n$Z \\sim P_Z, X \\sim P_{X|Z}, W \\sim P_{W|X,Z}, Y \\sim P_{Y|W,X}$.\n\nThis would produce synthetic data suitable for downstream analysis using instrumental variable methods, which do not require the unconfoundedness assumption. This represents a potentially fruitful direction for future work. We will add this to our discussion section, to make the assumptions present in STEAM transparent, and detail possible future works to address alternative scenarios.\n\n**Update:** We will **add the above discussion to Section 8** to clarify STEAM's assumptions and goals, while **outlining potential future work to address hidden confounding**.\n\n---\n\n**(B) Applications beyond medicine**\n\nYou are correct that the core principles of STEAM are broadly applicable to causal inference tasks in any domain featuring treatments or interventions. We chose to position the paper within the medical setting for two main reasons:\n\n1) **Clear practical need.** The medical field faces significant challenges with data access due to necessary privacy regulations, making high-quality synthetic data particularly valuable.\n2) **Motivating our desiderata.** The medical context helps justify the importance of our specific goals. For example, preserving the covariate distribution $P_X$ is motivated by the standard medical practice of reporting patient cohort statistics, and accurately modeling the treatment assignment mechanism $P_{W|X}$ is shown to be critical given the high stakes of misrepresenting treatment protocols.\n\nNevertheless, we agree that the STEAM architecture itself, and our proposed evaluation metrics, are domain-agnostic. To underscore this, we have already included an experiment on a non-medical dataset in Appendix H, on the Jobs dataset [11] which contains experimental data from a male sub-sample from the National Supported Work Demonstration used to evaluate the effect of job training on income. The results on this data confirm that STEAM provides the same benefits in this context as in the main body of the paper.\n\n**Update:** We will **expand Section 8 to highlight other domains where STEAM and our metrics can be potentially useful**, and clearly reference to our existing results in Appendix H.\n\n---\n\nThank you once again. We hope that we have addressed all your comments, and we greatly appreciate your feedback.\n\n---\n\n[1] Curth, Alicia, and Mihaela Van der Schaar. \"Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[2] Künzel, Sören R., et al. \"Metalearners for estimating heterogeneous treatment effects using machine learning.\" Proceedings of the national academy of sciences 116.10 (2019): 4156-4165.\n\n[3] Nie, Xinkun, and Stefan Wager. \"Quasi-oracle estimation of heterogeneous treatment effects.\" Biometrika 108.2 (2021): 299-319.\n\n[4] Kennedy, Edward H. \"Towards optimal doubly robust estimation of heterogeneous causal effects.\" Electronic Journal of Statistics 17.2 (2023): 3008-3049.\n\n[5] Wager, Stefan, and Susan Athey. \"Estimation and inference of heterogeneous treatment effects using random forests.\" Journal of the American Statistical Association 113.523 (2018): 1228-1242.\n\n[6] Shalit, Uri, Fredrik D. Johansson, and David Sontag. \"Estimating individual treatment effect: generalization bounds and algorithms.\" International conference on machine learning. PMLR, 2017.\n\n[7] Alaa, Ahmed, and Mihaela Van Der Schaar. \"Validating causal inference models via influence functions.\" International Conference on Machine Learning. PMLR, 2019.\n\n[8] Shi, Claudia, David Blei, and Victor Veitch. \"Adapting neural networks for the estimation of treatment effects.\" Advances in neural information processing systems 32 (2019).\n\n[9] Dennis Frauen and Stefan Feuerriegel. Estimating individual treatment effects under unobserved confounding using binary instruments. arXiv preprint arXiv:2208.08544, 2022.\n\n[10] Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. In The 22nd international conference on artificial intelligence and statistics, pages 2281–2290. PMLR, 2019.\n\n[11] Robert J LaLonde. Evaluating the econometric evaluations of training programs with experimental data. The American economic review, pages 604–620, 1986."}, {"review_id": "ruTj1hXdAD", "reviewer": "Reviewer_5zLf", "summary": "This paper addresses a gap for the generation of synthetic causal data. The authors identify that existing synthetic data generation methods and their evaluation metrics are primarily designed for predictive tasks and fail to capture the causal structure required for robust downstream treatment effect analyses. To remedy this, they propose (a) a set of properties that synthetic data for causal inference should preserve, (ii) new evaluation metrics that directly measure how well these desiderata are met, and (iii) a flexible, model-agnostic generation framework (STEAM) that explicitly targets these properties. Empirical results across multiple real and simulated medical datasets show that STEAM outperforms generic methods, especially as data complexity increases.", "questions": "- How sensitive is STEAM’s performance to the choice of base generative models, classifiers, and regressors? \n- The focus is on binary treatments. Can STEAM naturally extend to multi-valued or continuous treatments, and are there unique challenges for those settings?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Requiring the presentation of {covariate distribution, treatment assignment mechanism, and outcome generation mechanism} is not itself a novel realization, but by articulating those as tests for evaluating synthetic datasets is a valuable formalization.\n- This work proposes sound metrics for each of those properties. For instance, it employs the metrics from Alaa et al 2022 (as opposed to e.g. KL Divergence) to evaluate the covariate distribution, which is more robust to outliers and reveal different modes of failure (e.g., mode collapse or over-coverage). It uses a different, prediction-based approach to ensure the assignment mechanisms are present.\n- The evaluation is thorough and provides clear ablations of how STEAM compares to other models.\n- The advantage of STEAM (by designing it with the inductive biases that we want to preserve) becomes more pronounced as the data or the underlying relationships (e.g., treatment assignment, outcome generation) become more complex or high-dimensional.\n\nWeaknesses\n- Based on table 3, the STEAM approach doesn’t always seem to improve the covariate metrics (Pα,X, Rβ,X) much over baseline generative models; most gains are in treatment and outcome mechanisms.\n- Due to space limitations, the results and discussions sections are shorter than one would like. It largely boils down to a figure, a brief comment, and then the takeaway. There is very little analysis of Table 3.", "was_revised": true, "final_justification": "I thank the authors for addressing my questions on the discussion/analysis in the paper, STEAM performance on covariate metrics, model sensitivity, and future extensions.\n\nI maintain my score of accept.", "rebuttal": "Thank you for your thoughtful comments and suggestions. We give answers to each of the following in turn:\n\n- (A) STEAM performance on covariate metrics\n- (B) Sensitivity to component models in STEAM\n- (C) Extension to multiple or continuous treatments\n\n---\n\n**(A) STEAM performance on covariate metrics**\n\nThis is indeed an important observation from the results in Table 3. While STEAM consistently and substantially improves the preservation of the treatment assignment and outcome generation mechanisms, its impact on covariate preservation, as measured by $P_{\\alpha, X}$ and $R_{\\beta, X}$, is more modest and, in some cases, it performs worse. \n\nThis is in fact an expected outcome of STEAM's design. The core benefit of STEAM is not necessarily to be a better covariate generator, but rather to supply necessary model capacity to the treatment and outcome distributions, which are easily overshadowed in generic generative models that treat all variables equally. While STEAM does still isolate the modelling of the covariate distribution in $Q_X$, giving the generative model used for this component an ostensibly easier task than a generic, joint-level alternative, the modelling of $P_X$ is nearly of equivalent difficulty to modelling $P_{X,W,Y}$ when the number of covariates is high.\n\nThe ACIC dataset from Table 3 is a good example of this: with 60 variables, the task for STEAM's $Q_X$ (modelling 58 variables) is not substantially easier than the task for the corresponding generic generative models (modelling 60 variables), and STEAM's comparative performance in terms of $P_{\\alpha, X}$ and $R_{\\beta, X}$ is worst on this dataset. Contrast this with the datasets with fewer covariates (ACTG, IHDP) where STEAM more consistently shows a (modest) improvement in $P_{\\alpha, X}$ and $R_{\\beta, X}$. \n\n**Update:** We will **expand our discussion in Section 7.1** to give more **comprehensive analysis of Table 3**, including the above discussion.\n\n---\n\n**(B) Sensitivity to component models in STEAM**\n\nWe can see from Table 3 that STEAM is sensitive to the choice of generative model used for $Q_X$, as our metrics significantly differ between different STEAM configurations on the same dataset. The choice of model for $Q_X$ naturally affects the $P_{\\alpha, X}$ and $R_{\\beta, X}$ scores, but it also has significant ramifications on $JSD_\\pi$ and $U_{PEHE}$, since poor modelling of $X$ leads to inevitably poor modelling of the conditional distributions for $W|X$ and $Y|W,X$. \n\nOn the other have, we found that STEAM was much less sensitive to the choice of classifier and regressors used in $Q_{W|X}$ and $Q_{Y|W,X}$, respectively. In general, amongst a family of reasonable choices for these models, we did not identify substantial changes in the $JSD_\\pi$ and $U_{PEHE}$ metrics. This is unsurprising, as modelling these distributions is an easier task than modelling the (usually high dimensional) covariate distribution. \n\n**Update:** We will **expand our discussion in Section 7.1** to include these **takeaways on STEAM's sensitivity to the model choice for $Q_X$**. We will also **add a new ablation** in the appendix examining STEAM's sensitivity to model choices for $Q_{W|X}$ and $Q_{Y|W,X}$.\n\n---\n\n**(C) Extension to multiple or continuous treatments**\n\nSTEAM's modular design does indeed allow for natural extensions beyond the binary treatment setting.\n\n- **Multiple treatments**. STEAM can easily extend to the multiple treatment setting, as a multi-class classifier can be used for $Q_{W|X}$, and PO regressors compatible with $>2$ treatment arms can be used for $Q_{Y|W,X}$. A key practical challenge that can arise here is data sparsity. If some treatment arms have few subjects, which becomes more likely in multiple treatment settings, then certain classes of PO regressors can perform poorly. For example, a T-Learner, which trains separate regressors for each treatment arm using only the subset of training data that receives the relevant treatment, will become impractical when treatment cohorts become too small. This can be combatted by using POs that share some level of representation between treatment groups, e.g. an S-Learner.\n- **Continuous treatments**. More significant architectural changes would be required for this setting. Firstly, $Q_{W|X}$ would have to be replaced by a _regressor_ rather than a classifier. For $Q_{Y|W,X}$, the treatment space could be discretised and then treated like the multiple treatment case above. Alternatively, if we wish to leave the treatment as continuous we could replace the separate PO regressors with methods designed for continuous treatments, such as those based on the generalised propensity score [1].\n\n**Update:** We will **expand Section 8** to include this discussion on **potential extensions of STEAM to multiple and continuous treatment settings**.\n\n---\n\nThank you once again. We hope that we have addressed all your comments, and we greatly appreciate your feedback.\n\n---\n[1] Hirano, K. and Imbens, G.W. (2004). The Propensity Score with Continuous Treatments. In Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives (eds W.A. Shewhart, S.S. Wilks, A. Gelman and X.-L. Meng). https://doi.org/10.1002/0470090456.ch7"}, {"review_id": "IsV9aQnCBy", "reviewer": "Reviewer_SPcu", "summary": "The authors proposed a new framework (metrics + method) for evaluating the fidelity of an estimated distribution using synthetic data in the contexts of treatment data. In many scenarios in the real world (for example healthcare) the data is precisely treatment data, that is, a tuple (Y,X,W) \\sim P where Y is the outcome, X the feature vector and W is an ascertainment binary variable that models whether or not a particular treatment was applied. \n\nThe authors bring to attention the problem that P is estimated via some marginals using a mix of real and synthetic data. Synthetic data is very common in the settings the paper goes over because not only data is scarce but the questions are counterfactual in nature, real data only has a single outcome observed.  Then they proceed to expose sufficient and necessary desiderata to guarantee the fidelity of the estimated marginals Q and propose metrics that guarantee so. In particular they make the point that the standard methods are based on KL divergence and they prove that in high dimensions the error of such estimates is the same for infinitely many estimates.\n\nThen the authors proceed to introduce STEAM, a framework that performs better under the proposed metrics than standard approachs to estimate Q.", "questions": "- Why do we only care about the support as the justification for the metrics for the coovariate distribution? what about the Kurtosis?\n- It is not clear to me why  boundedness is part of the desiderata for the distance.\n- Is there a result analogous to Theorem 1 for the proposed metrics? If such a result exists, some of the arguments currently used to motivate the new metrics may be redundant. Additionally, is Theorem 1 truly a failure mode specific to KL-based estimations? It seems natural for a distance function to have infinitely many points at a fixed distance from the center—for example, all points on the surface of a ball of fixed radius.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Weakness\n- Some of the metrics proposed but the authors are based on unmentioned metrics just mentioned as aa citation, thus not providing neither motivation and making the reading of the paper more difficult. (pg 5)\n- The choice of the Jensen-Shannon distance is not justified in the paper, the authors claim symmetry, smoothness, and\nboundedness is enough desiderata for this but for me is not self-evident that makes this distance the best way to compare Bernoulli distributions\n- The claim \"Assessment should therefore centre on a complex task, in which comparable performance will likely imply the same for simpler tasks\" is not self evident to me.\nStrengths\n- The proposal of metrics that do not depend on oracle value and thus do not depend on semisynthetic benchmarks.\n- Comparing performance between simulated data against real data as the way to measure the fidelity of the outcome mechanism is a very interesting idea.\n- The empirical analysis was clear and well thought as a fair benchmark between the standard methods and STEAM, this is relevant as they are evaluating on their own proposed metrics.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions. We give answers to each of the following in turn:\n\n- (A) Covariate metrics\n- (B) Jensen-Shannon distance\n- (C) Using a complex task for outcome evaluation\n- (D) Theorem 1\n\n---\n\n**(A) Covariate metrics**\n\nWe are happy to clarify our motivation for choosing $\\alpha$-precision and $\\beta$-recall as the metrics for covariate evaluation. It is important to note that these metrics do not just compare the supports of the synthetic and real distributions, but they are explicitly _density-aware_, as they examine synthetic and real $\\alpha$-supports across a range of $\\alpha$. Underlying these metrics is the definition of an $\\alpha$-support: the minimum volume subset that contains probability mass $\\alpha$. By evaluating how much synthetic data lies in the  $\\alpha$-support of the real data, and vice versa, for $\\alpha \\in [0,1]$, we can obtain a precision-recall curve which, when compared to the ideal, diagonal line achieved only by identical distributions, results in the numerical integrated $\\alpha$-precision and $\\beta$-recall scores used for model evaluation. These metrics offer a more comprehensive evaluation of distributional alignment than simple support comparison, or comparison of a single moment like kurtosis.\n\nIntuitively, $\\alpha$-precision measures the _fidelity_ of the synthetic data, where a high score indicates that the model is not generating unrealistic, low-density samples. $\\beta$-recall, on the other hand, measures synthetic data diversity, where a high score indicates that the model has successfully generated samples that cover all the major modes of the real distribution.\n\nWe choose these metrics because they allow disentangling of two failure modes—generating unrealistic samples, and mode collapse—which unidimensional metrics, such as KL divergence, do not. Furthermore, they are popular choices in general synthetic data literature [1,2,3,4,5,6].\n\n**Update:** We will **expand Section 5.2.1** to include a more **in-depth explanation of these metrics**. \n\n---\n\n**(B) Jensen-Shannon distance**\n\nThere are many potential distances and divergences that could be used to compare $P_{W|X}$ and $Q_{W|X}$, and, while we choose Jensen-Shannon distance (JSD) in our manuscript, it is by no means that only reasonable choice. On our desire for boundedness, specifically, we see that a key practical advantage of a bounded distance metric is its interpretability. Since JSD is bounded between [0,1], it allows for more immediate, intuitive interpretation of results than unbounded metrics, such as KL divergence, as it is quite clear what is a 'good' and 'bad' score, without requiring further context, such as comparison between different proposed generative models. Given its boundedness, if $JSD_\\pi$ is close to 1, we can be sure that the treatment assignment mechanism is modelled well, and vice versa if it is close to 0. With unbounded metrics, such interpretation can be more difficult, as it can be unclear how big of an (e.g. KL) score is to be deemed 'bad'. Indeed, we enjoy the fact that our metrics $P_{\\alpha, X}$ and $R_{\\beta, X}$ are also bounded in $[0,1]$, allowing similar ease of interpretation.\n\nDespite this, we acknowledge that no metric is universally superior, and quantifying distributional distance is a difficult task in general. We discuss alternatives to JSD in Appendix E. We note that, from our experimental experience, we did not see large differences in the level of information offered by alternative distance metrics, i.e. comparative model rankings tended to remain similar across different distance metrics. Nevertheless, if a completely comprehensive evaluation is desired, it would be wise for a user to perform comparison using a suite of relevant distance metrics.\n\n**Update:** We will **expand Section 5.2.2 to explain our choice of JSD**, while making our discussion on alternatives in Appendix E more clearly referenced.\n\n---\n\n**(C) Using a complex task for outcome evaluation**\n\nWe agree that our original claim, \"comparable performance [on a complex task] will likely imply the same for simpler tasks,\" could be more precise. Our reasoning for this statement is specifically based on the relationship between different causal estimands that are particularly relevant in medical analysis: the Average Treatment Effect (ATE) and the Conditional Average Treatment Effect (CATE). Notably, ATE is the expectation of CATE over the covariate distribution: $ATE = \\mathbb{E}_X[CATE(X)]$.\n\nThis relationship implies that successfully modelling the entire CATE(X) function is a more comprehensive and stringent test of a synthetic dataset's utility. A dataset that yields accurate CATE estimates across the full patient population will, by definition, also yield an accurate ATE. The reverse is not true; a dataset could reproduce the correct ATE and yet contain incorrect CATEs (e.g., by overestimating the effect for one subgroup and equally underestimating it for another). Therefore, by centering our evaluation on the more complex CATE estimation task, we are performing a more rigorous \"stress test\" of the outcome generation mechanism $P_{Y|W,X}$. Of course, it is not strictly true that improved CATE preservation in synthetic data will necessarily improve ATE preservation, but it seems a nice heuristic. \n\n**Update:** We will **revise Section 5.2.3 to replace the original statement with this more precise justification** based on the relationship between ATE and CATE.\n\n---\n\n**(C) Theorem 1**\n\nOur proposed set of metrics is specifically designed to be immune to the failure mode identified in Theorem 1. Since we disentangle the evaluation of the distributions of $X$, $W|X$, and $Y|W,X$, our metrics—$JSD_\\pi$ and $U_{PEHE}$ in particular—will maintain _sensitivity_ to the modelling of the treatment assignment and outcome generation mechanisms as the dimensionality of $X$ increases. While Theorem 1 applies to KL divergence (and we see similar empirical behaviour in other joint-level metrics in Appendix D), since we individually evaluate $Q_{W|X}$ and $Q_{Y|W,X}$, our metrics will be able to identify which, amongst a suite of proposal models, best preserves these distributions, even in high-dimensional scenarios. We see this empirically demonstrated in Appendix D, where our metrics allow better identification of modelling differences than any joint-level metric.\n\nWe suspect that extensions to Theorem 1 for more joint-level metrics, beyond just KL divergence, are possible. The underlying issue of such metrics is related to the curse of dimensionality, as distances can lose meaning in high dimensions, where all points tend to be equally far away from each other, and the contribution of the low-dimensional W and Y components to the overall distance becomes negligible as d increases, causing the metric to \"lose sight\" of them.\n\nThe ball analogy is an interesting intuition, however it is not quite the same as the point proven by Theorem 1. We prove that, in high dimensions, _any_ differences in the modelling of $Q_{W|X}$ or $Q_{Y|W,X}$ will not be detected by KL divergence, as scores for different models will become increasingly similar. This is not quite the same as saying that there are infinitely many potential distributions that result in the same KL divergence (which is also true, and related to the ball metaphor), as we instead are saying that _any_ models that differ only in these specific conditionals will be equidivergent from the real distribution.\n\n---\n\nThank you once again. We hope that we have addressed all your comments, and we greatly appreciate your feedback.\n\n---\n\n[1] Liu, Tennison, et al. \"Goggle: Generative modelling for tabular data by learning relational structure.\" The Eleventh International Conference on Learning Representations. 2023.\n\n[2] Zhang, Hengrui, et al. \"Mixed-type tabular data synthesis with score-based diffusion in latent space.\" arXiv preprint arXiv:2310.09656 (2023).\n\n[3] Truda, Gianluca. \"Generating tabular datasets under differential privacy.\" arXiv preprint arXiv:2308.14784 (2023).\n\n[4] Ramachandranpillai, Resmi, et al. \"Bt-GAN: generating fair synthetic healthdata via bias-transforming generative adversarial networks.\" Journal of Artificial Intelligence Research 79 (2024): 1313-1341.\n\n[5] Ramachandranpillai, Resmi, Md Fahim Sikder, and Fredrik Heintz. \"Fair latent deep generative models (fldgms) for syntax-agnostic and fair synthetic data generation.\" ECAI 2023. IOS Press, 2023. 1938-1945.\n\n[6] Fang, Zhengyu, et al. \"Understanding and mitigating memorization in diffusion models for tabular data.\" arXiv preprint arXiv:2412.11044 (2024)."}]

Author rebuttal:

Meta-review: The paper introduces a simple method for synthetic data generation that is tailored to causal inference tasks. While positioned in a medical setting, the method is general enough to apply to other settings as well. The paper was well-received by almost all reviewers and the author-reviewer discussion suggested an avenue for minor improvements to make this paper ready for publication. To meet the expectations of the reviewers, the authors should incorporate the rebuttal in their final paper, including improving the clarity of the paper, moving the GCM results to the main paper, and considering the additional baselines brought up by reviewer 2zJC. 

An important but overlooked by the reviewers component of this work is the relationship of this work to privacy since synthetically generated data should not leak private information from the original data. I see that the authors present an analysis of their proposed method with respect to differential privacy in the Appendix. I would like to encourage the authors to move the main results on privacy to the main paper. This will provide a more complete picture of the constraints and guarantees of the proposed method.
Decision: Accept (poster)