Generating High-Fidelity Privacy-Conscious Synthetic Patient Data for Causal Effect Estimation with Multiple Treatments
Authors: Jingpu Shi
Dong Wang
Gino Tesei
Beau Norgeot
Venue: ICLR 2022 Submitted (2022)
Keywords: synthetic data
causal inference
EHR
healthcare
deep generative modeling
treatment effects
model validation
observational patient data
patient privacy

Abstract: A causal effect can be defined as the comparison of outcomes from two or more alternative treatments. Knowing this treatment effect is critically important in healthcare because it makes it possible to identify the best treatment for a person when more than one option exists. In the past decade, there has been exponentially growing interest in the use of observational data collected as a part of routine healthcare practice to determine the effect of a treatment with causal inference models. Validation of these models, however, has been a challenge because the ground truth is unknown: only one treatment-outcome pair for each person can be observed. There have been multiple efforts to fill this void using synthetic data where the ground truth can be generated. However, to date, these datasets have been severely limited in their utility either by being modeled after small non-representative patient populations, being dissimilar to real target populations, or only providing known effects for two cohorts (treated vs control). In this work, we produced a large-scale and realistic synthetic dataset that supports multiple hypertension treatments, by modeling after a nationwide cohort of more than 250,000 hypertension patients' multi-year history of diagnoses, medications, and laboratory values. We designed a data generation process by combining an adapted ADS-GAN model for fictitious patient information generation and a neural network for treatment outcome generation. Wasserstein distance of 0.35 demonstrates that our synthetic data follows a nearly identical joint distribution to the patient cohort used to generate the data. Our dataset provides ground truth effects for about 30 hypertension treatments on blood pressure outcomes. Patient privacy was a primary concern for this study; the $\epsilon$-identifiability metric, which estimates the probability of actual patients being identified, is 0.008%, ensuring that our synthetic data cannot be used to identify any actual patients. Using our dataset, we tested the bias in causal effect estimation of three well-established models: propensity sore stratification, doubly robust approach (DR) with logistic regression, DR with random forest (RF) classification. Interestingly, we found that while the RF DR outperformed the logistic DR as expected, the best performance actually came from  propensity score stratification, despite the theoretical strength of statistical properties of the DR family of models. We believe this dataset will facilitate the additional development, evaluation, and comparison of real-world causal models. The approach we used can be readily extended to other types of diseases in the clinical domain, and to datasets in other domains as well.

Under review as a conference paper at ICLR 2022
GENERATING HIGH-FIDELITY PRIVACY-CONSCIOUS SYNTHETIC PATIENT DATA FOR CAUSAL EFFECT ESTI- MATION WITH MULTIPLE TREATMENTS
Anonymous authors Paper under double-bling blind review
ABSTRACT
In the past decade, there has been exponentially growing interest in the use of observational data collected as part of routine healthcare practice to determine the effect of treatment with causal inference models. Validation of these mod- els, however, has been a challenge because the ground truth is unknown: only treatment-outtoome pair for each person can be observed. There have been multiple efforts to fill this void using synthetic data where the ground truth can be generated. However, date, these datasets have been severely limited in their utility either by being modeled after small non-representative patient populations, being dissimilar real target populations, or only providing known effects for two cohorts (treated vs control). In this work, we produced a large-scale and real- istic synthetic dataset that supports multiple hypertension treatments, by modeling after nationwide cohort of more than 250, 000 hypertension patients' multi-year history of diagnoses, medications, and laboratory values. We designed data generation process by combining an adapted ADS-GAN model for fictitious pa- tient information generation and neural network for treatment outcome genera- tion. Wasserstein distance of 0.35 demonstrates that our synthetic data follows a nearly identical joint distribution to the patient cohort used to generate the data. Our dataset provides ground truth effects for about 30 hypertension treatments blood pressure outcomes. Patient privacy was a primary concern for this study; the e-identifiability metrie, which estimates the probability of actual patients being identified, is 0.008%, ensuring that our synthetic data cannot be used to identify any actual patients. To demonstrate its usage. we tested the bias in causal effect estimation of five well-established models using this dataset. The approach we used can be readily extended to other types of diseases in the clinical domain, and to datasets in other domains as well.
INTRODUCTION
In health care, studying the causal effect of treatments patients is critical advance personal- ized medicine. Observing an association between a drug (exposure or treatment) and subsequent adverse or beneficial event (outcome) not enough claim the treatment is indeed the cause of the observed outcome due to the existence of confounding variables, defined as factors that affect both the treatments and outcomes. Randomized clinical trials (RCT's have been the gold standard for estimating causal relationships between intervention and outcome. However, RCTs are sometimes not feasible due to logistical, ethical, or financial considerations. Further, randomized experiments may not always be generalizable, due to the restricted population used in the experiments. In the past decade, observational data has become a viable alternative to RCTs to infer causal treatment effects due to both the increasingly available patient data captured in Electronic Health Records (EHRs) (Henry et al. (2016)) and the remarkable advances of machine learning techniques and ca- pabilities. Typically. EHRS capture potential confounding factors such race, gender, geographic location, eventual proxies of social determinants of health, as well as medical characteristics such comorbidities and laboratory results.
Many causal inference models have been proposed to estimate treatment effects from observational data. Validation of these models with realistic benchmarks, however, remains fundamental chal-

---

Under review as a conference paper at ICLR 2022
lenge due to three reasons. First, the ground truth of treatment effects in realistic setting is un- known. In real world, we can not compute the treatment effect by directly comparing the potential outcomes of different treatments because of the fundamental problem of causal inference: for given patient and treatment, we can only observe the factual, defined as the patient outcome for the given treatment, but not the counterfactual, defined as the patient outcome if the treatment had been different. Second, legal and ethical issues around un-consented patient data and privacy created a significant barrier in accessing routinely EHRs by the machine learning community. In order to mit- igate the legal and ethical risks of sharing sensitive information, de- de-identification of patient records a commonly used practice. However, previous work has shown that de-identification is not suffi- cient for avoiding re- re-identification through linkage with other identifiable datasets [Sweeney] (1997); Emam et al. (201 1): Malin & Sweeney (2004). Third, most publicly available datasets support either binary or very few treatments, while there has been growing literature developing techniques with multiple treatments in recent years (Lopez & Gutman (2017))
To address these challenges, in this work we generated a large-scale and realistic patient dataset that mimics real patient data distributions, supports multiple treatments, and provides ground truth for the effects of these treatments. The datasets generated are synthetic patients with hypertension modeled on a massive nationwide cohort patients' history of diagnoses, medications, and laboratory values. We designed a data generation process by adapting an Anonymization Through Data Synthe- sis Using Generative Adversarial Networks (ADS-GAN by Yoon et al. (2020) for fictitious patient information generation and using neural network for treatment outcome generation. The synthetic dataset demonstrates strong similarity to the original dataset as measured by the Wasser- stein distance. In addition, we ensure that the original patients' privacy is preserved so that our dataset can be made available to the research community to evaluate causal inference models.
We demonstrated the use of the synthetic data by applying our dataset to evaluate five models: one inverse probability treatment weighting (IPTW) model, one propensity matching model, one propensity score stratification model all introduced by |Rosenbaum Rubin (1983), and two models in the doubly robust family (Foster& & Syrgkanis (2020)).
To our knowledge, this the first large scale dataset that mimics real data joint distributions with multiple treatments and known causal effects. The approach we used can be readily extended to other types of diseases the clinical domain, and to datasets in other domains as well.
The rest of this paper is organized as follows. In Section 2 we discuss related works. The details of our method are presented in Section 4 In Section we discuss the evaluation metrics for the quality of the data and results of evaluating several established causal inference models with the data. We discuss the limitations of the work in Section and conclude the paper in Section 7
2 RELATED WORK
Our work is related to several existing works on publicly available databases, fictitious patient record creations, and data generation processes.
2.1 PUBLICLY AVAILABLE DATASETS
First used in Hill (01II), the Infant Health and Development Program (IHDP) is randomized con- trolled study designed to evaluate the effect of home visit from specialist doctors on the cognitive test scores of premature infants. It contains 747 subjects and 25 variables that describe the charac- teristics of the infants and their mothers. The Jobs dataset by LaLonde (1986) is a benchmark used by the causal inference community, where the treatment is job training and the outcomes are income and employment status after training. The Twins dataset, originally used for evaluating causal infer- ence in (Louizos et al. (2017): 7): Yao et al.] (2018)), consists of samples from twin births the U.S. between the years 1989 and 1991 provided in (Almond et al. (2005)). The Annual Atlantic Causal Inference Conference (ACIC) data challenge provides an opportunity to compare causal inference methodologies across variety of data generating processes.
Some of the above mentioned datasets do not provide true causal effects. Others are small in size so the models validated on such datasets may perform very differently in more general roal-world setting. All the datasets created for ACIC challenge were designed specifically for competitions. The covariates in the data are either drawn from publicly available databases, or simulated. In the former
2

---

Under review as a conference paper at ICLR 2022
case, the datasets are limited by small populations (Du et al. (2021) and arbitrarily designed data generation processes, which did not aim to capture any real-world causal relationships (Karavani et al. (2019). In the latter case, the distribution of the data is not realistic, i., dissimilar to the distribution of any real dataset.
2.2 EHR DATA GENERATION
Walonoski al. (2018) generated synthetic EHRs based on publicly available information. The focus of their work is on generating the life cycle of a patient and how a disease evolves over time. Goncalves et al. (020)) evaluated three synthetic data generation models- pronel--pbbabiiiticcc models, classification-base
imputation models, and generative adversarial neural networks-in generating realistic EHR data. Tucker al. (2020) used Bayesian network model generate synthetic data based on the Clinical Practice Research Datalink (CPRD) in the UK. [enaii et al. (2020) evaluated synthetic data produced from 5 contemporary studies using MDClone, Waag et al. (2021) proposed framework to generate and evaluate synthetic health care data, and the key requirements of synthetic data for multiple purposes. All of these works focus on EHR data generation producing patient variables but without ground truth for causal effects. In contrast, the focus of our work is not only on generating patient variables, but producing ground truth for causal effects as well.
2.3 POTENTIAL OUTCOME GENERATION
To validate their models, many researchers such as (Schuler & Rose (2017) )) created synthetic co- variates and produced potential outcomes with a designed data generation process. Such datasets are not designed to approximate any real data distributions. |Franklin et al. (2014); Neal et al. (2020) generated potential outcomes from covariates with known causal effects, but without any regard to patient privacy. We addressed the critical issue of patient privacy concerns so our data can be made available for the research community to evaluate their models.
3 PATIENT CLAIM DATA AND INCLUSION EXCLUSION CRITERIA
To make our synthetic data realistic, we generate the data based on real-world patient claim database from a large insurance company in the United States. This database contains 5 billion insurance claims (diagnoses. procedures, and drug prescriptions or refills) and lab test results from 56.4 million patients who subscribed to the company's 's service within 5-year time period between December 2014 and December 2020. From this database, we extracted subset of patients affected by hypertension. We chose hypertension because there are large number of related claims, making easier to learn the data distribution. In addition, since it is a condition affecting nearly half of adults in the United States (116 million, 47%), our generated dataset can be directly used for clinical researchers to develop and evaluate their models for this important disease. Patients are included in the dataset if they have medical claim indicating hypertension (ICD code I10, I11.9, 112.9, and 113.10) or currently treated with aati-bypertensive medications. We exclude patients from the dataset if they age <18< or age >85, affected by white coat hypertension, secondary hyperten- sion, malignant cancers, dementia, or are pregnant. After applying the above mentioned inclusion and exclusion criteria, we have about 1.6 million patients included in this study. We further exclude patients treated with a combination of drugs rather than a single drug. We then rank the drugs by the number of patients treated with each drug, and only keep patients treated with one of the 28 most popular drugs. These filtering steps produce about 262, 000 patients in the study. The distribution of this dataset is then learned and used generate synthetic patients, viewed samples drawn from the learned distribution.
The patients' diagnoses and treatment history and how their conditions evolve over time are captured by trajectory data consisting of labs, diagnoses and their corresponding dates. For the convenience of data processing and analysis, we convert the trajectory data into tabular data with rows representing different patients (samples) and columns representing patient features (variables) including patient demographics, diagnoses, medications and labs. In Table. I we list and briefly describe these 60 patient variables: 2 variables (F1) describing the systolic blood pressure before the treatment and the date was measured, variables (F2) describing the same metric but after the treatment, 3 variables (F3) indicating current and prior drug usage and refill information, 4 variables (F4) describing patient
3

---

Under review as a conference paper at ICLR 2022
Family var.
Var. names
Description
Fl
date-, lab-
First lab result and date
F2
date+, lab+ drugs, prior srior, last_refill age, gndr_ed _ed, race_ed, ethncty lab measurement results and date safety-morbe morbs,, morbs pprior zip ed, total. pop, female, median income etc trajectory index, mcid
Second lab result and date
F3
Drugs' info
F4
Age/GenderrEthnnicity
F5
11 lab measurements and date
F6
Current and previous comorbidities
F7
Zip code and related statistics Meta-information
F8
Table 1: The patient claim dataset hypertension contains 1, 1,,18,300 363 observations and 60 vari- ables. Above the family such variables are listed and briefly described.
basic information (age, gender, ethnicity), 30 variables (F5) indicating laboratory measurements, variables (F6) indicating the presence or absence of comorbid conditions defined by the Charlson Comorbidity Index (Charlson et al. (1987)), 15 variables (F7) describing the patient's zip code, the racial makeup and income levels in the patient's zip code tabulation area (ZCTA), 2 variables (F8) indicating meta information.
In this study, interested in the causal effects of anti- unti-hypertensive drugs (current drugs of F3) on patient outcomes, measured as the difference between the first (FI) and second lab results (F2).
4 APPROACH
To generate the synthetic data, we first generate the patient variables using adapted ADS-GAN model, then generate the treatment outcomes using a neural network, Our approach can be concep- tually decomposed into four steps described below.
4.1 STEP DATA PREPROCESSING
Since we generate the synthetic data from the patient claim data extracted in Section 3 in this step we preprocess the patient claim data and prepare for subsequent steps. Described Table this dataset are of mixed data types: integers (e.g., age), floats (e.g., lab values), categorical values (e.g.. drugs), and dates. Further, the values and dates of a lab test are missing for some patients the lab test not ordered by the doctors for these patients. In this step. clean, one-hot encode, transform and standardize the data so that all the variable are transformed into numerical values in [0, range. We also add binary feature for each lab test indicate missing lab values and dates. The resulted dataset has 221 features and we call the original dataset, to be distinguished from the synthetic dataset. The details of this step described in Appendix A.I
4.2 STEP 2: GENERATION OF OBSERVED VARIABLES USING ADS-GAN
In this step we generate synthetic patients characterized by the same variables as listed in Table I We want to achieve two goals in this step: to make the synthetic data as realistic as possible and to make sure the probability of identifying any actual patients in the original dataset from the synthetic dataset is very low. We quantitatively define the identifiability in Definition and the realisticity as the Wasserstein distance (Gulrajani al. (2017) between the distribution the synthetic dataset and that of the real dataset it is modeled after.
There trade-off between identifiability and realisticity of generated data. Frameworks like MedGan (Choi et al.] (2018)) and WGAN-GP (Arjovsky et al. (207)) do not explicitly define and allow to control the identifiability levels. Therefore, we evaluated generative models that allow to explicitly control such trade-off, e.g. the ADS-GAN (Yoon et al. (2020)) PATE-GAN (Jordon et al. ) and DP-GAN (Xie et al. (2018)). ADS-GAN proved to consistently outperform the bench- marks across the entire range of identifiability levels on both the MAGGIC and the three UNOS transplant datasets. It is also based on a measurable definition for identifiability. Another advan- tage of ADS-GAN is the of Wasserstein distance measure the similarity between two high

---

Under review as a conference paper at ICLR 2022
dimensional joint distributions. We finally selected ADS-GAN and adapted it to generate the patient variables in our study.
Let us denote the original dataset obtained in Section 4.1 as D = =xx where x, E X C E and
x; = 2 2 x...,,,.) with 2 E x2) C R representing the j-th feature of a patient. N is
the number of samples and d the number of features of each sample.
Let Px denote the underlying distribution from which each is drawn and let ~ Pz be drawn from multi-variate Gaussian distribution. The framework of ADS-GAN is to train generator G: x Z + X and a discriminator D X + R in an adversarial fashion: the generator G which produces synthetic patient variables X; = G (x, z) where x; E X C TRd ensures that the synthetic dataset D {S;; } are not too close D measured by the e-identifiability defined below; on the other hand, the discriminator D which measures the distance between two distributions ensures that the distribution of generated variables Px is indistinguishable from the distribution of real variables Px.
Definition 1. We define the weighted Euclidean distance U (x,, Xj) between x, and x, as U (x, x,) = || (x: - xj)ll
where weight vector. Then r; is defined as
min U (x;, E/D/
where D /xj represents the dataset D without Xj. From the definition, T; is the weighted minimum distance between x; and any other observations D.
We similarly define r; as
min U X,), x
which is the weighted minimum distance between x₁ and the synthetic samples in D.
Definition 2. The c-identifiability of dataset D from Ó defined as
c c <=I(D,D)
(D,
1 Σ [[roee]] = N
where I is indicator function.
To calculate the weight vector w, we first calculate the discrete entropy of the j-th feature, i.e.
(x -- Σ (((()==00) log [ (x = =(ti]
The weight for feature is then calculated as the inverse of H (X6)). In reality, if patient can be re-identified, the re-identification is more likely through rare characteristics or medical conditions of patient. Calculating the weight this way ensures that the rare features of a patient are given more weight, correctly reflecting the risk of re-identification associated with different features.
We base the diseriminator D on Wasserstein GAN with gradient penalty (Gulrajaní et al. 2017 (WGAN-GP), which adopts Wasserstein distance between Px and Px, and defines the loss Lp for discriminator as
CD= Ex
[xx) D -xx) (x)]l2 1)²]
where x belongs random interpolation distribution between Px and Px and μ is further hyper- parameter that we set to 10 based on previous work (Gulrajani et al. (2017)). We implement both the generator and the discriminator using multi-layer perceptrons.
To train the generator G, we need to compute the e-identifiability by computing r; and f; for every sample, which is computationally expensive. To solve the problem, Yoon et al. (2020) made simplifying assumption that (x, 2) the closest data point to x. However, this assumption can be violated during the training of the network that maximizes the distance between G (x, 2) and x.
5

---

Under review as a conference paper at ICLR 2022
We here introduce a contrastive loss (triplet ranking loss. |Schroff al. (2015) item, which is defined
as
Ucon (x, x", 2) = max (0, U (x, Then, the final identifiability loss function Li is
U (x', (x, z)))
(2)
Ly PExxPxxxxx. P [-U (x, G x))] + /Bxx'' (Veon (x,'',20 =]]
(3)
Similar to Yoon al. (2020)), this loss also assumes that G (x, 2) is the closest data point to x. However, penalty on the loss function will be imposed if this assumption is violated when the generated sample G (x, 2) closer to x', a randomly drawn sample from dataset D, than x. The
strength of the penalty term is controlled by B.
In the final optimization problem, we minimize and maximize D simultaneously, written as G D = arg min max (CD + Arr]
(4)
G D
where À is a hyper-parament that controls the trade-off between the two objectives. Once trained, the adapted ADS-GAN model can be used to produce synthetic data set D.
4.3 STEP 3: DATA GENERATION MODEL AND CAPTURED CAUSAL EFFECTS
We need to determine data generation model and proper treatment effects to use in the data gen- eration process to produce the potential outcomes of the synthetic data, i.e, the factuals and coun- terfactuals. Many researchers use arbitrary functions and arbitrary treatment effects. For example, Schuler Rose (017)) used linear function as the data generation process and set the treatment effects arbitrarily. To improve upon such an approach, in this work we train neural network model from the original dataset aiming to capture the mapping from patient covariates to outcomes as well as the treatment effects. The captured treatment effects not necessarily the true effects, but serve as the ground truth in the synthetic data when the data is used to evaluate causal inference models because the patient outcomes in the synthetic data generated from these causal effects.
We first partition the domain of observed variables X C R into the covariate domain Xc S R&,,, the treatment domain Xr C Rdd and the outcome domain X. IR, that d > de + d, + 1. The covariates are all the patient variables excluding drugs, prior drugs, zip code, and lab+. Treatments the drugs. Outcome is the difference between lab+ and lab-. In this step, each treatment is one-hot encoded and represented by de dimensional vector, where de is the number of treatments. use Y; (1,) & denote the potential outcome of the i-th
X
Given collection of N observations, individual, if treated with the treatment f; E Xx. We assume that (Y, t,, x,) IR x Xc x Xc are independent and identically distributed, and that the three fundamental assumptions for causal inference (Rosenbaum & Rubin (983)) in Appendix A.2 are satisfied.
Following (Lopez Gutman 2017) ; Shalit al. (017 ), given E X and 1, E Xc, where to is the placebo, the individual- treatment effect (ITE) of i; can be defined as
T₁ (x) =E YY (t+) - Y (to) |x)]
(5)
Hence, the population average treatment effect for treatment f, can be defined
ATE, EY (1.) (1o)) = / Thy (x)
(6)
X
So the data generation process can be modeled Y Ω (x, t), where Ω XT The true form of S2 is unknown and can be complicated. Here we make simplifying assumption that the representation learned from the covariate domain is separated from the representation learned from the treatment domain. Specifically, let Tb X + 'R be a representation function and R be the representation space. We define Q RxT IR that Ω (x, 4)) = (& (x). Also, we denote with 7 (x) the treatment effect estimate Q.
With simplified J we propose neural network architecture shown in Figure that is able to capture f2, C, and at the same time, calculate the treatment effects. For the covariate domain Xc. the network is fully connected feed-forward neural network with Relu the activation function for all the neurons. For the treatment domain XT, the inputs are encoded treatments directly connected to neuron with a linear activation. The loss function is the standard mean square error (MSE). We apply a dropout to all the layers and apply a gentle L2 regularization to all the weights the neural network.

---

Under review as a conference paper at ICLR 2022
The model Q is trained on the original dataset described in sec- tion [ where we have one factual for each observation. Due to the separation of the covariate domain and treatment do- main, and with the particular architecture of the ANN shown in Figure [1 the neural network weights for treatment connec- tions can be interpreted as the causal treatment effects. Since there is no interaction between the covariates and treatments, the individual treatment effects and population average treat- ment effects are the same. Indeed, suppose wy is the weight for treatment li one can show that w; is indeed the Tı, in Equation 5 and the ATE,, in Equation
X
X
Figure 1: Neural network architec- ture for patient outcome generation and causal effect calculation.
4.4 STEP 4: GENERATION OF FACTUALS AND COUNTERFACTUALS ON SYNTHETIC DATA
The domain of variables and all its partitions are the same for the real dataset and the synthetic dataset D = {x: x; = G (x,, 2) x; E D, ~ P2) Hence, the neural network trained the original dataset in Step 4.3 can be fed with the synthetic patient variables generated in Step 4.2 The neural network outputs are served as the treatment outcomes for the synthetic data.
Once trained, this neural network is capable of generating all factual and counterfactual treatment outcomes for the synthetic data. For any synthetic patient with covariate Xj X.. the potential outcome of any treatment t, E Xc can be generated as Y, (t,) = Ω (x,,t, = Q ( (xg) t.). However, instead of generating the potential outcomes of all possible treatments in Xc. in this work we only generate two potential outcomes for each patient: the factual outcome corresponding to the treatment produced by the ADS-GAN model, and the counterfactual outcome if the patient had not received any treatment. The reason that we do not want to produce all the counterfactual outcomes is that we want limit the treatment for each patient only to the one generated by the ADS-GAN model, in order to preserve the treatment assignment mechanism learned from the original dataset.
There is distinction between the assumptions made in Section in determining the treatment effects and the assumptions that our synthetic dataset actually satisfies. Specifically, our synthetic dataset satisfies the SUTVA and unconfoundedness assumption, as we did not model the interac- tions between patients and we provided all the patient variables in the dataset used to generate the outcomes. Whether it satisfies the positivity assumption, however, depends on the original dataset because the patient assignment mechanism of the synthetic data is learned from the original dataset.
RESULTS
In this section we evaluate the quality of our synthetic dataset. We show that there is strong sim- ilarity in both marginal and joint data distributions between the original and synthetic dataset, and that patient privacy is preserved. We evaluate several causal inference models using our dataset to demonstrate the usage of it.
5.1 ANALYSIS OF SYNTHETIC AND ORIGINAL DATA DISTRIBUTIONS
We first show how well the generated synthetic data preserves the joint distribution of the original data. We trained the adapted ADS-GAN model for 10,000 epochs and the outcome neural network for 20,000 epochs to generate the synthetic dataset. We calculated the Wasserstein distance (Vil- lani (2008)) between the joint distribution of this synthetic data and that of the original data to be 0.35. To put this value in the correct perspective, we measured the Wasserstein distance between the original dataset and a randomly generated dataset of the same dimensions. This serves as the baseline scenario. In addition, we randomly split the original dataset into two datasets and measured the Wasserstein distance between them, which is essentially the Wasserstein distance between the dataset and itself and serves as the best case scenario. Our results show that the Wasserstein distance between the synthetic and original data distribution (0.35) is almost as good as that in the best case scenario 17), and far more lower than the value in the baseline scenario (8.6).

---

Under review as a conference paper at ICLR 2022
A] Original data
Synthetic data
Fateent values
values
Pat Pant values
Figure 2: Heatmaps of correlation matrices of patient variables for the original (left) and synthetic data (right), respectively.
2.0
Original Synth
Origina Syrthh
50
Origi Synth
40
Original Synth
Origina
150
Syrth
40
Value
Velue
30
140
1.0
20
20
20
130
0.5
10
10
10
120
creatinine
carbon dioxide
urea_ _ee_niirogennnn
urea_ creatinine
sodium
Figure 3: Comparison of marginal distribution of lab values between original and synthetic data. The three horizontal dotted lines in each violin plot from top to the bottom represent the third quartile, median, and the first quartile respectively.
We then compare the joint distributions visually. In Fig. 2 the correlation among all patient attributes in the original (synthetic) dataset is visualized by the heatmap the left (right). In the heatmap, the brighter the color diagonal is the brightest in the map, each pixel the diagonal represents the correlation between variable and itself. The two heatmaps show almost identical patterns, indicating the joint distribution of the original data is well preserved in the synthetic dataset.
pixel is, the more correlated the two variables are with each other. The
In Fig. 3 we compare qualitatively the marginal distributions of individual variables of the generated synthetic data (orange) with the related ones from the original data (blue). The figure shows strong similarity between the original and synthetic dataset in both basic statistical summaries (e.g., median and quartiles) and overall shape of these distributions.
5.2 IDENTIFIBILITY OF SYNTUETIC DATA
Since the synthetic dataset we generated this study is meant to be made public, patient privacy has to be preserved to ensure that actual patients in the original dataset source can be identified through the synthetic dataset. All the synthetic samples in our dataset are conceptually drawn from a distribution, no single piece of information about any actual patients is directly carried over to dataset. We further calculate the e-identifiability as defined Definition to be 0.008%, indicating that the risk of any actual patient being identified is statistically zero.
5.3 RESULTS OF TESTING CAUSAL INFERENCE ALGORITHMS ON THE DATASET.
In this section, we evaluate the bias in causal treatment effect estimate of five well established models on our synthetic dataset: two models in the doubly robust (DR) family, one propensity score strati- fication, one propensity matching, and one inverse probability treatment weighting (IPTW) model. Doubly robust approaches adopt outcome regression model to estimate the treatment outcome and a propensity model to estimate the probability of a patient being assigned with a treatment. In both the DR models we tested, random forest is used as the outcome regression model. The dif- ference between the two DR models in the propensity model, which can be logistic regression (DR-LR) random forest (DR-RF) classification. We Microsoft DoWhy (Sharma & Kiciman
8

---

Under review as a conference paper at ICLR 2022
) and EconML (Battocchi et al. (2019) causal inference packages for the implementation. When calculating the causal effect of a treatment, we remove the counter-factuall this treatment from the dataset to prevent the problem becoming trivial.
We adopted four metrics to evaluate the models: the Spearman's rank correlation coefficient to measure how well the models preserve the rank of the drugs by their treatment effects, Kendall rank coefficient similar to Spearman's coefficient but based on concordant and discordant pairs, correlation between the estimated effects and the ground truth, and finally the R² to measure how much variance of the ground truth can be explained by the estimate.
To estimate how models perform in real-world setting, we generated additional dataset consist- ing of all patient variables of the original dataset and patient outcomes generated from the trained outcome neural network with patient variables and treatments of the original dataset as its inputs. We call this dataset the hybrid dataset because part of the data comes from the original dataset and part of the data is generated. We run the five causal inference models on the two datasets and re reported all the results in Appendix A.3
The results on the hybrid dataset show that the algorithms can grouped into two categories in terms of their performance: the propensity stratification, propensity matching, and two doubly robust models are in the first group and the IPTW model is the second group. The models in the first group produced much better results than the IPTW model. The results the synthetic dataset show a similar pattern. The variance of the performance metrics of the models in the first group is larger than that the hybrid dataset, but all these models perform much better than the IPTW model. Investigating why some models perform better than others on the two datasets is out of scope of this work. Here we show that the synthetic data preserves the relative performance of different models that would be achieved in a more realistic setting, simulated by the hybrid dataset.
6 DISCUSSION
In general, inclusion and exclusion criteria applied to the data may introduce selection bias. Our work was designed with target trial in mind in which patients are recruited at an initial qualifying measurement and then followed up after treatment assignments. We believe this minimizes the impact of selection bias from conditioning on the inclusion and exclusion criteria in our original data. In this work, we only produced one dataset for hypertension and evaluated five causal inference models. We leave it to future work to produce synthetic datasets for other diseases and evaluate and compare other causal inference models. Because hypertension affects almost half of adults in the United States, a synthetic dataset on hypertension of significant value by itself. For simplicity, in this study we did not consider treatment modifiers, i.c. interactions between treatments and patient variables. It is an interesting and important topic which we plan to address in the future.
7 CONCLUSION
To validate machine learning models, researchers have traditionally relied on labeled data, i.e. ground truth. Due to the fundamental problem of causal inference, however, the lack of realistic clinical data with ground truth makes it difficult to evaluate causal inference models. In this work, we produced large-scale and realistic synthetic data by adapting an ADS-GAN model generate patient variables and using a neural network to produce patient outcomes. The data we generated supports multiple treatments with known treatment effects. We demonstrated that this synthetic dataset preserves patient privacy and has strong similarity to the original dataset it is modeled after. We believe that it will facilitate the evaluation, understanding and improvement of causal inference models, especially with respect to how they perform in real-world scenarios.
REPRODUCIBILITY STATEMENT
To contribute to reproducibility, we provide in the supplementary material the code for replicating experiments. In section S we provide detailed description of the original real-world dataset and related inclusion and exclusion criteria. In section A.I we provide detailed description of the preprocessing procedure to make the generation process transparent and reproducible. Finally, we report the the link download our synthetic dataset in the final paper.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Douglas Almond, Kenneth Y. Chay, and David S. Lee. The costs of low birth weight. The Quar- terly Journal of Economics, 1206):1031--085 2005. Uttpttss://ccnPapee org/RePEc: oup: g 120: :::::::::::::11111111:::
repec. 2.1
M. Arjovsky, S. Chintala, and I.. Bottou. Wasserstein generative adversarial networks. In Interna- tional Conference on Machine Learning, 2017.
4.2
Keith Battocchi, Fleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estima- 2019. Version 0.x. 5.3
tion.
Anat Reiner Benaim, Ronit Almog, Yuri Gorelik, Irit Hochberg, Laila Nassar, Tanya Mashiach, Mogher Khamaisi, Yael Lurie, Zaher S Azzam, Johad Khoury, et al. Analyzing medical research results based on synthetic data and their relation to real data results: systematic comparison from five observational studies. JMIR medical informatics, 8(2)ee64922 2020. 2.2
Mary E Charlson, Peter Pompei, Kathy Ales, and Ronald MacKenzie. A new method of clas- sifying prognostic comorbidity in longitudinal studies: development and validation. Journal of chronic diseases, 40(5):373-383, 1987.
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks, 2018. 4.2
Xin Du, Lei Sun, Wouter Duivesteijn, Alexander Nikolaev, and Mykola Pechenizkiy. Adversarial balancing-basee representation learning for causal effect inference with observational data. Data Mining and Knowledge Discovery, pp. 1-26, 2021. 2.1
Khaled El Emam, David L. Buckeridge, Robyn Tamblyn, Angelica Neisa, Elizabeth Jonker, and
Aman Verma. The re-identification risk of canadians from longitudinal demographics. BMC Medical Informatics Decis. Mak., 1::6, 2011. doi: 10.1186/1472-6947-111 URL https:
//doi. aidoi.org/10. www.orrg........... 1186 172 6947-11-44
Dylan J. Foster and Vasilis Syrgkanis. Orthogonal statistical learning, 2020.
Jessica M Franklin, Sebastian Schneeweiss, Jennifer M Polinski, and Jeremy A Rassen. Plas- mode simulation for the evaluation of pharmacoepidemiologii methods complex healthcare databases. Computational statistics & data analysis, 12:219-226, 2014. 2.3
Andre Goncalves, Priyadip Ray, Braden Soper, Jemnifer Stevens. Linda Coyle, and Ana Paula Sales. Generation and evaluation of synthetic patient data. BMC medical research methodology, 20(1):
1-40, 2020. 2.2
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:ttt///oocestseete http:::::////ppppeeeeeeeeoeeeeee
http:eeddiinnnn.nnnnnnneeee newrips. cc/
pdf
J Henry, Yuriy Pylypchuk, Talisha Searcy, and Vaishali Patel. Adoption of electronic health record systems among non-federal acute hospitals: 2015-2015. ONC data brief, 35:1-9, 2016.
Jennifer I.. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 01 2011. doi: 10.1198//gg:20100.8883 162. URL https:
//doi. oedbi.org/10,111 1ttp:/11..199//jggge/ 2010. 08162 2.1
James Jordon, Jinsung Yoon, and M. Schaar. Pate-gan: Generating synthetic data with differential
privacy guarantees. In ICLR, 2019. 4.2
Ehud Karavani, Tal El-Hay, Yishai Shimoni, and Chen Yanover. Comment: Causal inference com-
petitions: Where should we aim? Statistical Science, 34(1):86-95) 2019. 2.1
10

---

Under review as a conference paper at ICLR 2022
Robert LaLonde. Evaluating the econometric evaluations of training programs with experimen- tal data. American Economic Review, 76(4):604-22) 20, 1986. URL https: /Econnapprrs......eeee
2.1
Michael J. Lopez and Roce Gutman. Estimation of causal effects with multiple treatments: A review and new ideas. Statistical Science, 32(3), Aug 2017. ISSN 0883-4237. doi: 10.1214/17-tt6112.
URL http: http://xxx.do...r/11111.1
doi wtt.io.rg/1...0 02000.11141110 7-STS612
4.3
Christos Louizos, Uri Shalit, Joris Mooij, David Sontag. Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models, 2017.
2.
Bradley Malin and Latanya Sweeney. How (not) to protect genomic data privacy in a distributed network: Using trail re-identification evaluate and design anonymity protection systems. J. of Biomedical Informatics, 37(3):17--192, June 2004. ISSN 1532-0464. doi: 10.1016ff.bi.2004.
04.005. htLhttp::: /ooi 1tt.///...111//1111 jei 2004. 04 005
Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. Realcause: Realistic causal inference bench- marking. arXiv preprint arXiv:2011.15007 2020. 2.3
Paul Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41 04 1983. ISSN 0006-3444, doi: 10. 10.1093/ biomet/70.1.41
41
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 815-823, 2015. 4.2
Megan S Schuler and Sherri Rose. Targeted maximum likelihood estimation for causal inference in observational studies. American journal of epidemiology, 185(1):65-777 2017. 2.3
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: general-
ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine
Learning. pp. 3076-3060000 3085, 2017. 4.3
Amit Sharma and Emre Kiciman. Dowhy: An end-to-end library for causal inference, 2020.
5.3
Latanya Sweeney. Weaving technology and policy together to maintain confidentiality. The Journal of Law, Medicine & Ethics, 25(2-3):98 110, 1997. doi: 10.1 10.1111/j.778--720X 1997.
X. URL |Lt.:////////...........11.11111 720x. 1997. wttp:/0..........111111..1. 11066504.
PMID:
Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 3(1):1-13, 2020. 2.2
C. Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008
Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, and Scott McLachlan. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association, 25(3):230-238, 2018. 2.2
Zhenchen Wang, Puja Myles, and Allan Tucker. Generating and evaluating cross-sectional synthetic electronic healthcare data: Preserving data utility and patient privacy. Computational Intelligence, 37(2):819-851, 2021. 2.2
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network, 2018. 4.2
11

---

Under review as a conference paper at ICLR 2022
Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Repre- sentation learning for treatment effect estimation from observational data. In S. Ben-
gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett
(eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL [Rt.ittts:////rcceeee nee
B5Cabba8132as 2bcabbee22a7717713330033333222e2 http://sss..pppooo...ppppe/////. pcs.cc//ppper/201//11el
nwwries
2.1
Jinsung Yoon, Lydia N. Drumright, and Mihaela van der Schaar. Anonymization through data syn- thesis using generative adversarial networks (ADS-GAN). IEEE Biomed. Health Informatics,
24(8):2378-2388 2020. doi: 10.1 (100///HH22000299900000
URL https: //doi org/10.
1109/JBHI 2020. 2980262 2002822222222222222222222222
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1
STEP 1: DETAILED DESCRIPTION OF DATA PREPROCESSING
Here we provide description of the preprocessing procedure to clean and transform the patient claim data described in Table The goal of this step is to prepare the data ready for subsequent steps.
We first one-hot encode all the categorical variables in the table, such as gender and race. For each lab test there are two variables: lab values and lab dates. Since all the lab dates were after the 'date-' date, which was the date when the patient's s systolic blood pressure was measured before treatment, we convert all the lab dates to the number of days since 'date-', excluding holidays and weekends. The reason for excluding these dates in the convert ion is that eventually reverse the exact process to convert the synthetic lab dates in number of days back to the original date format, we make sure that such synthetic dates do not fall holidays and weekends. Note that the exclusion of holidays and weekends does not apply to the prescription dates, assuming prescriptions can be filled on any day. When transform the 'date-" variable itself, we arbitrarily pick date proceeding all the "date-' values, and convert the 'date-' values to the number of days since this picked date, excluding all holidays and weekends.
There are many missing lab dates and values, as expected. If doctors order certain lab test for small portion of patients, the lab and dates for the majority of the patients would be missing. We fill the missing values with the mean of the values that are present. However, this induces data distribution that significantly deviates from Gaussian distribution: the probability of the mean is much higher than the probability of other values. Our experiments showed that learning such distribution was difficult. To address this problem, we created binary indicator for each lab test, which is set to if the lab test is not missing, and 0 otherwise. The intuition is that it is easier for the model to learn not only the distribution of the indicator, but also the join distribution of the indicator and its corresponding lab results, thus enhancing the synthetic data's 's ability to more faithfully reflect the original data distribution. Another benefit of doing this is to directly preserve the information on whether lab test was ordered for each patient. Our experiments show that this approach works well in practice. The final step of data preprocessing is to standardize all the values to the [o, 1] range.
A.2 FUNDAMENTAL ASSUMPTIONS FOR TREATMENT EFFECT ESTIMATIONS
We use E (2,,2,....) Xe denote treatments, Y; (1) to denote the potential outcome the patient is treated with the treatment E Xc, and x; to denote the variables for patient i.
SUTVA (Stable Unit Treatment Value Assumption) Potential outcomes for one individual are unaffected by the treatment others.
Unconfoundedness The treatment distribution is conditional independent of the potential outcomes given covariates, i.e. (vi) (14 1L (Y, (4),, YY (t2), ...,,, (4ixel) xx:).
Positivity Every individual has non-zero probability to receive each one of the treatments, for given level of covariates, i.e. (Vi) (Vs) (s E TAO < P (4; (x,) 1).
A.3 RESULTS OF EVALUATING DIFFERENT CAUSAL INFERENCE MODELS
13

---

Under review as a conference paper at ICLR 2022
spearmanr
kendalltau 0.92
correlation
R2 score
Propensity Stratification
0,99
0.99
0.99
Propensity Matching
0.92
0.8
0.98
0.94
Doubly Robust RF
0.95
0.93
0.99
0.95
Doubly Robust LR
0.99
0.96
0.99
0.99
IPTW
0.59
0.48
-0.46
-5.9
Table Model evaluation on hybrid data. The results show that the first four models perform much better than the IPTW model.
spearmanr
kendalitau
correlation
R2 score
Propensity Stratification
0.81
0.68
0.87
0.6
Propensity Matching
0.52
0.37
0.47
-I1
Doubly Robust
0.92
0.78
0.97
0.94
Doubly Robust LR
0.74
0.63
0.88
0.75
IPTW
0.09
0.04
0.06
-34
Table 3: Model evaluation on synthetic data. The results show a pattern similar to the table for the hybrid dataset: the first four models perform much better than the IPTW model. The performance variance the first four models larger than that on the hybrid dataset.
14


Reviews:
[{"review_id": "kLHP4aSLLS9", "reviewer": "Reviewer_eqMK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenth: \n+ The paper is easy to follow and codes are provided for review.\n\nWeaknesses:\n- The proposed problem is not well addressed in this paper. The purpose to simulate patient data is to evaluate causal inference models with generated ground truth. However, it is hard to guarantee that the generated treatment effects are close to the original treatment effects (both the effect size and sign): one can only guarantee the similarity between observed outcomes but not counterfactuals. \n- The technical contribution of this paper is not enough. The entire generation framework is based on the existing ADS-GAN model. The only difference is an additional contrastive loss to add penalty on the original loss function when the generated sample is much closer to x' than to x. The following potential outcome generation is based on a feed-forward neural network, which is a widely used backbone in causal effect prediction. \n- The potential outcomes are generated directly through a neural network without any adjusting for confounding variables. Since the data is generated from real-world observational data that includes various confounding factors. \n- Only three causal inference models are included in the experiments and two of them are from the same family (DR). More baselines (e.g., IPTW, matching-based methods) should be considered to examine the goodness of created dataset. \n- Why propensity score stratification is much better than DR methods? When the results are not consistent with most studies, can we still trust the generated data as well as the ground truth?\n- Only Wasserstein distance is used to measure the distance between original and synthetic dataset distribution. More metrics such as KL-divergence should be considered for comprehensive evaluation. \n- What's the performance of three causal inference models on the original dataset? Are the results on the original dataset consistent with the results on the synthetic dataset?\n- What's the influence of \\beta in Eq.(3) to model performance? An ablation study should be conducted to show this influence.", "summary_of_the_paper": "The paper studies the problem of generating synthetic patient data for the evaluation of causal inference models. The generated patient data is expected to highly mimic the distribution of the original dataset while also taking patient privacy into consideration. Experiments are conducted on the synthetic dataset with three well-established causal inference models.", "summary_of_the_review": "An interesting paper, but it seems that the proposed problem is not well-addressed by the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I2aOO4kwBxh", "reviewer": "Reviewer_46uW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- The problem tackled is very relevant for causal inference problems, as there is almost always a lack of ground truth available to assess how well different causal inference methods perform.\n- The paper is well-written and virtually error-free.\n\n**Weaknesses**:\n- When the inclusion and exclusion criteria are applied to the original dataset, the authors are potentially introducing selection bias by conditioning on downstream variables such as hypertension. This can then introduce spurious correlations between any common causes of the selected variables, which may for example include treatment variables if one of the treatments has an effect on hypertension. I was surprised to see that the authors have not at least discussed a potential selection bias issue. Furthermore, other data preprocessing steps such as filling in missing values or standardizing the values have the potential to make or break existing causal relationships.\n- The experimental section is rather limited. The authors only consider a single starting dataset, and evaluate only a few methods for causal effect estimation on the synthetic data.\n\nOther comments:\n- It is possible I am missing something, but it seems to me that the generator and discriminator roles are switched. Based on the explanation before Definition 1, it appears as though the generator is maximizing the loss so as to ensure that the synthetic dataset is not too close to the original one in terms of $\\epsilon$-identifiability, while the discriminator is minimizing the loss to ensure that $P_{\\hat{X}}$ and $P_{X}$ are indistinguishable. Shouldn't it be the other way around?\n- The ADS-GAN model and $\\epsilon$-identifiability should have corresponding citations to Yoon et al. (2020) when first mentioned. Furthermore, it would be helpful to write what the acronym ADS-GAN stands for.\n- I share the authors' concern that propensity score stratification performs better than the doubly robust methods. Is the propensity model the same for both methods? Perhaps there is some particular structure in the data that favors the simpler propensity score stratification.\n- page 6, subsection 4.3, second paragraph: Treatment domain is introduced as $X_T$, yet is almost immediately replaced with $\\mathcal{T}$? Are these two sets the same? If not, what is the difference? I cannot find where $\\mathcal{T}$ is defined.\n- page 6, last paragraph: New domain set $\\mathbb{Y}$ for $Y$ is introduced out of the blue. Perhaps the authors meant $\\mathbb{R}$ instead?\n- page 8, third line after Figure 3: small typo, \"We\" should not be capitalized", "summary_of_the_paper": "The authors propose a method for using real-world patient data to generate (semi-)synthetic privacy-preserving data on which to evaluate methods for causal effect estimation. For this purpose, they adapt the ADS-GAN (Anonymization through Data Synthesis using Generative Adversarial Networks) model introduced by Yoon et al. (2020) to produce a synthetic dataset that is identical in distribution to the original dataset, yet cannot be used to identify any of the original patients. To ensure the former desideratum, the authors calculate the Wasserstein distance between $P_{\\hat{X}}$ and $P_X$; to ensure the latter, they use $\\epsilon$-identifiabilty (Yoon et al., 2020). The authors show empirically that the synthetic dataset satisfies these desiderata, and then evaluate a few causal effect estimators on the synthetic data using the generated ground truth.", "summary_of_the_review": "The authors have presented a simple and potentially interesting approach for generating privacy-preserving synthetic data from a real-world dataset, but I am concerned by the fact that selection bias was introduced when the authors applied inclusion and exclusion criteria on the patients from the original dataset. Furthermore, the method is only applied to one real-world dataset, so it remains unclear to me how well it can be applied to datasets with different characteristics (e.g. size, amount of missing data, data types).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BFalhtOvk6S", "reviewer": "Reviewer_Jr99", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to read. It also addresses a very important, long-lasting problem in the literature; that is, generating datasets that include realistic counterfactuals. I really liked the idea of generating pseudo-patients to preserve privacy. \nThat being said, I think there is a fundamental problem with generating the counterfactuals from a model trained on data: this is the whole reason why we develop causal inference methods. If we already have an algorithm that can predict the true counterfactuals, then the problem has been solved (and we no longer need datasets for evaluating causal inference methods)! So we agree that we cannot be sure that the proposed algorithm predicts the true counterfactuals. Then, training causal inference methods using the generated dataset means they merely try to learn the underlying function that the proposed outcome neural network models. I’m not sure how this is different from adopting any arbitrary response surface as Hill et al. and ACIC organizers have done. \nPlease see my other comments below:\n- Please provide a reference for the statement that the ACIC dataset is “limited by non-representative populations”. As far as I know, ACIC datasets are all adopted from large medical studies.\n- Please define “identifiability” and “realisticity” formally.\n- For calculating the weights, the authors have used the inverse of H. I think it’s wrong though: imagine a feature is the same for all patients, then the entropy for that feature would be zero, and the inverse of zero is infinity. So a feature that doesn’t matter at all (for identification) gets a large weight. Or am I missing something here?\n- The paper provides little discussion on many design choices (e.g., why ADS-GAN or WGAN-GP was selected, etc.). \n- I couldn’t understand why “the neural network weights for treatment connections can be interpreted as the causal treatment effects”. Also, the true individual treatment effects are often different from the average treatment effect (think personalized medicine)!\n\nMinor comments:\n- The abstract is too long; consider reducing it to one third.\n- Wrong citation format; in most cases, both name and year must be inside parentheses.\n- Citation for ADS-GAN is missing in the text.\n- Page 5, line 2: don't → do not\n- The “subset of” sign in page 5, lines 9, 10, and 15 should be reversed.\n", "summary_of_the_paper": "This work proposes a method to generate a set of binary datasets that are realistic and include counterfactual outcomes so that they can be used for evaluating causal inference algorithms. Moreover, the proposed method ensures that the original patients’ privacy is preserved.\n", "summary_of_the_review": "Neither of the two components of the paper (i.e., privacy preservation and counterfactual prediction) are novel (they are adopted from previous works). The claim that the proposed counterfactual prediction component generates true counterfactuals is not supported (and my understanding is that it cannot be supported either). For these reasons, I think the proposed algorithm is not practical and therefore, I recommend rejecting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a method for generating high quality synthetic datasets, and use their methods to evaluate a variety of causal effect estimators.

In general, the paper was not received very favorably by reviewers.  The primary concerns were: (a) issues with built-in bias in the algorithm that generates synthetic data (due to collider stratification bias induced by conditioning on causally "downstream" variables, (b) issues with "replicating underlying counterfactuals," which indeed is a difficult problem, and (c) lack of "technical novelty."

First, I am personally very sympathetic to what the authors are trying to do.  Regardless of current reviewer reception, I think the causal inference community really needs more high quality benchmarks, and (semi)synthetic datasets, and validation approaches.  I urge the authors to continue this line of work.

That said, I think it is important (for causal benchmarks) to be clear about the distinction between the observed data distribution (e.g. p(C,A,Y) for the backdoor model), and the full data distribution (e.g. p(C, A, Y(0), Y(1)) for the backdoor model with a binary treatment).

Generally what makes a benchmark interesting is preserving some features of the _full_ data distribution, and allowing "knobs" that make the problem easier and harder.  Much of what the ACIC competition organizers did was provide such knobs.  Mimicking features of just the observed data distribution, even if they are complicated, isn't enough to make a causal benchmark interesting, since the problem is all about how full and observed data relate.

When revising the paper, please keep this difference in mind, and consider what features of p(C, A, Y(0), Y(1)) (or more complex versions of this) make for an interesting benchmark, while also generating p(C,A,Y) that "mimics observed data" in some way.
Decision: Reject