Generating High-Fidelity Privacy-Conscious Synthetic Patient Data for Causal Effect Estimation with Multiple Treatments

1 INTRODUCTION . In health care , studying the causal effect of treatments on patients is critical to advance personalized medicine . Observing an association between a drug ( exposure or treatment ) and subsequent adverse or beneficial event ( outcome ) is not enough to claim the treatment is indeed the cause of the observed outcome due to the existence of confounding variables , defined as factors that affect both the treatments and outcomes . Randomized clinical trials ( RCTs ) have been the gold standard for estimating causal relationships between intervention and outcome . However , RCTs are sometimes not feasible due to logistical , ethical , or financial considerations . Further , randomized experiments may not always be generalizable , due to the restricted population used in the experiments . In the past decade , observational data has become a viable alternative to RCTs to infer causal treatment effects due to both the increasingly available patient data captured in Electronic Health Records ( EHRs ) ( Henry et al . ( 2016 ) ) and the remarkable advances of machine learning techniques and capabilities . Typically , EHRs capture potential confounding factors such as race , gender , geographic location , eventual proxies of social determinants of health , as well as medical characteristics such as comorbidities and laboratory results . Many causal inference models have been proposed to estimate treatment effects from observational data . Validation of these models with realistic benchmarks , however , remains a fundamental chal- lenge due to three reasons . First , the ground truth of treatment effects in a realistic setting is unknown . In real world , we can not compute the treatment effect by directly comparing the potential outcomes of different treatments because of the fundamental problem of causal inference : for a given patient and treatment , we can only observe the factual , defined as the patient outcome for the given treatment , but not the counterfactual , defined as the patient outcome if the treatment had been different . Second , legal and ethical issues around un-consented patient data and privacy created a significant barrier in accessing routinely EHRs by the machine learning community . In order to mitigate the legal and ethical risks of sharing sensitive information , de-identification of patient records is a commonly used practice . However , previous work has shown that de-identification is not sufficient for avoiding re-identification through linkage with other identifiable datasets Sweeney ( 1997 ) ; Emam et al . ( 2011 ) ; Malin & Sweeney ( 2004 ) . Third , most publicly available datasets support either binary or very few treatments , while there has been growing literature developing techniques with multiple treatments in recent years ( Lopez & Gutman ( 2017 ) ) . To address these challenges , in this work we generated a large-scale and realistic patient dataset that mimics real patient data distributions , supports multiple treatments , and provides ground truth for the effects of these treatments . The datasets we generated are synthetic patients with hypertension modeled on a massive nationwide cohort patients ’ history of diagnoses , medications , and laboratory values . We designed a data generation process by adapting an Anonymization Through Data Synthesis Using Generative Adversarial Networks ( ADS-GAN by Yoon et al . ( 2020 ) ) model for fictitious patient information generation and using a neural network for treatment outcome generation . The synthetic dataset demonstrates strong similarity to the original dataset as measured by the Wasserstein distance . In addition , we ensure that the original patients ’ privacy is preserved so that our dataset can be made available to the research community to evaluate causal inference models . We demonstrated the use of the synthetic data by applying our dataset to evaluate five models : one inverse probability treatment weighting ( IPTW ) model , one propensity matching model , one propensity score stratification model all introduced by Rosenbaum & Rubin ( 1983 ) , and two models in the doubly robust family ( Foster & Syrgkanis ( 2020 ) ) . To our knowledge , this is the first large scale dataset that mimics real data joint distributions with multiple treatments and known causal effects . The approach we used can be readily extended to other types of diseases in the clinical domain , and to datasets in other domains as well . The rest of this paper is organized as follows . In Section 2 , we discuss related works . The details of our method are presented in Section 4 . In Section 5 , we discuss the evaluation metrics for the quality of the data and results of evaluating several established causal inference models with the data . We discuss the limitations of the work in Section 6 and conclude the paper in Section 7 . 2 RELATED WORK . Our work is related to several existing works on publicly available databases , fictitious patient record creations , and data generation processes . 2.1 PUBLICLY AVAILABLE DATASETS . First used in ( Hill ( 2011 ) ) , the Infant Health and Development Program ( IHDP ) is a randomized controlled study designed to evaluate the effect of home visit from specialist doctors on the cognitive test scores of premature infants . It contains 747 subjects and 25 variables that describe the characteristics of the infants and their mothers . The Jobs dataset by LaLonde ( 1986 ) is a benchmark used by the causal inference community , where the treatment is job training and the outcomes are income and employment status after training . The Twins dataset , originally used for evaluating causal inference in ( Louizos et al . ( 2017 ) ; Yao et al . ( 2018 ) ) , consists of samples from twin births in the U.S. between the years 1989 and 1991 provided in ( Almond et al . ( 2005 ) ) . The Annual Atlantic Causal Inference Conference ( ACIC ) data challenge provides an opportunity to compare causal inference methodologies across a variety of data generating processes . Some of the above mentioned datasets do not provide true causal effects . Others are small in size so the models validated on such datasets may perform very differently in a more general real-world setting . All the datasets created for ACIC challenge were designed specifically for competitions . The covariates in the data are either drawn from publicly available databases , or simulated . In the former case , the datasets are limited by small populations ( Du et al . ( 2021 ) ) and arbitrarily designed data generation processes , which did not aim to capture any real-world causal relationships ( Karavani et al . ( 2019 ) ) . In the latter case , the distribution of the data is not realistic , i.e , dissimilar to the distribution of any real dataset . 2.2 EHR DATA GENERATION . Walonoski et al . ( 2018 ) generated synthetic EHRs based on publicly available information . The focus of their work is on generating the life cycle of a patient and how a disease evolves over time . Goncalves et al . ( 2020 ) evaluated three synthetic data generation models–probabilistic models , classification-based imputation models , and generative adversarial neural networks–in generating realistic EHR data . Tucker et al . ( 2020 ) used a Bayesian network model to generate synthetic data based on the Clinical Practice Research Datalink ( CPRD ) in the UK . Benaim et al . ( 2020 ) evaluated synthetic data produced from 5 contemporary studies using MDClone . Wang et al . ( 2021 ) proposed a framework to generate and evaluate synthetic health care data , and the key requirements of synthetic data for multiple purposes . All of these works focus on EHR data generation producing patient variables but without ground truth for causal effects . In contrast , the focus of our work is not only on generating patient variables , but producing ground truth for causal effects as well . 2.3 POTENTIAL OUTCOME GENERATION . To validate their models , many researchers such as ( Schuler & Rose ( 2017 ) ) created synthetic covariates and produced potential outcomes with a designed data generation process . Such datasets are not designed to approximate any real data distributions . Franklin et al . ( 2014 ) ; Neal et al . ( 2020 ) generated potential outcomes from covariates with known causal effects , but without any regard to patient privacy . We addressed the critical issue of patient privacy concerns so our data can be made available for the research community to evaluate their models . 3 PATIENT CLAIM DATA AND INCLUSION EXCLUSION CRITERIA . To make our synthetic data realistic , we generate the data based on a real-world patient claim database from a large insurance company in the United States . This database contains 5 billion insurance claims ( diagnoses , procedures , and drug prescriptions or refills ) and lab test results from 56.4 million patients who subscribed to the company ’ s service within a 5-year time period between December 2014 and December 2020 . From this database , we extracted a subset of patients affected by hypertension . We chose hypertension because there are a large number of related claims , making it easier to learn the data distribution . In addition , since it is a condition affecting nearly half of adults in the United States ( 116 million , or 47 % ) , our generated dataset can be directly used for clinical researchers to develop and evaluate their models for this important disease . Patients are included in the dataset if they have a medical claim indicating hypertension ( ICD code I10 , I11.9 , I12.9 , and I13.10 ) or currently treated with anti-hypertensive medications . We exclude patients from the dataset if they are age < 18 or age > 85 , affected by white coat hypertension , secondary hypertension , malignant cancers , dementia , or are pregnant . After applying the above mentioned inclusion and exclusion criteria , we have about 1.6 million patients included in this study . We further exclude patients treated with a combination of drugs rather than a single drug . We then rank the drugs by the number of patients treated with each drug , and only keep patients treated with one of the 28 most popular drugs . These filtering steps produce about 262 , 000 patients in the study . The distribution of this dataset is then learned and used to generate synthetic patients , viewed as samples drawn from the learned distribution . The patients ’ diagnoses and treatment history and how their conditions evolve over time are captured by trajectory data consisting of labs , diagnoses and their corresponding dates . For the convenience of data processing and analysis , we convert the trajectory data into tabular data with rows representing different patients ( samples ) and columns representing patient features ( variables ) including patient demographics , diagnoses , medications and labs . In Table . 1 , we list and briefly describe these 60 patient variables : 2 variables ( F1 ) describing the systolic blood pressure before the treatment and the date it was measured , 2 variables ( F2 ) describing the same metric but after the treatment , 3 variables ( F3 ) indicating current and prior drug usage and refill information , 4 variables ( F4 ) describing patient basic information ( age , gender , ethnicity ) , 30 variables ( F5 ) indicating laboratory measurements , 2 variables ( F6 ) indicating the presence or absence of comorbid conditions defined by the Charlson Comorbidity Index ( Charlson et al . ( 1987 ) ) , 15 variables ( F7 ) describing the patient ’ s zip code , the racial makeup and income levels in the patient ’ s zip code tabulation area ( ZCTA ) , 2 variables ( F8 ) indicating meta information . In this study , we are interested in the causal effects of anti-hypertensive drugs ( current drugs of F3 ) on patient outcomes , measured as the difference between the first ( F1 ) and second lab results ( F2 ) .

Summary: The authors propose a method for using real-world patient data to generate (semi-)synthetic privacy-preserving data on which to evaluate methods for causal effect estimation. For this purpose, they adapt the ADS-GAN (Anonymization through Data Synthesis using Generative Adversarial Networks) model introduced by Yoon et al. (2020) to produce a synthetic dataset that is identical in distribution to the original dataset, yet cannot be used to identify any of the original patients. To ensure the former desideratum, the authors calculate the Wasserstein distance between $P_{\hat{X}}$ and $P_X$; to ensure the latter, they use $\epsilon$-identifiabilty (Yoon et al., 2020). The authors show empirically that the synthetic dataset satisfies these desiderata, and then evaluate a few causal effect estimators on the synthetic data using the generated ground truth.