A Flexible Generative Model for Heterogeneous Tabular EHR with Missing Modality
Authors: Huan He
William hao
Yuanzhe Xi
Yong Chen
Bradley Malin
Joyce Ho
Venue: ICLR 2024 poster (2024)
Keywords: Generative Model
Synthetic EHR
Primary area: generative models

Abstract: Realistic synthetic electronic health records (EHRs) can be leveraged to acceler- ate methodological developments for research purposes while mitigating privacy concerns associated with data sharing. However, the training of Generative Ad- versarial Networks remains challenging, often resulting in issues like mode col- lapse. While diffusion models have demonstrated progress in generating qual- ity synthetic samples for tabular EHRs given ample denoising steps, their perfor- mance wanes when confronted with missing modalities in heterogeneous tabular EHRs data. For example, some EHRs contain solely static measurements, and some contain only contain temporal measurements, or a blend of both data types. To bridge this gap, we introduce FLEXGEN-EHR– a versatile diffusion model tai- lored for heterogeneous tabular EHRs, equipped with the capability of handling missing modalities in an integrative learning framework. We define an optimal transport module to align and accentuate the common feature space of hetero- geneity of EHRs. We empirically show that our model consistently outperforms existing state-of-the-art synthetic EHR generation methods both in fidelity by up to 3.10% and utility by up to 7.16%. Additionally, we show that our method can be successfully used in privacy-sensitive settings, where the original patient-level data cannot be shared.

Emory University joyce. emory edu Published as a conference paper at ICLR 2024
A FLEXIBLE GENERATIVE MODEL FOR HETEROGE- NEOUS TABULAR EHR WITH MISSING MODALITY
Huan He
William Hao
Department of Biostatistics
Department of Computer Science
University of Pennsylvania
Emory University wi Hilliam. hao@emory.ccmmm edu
huan, hew.oneee@ennmmeeiiiinn.eeeee www.:iiciinee.nneee...mmm edu
Yuanzhe Xi
Yong Chen
Department of Mathematics
Department of Biostatisties University of Pennsylvania
Emory University yuanzhe x xi @emory. edu
ychen 2 upenn. edu
Bradley Malin
Joyce Ho
Department of Biomedical Informatics Vanderbilt University
Department of Biostatistics
b. A.maling amaainnvumee.ommmmmmmm org
ABSTRACT
Realistic synthetic electronic health records (EHRs) can be leveraged to acceler- ate methodological developments for research purposes while mitigating privacy concerns associated with data sharing. However, the training of Generative Ad- versarial Networks remains challenging, often resulting in issues like mode col- lapse. While diffusion models have demonstrated progress in generating qual- ity synthetic samples for tabular EHRs given ample denoising steps, their perfor- mance wanes when confronted with missing modalities in heterogeneous tabular EHRs data. For example, some EHRs contain solely static measurements, and some contain only contain temporal measurements, or a blend of both data types. To bridge this gap. we introduce FLEX -EHR---E------ a versatile diffusion model tai- lored for heterogeneous tabular EHRs, equipped with the capability of handling missing modalities in an integrative learning framework. We define an optimal transport module to align and accentuate the common feature space of hetero- geneity of EHRs. We empirically show that our model consistently outperforms existing state-of-the-arr synthetic EHR generation methods both in fidelity by up to 10%0% and utility by up to 7. 7.1.% Additionally, we show that our method can be successfully used in privacy-sensitive settings, where the original patient-level data cannot be shared.
INTRODUCTION
The widespread digitization of health data has enabled the training of deep learning models for precision medicine, in the form of personalized prediction of risks and health trajectories (Rajkomar et al., 2018; Miotto et al., 2016). However, there various concerns over patient privacy that need to be accounted for order to collect the large quantities of data needed to train robust models. As such, it is challenging for researchers to obtain access real electronic health records (EHRs). One approach to mitigate privacy risks is through the practice of de-identification in the form of data perturbation and randomization (EI Emam al., 2015; Mcl McLachlan et al., 2016). However, the blind application of de-identification leads to records that are vulnerable to re-identification attacks (Narayanan & Shmatikov, 2008). An alternative approach that receiving increasing attention the creation and dissemination of synthetic datasets that aim to capture many of the complexities of the
"Equal Contribution

---

Published as a conference paper at ICLR 2024
original data set (e.g., distributions, non-linear relationships, and noise). Synthetic data can yield records that are robust against re-identification. To support the creation of realistic and synthetic data, generative models have emerged the key element to advance precision medicine.
Although several distinguished efforts for synthetic EHR generation exist (Bing et al.. 2022; Choi et al., 2017; Torfi & Fox, 2020; Yan et al., 2020; Li et al., 2021a; Theodorou et al., 2023), designing an effective generative model for EHRs remains a challenging task for two key reasons: (1) hetero- geneous features encompassing both static and temporal measurements and (2) missing data. EHRs consist of diverse and multi-dimennional data that contain static features (e.g.. race and gender) and temporal measurements (e.g.. heart rate, lemperature, blood pressure). We refer this heteroge- neous tabular EHRs (i.e., not including unstructured text). Existing works (Choi et al., 2017; Torfi & Fox, 2020; He et al., 2023) extended classical deep generative models to EHRs, building upon generative adversarial networks (GANs), autoencoders, or diffusion models. However, these meth- ods are limited to generating static measurements including billing codes, and ignore the temporal features (e.g., lab results repeated over time), hindering their utility for downstream tasks, On the other hand, (Biswal et al., 2021) I) focused on generating temporal features only. EHR-M-GAN (Li et al., b021b) circumvents this problem by training two separate encoders for each modality (static and temporal features), but it lacks the capability of training using data with missing modality.
Missing data, is complex and pervasive problem in medical records and public health research, affecting both randomized trials and observational studies (Haneuse et al., 2021). Nevertheless, existing methods assume completeness of the data, yielding less reliable generation when facing missing data. Reasons for missing data can vary substantially across studies because of dropout or loss follow-up, missed study visits, or an unrecorded measurement during an office visit. Merely removing missing data is not feasible because the learned generative model is likely to suffer from selection bias, whereby the results downstream utility (ie, to the intended or target patient popula- tion) is compromised). This underscores the need for generative methods for EHRs that 1) handle heterogeneous (or mixed-type) EHRs and 2) expand the scope of existing generative methods by supporting missing observations of EHRs. A clear articulation of assumptions is critical because of the different mechanisms that can induce missing data. We consider the scenario of missing modality not at random (MMCAR) in work suggested by Tang et al. (2020); Haneuse et al. (2021); Gianfrancesco & Goldstein (2021). In this scenario, not every record will have data associated with each modality. For example, medications (documented static data) may not have corresponding records of the measurements of patients physiological status due to loss of records. Another exam- ple is protecting patient privacy by omitting one modality where they can be easily identified (e.g.. a woman above 100 with otherwise similar temporal features to the population).
Present Work. To address the aforementioned limitations, for the first time, we introduce FLEXGEN-EHR, a flexible generative framework for simultancously synthesizing heterogencous longitudinal EHR data. Specifically, we focus on generating both static and temporal records jointly. Patient trajectories with high-dimensionallty and heterogeneous data types (both continuous- valued and sinerete-valued timeseries) are generated while the underlying temporal dependencies and temporal-static correlations are captured.
In summary, our contributions are as follows: ! We formalize the challenge of generating hetero- geneous Electronic Health Records (EHR) in the presence of missing modality.
② We present FLEXGEN- -HHR, a latent diffusion method demonstrating superior generation fidelity, evidenced by reduction of Maximum Mean Discrepancy (MMD) by up to 3. 3.10%. and enhanced utility, with an increase Area Under the Precision-Recal Curve (AUPR) by up to 7.16%.
3 We introduce an innovative solution to the missing modality issue by formulating an optimal transport problem the embedding space, enabling the construction of meaningful and reasonable latent embedding pairs to solve the missing correspondence in the data.
④ We empirically verify that FLEXGEN-EHR maintains high standards of generation and utility even in instances of missing modality, solidifying its applicability and reliability in practical, real- world scenarios involving incomplete data.
2

---

Published as a conference paper at ICLR 2024
Temporal Features
Encoder
Decoder
Hoart.oto
poryy
Enc
Dec
2emp
Sien cemporuture
Liatent Unified
Static Features
LDM
Gander
N
E
Dec
2,tat
Figure 1: Conceptual model illustration FLEXGEN-EHE allows training using data with missing modality (details in 4.3).
RELATED WORK
Here, we discuss the related works on synthetic EHR generation and diffusion-based models.
In this context, within defined categories and remain consistent. Similarly, "numerical" and 'temporal' are used synonymously, denoting features that can vary over time.
"categorical' and 'static' interchangeably. both referring to features that fall
Synthetic EHR Generation.
The predominant approaches for EHR tabular generation currently rely on GAN-based methods, which utilize two neural networks: the generator and the discrimina- tor. MedGAN (Choi et al., 2017), CorGAN(Tori & Fox, 2020), and EHR-WGAN (Zhang et al., 2019) utilize GANs for generating patient feature matrices (diagnosis only). However, challenges still remain for GAN-based methods, such as the mode collapse problem (Li et al., 2021c) and the alignment between heterogeneous data types.
While several works focus static data generation such as MedGAN and CorGAN, it's essential note that real-world EHRs encompass mixture of heterogeneous data types. These include temporal features (e-g.. blood test results) and static features (e.g., sex and ethnicity), also denoted as numerical and categorical features. Although VAE performs the common method for static feature generation, when it comes to synthesizing temporal features, current GAN-based models, such as C-RNN-GAN (Mogren, 2016), TimeGAN (Yoon et al., 2019), rely heavily on adopting recurrent neural networks (RNNs) for both their generator and discriminator components. Therefore, in the context heterogeneous Tabular EHR data generation, GAN-based methods face challenges in learning joint distribution representation due the diverse structures of their generators.
Very recently, EHR-M-GAN (Li et al., 2021b) is proposed as a method for synthesizing heteroge- neous EHR data. However, it operates under the assumption that representations learned by autoen- coders can seamlessly integrate with those from bidirectional Long Short- Term Memory(BLSTMM (Zhou al., 2016), a premise that might not be reliable given the inherent structural differences between the models. In contrast, FLEXGEN-EHR introduces integrative learning framework. For both static and temporal features, they are learned under separate VAE architecture, which naturally aligns common feature space of heterogeneity for EHR Tabular datas.
Tabular Diffusion. First proposed in Soohl-Dickstein al., 2015), diffusion models are family of latent variable generative models characterized by forward and reverse Markov process. Dif- fusion models have excelled in various mmage-generation tasks and now extended beyond computer visions (Chen al., 2021; Vahdat et al., 2021). However, only few works have been proposed to introducing diffusion models for tabular data generation so far, especially when encompassing with heterogenous datasets like EHRs.
As an early adopter of diffusion models for EHR generation, MedDiff (He et al., 2023) primarily focuses utilizing Gaussian diffusion process to generate numerical EHR data. But it is important to note that real-world EHRs is formed by a mixture numerical features (eg., respiration rate) and categorical features (eg., LOS and mortality). MedDiff generates diagnosis count matrix only, leav-
3

---

Published as a conference paper at ICLR 2024
ing out numerical features unsolved. The more recent model, TabDDPM (Kotelnikov al., 2023), addresses this problem by generating heterogeneous tabular data encompassing both static and tem- poral features, which could be denoted numerical and categorical well. It employs Gaussian quantile transformation for temporal features, and a one-hot encoder for each static features. A re- cent study (Ceritli al., 2023) evaluates the performance of implementing TabDDPM into EHR generation, which out performs other contemporary models.
common observation among existing diffusion-based EHR models is their inability to generate categorical features or their tendency to treat numerical and categorical features independently. Yet, in practical scenarios like bedside data analysis in hospitals, numerical features (eg.. respiration rate) and categorical features (eg., diagnosis and admission type) often have intrinsic logical re- lationships. For a synthetic model to generate realistic heterogeneous tabular EHR dataset, must adeptly discern and represent the underlying relationships between numerical and categorical features. Different from TabDDP that concatenates the numerical and categorical features to the net- work, FLEXGEN-EHIE adeptly discerns and represents the underlying relationships between static and temporal features.
PRELIMINARIES
Diffusion. Diffusion models leverage a pre-defined forward process in training, where clean dis- tribution (ar) can be corrupted to noisy distribution q( (z)) at a specified timestep t. Given pre-defined variance schedule {:rh the noisy distribution at any intermediate timestep is given by:
((<1(=0) = N
(Vorso (1 ((n))o))NN((aa)))
a.) de II (1-))).
To reverse such forward process, generative model 0 learns to estimate the analytical true posterior in order to recover -t-1 from x, as follows:
min D ( =o) Vt E [1....,T}, T}, and such an objective can be reduced a simple denoising estimation loss []: LDEPM = [o, [|c- co(aao + /1 - āgc, t)(?
#
(1)
For the case where label information is available, the model is trained to estimate the noise as above in both conditional cases co(x,, y, t) with data-label pairs (ro, y) and unconditional case ca (2x, t). In the sampling. the label-guided model estimates the noise with a linear interpolation
= (1 w)ea (Ze, y, wea(Ze,) ()
to recover 11- which is often referred to Classifier-Fres Free Guidance (CFG) (Ho, 2022).
Latent Diffusion Model. To improve the efficiency, the Latent Diffiusion Model (LDM) (Rombach al., 2022b) introduces explicit separation hetween the compressive and generative learning phases of training diffusion models. Central to this approach is the use of an autoencoding model, consisting of encoder E--) and decoder (-)). This autoencoder is designed capture com- pressed latent space for diffusion model, which, upon decoding, closely resembles the original data space its perceptual attributes.
Given a tabular data entry æ with dimensions x E If' WW the encoder E maps z into latent representation denoted by z E(x). Subsequently, the decoder D reconstructs the original data entry from 27, resulting in J = D)z) D(E(x)). This latent space, with dimensions 2 E A offers the advantage reduced computational demands while preserving the perceptual integrity of the regenerated samples.
The objective function tailored for training the LDM is expressed as:
L(l)= = B
[ (2 - 2
(2)
Here, the latent representation z is derived during the training process using Encoder E. Once generated, can be decoded back to its original data form using 1).
4

---

Published as a conference paper at ICLR 2024
3.1 OPTIMAL TRANSPORT
Optimal transport formalizes the problem of finding minimum cost mapping between two point sets, viewed discrete distributions. Specifically, assume two empirical distributions over em- beddings, e.g., embeddings of static features z and embeddings of temporal features z!
11 = Σ pzd and Σ q 1=1
(3)
Here, p and q are non-negative vectors of length T and J that up 1. We denote their proba- bilistic couplings, and cost matrix C, as: We find a transportation map realizing:
inf { c(x, ((x)))d(xx) T
[ Tw
(4)
where the cost c(x, r(x)) is typically just ||x- r(x)] and T implies that the source points must exactly map to the targets.
FLEXGEN-EHR
4.1 PROBLEM FORMULATION
Heterogeneous Tabular EHR Generation. Given heterogeneous tabular EHR data D {xf,, x7 v:) of N electronic records where x? E RTT contains time-invariant features, x! R contains time-dependent features, and y: represents the label information of inter- est. Here, N refers to the number examples (unique IDs in the data table), is the number of timesteps after "discretizing" the observation period into time bins of size. The dimensionalities of the time-invariant and time-dependent features are denoted by m and d, respectively. The goal is to generate synthetic EHR data D such that L(D, D) is minimized where L is any divergence measurement, such as mean-squared loss and maximum mean discrepancy (MMD).
The above problem can be extended to the setting where every record will have data associated with each modality. Thus, the heterogeneous tabular EHR data can take the form D = {xf v1) (x} 33) (xę. xX. 3k.) for i 1.... J.j = 1,... k 1,... K. Here, J, K represent the number of records containing only static, only temporal, and both types of information, respectively.
Heterogeneous Tabular EHR Generation with missing modalities.
4.2 LATENT DIFFUSION MODEL ON EHRS
Given a sample (x. x 3i). FLEXGEN-EHR utilizes dual encoder-decoder framework to in- dependently obtain static and temporal latent embeddings, denoted as 2i and z?" Specifically, an encoder Encs operates on static features, embedding the patient information as z = Enc (x²), while another encoder, EncT, is specialized for temporal features, representing repeated measure- ments. The implementation of two distinct encoders motivated by the fact that single encoder is inadequate to capture meaningful embeddings for both feature types due to the distinctive domains of static and temporal features. Our empirical investigations confirm that utilizing an LSTM encoder for temporal features and an MLP encoder for static ones yields superior quality embeddings. Fol- lowing the encoding phase, these separately constructed latent representations are concatenated to construct fused latent representation, represented as 7, 1 : A latent diffusion model G is then trained on this unified representation 74-
The choice of using a latent diffusion model is significant not only because generation with diffusion on a compact scale is more expedient, but also due to our observations that, given the high-dimension of EHR data (d > 5000), leveraging diffusion models even the recently proposed TabDDPM tends to encounter training failures and generates samples at least 5 times slower. Thus the integration of lanal-encoder-dccodee the achievement of hyper-paramete robustness, high-quality synthetic samples, and more efficient generation procedures. This optimized combination ensures the precise and coherent embedding of
framework and latent diffusion model is synergistic, culminating in

---

Published as a conference paper at ICLR 2024
both static and temporal features, and provides comprehensive and nuanced representation of the complex, multifaceted data encountered in healthcare settings.
4.3 LATENT SPACE ALIGNMENT
The above section assumes that both modalities are present, which cannot readily handle the missing modality. Here, we define the optimal transport problem for solving data with missing modalities such that x; (x., ,NA, yi) or x; (NA. y.)). where NA denotes not available. Instead of mapping-based methods, which rely on nearest-neighhor computation to infer the NA data, we observed that latent space embedding models, trained on disparate features, manifested analogous geometric patterns and behaviors. This drives to posit that the latent embedding spaces of hetero- geneous EHRs can potentially be transformed reciprocally through linear transformations.
Specifically, suppose the sample size of 2 and z!! are identical, let Z" [2.....,,2 2 E /R and z [xf.... ......... 2 Rllsx represent the embedding matrices of temporal and static features, respectively. Here, It and is denote the size of the embeddings and I represents the number of samples. A viable solution entails solving the following linear system:
min AEO())
I AZ 1/:3
(5)
where O(1) = (A Risxit A This formulation has a closed-form solution that can be easily obtained. However, the linear system approach only solves the supervised version of the problem as it requires known correspondence between the columns of z and ZT_ Clearly, it cannot solve the alignment problem without the correspondence caused by the missing modalities.
We solve this issue by "imputing" the correspondence via modified Gromov-Wasseerteinn wwssersteis-baaaaeee based manifold alignment algorithm. The algorithm exploits the inner structure of the embedding space and uses the available label information as global correspondence between two spaces. We consider two measure spaces expressed in terms of within-embedding space similarity (cosine similarity used here) matrices CT E gx'x and CS E R/ For clarity, we represent i, J. k and as indices for individual samples, acknowledging a slight abuse of notation herein. We now define a loss function
between similarity pairs: IR R R. where L (Ch, C2 measures the discrepancy between
the distances T%e and C. In this work, we define L CT can be understood the cost of "matching" i to and k to l. In addition, it avoids misalignment by penalizing the mis-correspondence when pairs have similarities between the embedding space but have different outcomes. All the relevant values of L(.,.) can be put in 4-th order tensor
We: C, = H y C))
LE RERIIIXXX where Lijut = L (C,, C.)). Similar to the definition 3.1, we seek a coupling I' e
[XX specifying how much mass transfer between each pair of points from the two embedding spaces. The Gromov- Wasserrtenn problem is then defined as solving
GW (CT, C ppp..)
min rell(P-g)
Lighl iyT -TH(T) L.,.
Σ
where Tay is the relative probability that matches embedding x. to z5. We add an entropic regular- ization penalty to solve the problem faster. Once we have solved equation 6, the optimal transport coupling I' provides an explicit (soft) matching between temporal and static features, which can be interpreted as probabilistic translation: for every pair of embeddings, it provides a likelihood that these two embeddings are correspondent of each other. Now, we can solve the least square problem with full correspondence information:
min 1 AZT
Instead of equation 5, this formulation can achieve alignment when the number of samples with missing modalities does not necessarily to be equal to that of non-missing data becasue the learning problem is not supervised correspondence problem. Suppose static features in x; are not available, we can obtain the most likely latent embedding z by running a simple transformation A r= To obtain 2! from z5. resolving the problem considering z as the source and 2? as the target is necessary to obtain reliable and accurate solution.

---

Published as a conference paper at ICLR 2024
Generation. Next, elucidate the procedure for generation. Whether modality is missing or present, we demonstrate that FLEXGEN- EHR can train a latent diffusion model using fused rep- resentations z; = =z]= shown in equation 2. If there is missing data, "impute" the full representations by solve the OT problem equation to find reasonable correspond- ing features.
Upon the successful training of the diffusion model, our first step is to generate synthetic fused representations. Subsequently, decoupled embeddings (by simply splitting) into respective decoders as follows:
x Dec (z"), X Dee (z9)
The overview of FLEXGEN-EHRE presented in Figure 1.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines. In our study, we consider six methods as baselines: i) VAE (Kingma & Welling, 2022), Variational Autoencoder model, traditionally used in generating high-dimensional data; ii) MEDGAN (Choi et al., 2017), a GAN-based model that generates low-dimensional synthetic records and decoded with an autoencoder; iii) CORGAN (Torfi Fox, 2020), another GAN-based model that, akin to MedGAN, amalgamates Convolutional Generative Adversarial Networks (Con- vGANs) and Convolutional Autoencoders to synthesize and reconstruct medical records; iv) EHR- M-GAN (Li et al., 2021b), GAN-based model tailored for longitudinal heterogeneous EHRs; v) TABDDPM (Kotelnikov et al., 2023), a diffusion model specialized handling tabular data with the unique capability of addressing the challenges presented by the heterogeneous nature of EHRs; and vi) LDM (Rombach et al., 2022a), latent diffusion model that decomposes the generation process into a sequence of autoencoders and diffusion models (DMs).
These models introduce unique methodologies and focus on varying aspects of EHR data synthesis and reconstruction, providing comprchensive perspective the potential approaches and their efficacy handling heterogeneous medical records. More details are available in Appendix B.I.
Datasets. We use woaldw de-identified EHR datasets, MIMIC-III (Johnson et al., 2016) and eICU (Pollard et al., 2018). Both are inpatient datasets that consist of varying lengths of sequences and include multiple static and temporal features with missing components.
We adopted the preprocessed datasets from FIDDLE (Tang et al., 2020). The cohort numbers and dimensionalities of extracted features summarized in Table 1, where and d denote the dimen- sions static and temporal features respectively. More details on datasets and preprocessing are available at Appendix B.2.
Table 1: Summarization of the sample size and dimensionalities of extracted features. ARF means acute respiratory failure.
MIMIC-III
N
eICU
N
d
In-hospital mortality
In-hospital mortality
8,577
96
7.307
77,066
146
2,382
(48 h)
(48 h)
ARF (4h)
15,873
98
4,045
ARF (4h)
138,840
717
5,854
Evaluation.
We evaluate the effectiveness of FLEXGEN-EHR on fidelity, utility on downstream machine learning tasks, and also demonstrate its ability to preserve privacy. Fidelity: We evaluate the quality of synthetic data through various metries that assess how closely the synthetic data resembles real data. Following Li et al. (2021b); Yoon et al. (2023), we report R² values (the higher the better) and maximum mean discrepancy (the lower the better). The details can be found B.3. Utility: We focus on prediction tasks and train a random forest (RF). We report the area under the ROC and PR curves (AUROC and AUPR, respectively) using test datasets. Privacy: Unlike de-identified data, there is no straightforward one-to-one mapping between real and synthetic data (generated from random vectors). However, there may be some indirect privacy leakage risks built on correlations between the synthetic data and partial information from real data. Following Torfi & Fox (2020);
7

---

Published as a conference paper at ICLR 2024
Theodorou et al. (2023), we consider the membership inference attack that adversaries may apply to de-anonymize private data.
Implementation Details.
We perform experiments on two settings: i) an easier setting where we train models on data without missing modality, and ii) harder setting where we randomly delete p% static samples and 9% temporal features. We implemented FLEXGEN- LLEXGEN-EHR with PyTorch. For training the models, we used Adam (Kingma& & Ba, 2015) with the learning rate set to 0.001, and a mini-batch of 128 on a machine equipped with one Nvidia GeForce RTX 3090 and CUDA 1 11.2. Hyperparamters of FLEXGEN-EHR are selected after grid search. We use timestep of 50 and a noise scheduling from 1 x 10 to 1 x 2
5.2 Q1: RESULTS- RESULTS-COMPARISON IN FIDELITY
We first evaluates the statistical similarity of the generated and real data. For each method, we gen- erate synthetic dataset of the same size as the training dataset. We calculate the probabilities for each feature (dimension values. Results are presented in Table 2. Across all methods, we find that FLEXGEN-EHE consis- tently outperforms baselines by making an average improvement of R² (0.025) and MMD (0.031). In terms of correlation, FLEXGEN-EHR improves Rs by 0.027, 0.019, 0.019, and 0.032 over the strongest baseline on each dataset respectively. In terms of distance, FLEXGEN-BE -HEREN-EHRRR reduces MMD by 0.030, 0.028, 0.014, and 0.049 over the strongest baseline each dataset respectively. Although it is not our main contribution, results demonstrated that the improvement can be attributed to the dual encoder-decoder with latent diffusion.
within the real and synthetic datasets and then compute R2 and MMD
Table 2: Generation Fidelity, R² (f) correlation and MMD (L) distance between synthetic and real the highest correlation.
datasets.
Dalasel
AIMIC-Mottalty
MIMIC-ARF
Mortaliyy
eICU-ARF
R² (t)
MMD (J)
R²
MMD
R²
MMD
R²
MMD
VAE
0.632 =0.052
1,514
+0.048
0.577
+0.074
0.981
+0.048
0.694 +0.031
0,857
+00029
0.655
20.034
0.802
10.016
MEDGAN
0.639 00.047
1.108
10.037
0.595
10.069
0,831
10.052
0.712
+0.037
0,832
1:0044
0.662
10.044
0.786
10.019
CORGAN
0.656 =0.051
1.093
+0.441
0,612
+0.055
0,830
+00059
0.724
+0.054
0,712
+0.037
0,671
=0.031
0.760 +0.015
TABDDPM
0.395 10.60
1.636
100067 0.525
10.059
1.182
10.041
0.653
1:0.46
0,774
1:0.045
0.617
110.088
0.784 10.012
LDM
0.694 10.03
0.755
10.042 0.658
10.052
0.635
10.039
0.758
1.0049 0.662 1.0.038
0.736
100033
0.710 10.017
EHR-M-GAIN
=0.04
0.711
=0.045
0.676
#0.047
0.614
+0.043
0.781
+0.022
0.617
#0.033
0.762
#0.019
0.667 #0.020
FLEXGEN-EHR
0.739 =0.05
=0.042
0.695 1:0051
0.586
0:00445
0.800
+:0037
=10.035
=0.024
0.0019
5.3 Q2:
IN UTILITY
Following the experimental setup in (Tang et al., 2020), we trained ML models to predict inhospi- tal mortality and acute respiratory failure (ARF) in our evaluation of utility. Overall, FEN- EHR has won 4 out of 4 tests (2 metrics in 4 datasets) and makes an average improvement of AUROC (6.37%) and AUPR (7.16%) over with the strongest baseline across datasets.
Table 3: Generation Utility. The downstream classifier is trained using either real data (the first row) or synthetic data (2-8 rows). We report AUROC (t) and AUPR (1) evaluated on real test the highest comparable performance with real data.
Dataset
MIMIC-Mortality
MIMIC-ARF
eICU-Mortalit
cICU-ARF
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
Real data (goul)
0.856 =0.032
0.445
1:0111
0.817
10.023
0.657
+0.029
0841 +0.012 0.401
+0.027
0.714
100014
0.269
10.032
VAE
0.731 =0.031
0.394
=
102
0.693
+0.019
0,582
+0.025
0.732
+0.012
0.298
+0.028
0.634
10.012
0.188 #0.035
MEDGAN
0.754 =0.030
0.401
+0.109
0.766
+0.021
0.602
+0.021
0.743 +0.015
+0.029
0.647
=0.015
0.211 #0.022
CORGAN
0.759 10.027
0.400
10.114 0.783 10.025
0.613 10.023
0.752
1.0.00
0.317 100024
0.649
100017 0.214 10.025
TARDDPM
0.712 00022
0.375
10.135
0.788.10.024
0.626 10.027
0.741
100012 0.305 10.022
0.615
10.015 0.205 10.028
LDM
=0.03
12.12
0.775:10.000
0.621 #0.022
0.749 +0,011
0.322
#0.022
0.668
=0.019
0.223=0.029
EHR- T-R-N-GAN
0.778
20.05
0.409
10.
0.787110.25
0.645
4..024
40,010
1:0.028
10022
0.216 10.031
FLEXGEN-EHR
0.794 00.35
0.428
10.116
0.792
+00017
0.674 0.0025
0.764
+.0013
0.356
1-0.028
0.689
-0.014
0.254 00.02
8

---

Published as a conference paper at ICLR 2024
Table 4: Generation Privacy. Accuracy reported as the membership inference metrics.
Dataset
MIMIC Moric-Moriality MIMIC-ARF elCU-Monality eICU-ARF
Real (Ideal)
0.5
0.5
0.5
0.512
0.519
MEDGAN
CORGAN
0,513
0.481
0,515
0.483
TARDDPM
0.483
0.521
0.521
0.481
LDM
0.482
0.518
0.519
0.483
EHR-M-GAN
0.485
0,514
0.517
0.482
FLEXGEN-EHR
0.483
0.484
0.519
0.481
5.4 Q3:
IN PRIVACY
In this section, we quantify the vulnerability of all methods to adversarys membership inference attacks (Hayes et al., 2018). As shown in Table there is not much difference in the privacy metrics (e.g., the accuracy of am attacker changed from 0.482 to 0.513 on MIMIC-Mortality) It shows that FLEXGEN-EHR maintains level of privacy preservation that aligns with existing methods, while achieving superior data fidelity. is essential to recognize that this trade-off between generation quality and privacy guarantee is shared characteristic among all Liffusiion-bsse generative models.
5.5 Q4: RESULTS- HETEROGENEOUS TABULAR EHR GENERATION WITH MISSING MODALITY
We investigate the capability of FLEXGEN-EHHE generate with missing modality effectively. We randomly designate 5% of samples as lacking temporal features and 9% samples as lacking tem- poral features. We compare with EHR-M-GAN, the strongest baseline selected for this study. In the case of the baseline, we impute samples with absent modality using kNN and proceed to train both the generative and classification models, as EHR-M-GAN is not able to handle such scenarios. Investigation into Generation Fidelity. We report the fidelity metrics (R²) in Figure 2 and 3. It can be observed that FLEXGEN- EHR maintains high generation fidelity even when number of samples assess
with missing modality increases. Examination of Generation Utility. Subsequently, the generation utility. The outcomes shown in Figure 4 demonstrate that the simple imputation of samples notably diminishes discriminative capability, whereas FLEEGEN--EHR problem, delivering performance that is comparable with full samples.
alleviate this
2
AUROC
6033 6203 6204 MA0
3033
Massing
Maxing percantage (p.0)
Mising percentage (p.0)
Figure/ 2: R on MIMIC- Mortality
Figure 3: R² on elCU-ARF Figure AUROC on eCCU-ARF
CONCLUSION
We introduce FLEX EHR, a novel generative model that both flexible and easy-to-use, and can be trained data with missing modality. We introduce an innovative solution to the missing modality issue by formulating an optimal transport problem in the embedding space, enabling the construction of meaningful and reasonable latent embedding pairs to solve the missing correspon- dence in the data. By constructing reasonable fused representations for data with missing modality, FLEXGEN-EHR is able to training using all samples without requiring removing then other im- putation techniques. Results demonstrate the potential of FLEXGEN EHR as a general strategy for EHR generative models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded in part by the National Science Foundation (NSF) awards IIS-214541,, CNS- 2124104, DMS-2208412, and National Institutes of Health (1R0ILM014344, IR01AG077820. ROILMO1 RO1AI130460, A01733333 R56AG074604, RO1LM013519, R56AG069880, UOITR003709, RFI R21AI167418, R21EY034179)
REFERENCES
Brett K. Beaulieu-Jones, Daniel R. Lavage, John W. Snyder, Jason Moore, Sarah Pendergrass, and Christopher R. Bauer. Characterizing and managing missing structured data in electronic health records. 2017. doi: 10.1101/167858.
Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab. Conditional generation of med- ical time series for extrapolation to underrepresented populations. PLOS Digital Health, 1(7): e0000074, 2022.
Siddharth Biswal, Soumya Ghosh, Jon Duke, Bradley Malin, Walter Stewart, Cao Xiao, and Jimeng Sun. Eva: Generating longitudinal electronic health records using conditional variational autoen- coders. In Machine Learning for Healthcare Conference, pp. 260 280,2282-00 2021.
Taha Ceritli, Ghadeer O. Ghosheh, Vinod Kumar Chauhan, Tingting Zhu, Andrew Creagh, and David A. Clifton. Synthesizing mixed-type electronic health records using diffusion models. 2023.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In Proc. of ICLR, 2021.
Edward Choi. Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Gen- crating multi-labe label discrete patient records using generative adversarial networks. In Proceedings of the 2nd Machine Learning for Healthcare Conference, 2017.
Khaled El Emam, Sam Rodgers, and Bradley Malin. Anonymising and sharing individual patient data. BMJ, 350, 2015.
Milena A Gianfrancesco and Neal D. Goldstein. A narrative review the validity of electronic health record- based research in epidemiology. BMC Medical Research Methodology, 21, 2021. URL http://pppeeemaannccccccoooroooo
Sebastien J-P A Haneuse, David E Arterburn, and Michael J. Daniels. Assessing missing data assumptions in ehr-based studies: A complex and underappreciated task. JAMA network open, 2:e210184, 2021. URL http::/pp..sommaasssss....o.ooop
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Membership infer- ence attacks against generative models. 2018. URL http:://pp..smmmnicccooooorogggg 202588705.
Huan He, Shifan Zhao, Yuanzhe Xi, and Joyce C Ho. Meddiff: Generating electronic health records using accelerated denoising diffusion model. 2023.
Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207.12598, 2022. URL https://api. semanicssholarrrrgCopssDDDDD.
Alistair Johnson, Tom Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Celi, and Roger Mark. Mimic-ili, freely accessible critical care database. Scientific Data, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2022.
10

---

Published as a conference paper at ICLR 2024
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564 17579. PMLR, 2023.
Jin Li, Benjamin J. Cairns, Jingsong Li, and Tingting Zhu, Generating synthetic mixed-type longi- tudinal electronic health records for artificial intelligent applications. NPJ Digital Medicine, 6, 2021a. URL
Jin Li, Benjamin J. Cairns, Jingsong Li, and Tingting Zhu, Generating synthetic mixed-type longi- tudinal electronic health records for artificial intelligent applications. NPJ Digital Medicine, 6,
2021b. URL
Wei Li, Li Fan, Zhenyu Wang, Chao Ma, and Xiaohui Cui. Tackling mode collapse in multi- generator gans with orthogonal vectors. Pattern Recognit, 110:107646, 2021c. URL hitps:
Scott McLachlan, Kudakwashe Dube, and Thomas Gallagher. Using the caremap with health in- cidents statisties for generating the realistic synthetic electronic healthcare record. 2016 IEEE International Conference Healthcare Informatics (ICIII), 2016
Riccardo Miotto, Li Li, Brian A. Kidd, and Joel T. Dudley. Deep patient: An unsupervised repre- sentation to predict the future of patients from the electronic health records. Scientific Reports, 6, 2016.
Olof Mogren. C-rnn-gan: Continuous recurrent neural networks with adversarial training. ArXiv, abs/ bbs/1611 1.09904, 2016. URL
Arvind Narayanan and Vitaly Shmatikov. Robust de-anonymization of large sparse datasets. In 2008 IEFE Symposium on Security and Privacy, 2008
Tom J. Pollard, Alistair E. W. Johnson, Jesse Daniel Raffa, Leo Anthony Celi, Roger G. Mark, and Omar Badawi. The eicu collaborative research database, freely available multi-center database for critical care research. Scientific Data, 5, 2018. URL 52182706.
Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan Shean-Sheeggaaa Foo, Zhenghua Chen, Min Wu, C. Kwoh, and Xiaoli Li. Adatime: A benchmarking suite for domain adaptation on time series data. ACM Transactions on Knowledge Discovery from Data, 17:1 - 18, 2022. URL
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj. Michaela Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Yi Zhang, Ger- ardo Flores, Gavin E. Duggan, Jamie Irvine, Quoe Le, Kurt Litsch, Alexander Mossin, Justin Tan- suwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Kather- ine Chou, Michael Pearson, Srinivasan Madabushi. Nigam H. Shah, Atul J. Butte, Michael D. Howell, Claire Cui, Greg S. Corrado, and Jeffrey Dean. Scalable and accurate deep learning with electronic health records. upj Digital Medicine, 1(1):18, 2018. doi: 10. 00.1038/44746-000 01077744--00--275 018-0029-1. URL https://doi.org wwtp:/ooooor///// 1tt.110.118800 10000001114000 114460018-0222
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10693 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2022b.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proc. of ICMI,, 2015.
Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W. Sjoding, and
Jenna Wiens. Democratizing ehr analyses with fiddle: flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association
JAMIA, 27:1921 1934, 2020. URL

---

Published as a conference paper at ICLR 2024
Brandon Theodorou, Cao Xiao, and Jimeng Sun. Synthesize high-dimensional longitudinal elec- tronic health records via hierarchical autoregressive language model. Nature Communications, 14, 2023. URL https:://ppi.semmaatt www.aaacccccccaoooroooosoooooooa
Amirsina Torfi and Edward A. Fox. Corgan: Correlation-capturiin convolutional generative adver- sarial networks for generating synthetic healthcare records. In FLAIRS Conference, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based based generative modeling in latent space. In NeurlPS, 2021.
Tianshi Xu, Anthony Austin, Vasileios Kalantzis, and Yousef Saad. A parallel algorithm for comput- ing partial spectral factorizations of matrix pencils via chebyshev approximation. SIAM Journal on Scientific Computing, 0(0):S324S3351) S351, 2022.
Chao Yan, Ziqi Zhang, Steve Nyemba, and Bradley A. Malin. Generating electronic health records with multiple data types and constraints. In AMIA Annual Symposium Proceedings, pp. 1335, 2020.
Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D. Mooney, and Bradley Malin. multifaceted benchmarking of synthetic electronic health record genera- tion models. Nature Communications, 13, 2022. URL http:://pp.ccmmnniicssoooooogoop 251253032.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative advensarial net- works. In Neural Information Processing Systems, 2019. URL https:/app.semmnticcholaaaa CorpusID:202774781.
Jinsung Yoon, Michel Mizrahi, Nahid Farhady Ghalaty, Thomas Dunn Jarvinen, Ashwin S. Ravi, Peter Brune, Fanyu Kong, Dave Anderson, George Lee, Arie Meir, Farhana Bandukwala, Elli Kanal, Sercan Ó. Arik, and Tomas Pfister. Ehr-safe: generating high-fidelity and privacy- preserving synthetic electronic health records. NPJ Digital Medicine, 6, 2023. URL hitps: http:/isshhooooooooooooooooooooe
Ziqi Zhang, Chao Yan, Diego A. Mesa, Jimeng Sun, and Bradley A. Malin. Ensuring electronic med- ical record simulation through better training, modeling, and evaluation. Journal of the American Medical Informatics Association JAMIA, 2019. URL 203924569.
P. Zhou, Wei Shi, Jun Tian, Zhenyu Qi, B. Li, Hongwei Hao, and Bo Xu. Attention-baser bidi- rectional long short-term memory networks for relation classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016. URL
12

---

Published as a conference paper at ICLR 2024
A FURTHER DETAILS
A.1 DIFFUSION FORWARD PROCESS
Given a data distribution x0)(x0)xo ~ g(X)), diffusion models gradually add noise to the original data distribution until loses all the original information and becomes an entirely noisy distribution (as shown by the xTTr sample in Figure 1). DDPM convolves ((x)) with an isotropic Gaussian noise (0, of in steps produce noise corrupted sequence X1, Xg...., Xy. Xy will converge to an isotropic Gaussian distribution as 7 - co. However, for DDIM, different variance schedule used to produce X1, X2, ......... XT.
.2.....TT: The explicit input distribution q of DDIM is derived as
T
((x1:|(0)) == ((x x Il 1x,,
(8)
where q(xT) (xo) N(Vāxx, (1 -r II and for all 1,
x0), = Ni(V + VI v1-a,-1 x²xt - VaxXD o²"I
VI- aj
Here ā controls the scale of noise added at each time step. At the beginning, noise should be small so that it is possible for the model to learn well, c.g., ã1 02 aT.
The DDIM distribution parameterized to guarantee the marginal density is equivalent to DDPM. However, the key difference between DDPM and DDIM is that the forward process of DDIM is no longer a Markov process. This allows acceleration of the generative process as multiple steps can be taken. Moreover, different reverse samplers can be utilized by changing the variance of the reverse noise. This means DDIM can be compatible with other samplers.
A.2 DIFFUSION BACKWARD PROCESS
For DDPM, the forward process is defined by Markov chain, and thus the true sample can be recreated from Gaussian noise Xy NO, I) by reversing the forward process. However, as noted above, DDIM relies on a family of non-Markovian processes. Denote pe as parameterized neural network, the reverse process with a prior po (xr) = -N I) can be computed as
px(Xo|xi) = N (N(1 - (1o( 1 ool).
px xx)) = 9(xx-1|X1, X1 VI des (xi,t)
Vai
Von.
A.3 DIFFUSION CONDITIONED SAMPLING
To incorporate guidance in the absence of independent classifier fer one could leverage the scores from both conditional and unconditional diffusion models as suggested. Specifically, the uncondi- tional denoising diffusion model, represented as (x),, is parameterized through a score estimator 6x(x,, (). On the other hand, the conditional model po(xly) utilizes score estimator 6a(x,, 4, v). These two models can be learned through single neural network.
To elucidate, conditional diffusion model pe(xly) is trained corresponding data sets (x, y). Through the training process, the conditioning variable is discarded intermittently at random inter- vals. This procedure ensures the model's capability generate images without explicit conditioning, as expressed by the relation: 6o (xx, t) = co [x,, f, =
Furthermore, the gradient of implicit classifier unconditional score estimators. Once incorporated into the classifier classifier-guided guided modified score, there's 's reliance on independent classifier, as shown in the equations below:
be expressed utilizing both conditional and
13

---

Published as a conference paper at ICLR 2024
Vx, logp (v/ xx) Xx, log (x, 1 v) Vxx log (xx)
VI at (Ea (X2,1111 co (xx, 1))
(10)
Eo (x,,t, v) = (x,, v) - v1 a. V log p (y xx)
co (x,, t, + w (co 2, co (xe,
(w - 1)co t, weo (x4, t)
The parameter emerges weighting coefficient in the equation. Its role is to modulate the difference between the conditional and unconditional score estimators, effectively controlling the balance between the two. In essence, determines the extent which the conditional guidance is emphasized over its unconditional counterpart.
A.4 ALGORITHM FLEXGEN-EHR
Algorithm 1: Training of FLEXGEN- EHR
Input: D = {x{xxxx V1)) (x] .y)) (x x yk) for i 1, ,,, = 1, J, 1,... K
A static encoder Enc.. temporal encoder, Ence
static decoder Dec a temporal decoder, Dec
A latent diffusion model G
for e < to E do
Compute static and temporal latent embeddings 2 Enc (xf), Encc (x²)
Learn correspondece information I by solving equation 6
Learn A by solving minA I AZ
Impute missing embeddings by ب A P=1
Train G using fused latent embeddings z; = [z]
Train encoders and decoders based on G(zz)
end
Return EncS, EncT Dec Dec G
B EXPERIMENTAL DETAILS
B. IMPLMENTATIONS
Encoder Selection. we carefully selected the appropriate encoder. This consideration was applied to all our comparisons. For the extraction of temporal features, we utilized 1d neural network (CNN) as the encoder. This configuration was kept consistent across all datasets to ensure a fair comparison, where differences in prediction performance could be attributed to the generative algo- rithm itself. The implementation of the CNN architecture was adapted from a recently published benchmark codebase in the literature Ragab et al. (2022). The ID-CNN architecture consists of three blocks, each consisting of a ID convolutional layer, followed by ID batch normalization layer, rectified linear unit (ReLU) function for non-linearity, and finally, a ID max-pooling layer. VAE: It's generative model that learns to encode and decode data in an unsupervised manner. The model constructed with an encoder, a decoder, and loss function designed to reconstruct data while enforcing constraint the encoding space. The encoder converts input data into set of parameters in a latent space, typically representing the mean and variance of probability distri- bution. These parameters are then sampled to generate new data points in the latent space, which the decoder reconstructs back into the original data space. MEDGAN. It's a discrete-valuee EHR generative model. We flatten the input and use two layer MLP to train the model. The first layer and second layer have hidden size 512 and 128 respectively. CorGAN, Similar to MedGAN, proposed generate discrete EHR matrices. For all other baselines, directly adopt implemen-
14

---

Published as a conference paper at ICLR 2024
Table 5: Summary of MIMIC-III tables used in our analysis
Table name
Description
Example variables
PATTENTS
Information unique pa-
Age, Sex
lients
ADMISSIONS
Information on unique hos- pitalizations
Admission type, Admission location
ICUSTAYS
Information unique ICU stays
Care unit, Ward ID, Admission-to-ICU time
CHARTEVENTS
Charted data, including vi- tal signs, and other infor-
Heart rate, Pain location, Daily weight
mation relevant to patients
care
LABEVENTS
Laboratory
results from the hospital database Fluid intake administered, including dosage and route (c.g., oral or intravenous)
Lactate, WBC
INPUTEVENTS MV
NaCI 0.45%, Whole blood
OUTPUTEVENTS
Fluid output during the ICU stay OR urine Patients procedures during the ICU stay Microbiology specimen from hospital database Documentation of dates and times of certain events
Stool
PROCEDUREEVENT...
CT scan, X-ray
MICROBIOLOGYEVVNTS
Sputum
DATETIMEEVENTS
Last dialysis, Pregnancy due
tations from the corresponding official websites and only modify necessary hyperparameters. Codes of baseline models are available online 4
B.2 DATASETS
MIMIC-III is extensive database that houses anonymized data related around sixty thousand critical care unit admissions from Beth Israel Deaconess Medical Center, gathered between 2001 and 2012. eICU is large-scale, multi-center collection of anonymized health-related data, encompass- ing over 200,000 admissions to intensive care units throughout the United States from 2014-2015. We adopted the preprocessed datasets from FIDDLE (Tang et al., 2020). Table 5 and 6 are directly reused from (Tang et al., 2020) here. In table 5, files PPATIENTS, ADMISSIONS, ICUSTAYS' correspond to static features and others are temporal features. table 6, files 'patient, treatment' correspond to static features and others temporal features
B.3 EVALUATION
In our model, R² serves as an indicator of how well the synthetic data generated by FLEXGEN- EHR corresponds to the real-world data it aims to replicate. This metric is particularly relevant for evaluating the model's performance in terms of data fidelity. In contrast to R², MMD/KS-statii are both parametric tests used for determining if two samples are drawn from different distributions. By quantifying the maximum distance between these functions, MMD/KS. --tttissttssss allow us to rigorously test the hypothesis that the synthetic and original data come from the same distribution. Fidelity Metric: Maximum Mean Diserepancy (MMD). Maximum Mean Discrepancy (MMD) is statistical test used to measure the difference between two probability distributions. For two
Code available
i "ode available at https//ghhaa.ccm/mmp22////aaa
"Code available
*Code available at http://gthaa.coo/ssttttiiinno
Coode available http://gghhbb.cco/////111chcaha
15

---

Published as a conference paper at ICLR 2024
Table 6: Summary of eICU tables used our analysis
heightTable name
Description
Example variables
Age, Sex
patient
Information unique patients, hospitalizations, and ICU stays
Hospital/ward ID
vitalPeriodie
Temperature
Vital signs measured through bedside monitors in- vasively
vitalAperiodic
End Tidal CO2
lab
CPK
Laboratory tests
customl_ab
troponin
medication
Morphine dosage
DrfusionDrug
Active medication orders, the intake of drug through infusions, and intake/output of fluids
Dialysis total
intakcOutput
Culture site (wound)
microLab
Microbiology cultures taken from patients
Organism
note
Abdominal pain
Assessmsennm
Documentation of physician/nurss assessment
Psychological status
nurseCare
Respiratory rate
nurseCharting
Transplant
pastHistory
Relevant past medical history
AIDS
Blood pressure
physicalExam
Results of physical exam (structured)
Verbal score
respiratoryCare
Airway position
Respiratory care data
respiratorCCaartin
Vent details
treatment
Structured data documenting specific, active treat- ments
T Thrombolytics
distributions P and Q, the MMD is defined as:
MMD(P.Q) sup (E==r([(x) Ey-olf())
(11)
JeF
where F is a class of functions.
The Gaussian kernel common choice for MMD and is defined as:
k(x, 9) exp (- ||x 311
(12)
20
where a is the bandwidth of the Gaussian kernel.
Privacy Evaluation. We incorporate the Privacy Assessment methodology delineated in CorGAN for our privacy evaluation paradigm Torfi & Fox (2020). Within this framework, Str designated as the subset of the original dataset utilized for the training of the FLEXGEN-EHR Concurrently, Ste epitomizes the portion that remains unengaged during the training process, and Syns stands as the synthesized dataset. A central component of our analysis is the computation of the Cosine Similarity Score between the aggregated dataset +S and Syns. This metric is elected due to its capacity to yield profound and meaningful correlation assessment. The distinction similarity evaluations is governed by a meticulously determined threshold, ensuring a rigorous and comprehensive analysis. Details on Membership Attack Implementation: We label the real records with the lowest hamming distance to the closest record in the synthetic dataset as positive. We pick hamming distance as our distance metric between patient records throughout our privacy evaluations accordance with (Yan et al., 2022). attack allow us to test the ability of the synthetic dataset to prevent an attacker from inferring whether a real record was used in the training dataset. Ideally, the accuracy around 50%, which is similar to random guess. This shows that neither the model nor the synthetic dataset reveals any meaningful or compromising information about the patient identity in the training dataset.
16

---

Published as a conference paper at ICLR 2024
C DISCUSSION
C.I PRACTICAL USAGE
We wish to clarify that the type of missingness addressed in our study akin structured miss- ingness, where specific patterns or conditions lead to data missingness. This scenario is frequently encountered in real-world healthcare datasets Beaulieu- Jones et (2017). We have expanded our experimental framework to include scenarios with missing features. This extension allows us to more comprehensively demonstrate the robustness of FLEXGEN-EHRin handling other realistic and complex missing data situations. The enhanced experiments showcase LEEXGEN-EHR' su- perior performance in comparison to other models like LDM and MedGAN. The results, clearly depicted in the table below, highlight our models effectiveness in managing various missing data patterns, thereby reinforcing its practical utility in real-world healthcare applications."
C.2 LIMITATIONS AND FUTURE WORK
In this section, we delineate the limitations of our proposed FLEXGEN-EHRmode particularly focusing on aspects related to privacy and generation speed. Acknowledging these limitations is cru- cial for a comprehensive understanding of the model's applicability and scope practical healthcare settings.
remarkable capabil- ity in generating nigh-fidelity synthetic EHR data, it's important to address the subtle yet significant privacy trade-offs associated with diffusion models. Our model, like many in its category, slightly compromises on privacy achieve the desired level of data utility and quality. This is consequence of the inherent characteristics of diffusion models that aim to capture complex data patterns in great detail. This detailed representation, while beneficial for data utility, can inadvertently encode sensi- tive information, potentially making the model susceptible to privacy risks such as re- re-identification or membership inference attacks. We have undertaken measures mitigate these risks, such as evaluating against membership inference attacks, yet the model's intrinsic properties suggest an inevitable balance between data utility and privacy. Future iterations of FLEXGEN-EHRcould ex- plore advanced privacy privacy-preserving techniques, possibly integrating differential privacy advanced anonymization methods to strengthen data confidentiality while maintaining the quality of synthetic data generation.
Privacy Concerns in Diffusion Models While BLEXGEN-EERRemoone
Generation Speed of Diffusion-Basse Models Another limitation for diffusion--asse generative models, like FLEXGEN-EHRO consider, is the generation speed. The inherent architecture of dif- fusion models necessitates reversion process that is inherently sequential and iterative. Each step in this process gradually denoises the data, methodical approach that, while effective in generating high-quality synthetic data, is inherently slower compared to other generative models. This charac- teristic can limiting factor, especially in scenarios requiring rapid data generation. In healthcare settings, where time efficiency can be crucial, this aspect of the diffusion model might pose practical challenges, which can potentially alleviated by (Xu et al., 2022). important for practitioners and researchers to be cognizant of this temporal trade-off when employing FLEXGEN- LEXGEN-EERin real-timo or near-real ceatrreal-ting -eml- applications.
17


Reviews:
[{"review_id": "mTtFw1LZIp", "reviewer": "Reviewer_ULHZ", "summary": "This paper presents a latent diffusion model for heterogeneous tabular EHR generation. More specifically, the method aims to generate static and temporal EHR data jointly while taking into account of the missing modalities of EHR data. The authors aim to capture intrinsic relationships between static and temporal data via a unified latent space and address missing modalities through an optimal transport problem in the embedding space. The model is evaluated against various baselines including VAE, GAN, and other diffusion models (DM). Their results show that their method outperforms other models in generation fidelity and utility.", "questions": "- Table 1 is a bit confusing to read. There is no label for the temporal feature dimensions (d) for MIMIC-III in the table. Does that mean it is the same as eICU? Similarly, is the T for eICU also the same? \n\n- Is the paper also claiming novelty in joint static and temporal diffusion?\n\n- What about considering generating samples that include missing modalities? Should generating missing modalities also be a point of consideration in terms of data fidelity?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main contributions of this paper include formalizing the problem of generating heterogeneous EHR data with missing modalities as well as the use of the optimal transport problem to solve this issue. It also discusses their latent space alignment, an important feature when dealing with missing modality data, in ample detail. The authors evaluate their results against multiple generation methods and uses data from the MIMIC-III and eICU database. The baselines include models of various architectures including VAEs, GANs, and other DMs. In addition, three areas of data evaluation are also considered, namely data fidelity, data utility, and data privacy. The paper presents their architecture in a clear manner that is easy to understand. There is also sufficient breadth covered in the Related Work section.", "weaknesses": "In the Related Work section, the authors bring up TabDDPM as a recent model that addresses heterogeneous tabular generation. They claim that existing diffusion-based EHR models are either unable to “generate categorical features or their tendency to treat numerical and categorical features independently.” However, if I am not mistaken, TabDDPM combines multinomial and Gaussian losses which implies that it treats categorical and numerical features dependently. In addition, as the authors mentioned, TabDDPM was also applied to EHR generation (Ceritli et al., 2023). If there is no misunderstanding, then this paper’s novelty lies in its approach of using a unified latent space to consider static and temporal correlations and dependencies (rather than being the first diffusion model to capture joint relationships). If this is the case, it was unclear as at first read, it could be misunderstood that the paper was also novel in capturing dependent relationships between static and temporal EHR amongst diffusion models. \n\nIt would also be helpful to clarify what feature dimensions (static and temporal) match to in the context of MIMIC-III and eICU in the Appendix. Even though a table is provided in the Appendix, the corresponding dimensionality is not the most clear. \n\nIn sections 4.1 and 4.2, the authors assume an individual patient EHR to take one of the three forms (all static features exclusively present, all temporal features exclusively present, or all features are present). However, in real EHR data, there are often patients with some (not all) static features present and some (not all) temporal features present. \nIn addition to designating p% and q% missing for static and temporal features, it would also help to see results from randomly sampling patients from the datasets to obtain a realistic distribution of missing data/modalities for training data (as to my knowledge, datasets such as MIMIC already have patients with missing measurements). \n\n\nSince addressing the problem of missing modality seems to be the primary goal of the paper, more experiments in general (perhaps showing more baselines other than EHR-M-GAN or other evaluation metrics/scores for fidelity and utility) would be helpful. Currently, only R-squared values are depicted for generation fidelity and only eICU for utility. More results in the Appendix would help support the claims. It would also be very helpful to include results from training on real data for Figures 2-4 to compare as it would provide more context. Currently, EHR-M-GAN is the only comparison. \n\nLastly, a couple of brief mentions of future work or limitations in the conclusion would also help clarify and reaffirm their current goals and progress. For example, addressing lower privacy scores or lack of ablation studies and their implications. \n\nSome minor edits found:\n\n- Section 4.1: First sentence “where ___ contains time-invariant features” should have “S” superscript for x instead of T? \n\n- Section 4.2: Second sentence “embedding the patient information as ___” should have “S” superscript for z instead of T?\n\n- Section 5.5: Second sentence “randomly designate p% of samples as lacking temporal features” should be “static”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXwnEq1T1T", "reviewer": "Reviewer_DnAy", "summary": "The authors address the challenge of generating realistic synthetic EHRs, especially in the presence of missing modalities in heterogeneous tabular data. Introducing FLEXGEN-EHR, a tailored diffusion model, they propose an optimal transport module to handle missing modality issues. Their empirical results indicate that this model surpasses existing methods in fidelity and utility, with promising implications for privacy-sensitive contexts.", "questions": "- In Table 2, could you elucidate how your method contrasts with baseline models that are solely focused on synthesizing Discrete codes? Specifically, how did you employ MedGAN in generating the Labevents data for MIMIC-III?\n- I noted the statement, \"we observed that latent space embedding models, trained on disparate features, manifested analogous geometric patterns and behaviors.\" Could you provide further clarity on the specific geometric patterns and behaviors that were identified during this observation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors present a practical problem definition that is aptly suited for real-world scenarios involving incomplete data.\n- Furthermore, they introduce an innovative approach to address the issue of missing modality, notably by formulating it as an optimal transport problem within the embedding space.", "weaknesses": "- While the manuscript underscores the significance of proficiently addressing missing modality, it lacks experimental validations. To effectively highlight the merits of optimal transport, I would suggest incorporating a comparative analysis between doing an imputation for missingl modalities via kNN in the FLEXGEN-EHR framework, and leveraging Optimal Transport for missing modality imputation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SeSpc8qFFW", "reviewer": "Reviewer_CG4L", "summary": "The author proposes diffusion model for generating EHR. They model the static measurements and temporal measurements separately, and using optimal transport module in the hidden space to deal with missing modalities.  They evaluate FLEXGEN-EHR on two datasets against six baseline methods.", "questions": "None", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea that aligning the temporal features and static features is novel. This method helps to deal with missing modality of EHR.", "weaknesses": "1. The author should bold the best results in Table 4 as Table 2 and Table 3. There exists a trade-off between generation quality and privacy guarantee. The method achieves worse performance on some datasets under some criteria is not a serious problem. However, the author should not neglect it.\n\n2. Minor\n* In the second line from the bottom, the author states \"their tendency to treat numerical and categorical features independently\". Although TabDDPM deals with numerical and categorical features separately, TabDDPM learns the dependency between them. This is because although the diffusion process is different, they concatenate the numerical and categorical features to the network, which is the same as the latent diffusion model used in this paper. \n* Typo in the second line of paragraph \"Heterogeneous Tabular EHR Generation\": x^{S}_i instead of x^{T}_i", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQTzcY7j1F", "reviewer": "Reviewer_z7JT", "summary": "The paper presents \"FLEXGEN-EHR\", a flexible generative model tailored for heterogeneous tabular Electronic Health Records (EHRs) with a specific focus on handling missing modalities. EHRs often contain a mix of static (e.g., race, gender) and temporal (e.g., heart rate, blood pressure) data, and there's a challenge in generating realistic synthetic data that can replace real EHRs while maintaining privacy. Addressing the prevalent issues of missing data and heterogeneity in EHRs, the proposed model leverages an optimal transport module to align and accentuate common feature spaces. Empirical evaluations demonstrate FLEXGEN-EHR's superiority over existing methods in terms of data fidelity and utility, even in scenarios with missing modalities.", "questions": "Q1. What are the computational costs associated with FLEXGEN-EHR, especially when scaling to larger EHR datasets?\n\nQ2. How does the model handle extremely diverse datasets, for instance, EHRs from different countries or medical practices?\n\nQ3. Can FLEXGEN-EHR be extended or adapted to handle other forms of medical data, such as medical images or unstructured clinical notes?\n\nQ4. Given the significance of missing data in EHRs, how does the model ensure that the synthetic data generated doesn't inadvertently introduce biases or inaccuracies, especially in predictive tasks?\n\nQ5. How can the model handle time-varying labels? (e.g. health status of patient at any point in time instead of mortality just at the end of the trajectory.)\n\nQ6. Please provide more information for Equation 6 and what is trying to be achieved in the paragraph above it.\n\nQ7. Have an algorithm section either in main text or appendix that shows a step by step overview of the model both for training and for generation.\n\nQ8. How are the uncertainties calculated for the tables?\n\nQ9. How can the model handle more realistic missingness? (see W5).\n\n\nMinor comments:\n\nq1. “Numerical and temporal” and “static and categorical” are not the same and using them synonymously is not correct. You could have numerical static features and categorical temporal features (e.g. binary indicator of using mechanical ventilator at different points in time).\n\nq2. There are multiple typos and clarity issues in the paper. \n“It’s” is not formal.\nEmpty brackets ([]) before equation 1.\n\tSection 4.2 line 3, typo for static embeddings.\n\tEnd of paragraph one of section 4.2 missing “i” index for z’s.\n\nq3. What is H and W in page 4? \n\nq4. What is Π  before equation 4?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1. The paper addresses a novel and crucial problem of generating synthetic EHRs that handle both heterogeneous data types and missing modalities.\n\nS2. Empirical results showing superior performance in terms of fidelity (by up to 3.10%) and utility (by up to 7.16%) compared to state-of-the-art methods validate the model's effectiveness.\n\nS3. The paper provides clear problem formalization, methodological details, and is well-structured, making the approach and its implications comprehensible.\n\nS4. With growing concerns over patient privacy and the challenges in obtaining real EHRs, a robust generative model like FLEXGEN-EHR holds potential to take a step for research in precision medicine without compromising patient confidentiality.", "weaknesses": "W1. While the model showcases empirical strength, discussions or case studies on how it might be applied in real-world healthcare settings would enhance its practical value. The paper lacks an important motivation beyond patient privacy on why generating synthetic EHR is useful in practice. Providing this motivation is essential for the paper to be useful in the healthcare setting and should not be high-level only and be supported by experiments. For example, is this useful for data augmentation? If so does having data augmentation provided by this paper’s model improve downstream prediction tasks? And other questions in a similar vain.\n\nW2. It would be beneficial to understand the model's performance across diverse EHR datasets to gauge its broad applicability. \n\nW3. One big challenge in generating synthetic data for EHR is the heterogeneity of the data with respect to specific groups (based on e.g. gender, age, race, etc). The paper is missing this analyses completely as it would benefit the validity of the generated data. Without identifying these spurious correlations, it is not at all clear where such fake patient data would be useful.\n\nW4. The paper combines various components, like optimal transport and diffusion models, which might make the model computationally intensive or challenging to implement.\n\nW5. The paper considers missingness in only static features or only temporal features. This is not a useful separation for missingness of modalities. Missingness in healthcare and specifically in EHR is not in this manner. Missingness is often not at random which in an on by itself provides information for prediction models (e.g. not doctors not ordering urine sample for a specific patient from a point onwards does not indicate missingness at random but rather urine sample is not required for the specific diagnosis that is being made). Additionally, missingness is not for the entire trajectory of the patient but rather at different points in time each covariate could either be missing or not and the method provided in this paper does not delve deep into these cases.\n\nW6. The paper does not handle irregular sampling property of the EHRs which is in fact more important for handling than the type of missingness handled in the paper.\n\nW7. The final results although beating the baselines, the uncertainty is large enough that the model’s performance improvement relative to LDM, EHR-M-GAN is not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTyq5Voakl", "reviewer": "Reviewer_Tnu1", "summary": "The authors introduce FLEXGEN-EHR, a new model designed to generate synthetic EHR data, capturing heterogeneity in tabular data. Unlike most existing models, which focus on generating either static or temporal features, FLEXGEN-EHR is a flexible approach that can generate both static and temporal features simultaneously, even in the presence of missing values in the original data. The model uses a latent diffusion process for EHR generation with a novel optimal transport objective for latent space alignment. In experiments, FLEXGEN-EHR demonstrates superior performance in terms of fidelity and utility while ensuring that privacy is maintained.", "questions": "- Could you explain what R^2 is? The papers referred to use MMD and KS-statistics.\n- What are the architectures of the decoder? Are they MLP and LSTM?\n- How much better is imputation with OT compared to nearest-neighbor imputation if implemented in FLEXGEN-EHR?\n- Could you provide more details about the statement \"[Equation 7] can achieve alignment when the sample sizes between static\nand temporal aren’t the same.\" Does this refer to the sample sizes of the non-missing data?\n- It would be great if the authors could provide details about the membership attack implementation (possibly in the Appendix).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the paper and the problem setting are realistic and convincing.\n- The authors provides a clear and comprehensive review of related works in the literature.\n- The method of latent space alignment is convincing and novel for the particular task.\n- The experiment results demonstrate superior performance compared to other EHR generation baselines.", "weaknesses": "- The paper needs ablation studies. For example, it would be more convincing if the authors were to include a comparison of the performance gap: (1) between scenarios with and without the use of OT on complete data; (2) between FLEXGEN-EHR and FLEXGEN-EHR - OT + KNN imputation on partially missing data.\n- There seem to be a few typos in the manuscript (e.g., column names in Table 1, MMCAR, the notation for time-invariant features, adversarys)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)