Guided Discrete Diffusion for Electronic Health Record Generation
Authors: Jun Han
Zixiang Chen
Yongqian Li
Yiwen Kou
Eran Halperin
Robert E. Tillman
Quanquan Gu
Venue: Accepted by TMLR ()

Abstract: Electronic health records (EHRs) are a pivotal data source that enables numerous applications in computational medicine, e.g., disease progression prediction, clinical trial design, and health economics and outcomes research. Despite wide usability, their sensitive nature raises privacy and confidentially concerns, which limit potential use cases. To tackle these challenges, we explore the use of generative models to synthesize artificial, yet realistic EHRs. While diffusion-based methods have recently demonstrated state-of-the-art performance in generating other data modalities and overcome the training instability and mode collapse issues that plague previous GAN-based approaches, their applications in EHR generation remain underexplored. The discrete nature of tabular medical code data in EHRs poses challenges for high-quality data generation, especially for continuous diffusion models. To this end, we introduce a novel tabular EHR generation method, EHR-D3PM, which enables both unconditional and conditional generation using the discrete diffusion model. Our experiments demonstrate that EHR-D3PM significantly outperforms existing generative baselines on comprehensive fidelity and utility metrics while maintaining less attribute and membership vulnerability risks. Furthermore, we show EHR-D3PM is effective as a data augmentation method and enhances performance on downstream tasks when combined with real data.

Published in Transactions on Machine Learning Research (05/2025)
Guided Discrete Diffusion for Electronic Health Record Generation
Jun Han* jun han@optum.ccm com
Optum AI, UHG
Zixiang Chen* Yongqian Li, Yiwen Kou chenar19@0e 9acs. ucla.edu Deportment of Computer Science, UCLA
Eran Halperin, Robert E. Tillman rob. fillmmn@@opummccom Optum AI, UHG
Quanquan Gu!
ggu@cs. vcla.edu.
Department of Computer Science, UCLA
Equal contribution Project Co-Lead
Reviewed on OpenReview:
Abstract
Electronic health records (EHRs) are pivotal data source that enables numerous applications in computational medicine, c.g., discase progression prediction, clinical trial design, and health economics and outcomes research. Despite wide usability, their sensitive nat raises privacy and confidentialityy concerns, which limit potential use cases. To tackle these challenges, we explore the of generative models to synthesize artificial, yet realistic EHRs. While diffusion- based methods have recently demonstrated data modalities and overcome the training instability and mode collapse issues that plague previous GAN-based approaches, their applications in EHR generation remain underexplored. The discrete nature of tabular medical code data EHRs poses challenges for high-quality data generation, especially for continuous diffusion models. To this end, we introduce a practical tabular EHR generation method, EHR-D3PM, which enables both unconditional and conditional generation using the discrete diffusion model. Our experiments demonstrate that EHR-D3PM significantly outperforms existing generative baselines comprehensive fidelity and utility metrics while maintaining less attribute and membership vulnerability risks. Furthermore, we show EHR D3PM is effective as a data augmentation method and enhances performance on downstream tasks when combined with real data.
sate-of-thhearr performance in generating other
1 Introduction
Electronic health records (EHRs) are rich and comprchensive data source, enabling numerous applications in computational medicine including the development of models for disease progression prediction and clinical
event medical models (Li et al. 2020 Rajkomar et al. 2018). clinical trial design (Bartlett et al. 2019) and
health economics and outcome research (Padula et al. 2022). In particular, many existing discase prodiction models primarily utilize tabular formats, often transforming longitudinal EHR data into binary or categorical
forms, rather than employing time-series forecasting methods (Lee et al. 2022
Huang et al. 2021 Rao et al.

---

Published in Transactions on Machine Learning Research (05/2025)
2023
Debal & Sitote 2022). However, the sensitive nature of EHRs, which includes confidential medical data. poses challenges for their broad use due to privacy concerns and patient confidentiality requirements (Hodge Jr et al.] 1999). In addition to these concerns, data scarcity also restricts their potential use in applications for rare medical conditions. To address these challenges, we consider using generative models to syuthesize artificial, but realistic EHRs, which has recently emerged as a crucial research area for advancing applications of machine learning to healthcare and other industrics with privacy and data scarcity challenges.
The primary goal of synthetic EHR, generation is to generate data that is (i) indistinguishable from real data to expert, but (ii) not attributable to any actual patients. Recent advancements in deep generative models, including Variational Autoencoders (VAE) (Vincent et al., 2008) and Generative Adversarial Networks (GAN) (Goodfellow et al. 2014), have demonstrated significant promise in generating realistic synthetic EHR data Biswal et al. 2021; Choi et al.) 2017]. In particular, GAN-based EHR generation has emerged as the most predominant and popular approach (Choi et al., 2017 Zhang et al. 2020 Torfi & Fox 2020), and achieved state-of- theattthhrtttt- performance in terms of quality and privacy preservation. However, the unstable training process of GAN-based methods can lead to mode collapse, raising concerns about their widespread application.
Recently, diffusion based generative models, initially introduced by Sohl-Dickstein et al. (015)), have demon- strated impressive capabilities in generating high-quality samples in various domains, including images (Ho et al.
2020 Song & Ermon 2020,, audio (Chen et al. 2021 Kong et al. 202I), and text (Hoogeboom et al. b021b Austin et al. [2021] Chen et al. 2023). diffusion model consists of a forward process, which gradually transforms training data into pure noise, and a reverse sampling process that reconstructs data from noise using a learned network. Compared to GANs, their training is more stable it only involves maximizing the log-likel of single neural network.
Due to the superior performance of diffusion models, recent methods have explored their application in generating categorical EHR data (Yuan et al. 2023 Ceritli et al. 2023). While these approaches demonstrate promising performance, their improvement over previous GAN- based methods is varied. Particularly, they struggle to generate EHR records with rare medical conditions at rates consistent with the occurrence of such conditions in real-world data. Furthermore, existing approaches offer limited support for conditional generation, which is crucial for many downstream tasks such as discase classification.
In this paper, we propose a practical EHR generation method that utilizes discrete diffusion (Sohl-Dickstein lt al. 2015 Hoogeboom et al.) 2021b Austin et al. 2021 Chen et al. 2023), a type of diffusion process tailored for discrete data sampling, as well as a flexible conditional sampling method that does not require additional model training. Our contributions are summarized as follows:
We introduce Discrete Denoising Diffusion model specifically tailored for generation of tabular medical codes in EHRs, dubbed EHR-D3PM. Our method incorporates an architecture that effectively captures feature correlations, enhancing the generation process and achieving EHR-D3PM excels in generating instances of rare conditions, an aspect where existing methods often face challenges.
performance. Notably,
We further extend EHR-D3PM to conditional generation, specifically tailored for generating EHR samples related to particular medical conditions. Given the unique requirements of this task and the discrete nature of EHR data, we have custom-dcsigned the energy function and applied energy-guided Langevin dynamics at the latent layer of the predictor network to achieve this goal.
We investigate the effectiveness of EHR-D3PM a data augmentation method in downstream tasks. We show that synthetic EHR. data generated by CHR-D3PM yiclds comparable performance to that of real data in terms of AUPRC and AUROC when used to train predictive models and when combined with the real data, EHR-D3PM can enhance the performance of predictive models.
Notation. We use the symbol q to denote the real distribution in a diffusson process, while pe represents the distribution parameterized by the NN during sampling. With its success probability inside the parentheses, the Bernoulli distribution is denoted by Bernoulli(). We further use Cat(p) to denote a categorical distribution over one-hot row vector with probabilities given by the row vector p.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2
Related Work
EHR. Synthesis. Various methods have been developed for generating synthetic EHR data. Buezak et al. (010)) proposed early data-driven approach for creating synthetic EHRs, but their approach offers limited flexibility and has privacy concerns. Recently, GANs have become prominent in EHR generation, including medGAN Choi et al. (0117), medBGAN (Baowaly et al. 2018), EHRWGAN ZZhang et al.) 2019). and CorGAN Torf & Fox (2020). GAN- based methods offer significant improvement in the quality of synthetic EHRs, but often face issues related to training instability and mode collapse
(Thanh-Tung et al.) 2018), restricting their wide use and the diversity of generated data. To address this, other methods, including variational auto-encoders (Biswal et al. ) and language models (Wang Sun 2022), have been explored. Very recently, MedDiff (He et al., 2023bb considered using diffusion models and proposed sampling techniques for high-quality EHR generation. Ceritli et al. (2023); Yuan et al. (2023) further extended the diffusion model to mixed-type EHRs. In this paper, we focus on developing guided discrete diffusion model tailored specifically for generating tabular medical codes in Electronic Health Records (EHRs), which has wide-ranging applications in healthcare (Debal& & Sitote 2022 Lee et al. 2022). Our model aims to improve the generation of ICD codes for rare conditions, which has been a challenge for previous methods.
Discrete Diffusion Models. Discrete diffusion models was pioncered by Sick-Dickstein al. (0055 which explored diffusion processes binary variables. The approach was further developed by Hoogeboom al. (2021b), who incorporated categorical random variables using transition matrices with uniform probabilities. Song et al. (2021) introduced a similar extension in their supplementary content, though they did not experiment with this model type. Subsequently, Austin et al. (2021) introduced a generalized framework named Discrete Denoising Diffusion Probabilistic Models (D3PMs) for calegorical random variables, effectively combining discrete diffusion models with Masked Language Models (MLMs). Recent advancements in this field include the introduction of oditing- operations [Jolicouu-Maartnnen
al. 2021 Reid et al.] 2022
auto-regressive diffusion models (Hoogeboom et. al. 2021a Ye et al.] 2023), a continuous-ting structure [Campbell et al. 2022), strides in generation acceleration (Chen et al. 2023 and the application of neural network analogs for learning purposes (Sun et al. 2023). Discrete diffusion and flow models (Lou et al. 2024
Campbell et al. 2024 Gat et al. 2024 are recently proposed for image, text and protein data (non-tabular)
Diffusion models (Kim et al. 2023 Lee el. al. 2023 Kotelnikov et al.) 2023 Zhang et al. 2024] are recently developed for tabular data generation but not specifically designed for sparse and high dimensional EHR gencration. Diffusion and flow-based gradient-boosted trees (Jolicocur- -arttinca al. 2024 are proposod for tabular data generation but turn out to be extremely slow in high dimension or on large datasets (Cresswell & Kim 2024). In this paper, we based D3PMs with multinomial distribution for FHR generation.
3 Background
In this section, we provide background on diffusion models.
Diffusion Model. Given Xo drawn from a target data distribution following qdata (-); the forward process is Markov process that maps the clean data Xy to noisy sample from prior distribution (-)ise The process Xo- - Xy is composed of the conditional distributions g(x, xx) where
T II ((xx|xx-1,x0))
(x1:r
(1)
By Bayes rule, 1 induces a reverse process XT - Xg that can convert samples from the prior quoise into samples from the target distribution Idata:
g(xr|Xe Xo) g(x 1 x,, Xn)
(2)
((x)
After training a diffusion model, the reverse process can be used for synthetic data generation by sampling from the noise distribution quoise and repeatedly applying learnt predictor (neural network) (-((x.) parameterized
3

---

Published in Transactions on Machine Learning Research (05/2025)
by 0:
/
Po (XT) Inaiso (XT): Po (x
g(x,-1 (x,, xo) )
(3)
%0
Training Objective. The neural network po (+() evidence lower bound (ELBO) (Sohl-Dickstein
that predicts Xo trained by maximizing the
2015
log (xu) 2 E Bx(x)))/x)) log p(xo:r)
q(x1:T xx)
Bx(xi (x)) [log po (xo [x]]
Σ Byix, xx) [KL(g(x:-1/xx, (x) [x,]]
[xx (L(g(x) (xr)),
Here KL denotes Kullback-Lieble divergence and the last term (xxx)1x)) (x)pl//////X)ll/usssxxxxxx equals approximately equals zero if the diffusion process q is properly designed.
Different choices of diffusion process 1 and will result in different sampling methods (3) There are two
popular approaches to constructing diffusion generative model, depending on the nature of the process.
Gaussian Diffusion Process. The Gaussian diffusion process assumes a Gaussian noise distribution quoise In particular, the prior is chosen to be Jnoise -0 I), and the forward process is characterized by
(x., xo) = (x V1 -ixx-1,
where 3 is the variance schedule determined by a pre-specified corruption schedule. The Gaussian diffusion process has achieved great success continuous-valuedd applications like image generation (Ho et al. 2020
Song et al. 2021). Recently, it has been applied tabular EHR data generation (He et al. 2023b Yuan
al. 2023
Discrete Diffusion Process. Discrete Denoising Diffusion Probabilistic Models (D3PMs) is designed to
generate categorical data from vocabulary {1,.... }}, represented as a one-hot vector x E {0. 1) The noise follows a categorical distribution Iunise The Multinomial distribution (Hoogeboom et al. 2021bb is
among the most effective noise distributions. In particular, Anoise is chosen to be a uniform distribution over the one-hot basis of the vocabulary (e,,..., and the forward process is characterized by
x(xx/xx-1,x00) Cat Bixi-1 + (1 (oo)aie
where Cal is the categorical distribution and Bg is the variance schedule determined by a pre-specified corruption schedule. Due to its discrete nature, D3PM widely used to generate categorical data like text Hoogeboom et al.) 2021b Austin et al. 2021) and categorical tabular data Kotelnikov et al. (2023): Ceritli et al. (2023). This paper uses a D3PM with a multinomial noise distribution to generate tabular medical codes in EHRs.
4 Method
In this section, we formalize the problem of tabular EHR data generation and provide the technical details of our method.
4.1 Problem Formulation
We consider medical coding data in EHRs, such as diagnose codes (ICD), procedure codes (CPT) and medication codes (GEN), which are standardized medical codes (Slee) 1978 In wide application of medical domains, it common convert continuous variables into discrete ones to enhance the performance of prediction models (Rasmy al.) 2021 Hill et al. (2023). For example, Hill et al. (0233 converts continuous lab codes to discrete tokens based deciles of each L.OIC code; prediction models built. on learnt representation
4

---

Published in Transactions on Machine Learning Research (05/2025)
of tokenized discrete data significantly outperform ML models which directly trained on the mixed-type tabular data. Therefore, our paper focus on the generation of discrete medical codes, e.g., ICD, CPT, GEN and LOINC codes.
For a given set Ω of medical codes interest, we encode the set as N := [2]] categories (1.2....,.Vz A sample patient EHR is then encoded as a sequence x6) E {0, 112h is one-hot function. x(1) represents the occurrence of the i-th medical code in the patient EHR. Although the total number of medical codes is large, the number of medical codes each code is correlated with should be relatively small. Therefore, the joint distribution the intrinsic dimension of the data is low. We assume a sufficiently large set of patient EHRs is available to train transformer-based diffusion model to generate synthetic EHRs sequences x'.
....,,x(() x()], where each token tokens x = _x(1)..... ,...,..''......
& can be approximately factorized and
4.2 Unconditional Generation
In Section
3 we introduced multinomial diffusion with a single token, x E RK In the context of categorical
EHRs, we aim to generate a sequence of tokens with K = 2, denoted by x = [x(*), .x("),...,x( x.)].. Therefore, we need to extend the terminology from Section 3 We define the sequence of tokens at the t-th time step
as X, = [x] (1) ......; (N)),,
where x, represents the i-th token at diffusion step t. Multinomial noise Qnoise is added to each token in the sequence independently during the diffusion process,
N
(xx/|x=11,x0) = II
Cat (x) (0) (1-1) 3 + (1 - [noise)
1=1
The reverse sampling procedure uses the predictor po(- with the following neural network architecture with Zu,t = X [x, ....,,"' (N)
t 21-1, + Epss + Etines
Zi, = LinMSA([N((4L...)) + 2/-100
L1 = MLP(LN((L-1,))
Output = Softmax(zL.)),
(4)
where Epcs; Erime E R2XD represents the position embedding and time embedding respectively, the variable indexes the layers belonging to the set (L.....J.. LinMSA refers to the efficient multi-head self-attention block proposed by Wang et al. (2020). which has linear complexity with respect to the input dimension. LN is an abbreviation for layer normalization. For each dimension of &p, we apply a multilayer perceptron layer to obtain the logit, abbreviated as ParalleMMLP. The sofumax function transforms the last-layer latent variable zt, into the conditional probability po(-)x;), serving as the final softmax layer. The details of our denoise model are provided in Figure 4 in Appendix A.2
4.3 Conditional Generation with Classifier Guidance
The goal of conditional generation to generate (x(c) close qdata (x|c), where e denotes a context, such the presence of single group of medical codes in patient's Electronic Health Record (EHR). Since is not available during the training process, we cannot train the generator px(c)c directly. However, assume access to a classifier p(c|x) that approximates the conditional distribution (c|x). Given an unconditional EHR generator (x) and classificr p(c[x), we can proposc a training-free conditional generator as follows:
pe (x(c) x (x) p(c|x).
(5)
Since qrtata (x/c) x 9data(x) (clx)(c/)), we can expect (x(xic) in (5) is close to (x/c) provided the unconditional generator pe (x) is close to (x) and the classifier p(c(x) is close to (c)x)
To sample from equation (5) the most popular approach is applying Langevin sampling on the unnormalized joint density while injecting Gaussian noise (Ho & Salimans, 2021). However, there is challenge multinomial diffusion the state x lies in discrete space. In particular, the multinomial diffusion procedure
is follows:
px Xx, c)
- g(xx-1 X1 )po (So Xe, c),
5

---

Published in Transactions on Machine Learning Research (05/2025)
where &o is the latent variable that predicts Xo. It is not possible to directly apply Langevin dynamics in the space of &n- However, the last-layer latent variable az, in Equation ( (before the softmax layer) lies in a continuous space. We have the following:
(x xx, xx, = po(8o z1,1) po Xx, c) dzz,
Therefore, we can utilize the plug-and-play method, initially employed in text generation (Dathathri al. 2019) and recently applied to protein design (Gruver et al. 2023 with the latent space Z In particular, we introduce modified latent variable y(h) for žL,t, which is initialized as y()) <- ZL,-- This modification allows us to iteratively update the latent variable using Langevin dynamics, guiding it towards a desired target while maintaining the learned structure of the diffusion model. The iterative update is applied as follows:
(((+1) t y())) v y(m) (DKL(y - Vo(y(())] + 2
where the energy function (k)) = log(p(cly(*))) = log (Ex, pe pe(x y (p([(()) and (y(()) = ( (pa (8oly())) is the Kullback Leibler (KL) divergence for regularization of the guided Markov transition. The gradient of the energy term V. drives the hidden state y(() towards high probability of ((cly(*)), ensuring that the generated samples align with the desired target. The gradient of the regularization term V yco) DKL ensures that the guided transition distribution still maximizes the likelihood of the diffusion model, preserving the learned structure and preventing excessive deviation from the original model. For a more detailed discussion, see Appendix
5 Experiments
In this section, we apply method to three EHR datasets, including the widely used public MIMIC-III dataset and two larger private datasets from large health instituta We compare our method with state-of-the-art EHR generative models in terms of fidelity, utility and privacy.
5.1 Experiment Setup
Datascts. Public Datascts MIMIC-III (Johnson et al. 2016) includes deidentified patient EHRs from hospital slays. For each patient's ' EHR, we extract the diagnosis and procedure ICD-9 codes and truncate the codes to the first three digits. This dataset includes a patient population of size 46,520. Private Datasets We consider two industrial EHR datasets. D, includes a patient population of size 1,670,347 and has sparse binary features. D2 includes a population of size 1,859,536 and has relatively denser binary features from a different corpus. D2 contains diagnose codes (TCD), procedure codes (CPT) and medication codes (GEN).
Diseases of Interest. We consider the utility of the synthetic data in six chronic diseases: type-Il diabetes, chronic obstructive pulmonary disease (COPD), chronic kidney disease (CKD), asthma, hypertension heart and osteoarthritis.
5.2 Baselines
Med-WGAN EMR-WGAN and EHRMGAN A number of GAN models (Choi et al. 2017 Baowaly et al. 2018 Torfi & Fox 2020) have been proposed for realistic EHR generation. Med-WGAN and EMR- WGAN have best performance among based methods. EHRMGAN is one of most recent GAN-based method whose repository is public and incorporates the training of conditional generation. Therefore, we compare our guided generation with the conditional generation of EHRMGAN.
TP and TabSyn We compare our model with two recent diffusion models, TabDDPM (Kotelnkoov et al. 2023) and TabSyn (Zhang et al. 2024 for tabular data. When training TabSyn in first 100 dimensions of our datasets, TabSyn works well. But when training TabSyn in first 200 dimensions or all dimensions of
Too comply with the double-blind submission policy, withhold the name of the institution providing the datasets; should the paper be accepted, will provide these details.

---

Published in Transactions on Machine Learning Research (05/2025)
our datasets, the performance of TabSyn degenerates significantly. One reason could be that the feature in EHRs are extremely sparse, which is challenging for learning in the first component of TabSyn. Therefore we leave the comparison with TabSyn on Appendix B.1
EHRDiff Yuan et al. [023] is the only diffusion model directly designed for synthesizing tabular EHR with an open-source codebase. As the code of other diffusion models (Yuan et al. 2023 Ceritlii al. 2023 He
iet al. 2023a) for tabular EHRs are not available, we select EHRDiff as baseline. EHRDiff uses Gaussian distributions in DDPM and apply simple MLP networks in the denoising model.
DifestDiff The improved version (Cresswell & Kim 2024 of Fifest (Joliceur-Marrineau et al. 2024 is trained for comparison, which is still extremely slow in high dimension or large dataset as only CPUs can be used for training and sampling. Therefore, we only provide its comparison on MIMIC dataset on Appendix
5.3 Evaluation Metrics
Dimension-wise Prevalence We compute dimension-wise prevalence by taking the mean of the data in each dimension. Dimension-wise prevalence captures the marginal feature distribution of the data. We compute the Spcarman correlation between prevalence in the synthetic data and prevalence in the real data.
Correlation Matrix Distance (CMD) measures the difference between the covariance matrix of the synthetic data and the covariance matrix of real data. We first compute the empirical covariance matrices of the synthotic data and real data respectively and take the difference between these two matricos. Then we caleulate the Frobenius norm of the difference matrix distributional distance.
Maximum Mean Discrepancy (MMD) is one of most common metrics to measure the difference between two distributions in distributional space. We compute the MMD for a set of synthetic data and a set of real test data. We employ a mixture of kernel-based met hods (Li et al. 2017) to estimate MMD to improve its robustness. The detailed formula is given in Eq. (6) on Appendix A.4
Medical Concept Abundance Distance (MCAD) measures the synthetic EHR distribution on record level. It quantifies the discrepancy between the empirical distribution (histogram) of the unique positive code number on synthetic data and the empirical distribution of the unique positive code number on real data.
Downstream Prediction (AUPRC and AUROC) To evaluate the utility of the generated data, we evaluate the accuracy of classifiers trained to predict the diseases of interest mentioned above using synthetic data. We train the classifiers to predict the ICD code that corresponds to the discase of interest using all other available ICD codes as features. We train the classification model using syuthetic data and evaluate its performance on real test data. We adopt the state-of-the-ar robust classification model for tabular data given in (Ke et al., 2017).. The most reliable classification model one trained on real data; we use this as a benchmark to represent an upper bound for classification accuracy.
Precision and Recall To evaluate the quality and coverage of synthetic samples by conditional sampling methods, precision and recall metrics induced from Kynkäänniemi et al. (2019) is computed, where we use elementwise L1 distance to measure the distance between two samples.
Privacy Metrics Attribute inference risk (ATR) and membership inference risk (MIR) are used to evaluate the vulnerability risks. AIR measures the risk from the adversary ability to infer sensitive attributes of a targeted record when subset of features are exposed to the attacker. AIR is calculated as the weighted sum of Fl scores of the inferences of other sensitive attributes. The number of exposed attributes for all experiments is defaulted as 256. MIR evaluates the risk that an attacker can infer the real samples used for training the generative model given generated EHR data or the model parameters. For each EHR in this set of real training and evaluation data, we calculate the minimum L2 distance with respect to the synthetic EHR data. The real EHR whose distance smaller than preset threshold predicted as the training EHR. The predicted F1 score is computed evaluate membership vulnerability risk. The threshold set as for all experiments. We codebase of Yan et. al. (022)) to compute both AIR and MIR scores.
7

---

Published in Transactions on Machine Learning Research (05/2025)
EMR-WGAN
TabDDPM
EHRDiff
EHR-D3PM
prevalence synthetic data
0.6
0.5
0.5
0.5
0.5
0.4
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
8.0
0.1 0.3 0.5
8..
0.1 0.2 0.4
0.6
o
0.0
0.6 prevalence of real data
8.0
0.1 0.3 0.4 0.5 0.6
prevalence real data
prevalence real data
prevalence of real data
(a) Spearman corr 0.93
(b) Spearman corr
0.95
(c) Spearman corr
0.89
(d) Spearman corr 0.99
EMR-WGAN
TabDDPM
EHRDIff
EHR-D3PM
prevalen lenceeee synthetic data
0.05
0.05
0.05
0.05
0.04
0.04
0.04
0.04
0.03
0.03
0.03
0.03
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.00 0.02 0.03 0.000
0.00
0.02 0,03 0.04 0.00
0.00
0.05" 2
p.
0.00
prevalence of real data
prevalence of real data
prevalence of real data
prevalence of real data
(e) Spearman
0.90
Spearman
0.94
(g) Spearman
0.81
(h) Spearman corr 0.98
Figure 1: Comparison of prevalence on synthetic data and real data D2 with ICD, CPT and GEN codes, where the total dimension is 2683. The second row represents the prevalence of the first row in the low data regime. The prevalence is computed on 200K samples. The dashed diagonal lines represent the perfect matching of code prevalence between synthetic data and real EHR data. Pearson correlations are very high for all mothods and thus not used as metric to compare different methods.
EMR-WGAN
TabDDPM
EHRDIff
EHR-D3PM
0.030
0.025
Real
Real
0.030
Real
Real
0.020
Synthetic
0.025
Synthetic
0.025
Synthetic
0.020
Synthetic
0.020
0.020
0.015
0.015
0.015
0.010
0.010
0.010
0.005
0.005
0.005
0.005
0.000
0 25 50 75 100 125 150
175
0.000
0
25
50 100 125 150 175
0.000
25
50 75 100 125 150 175 0.000
25 50 75 100 125 150 175 Feature number per record
Feature number per
Feature number per record
Feature number per record
Figure 2: Density comparison of record feature number on synthetic data and real data D2 with ICD, CPT and GEN codes. The number features per record is the sum of ICD codes present in each sample. The number of bins 175, and the range of feature number values is (0, 175).
5.4 Experiment Results
Fidelity We first evaluate the accuracy of marginal distributions synthetic data in Figure I for dataset D2, Figure 5 for MIMIC and Figure 7 for Dj on Appendix B.I We compare prevalence in synthetic data with prevalence in real data for each dimension. In Figure 1 Figure 5 and Figure 7 we can see that the prevalence for our method EHR-D3PM aligus best with the real data. EHR-D3PM consistently has the highest Spearman correlation. We further observe that EMR- WGAN and EHRDiff fail to provide an unbiased ostimation of the distribution the low prevalence regime, which corresponds to rare conditions. This failure is mild when the dataset has dense features, as shown in Figure of Dz, but is obvious when the dataset has sparse features, as shown in Figure of Dj.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Next visualize the histogram of feature number per record, which is calculated by summing the medical codes in each sample, on dataset D2 in Figure 2 MIMIC-III dataset in Figure P and dataset DD in Figure S
on Appendix B.I. We compare feature numbers for synthetic data with that of real data. As we see in
Figure G Figure S and Figure 2 EMR-WGAN and EHRDiff demonstrate poor performance in estimating
the mode or the lail of the density. When the datasets are large, our method lends to provide a perfect estimation of the feature number for the real data.
Finally compare method with baselines in fidelity metrics, CMD, MMD and MCAD in Table The results in CMD metric show that our EHR-D3PM significantly outperforms all baselines, particularly with the larger datasets D1 and D2. This indicates our method can learn much better pairwise correlations between different feature dimensions. Table I also shows that the distribution of synthetic data learned by our method has the least discrepancy with real data distribution on all three datasets.
Table 1: Fidelity metrics (CMD, MMD and MCAD) on MIMIC and real dataset D2. These metrics on real dataset D1 are provided on Appendix due to space limit.
METRICS
CL)(
MMD(J)
MCAD(4)
DATASETS
MIMIC
D2
MIMIC
D2
MIMIC
D2
MED- WGAN
27.54020.628
28.94220.196
0.078
0.08640.013
0.189622..oo 0024
0.194420.0017
EMR -ANN
26.658+0.68
21.438 1..16
0.053
0.024+0.004
0. 10.46=000000 000017
0.1572+0.0015
TARDDPM
25.236+0.68
16.7288
0.010 0016
0.042+0.00
0. 0.13306=0.00m 0011s
0.1587=0.000
EHRDIFF
25.447
18.941 1 +0.092
0.009
0.046+0.00 0os
0. 0.14399 0015
0.1764
EIR- DTR-
21.128 +0.393
10.255 0.0037
0.003
0.019
0.1013 00.0000 0.1081 00.000
Table 2: Synthetic data utility. Disease prediction from ICD codes on real data D2. AUROC is reported. We use synthetic data of 160K to train the classifier and 200K real test data to evaluate different methods. 80% of test data are bootstrapped 50 times to compute 95% confidence interval.
T2D
ASTHMA
COPD
CKD
OSTEOARTHRITIS
REAL DATA
0.955
0.853
0.951 002
0.944 002
0.926
0.893=0.002
MED-WGAN
0.924=0.001
0.81920.002
0.853 003
0.805
0.500+0.001
0.820 003
EMR-WGAN
0.918
0.747=0.00z
0.888 003
0.90740.000
0.844
0.753
TABPPDM
0.945=0.003
0.846
0.931
0.915
0.837
0.868
EHRDIFF
0.950 0.001 0.001
0.843 0022
0.936 + .002
0.916
0.822+0.000
0.875 0.002
EHR- D3PM
0.952 10.001
0.85340.0m
0.947 20.002
0.944
0.911
0.889 00.002
Utility We now apply our method to discase classification downstream tasks. Since MIMIC-III contains much smaller patient population than the private datasets, which may not provide valid test data benchmark for disease classification, we focus on the private datasets D2 and Dz. In Table 5 we observe that the prevalence of most diseases is low in datasets D, and D2. The training and test data set sizes are 160K and 200K. From Table 2 and Table g on Appendix B.1 the absolute average increase in AUPRC and AUROC over baselines (EHRDiff and TabDDPM) 3.22% and 3.12% respectively for dataset D2- From Table 7 and Table S on Appendix B.1 the absolute average increase in AUPRC and AUROC over basclines (EHRDiff and TabDDPM) 3.90% and 2.57% respectively for dataset D,. The improvement in downstream prediction for our model benefits from the best correlation between different features our model has learned.
Privacy In Table 3 we evaluate privacy metrics AIR and MIR. It shows that our model has relatively low risks on both AIR and MIR. This indicates our method has mild vulnerability to privacy risk compared to existing baselines. Generally, there is a trade-off between privacy and fidelity for generative models. In particular, when a generative model fails to learn any information of the data distribution, AIR and MIR scores are expected be the lowest. Therefore, incorporating differential privacy to further reduce the privacy risk in discrete diffusion models is interesting direction, which is largely unexplored.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Privacy metrics (AIR and MIR) on MIMIC, dataset D1 and dataset D2.
ATTRIBUTE INFERENCE RISK(L)
MEMBERSHIP INFERENCE RISK(4)
MIMIC
D1
D>
MIMIC
Di
D2
0.019=0.0015
0.071 00.004
00.0056
0.080
0.440
0.339
0.398
EMR-WGAN
0.058=0.0027
0.116+0.000
0.132+0..0 0089
0.456++00000 0003
0.358+0.001
0.41 0.415=0.0020
TABDDPM
0.021 0.0013
0.085
0.00091
0.093 1 w.....0.2222
0.462 10.0037
0.351 10.0018
0.415 0.0020
EHRDIFF
0.022=0.0016
0.077
0.089 0.0044
0.445=0.000 0.0034
0.353
0.421++..000
EHR-D3PM
0.020
10681000046
0.078 1.0781000048
0.432 1..0034
0.344
0.406_0.0011
Diabetes
Asthma
COPD
Osteparthritis
0.85
0.95
0.950
0.94
0.84
0.945
0.93
AUROC
0.92
0.87
0.940
Roal Source
0.91
Uncond Sample
0.935
0.90
EHRMGAN
Real Sample
0.85
0.930
0.81
0.89
Guided Sample
0.88
10K
30K
50K
10K
30K
50K
10K
30K
50K
10K
50K
Augmented Data Size
Augmented Dala Size
Augmented Data Size
Augmented Data Size
Figure 3: Synthetic data augmentation for disease classifications from ICD codes based on dataset D2. The size of the real source data for training the LGBM classifier is 5000, as indicated by the dashed purple line. We augment the source training data with synthetic data to train the LGBM classifier. 'Uncond Samples' stands for the synthetic data generated by our unconditional sampler. Guided samples are synthetic data generated by our proposed guided sampler for each disease. To minimize noise from evaluation, we adopt 200K real test data to evaluate all experiments and report test AUROC for comparison. 80% of the test data are bootstrapped 50 times to compute 95% CT, which is visualized by the shaded region around each line.
5.5 Guided Generation
In this section, we apply our guided sampling method to generate conditional samples for different disease conditions. For each condition, we apply guided sampler and generate set of synthetic data. We evaluate the precision and recall of generated samples based on Kynkäänniemi et al. (2019) in Table 4 As we can see, our guided sampler has consistently higher recall in diabetes, COPD, Asthma and Ostcoarthritis. Compared with our unconditional method, samples of the conditional baseline EHRMGAN have slightly better recall score but much worse precision score.
Table 4: Precision (t) and Recall (t) for samples of different diseases on D2. 'Uncond' denotes samples generated by our unconditional method. 'Guided" stands for samples generated by our guided sampler.
DIABETES
ASTHMA
COPD PRECISION RECALL
OSTEOARTHRITIS PReciSiON RECALL
PRECISION RECALL
PREcISION RECALL
UNCOND EHRMGAN
0.785_.013
0.72
0.416 0.657
0.7084.009
0.406 007
0.436
0.212+.o1
0.636_.015
0.778+.016
0.516
0.4
0.592+.000
0. 2.183
0.335
0.239+.013
GUIDED
0.752_
0.863 .013
0.574+.01
0.591 + 008
0.6534.009
0. 0.617+.00
0.4044.009
0.3164 .098
In the following, we utilize synthetic samples to augment training data in downstream disease classifiers. The size of the real source data for training classifiers is 5000. We augment the original training data with data from three different groups: real data, synthetic data generated by our unconditional sampling method and our guided sampling method. We report AUROC in Figure and AUPRC in Figure g on Appendix B.2 to
10

---

Published in Transactions on Machine Learning Research (05/2025)
evaluate accuracy. We can see that classifiers trained with synthetic data augmentation always improve the vanilla baseline (classifier trained with the original source data). We also observe that the data augment ation by guided sampling consistently outperform the data augmentation by our unconditional sampling and the conditional bascline EHRMGAN. It is interesting to observe that the data augmentation by guided sampling has consistently higher AUROC than real data augmentation. We observe this to be because the synthetic samples generated by guided sampling contain richer information in cases of diseases of low prevalence.
6 Conclusion and Future Work
In this paper, we introduced practical generative model for synthesizing realistic EHRs EHR-D3PM. Leveraging the latest advancements in discrete diffuusin models, EHR-D3PM overcomes the challenges of GAN-based approaches and effectively generates high-quality tabular medical data. Compared with other diffusion- based approaches, EHR-D3PE enables high-quality conditional generation. Our experiment demonstrates that EHR-D3PM not only achieves statc-of-thheart performance terms of fidelity, utility, and privacy metrics but also siguificantly improves rownstream task performance through data augmentation. Incorporating longitudinal feature into our model is an interesting research direction. Further investigations of the vulnerability of diffusion-based generative models in EHR generation, particularly to attribute and membership inference attacks (Shokri et al., 2017), is a promising future direction as well as providing formal privacy guarantees, e.g. by incorporating differential privacy, which is largely unexplored in diffusion- models for discrete data.
Broader Impacts
The primary goal of this work to develop state-of-the-art generative model for medical code data in EHRs. The synthetic data generated by this model may be used to train, and augment the training of various downstream models for computational medicine. We believe this work may enable future applications of machine learning to healthcare which lead to positive impacts for patients, e.g., through early prediction of disease progression or matching patients to appropriate clinical trials that advance medicine. While the use of generative models and synthetic data can enhance palient privacy, they do not eliminate the risks of membership inference and other attacks, as we mention and demonstrate in the paper. Our experiments demonstrate that the proposed model achieves statc-of-the-art performance in terms of reducing these risks, but proper controls must still be used whenever working with patient data. Furthermore, any downstream models trained using synthetic data must go through the same rigorous review and testing process to prevent disparate impact as models trained on real data.
Limitations
Our proposed EHR-D3PM model focuses on generating synthetic tabular EHRs and achieves seate-of-thh-art performance, which has a significant impact various tasks. However, there certain limitations to our work that should be acknowledged.
Firstly, our model does not currently incorporate longitudinal features, which could potentially enhance the utility of the generated data. Incorporating temporal dependencies and modeling the evolution of patient records over time an important direction for future research. This extension would enable the generation of more realistic and comprehensive EHR data, capturing the dynamic nature of patient health over extended periods.
Secondly, the vulnerability of diffusion-basse generative models in EHR generation to more advanced privacy attacks is largely unexplored area that requires further rescarch. While our model demonstrates strong privacy preservation properties, it is crucial to investigate its resilience against sophisticated adversarial attacks specifically designed to target EHR data. Thorough analysis and development of robust. defense mechanisms are necessary to ensure the long-term security and privacy of the generated synthetic EHRs.
By acknowledging these limitations, we aim provide a transparent assessment of our work's scope and potential areas for future exploration.
11

---

Published in Transactions on Machine Learning Research (05/2025)
References
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete stato-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021.
Mrinal Kanti Baowaly, Chia-Ching Lin, Chao-Lin Liu, and Kuan-Ta Chen. Synthesizing electronic health records using improved generative adversarial networks. Journal of the American Medicol Informatics Association, 26(3):228 241, 12 2018. ISSN 1527-974X. doi: 11000//jama//oy1445
Victoria L Bartlett, Sanket S Dhruva, Nilay D Shah, Patrick Ryan, and Joseph S Ross. Feasibility of using real-world data to replicate clinical trial evidence. JAMA network open, 2(10)=1112269 e1912869, 2019.
Siddharth Biswal, Soumya Ghosh, Jon Duke, Bradley Malin, Walter Stewart, Cao Xiao, and Jimeng Sun. Eva Generating longitudinal electronic health records using conditional variational autoencoders. Tn Machine Learning for Healtheare Conference, pp. 260 282. PMLR, 2021.
Anna L. Buczak, S. Babin, and Linda J. Moniz. Data-driven approach for creating synthetic electronic medical records. BMC Medicul Informaties and Decision Making, 10:59 59, 2010.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 15:28266-28279, 2022.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protcin co-design. In Forty-fire first International Conference on Machine Learning. 2024.
Taha Ceritli, Ghadeer O Ghosheh, Vinod Kumar Chauhan, Tingting Zhu, Andrew P Creagh, and David A Clifton. Synthesizing mixed-type electronic health records using diffusion models. arXiv preprint 2rX2w:230. 00000..16672 14679, 2023.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. International Conference on Learning Represenlations, 2021.
Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kon, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint ar arXi::2312.091 091993, 2023.
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun. Generating multi-label discrete patient records using generative adversarial networks. In Machine learning for healtheare conference, pp. 286-38 305. PMLRR 2017.
Jesse C Cresswell and Taewoo Kim. Scaling up diffusion and flow-based xgboost models. In ICML 2024 AI for Science Workshop, 2024.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint Xi:::1991 02164, 2019.
Dībaba Adeba Debal and Tilahun Melak Sitole. Chronic kidney disease prediction using machine learning techniques. Journal of Big Data, (1):1009 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde- Warde-Farley, Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. sevenh- Conference Neural Information Processing Systems, 2023.
Jun Han and Qiang Liu. Stein variational gradient descent without gradient. In International Conference on Machine Learning, pp. 1900-190099 1908. PMLR, 2018
Huan He, Yuanzhe Xi, Yong Chen, Bradley Malin, Joyce Ho, al. A flexible generative model for heterogeneous tabular ehr with missing modality. In The Twelfth International Conference Learning Representations, 2023a.
Huan He, Shifan Zhao, Yuanzhe Xi, and Joyce C Ho. Meddiff: Generating electronic health records using accelerated denoising diffusion model, 2023b.
Brian I. Hill, Melikasadat Emami, Vijay S Nori, Aldo Cordova-Palomera,, Robert E Tillman, and Eran Halperin. Chiron: A generative foundation model for structured sequential medical data. In Deep Generative Models for Health Workshop NewrIPS 2023, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NewrIPS 2021 Workshop on Deep Generalive Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840- (18840-6551, 2020.
James G Hodge Jr, Lawrence O Gostin, and Peter D Jacobson. Legal issues concerning electronic health information: privacy, quality, and liability. Jama, 282(15):1466-1471, 1999.
Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXi::2110.00077 2021a.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neurol Information Processing Systems, 34:1 34:12444-111 1444-1 2021b.
Yinan Huang, Ashna Talwar, Satabdi Chatterjee, and Rajender R Aparasu. Application of machine learning in prodicting hospital readmissions: a scoping review of the litcrature. BMC medical research methodology, 21:1 14, 2021.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Mahdi Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3, 2016.
Alexia Jolicoeur-Martinean, Ke Li, Rémi TailefTail Tal Kachman, and Toannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint ar 14080, 2021.
Alexia Saicoour-Martineen Kilian Fatras, and Tal Kachman. Genorating and imputing tabular data via diffusion and flowe gradient-boosted trees. In International Conference on Artificial Intelligence and Statistics, pp. 1288-1296. PMI.R, 2024.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Jayoung Kim, Chacjeong Lee, and Noscong Park. Stasy: Scoro-based tabular data synthesis. The Eleventh International Conference on Learning Representations, 2023.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. International Conference on Learning Regresenlations, 2021.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564-17579. PMLR, 2023.
Thomas Kynkäännicmi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Chaejcong Lee, Jayoung Kim, and Noseong Park. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. In International Conference on Machine Learning, pp- 18940-18956. PMT.R, 2023.
Chanjung Lec, Brian Jo, Hyunki Woo, Yoori Im, Rac Woong Park, and ChulHyoung Park. Chronic discase prediction using the common data model: development study. JMIR AI, 1(1)e44 2022.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi- Khorshiio Behrt: transformer for electronic health records. Scientific reports, 10(1):7155, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Aduances in neural informalion processing systems, 29, 2016.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffiusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.
William V Padula, Noemi Kreif, David J Vanness, Blythe Adamson, Juvn-David Rueda, Federico Felizzi, Pall Jonsson, Maarten J IJzerman, Atul Butte, and William Crown. Machine learning methods in health economics and outcomes research- the palisade checklist: a good practices report of an ispor task force. Value in health, 25(7):1063 1080, 2022.
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine, 1(1):18, 2018.
Patike Kiran Rao, Subarna Chatterjee, K Nagaraju, Surbhi B Khan, Ahlam Almusharraf, and Abdullah Alharbi. Fusion of graph and tabular deep learning models for predicting chronic kidney disease. Diagnostics, 13119, 2023.
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):86, 2021.
Machel Reid, Vincent J Hellendoorn, and Graham Neubig. Diffuser: Discrete diffusion via edit-based reconstruction. arXie preprint arXiv:2210.16836 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatíkov. Membership inference attacks against machine learning models. In 2017 IEEE sympostum on security and privacy (SP), pp. 3 18. IEEE, 2017.
Vergil N Slee. The international classification of discases: ninth revision (icd-9), 1978.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using nonequiibrium thermodynamics. In International conference on machine learning, pp. 2256-2263 PMI.R, 2015.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International
Conference on Learning Represenlations, 2021.
Yang Song and Stefano Ermon. Improved techniques for training coore-based generative models. Advances in neural information processing systems, 14:12488-12444 2020.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based contimuous-time discrete diffusion models. The Eleventh International Conference on Learning Representations, 2023.
Hoang Thanh- Thanh-Tung, T. Tran, and Svetha Venkatesh. On catastrophic forgetting and mode collapse in generative adversarial networks. ArXiv, /t///0007.0411,, 2018.
Amirsina Torf and Edward A. Fox. Corgan: Correlation-capttring convolutional generative adversarial
networks for generating synthetic healtheare records. In The Florida AI Research Society, 2020.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising auloencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096 1103, 2008.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arx:w:2006.0 04006, 2020.
Zifeng Wang and Jimeng Sun. Promptehr: Conditional electronic healthcare records generation with prompt lcarning. In Conference on Empirical Methods in Natural Language Processing, 2022.
Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D. Mooney, and Bradley A. Malin. A multifaccted benchmarking of synthetic electronic health record generation models.
Nature Communications, 13, 2022.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language models can perform many tasks with scaling and Instruction-inetune finetuning. arXiv preprint arXiv:2308.122 12219, 2023.
Hongyi Yuan, Songchi Zhou, and Sheng Yu. Ehrdiff: Exploring realistic ehr synthesis with diffusion models. arXiv preprint ar::2.900 05656, 2023.
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. The Twelfth International Conference on Learning Representations, 2024.
Ziqi Zhang, Chao Yan, Diego Mesa, Jimeng Sun, and Bradley A Malin. Ensuring electronic medical record simulation through better training, modeling, and evaluation. Journal of the American Medical Informatics Association, 27(1):99-100 10 2019. ISSN 1527-974X. doi: 10.1093/jamia//cz// 16611...
Ziqi Zhang, Chao Yan, Thomas A. Lasko, Jimeng Sun, and Bradley A. Malin. Synteg: a framework for temporal structured electronic health data simulation. Journal of the American Medical Informatics Association JAMIA, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2025)
A Experiment Details.
In the following Table 5 International Classification of Diseases, Ninth Revision (ICD 9) codes. This table includes common conditions such as Type II Diabetes (T2D), Chronic Kidney Discase (CKD), Chromic Obstructive Pulmonary Disease (COPD), Asthma, Hypertension and Osteoarthritis. Each disease is associated with specific ICD 9 codes that are used for clinical classification and diagnosis purposes. this paper, we are interested in the diseases listed in Table
we present a concise summary of various diseases along with their corresponding
Table List of Diseases and Corresponding ICD 9 Codes.
Disease
ICD 9 Code
MIMIC
D1
D2
DIABETES
250,* *
0.214
0.261
0.068
CURONIC KIDNEY DISEASE (CKD)
585.1 9
0.106
0.119
0.015
CHRONIC OBSTRUCTIVE PULMONARY DISEASE (COPD)
496
0.069
0.136
0.017
ASTHMA
493.20 22
0.051
0.085
0.079
HYPERTENSION (HTN-HEART)
402.*
0.001
0.028
0.006
OSTEOARTHRITIS
715.96
0.0197
0.061
0.034
A.1 Dataset Details
MIMIC Dataset The MIMIC TIT dataset includes a patient population of 46,520. There are 651,047 positive codes within 64,314 hospital admission records (HADM IDs). We have implemented an 80/20 split for training and testing purposes. Specifically, this allocates 12,862 records for testing and the remaining 51,451 for training. The histograms in Figure 6 indicate the density distribution of feature number per record. The dimension is N - 1042.
Dataset Dj The first dataset, denoted by D1, includes a patient population of size 1,670,347. We split the whole dataset into 100K for valiclation, 2000K for testing and the rest 1,370, 347 for training. The number of codes per patient is relatively small, indicated by the histogram of feature number per record in Figure g We only consider medical codes with prevalence in corpus larger than 1.Ge-5. The dimension is N = 993.
Dataset D2 The second dataset, denoted by D2, includes a patient population of size 1,859,536. We split the whole dataset into 100K for validation, 2000K for testing and the rest 1,359,536 for training. The number of codes per patient is relatively large, indicated by the histogram of feature number per record in Figure 2 Although, datasct D: has relatively denser feature, the prevalence of six chronic discascs is pretly low. D2 contains diaguose codes (ICD), procedure codes (CPT) and medication codes (GEN). The number of ICD codes, CPT codes and GEN codes are 993, 690 and 1000 respectively. The total dimension is 2683.
intcrested in
A.2 Model Architecture Detail
The denoise model in this paper has uniform architecture, illustrated in Figure The architecture propose is tailored for tabular EHRs as it is non-sequential data. While the architecture proposed in multinomial diffusion Hoogeboom et al. (2021b) is designed for sequential data, where neighboring dimensions have semantic correlation. The tabular EHR datasets in our paper don't. have such property. Therefore, we propose a novel transformer-basee model for tabular EHRs. One bottleneck of transformer models is that the computational complexity of the attention module is quadratic to the dimension of input data. We adopt an efficient block based on Wang et al. (2020), whose attention operation has linear complexity with respect the dimension of input data.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Input Shape: (B, N)
Input
Time t Shape: (B, 1)
Layer Norm
Embedding Shape: (B, N, d)
Linear
t Embed
Pos Embed
Shape: (B, d) x
Shape: (B, N,
K
V
L
Transformer
Proj
Proj
Shape: (B, d)
Multihead Attention
Parallel Linear
Shape: (B, K)
Softmax
Output
Prob Shape: (B, N,
(a) Denoise Model
(b) Transformer Block
Figure Architecture of our denoise model. (b) provides the detail of transformer block which has linear complexity with respect the dimension of input. Axial positional embedding is employed encode the positional information. We employ sinusoidal positional embedding time to the time embedding and then use two-layer MLPs to map the time embedding into hidden state. In the first layer of the two-layer ML.,, use Softplus activation function. We apply I. times of such two-layer MLP to get the hidden state of time embedding to yield the input of each transformer block, as indicated in (a). Positional embedding is added to the embedding of discrete inputs. The input has dimension and means the batch size. For notation simplicity, we use all dimension tabular data has K categories. We use one-hot representation and therefore, the output of the denoisc model has shape (B, N, K). The shape of intermediato layers is provided in (a). In (b), 'Proj" denotes the projection operation proposed in Linformer Wang al. (0020), which induces the linear complexity of the attention module with respect to the input dimension N. The projection dimension is set as the default value 128 for all experiments in this paper.
A.3 Hyper-parametert
Hyper-parameters on MIMIC dataset Since the MIMIC III dataset is relatively small, use relative small model train EHR-D3PM to avoid overfitting to the training data. The hidden dimension 256. The number of multi-attention heads is 8. The number of transformer layers is 5. The number of diffusion steps is 500.
In the optimization phase, we adopt adamW optimizer, and the weight decay in adamW is L.e-5. The learning rate 1c-4 and batch size 256. The beta for expentialLR learning rate schedule 0.99. The number of training epochs is 100. It takes less than three hours finish training this model A6000 with 48G memory.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Hyper-parameters on datascts D1 and D2 The denoise model for datasets D1 and D2 are the same. As datasets D1 and D2 are large, we use relatively large model. The number of multi-attention heads is 8. The hidden dimension is 512. The number of transformer layers L is 8. The number of diffusion steps is 500.
The optimization parameters for both datasets D1 and D2 are also the same. In the optimization phrase, we adopt adamW optimizer. The learning rate 1.0e-4 and batch size is 512. The weight decay adamW is 1.0e-5. The beta for expentialLR in learning rate schdule is 0.99. The number of training epochs is 40. It takes and half day to train one model on A100 with S0G memory.
Syper-paramcters of bascline EHRDiff To have fair comparison with diffusion bascline EHRDiff, use the same hyper parameters steps in EHRDiff is also 500 and the number of layers in EHRDiff is also 5. The other hyper parameters use the default values in the github implementation EHRDiff.
proposed dilfusion model on all chree datasets. The number of diff'usion
Syper-parameters of baselines TabSyn and TabDDPM To have a fair comparison with baselines TabSyn and TabDDPM, the number of diffusson steps in TabSyn and TabDDPM are also 500 and the number of layers in denoising models of TabSyn and TabDDPM also 5. For other hyper parameters, we use the default parameters provided on the official implementation of TabSyn and TabDDPM. When training TabSyn on our datasets, we found that TabSyn works quite well in the first 100 dimensions of our datasets. But training TabSyn in the first 200 dimensions of our datasets, we found the performance significantly degenerates. The dimensions of seven datasets used in the paper of TabSyn are all less than 50.
A.4 Evaluation Metrics
MMD
The empirical MMD between two distributions P and Q is approximated by
MMD(P.Q) = 1 m 7=1
Σ (P.Q),
(6)
where kiy is a kernel function; m is number of kernels; MMDA, (P,Q) is estimated by samples [x:)== P and {x ~ Q as follows,
MULD,, n(n- 1)
Σ (x), +
Σ
Je (x((x))))) Σ (x,.
12 i
In our evaluation, we use Gaussian RBF kernel kiys
||x x(122)
(x, = =exp
2
with bandwidth h, = Avg * 2(7-m/2), where Avg the average of pairwise L.2 distance ween all samples. We choose m and thus 7 3,,,,,
A.5 Hyper- Hyper-paramenn parameters of Classifier Models on Downstream Tasks
For the downstream tasks, we used a light gradient boosting decision tree model (LGBM) (K et al. 2017 as it had uniformly robust prediction performance on all rewmstream lasks. In all experiments, we set the typer-parameten
of LGBM as follows: n_ estimators 1000, learning rate 0.05 max depth 10, rog_laame lambda 0.5, scale pos_______________ lasp_wsisssht____eeeeee.pppppeee with various sets of hyper parameters which will induce the same conclusion we have in this paper.
reg_ ass_aphaa 0.5, reg
min data in bin 128. We also experiment
B Additional Experiments
Due to space limit, leave bunch of experiment results on appendix.
18

---

Published in Transactions on Machine Learning Research (05/2025)
FofestDiff
TabDDPM
EHRDIff
EHR-D3PM
0.6
prevalence synthelic
0.5
0.5
0.5
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.2
0.2
0.2-
0.2-
0.1
0.1
0.1
0.1
0.000 6.1 6.2 0.3 0.4 0.5 0.6
8.0
0.1 0.2 0.3 0.4 0.5
8.0.0
0.1 0.2 0.3 0.4 0.5
0.6
0.0
0.1 0.2 0.3 0.4 0.5 0.6
prevalence of real data
prevalence of real data
prevalence of real data
prevalence of real data
(a) Spearman corr 0.91
(b) Spearman cort
0.89
(c) Spearman corr EHRDiff
0.75
(d) Spearman corr 0.94
ForestDiff
TabDDPM
EHR-D3PM
0.05
0.05
0.05
0.05
prevalence synthelic
0.04
0.04
0.03
0.03
0.03
0.03
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.00
0.00 0.01 0.02 0.03 0.04 0.05
0.00.
0.00
0.01
0.02 0.03 0.04 0.b.
0.00
0.01 0,02 0.03 0.04 0.05
0.00
0.00 0.01 0.02 0.03 0.04 0.05 prevalence real data
prevalence real data
prevalence real data
prevalence of real data
(e) Spearman corr = 0.82
Spearman corr 0.88
(g) Spearman corr 0.71
(h) Spearman corr = 0.93
Figure 5: Comparison of prevalence synthetic data and real data (MIMIC). The second row represents the prevalence the first row in the low data regime. The prevalence is computed on 10K samples as the MIMIC dataset is relatively small. The dashed diagonal lines represent the perfect matching of code prevalence between synthetic data and real EHR data. Pearson correlations very high for all methods and thus not used as a metric to compare different methods.
ForestDiff
TabDDPM
EHRDIff
EHR-D3PD
0.10
Real
0.10
Real
0.10
Real Synthetic
0.10
Real Synthetic
0.08
Synthetic
0.08
Synthetic
0.06
0.06
0.06
0.06
0.04
0.04
0.04
0.04
0.02
0.02
0.02
0.00
0 5 10
15
20 25 30
35 40
0.00
5
10
15
20
25
30
35
40
0.00
5
10
15
20
25
30 35
40
0.00
5
10
15
20 25 30 35 40
Feature number per record
Feature number per record
Feature number per record
Feature number per record
Figure 6: Densily comparison of per-record feature number on synthetic and real data for the MIMIC dataset. The number of features per record the sum of ICD codes present in each sample. The number of bins is 40, and the range of feature number values is (0, 40).
Table 6: Fidelity metrics (CMD, MMD and MCAD) on dataset MIMIC.
METRICS
CM)(4)
MMD(L)
MCAD(L)
27.540+o.622
0.078
0.189640.omm
EMR-WGAN
26.658 63.
1.05344..000 0054
0.15-1620.0017
TABDDPM
25. 25.236
0.010 0016
0.1306
EHRDIFF
25.447 40.485
40.01
0.1439 o0001
FORESTDIFF
24.173
0.008 00.005
0.1375
DHR-D3PM
21.128 0.393
0.003 00.0004
0.1013 0000010
19

---

Published in Transactions on Machine Learning Research (05/2025)
B.1 Additional experiments unconditional generation
Fidelity Figure 5 and Figure provide additional comparison of marginal distribution matching dataset MIMIC and dataset D1. We have found that TabSyn works well in low dimension but has poor performance on our high dimensional datasets with sparse features. From the Spearman correlation in low prevalence regime, we can still see that our method significantly outperform baselines. Based on results in Figure b we observe that Fofest fails to provide an unbiased estimation of the distribution the low prevalence regime (<= 0.05), which corresponds to rare conditions. From Figure 5 Figure / and Figure we consistently observe the performance of EHRDifT significantly degenerates in the low prevalence regime. One reason we articulate is that the foundation of EHRDiff is designed for continuous distribution and cannot be readily applied to the generation of discrete data particularly on data of low prevalence regime. From Figure G and Figure [ we can see that the histogram of feat ure number per record on synthetic data by our method provides the best matching to that of real data.
prevalence synthetic data
0.6
TabSyn
0.6
TabDDPM
0.6
EHRDIff
0.6
EHR-D3PM
0.5
0.5
0.4
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1
8.0
0.1 0.3 0.4
0.6
0.0
0.1 0.2 0.3 0.4 0.5
0.6
prevalence of real data
prevalence of real data
0.00 0.1
0.2
0.3
0.4
0.5
0.6
08.0
0.1
0.2
0.3
0.4
0.5
0.6
(a) Spearman corr = 0.69
(b) Spearman corr TabDDPM
0.97
(c) Spearman coir
0.85
(d) Spearman corr = 0.99
TabSyn
EHRDiff
EHR-D3PM
0.05
0.05
0.05
0.05
prevalence synthetic
0.04
0.04
0.04
0.04
0.03
0.03
0.03
0.03
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.08 0.00 0.01 0.02 0,03 0.04 0.05
0.00
0.00
0.01
0,02 0.03 0.04 0.05
0. 0.00
0.00
0.01 0.02 0.03 0.04 0.05
0.00
8.00
0.01 0.02 0.03 0.04 0.05 prevalence real data
prevalence real data
prevalence real data
prevalence real data
(c) Spearman corr 0.61
(f) Spearman corr 0.95
Spearman corr 0.79
(h) Spearman corr 0.97
Figure 7: Comparison of prevalence synthetic data and real data D1. The second row represents the prevalence the first row in the low data regime. The prevalence is computed on 200K samples. The dashed diagonal lines represent the perfect matching of code prevalence between synthetic data and real EHR data. Spcarman correlations between synthetic data and real data are reported.
TabSyn
TabDDPM
EHRDIff
EHR-D3RP
0.06
0.06
0.05
0.20
Real
0.05
Real
Real
Real Synthetic
Synthetic
Synthetic
Synthetic
0.15
0.04
0.04
0.03
0.03
0.03
0.10
0.02
0.02
0.01
0.00
0 10
20
30
50
60
0.00
10
20
30
50
0.00
10
30
40
50
60
0.00
10
20
30
40 50 60
Feature number per record
Feature number per record
Feature number per record
Feature number per record
Figure 8: Density comparison per-record feature number between synthetic data and real data D1. The number of features per record is computed summing the ICD codes present each sample. The number of bins is 65 and the range of feature number values (0, 65).
20

---

Published in Transactions on Machine Learning Research (05/2025)
Utility We also apply our methods to downstream prediction tasks on dataset D1, where the prevalence of six chronic diseases is much lower. From Table i and Table g can see that the accuracy of our prediction is still close to the prediction of classifier models trained on real data, which is the target classifier baseline. While other basclines have a much larger performance gap when compared with the idcal classifier. Particularly on rare diseases such as hypertension heart, the classifier trained on synthetic data by our model has 8% absolute improvement AUPRC and AUROC over the strongest baseline on both D1 and D2. From the confidence intervals provided in Table 7 and g we confirm that such improvement over the baselines on both dataset D1 and dataset D2 are statistically significant.
B.2 Additional experiments on guided generation
We provide additional experiment results guided generation. We augment the real data with synthetic data generated by sampling method and train downstream classifier. We measure the performance of all classifiers on real test data. From Figure 9 and Figure 3 we see that the classifier trained on augmented training data with synthetic data, either by our unconditional sampling met hod or by our guided sampling method, consistently outperforms the classifier trained with original source data (vanilla baseline) and the conditional bascline EHRMGAN. all cases, the relative increase of AUPRC over vanilla. baseline is more than 3%; in the classification of COPD, the relative improvement over the vanilla baseline is more than 30%. This elearly indicates that method classification tasks when the real dataset is scarce. More importantly, the data augmentation by guided sampler consistently outperforms the data augmentation with unconditional sampler. We observe this to be because the synthetic samples generated by guided sampling contain richer information in diseases of low prevalence. most balanced training data with positive label will enhance the performance of classifiers and reduce the risk of over fitting to negative class.
be applied to augment the training data of downstream
Diabetes
Asthma
COPD
Osteoarthritis
0.83
0.57
0.60
0.33
0.82
0.56
0.55
0.81
0.55
0.45
0.31
0.30
0.80
0.54
Uncond Sample
0.35
EHRMGAN
0.79
0.30
Sample
0.28
Guided Sample
0.78
0.52
0.20
0.26
0
10K
30K
50K
10k
30K
50K
10K
30K
50K
10K
30K
50K
Augmented Data Size
Augmented Data Size
Augmented Data Size
Augmented Data Size
Figure 9: Synthetic data augmentation for disease classification from ICD codes based on dataset D2. The size of real source data for training LGBM classifier is 5000, indicated dashed line. We augment the original source training data with synthetic data to train LGBM classifier. 'Uncond Samples' stands for the synthetic data generated by our unconditional sampler. Guided samples synthetic data generated by our proposed guided sampler for each disease. To minimize noise from evaluation, we adopt 200K real test data to evaluate all experiments and report test AUROC for comparison. 80% of the test data are bootstrapped 50 times to compute 95% confidence intervals (CT), which is added as shaded region.
C Additional Details of EHR CHR-D3PM
If x is continuous variable, the most common way to sample from the postcrior pe(x(c) x po(x) p(c|x) is using the following Langevin dynamics,
(k+1) + (1) + m V logp(c)x) + v/V log (x()) VITT2E,
where AO I). (7 has been appliod image DDhariall & Nichol 2021 and recently be applicd EHR generation with Gaussian diffusion (He al. 2023b). In practice, 72 always chosen to be zero in practice, and we will generally use V(x)= log(p(c(x) to replace the likelihood that want. to maximize in
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table Synthetic data utility. Disease prediction from ICD codes on real data Dj. AUPRC reported. We use synthetic data of 160K train the classifier and 200K real test data evaluate different methods. 80% of test data are bootstrapped for 50 times compute for 95% confidence interval.
T2D
ASTHMA
COPD
CKD
OSTEOARTHRITIS
REAL DATA
0.702=0.000
0.288+0.004
0.67540.000 002
0.806
0.253
0.296=0.003
MED-WGAN
0.628=0.002
0.149900.02
0.578+0.002 002
0.722
0.11440.000
0.192=0.003
0.656=0.000
0. 0.193
0.603 000
0.753 +0.002
0.151 000003
0.219 =0.00
TABPPDM
0.669
0. 0.238
0.625
0.778
0.176
0.238 005
EHRDIFF
0.670=0.002
0.202
0.642 0002
0.782210.002
0. 0.1
0.245
EHR D3PM
0.693 000
0.263 20.003
0.655
0.796 20.002
0. 0.229
0.278 _0.7800000
Table 8: Synthetic data utility. Disease prediction from ICD codes on real dataset Dj. AUROC is reported. We use syuthetic data 160K to train the classifier and 200K real test data to evaluate different methods. 80% of Lest data are bootstrapped for 50 times compute 95% confidence interval.
T2D
ASTHMA
COPD
CKD
HTN--EART
OSTEOARTHRITIS
REAL DATA
0.808=0.000
0.759
0.867
0.913
0.832
0.789
BED-WGAN
0.757 =0.001
0.595
0,80640.00 0000
0.873
0.625
0.661
0.770=0.00
0.6422+..0m 002
0.815 001
0.885
0.686.10.00
0.689 0000
TABPPDM
0.787
0.728
0.851
0.898
0.74640.004
0.747
EHRDIFF
0.789
0.7222..002
0,856
0.90220.001
0.71440.002
0.759=0.000
EHR-D3PM
0.801 : 001
0.748 00.02
0.860
0.908 00.01
0.821 +0.002
0.782 00002
Table 9: Synthetic data utility. Disease prediction from ICD codes on real data D2. AUPRC is reported. We use synthetic data of 160K to train the classifier and 200K real test data to evaluate different methods. 80% of test data are bootstrapped 50 times to compute 95% confidence interval.
T2D
ASTHMA
COPD
CKD
HTN-HEAAT
OSTEOARTHRITIS
REAL DATA
0.834=0.000
0.581 00000
0.622 009
0.733 +0.0000000 00066
0.278
0.373
MED-WGAN
0.725=0.003
0.496 005
0.203 0.7
0. 0.1664..00 005
0.008
0.223
EMR-WGAY
0.734=0.003
0.411
0.402+0.00 007
10.88
0.134
0.210=0.000 0.000
TABPPDM
0.795
0.558
0.544 008
0.692
0.156
0.327 00.00s
EHRDIFF
0.807 =0.003
0.549
0.548 00.007
0.690 00s
0.141
0.319
EHR- D3PM
0.821 00.0
0.57220.004
0..07
0.71440.007
0.2264.0.00s
0.348 20.06
Table 10: Fidelity metrics (CMD, MMD and MCAD) on dataset D1.
METRICS
CMD(L)
MMD(4)
MCAD(L)
MED-WGAN
18.107
0.07540.011
0.1871 #0.0016
EMR-WGAN
11.8692a.100 108
0.01840.oms
0.1625
TARDDPM
11.204=0.164
0.021
0.15442=o.001
EHRDIFF
23.208 o..08
0.022310.000
0.1687 1.00014
RHR-D3PM
7.692 0.006
0.012
0.0873 0.00
For discrete data, is intractable since we can't get the gradient backpropagation via log In addition, 7 can't guarantee that (k++1) lies in the category {1,..., ., K} after update. Therefore, we need to
22

---

Published in Transactions on Machine Learning Research (05/2025)
do Langevin updates on the latent space Z1,
yl + y(h) V (yL(Y(()) - Vo(y(())] + 2TTE,
where y(h) is the modification of Zr,t, and DrL (y (KL (y Ilpo (Kn (yl is the KL divergence for regularization of the guided Markov transition. The gradient of the KL term plays similar role (x) in (7). It will be interesting to leverage deterministic updates Liu & Wang 2016 Han & Liu (2018) to accelerate this process.
Example: Suppose that the context e is to generate a EHR x such that x has diabete disease, i.e., the k-th token of x equals [1, 0), where k 156 for MIMIC data set. Then we have p(c[&a) = 1 if k-th token of x equals [1,0] and p(c]2o) otherwise. In addition ( is the k-th position output of softmax layer when input (()) Then can compute the energy function follows,
Vo( = log(p(cly(8)) = log (ro (1)(8/))()))((880))
In all experiments of guided generation, the number of Langevin update steps is 10, 1) 0.1 and 0.01.
23


Reviews:
[{"review_id": "y9XOTqYt5H", "reviewer": "Reviewer_pKPR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The use of a discrete diffusion model for EHR data makes sense.\n2. The proposed approach targets a practical problem.\n\nWeaknesses:\n1. Lack of novelty. Discrete diffusion and the classifier guidance technique in 4.3 are already proposed in previous works. The paper uses them to solve the EHR generation task. While this is a reasonable solution, it does not offer exciting new insights/approaches.\n2. I am a bit skeptical about solving data scarcity problem (in EHR) by synthesizing. Learning an accurate generative model itself is data-hungry, as it requires accurate modeling of the data distribution. When there is a shortage of data, how to make sure the learned generative model approximates the (mostly) hidden data distribution well?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method for synthesizing electronic health records using a discrete diffusion model tailored for tabular medical code data. It introduces a flexible conditional generation mechanism that does not require training on data with context label. Extensive evaluation shows that the proposed approach outperforms various generative baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please discuss in main text how the proposed differs from EHRDiff (besides that it does not offer conditional generation), as this seems like a very related baseline.\n2. I think it is necessary to address my 2nd weakness in the main text. Otherwise the proposed approach cannot be justified.", "broader_impact_concerns": "Nothing in particular.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mg33OFMQda", "reviewer": "Reviewer_1i4h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors conduct a lot of experiments.\n\nWeaknesses: D3PM type methods were proposed three years ago, new methods like the work by Lou et al. (2024) proposed learning probability ratios, extending score matching (Song and Ermon, 2019) to discrete spaces, or discrete flows by Campbell et al. (2024) and Gat et al. (2024) have been proposed and provide better results than D3PMs. So why choose D3PM for electronic health record generation?\n\nAndrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design, ICML 2024.\n\nItai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman, Discrete Flow Matching, NeurIPS 2024.\n\nAaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. ICML 2024.\n\nIt is well known that GPT like autoregressive models has the best results for discrete data, but the authors didn't provide GPT type models' results as baseline.\n\nThe statement on the first paragraph of page 3 \"The approach was further developed by Ho et al. (2020); Song et al. (2020), which incorporates categorical random variables using transition matrices with uniform probabilities.\" is wrong, since these two papers didn't address discrete data. \n\nMany references are double cited such as:\n\nEdward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun. Generating\nmulti-label discrete patient records using generative adversarial networks. In Machine learning for healthcare\nconference, pp. 286–305. PMLR, 2017a.\n\nEdward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Generating\nmulti-label discrete patient records using generative adversarial networks. In Finale Doshi-Velez, Jim\nFackler, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (eds.), Proceedings of the 2nd\nMachine Learning for Healthcare Conference, volume 68 of Proceedings of Machine Learning Research, pp.\n286–305. PMLR, 18–19 Aug 2017b.\n\nDibaba Adeba Debal and Tilahun Melak Sitote. Chronic kidney disease prediction using machine learning\ntechniques. Journal of Big Data, 9(1):109, 2022a.\n\nDibaba Adeba Debal and Tilahun Melak Sitote. Chronic kidney disease prediction using machine learning\ntechniques. Journal of Big Data, 9(1):109, 2022b.\n\nChanjung Lee, Brian Jo, Hyunki Woo, Yoori Im, Rae Woong Park, and ChulHyoung Park. Chronic disease\nprediction using the common data model: development study. JMIR AI, 1(1):e41030, 2022a.\n\nChanjung Lee, Brian Jo, Hyunki Woo, Yoori Im, Rae Woong Park, and ChulHyoung Park. Chronic disease\nprediction using the common data model: development study. JMIR AI, 1(1):e41030, 2022b.\n\nMany references are officially published at leading conferences but cited as arxiv papers.\n\nThere are grammatical errors, for example, \"but their approach offers limited flexibility has privacy concerns.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper applies D3PM for electronic health record generation, and compares the results with those by other generative methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the authors should provide experimental results for electronic health record generation by recent discrete diffusion models and flow models as well as GPT type autoregressive models.", "broader_impact_concerns": "No concerns on the ethical implications", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DIA471u5Qc", "reviewer": "Reviewer_wcsW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n*) The paper tackles and important question as to how useful ML models can be trained in the medical domain without endangering the anonymity of the patients.\n\n*) The paper is well written.\n\n*) The model is thoroughly evaluated on a range of benchmarks. It is especially interesting that models trained on data augmented with synthetic data perform better on downstream tasks.\n\nWeaknesses:\n\n\n*) The main concern I have about the paper lies in the novelty of the method. Does the difference of the presented model with respect to standard D3PMs lie in the specific way in which the data is represented, as outlined in Section 4.1? Or has the model architecture described in (4) also been newly developed for this model? I think it would be good to emphasize the differences more in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the paper, a discrete diffusion model is described which allows for the generation of synthetic electronic health records. Both unconditional and conditional generation are supported. The model performs well on a number of benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses and minor issues.\n\n\nMinor issues:\n\nAbstract, line 4: confidentially -> confidentiality\n\nPage 2, first point in list: tailed -> tailored\n\nEquation after (3):\n\n-first line: dx_{1:T} should be removed\n\n-last line: square brackets after expectation value missing", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a discrete diffusion model to generate synthetic electronic health records. Reviewers appreciated the important practical motivation on generating synthetic data that can be used for analysis and training ML models. The evaluation on multiple benchmarks is thorough. 

However, reviewers noted that many newer methods have been proposed since D3PM that may provide better results. In addition to the feedback from the reviewers, I would request two changes:
1. As the authors write in the rebuttal, consider editing the title and abstract/intro to reflect that the primary goal is to solve a practical problem of generating tabular EHR. Avoid using terms such as "novel" method in the abstract and instead focus on the applied usecase.
2. Acknowledge and discuss the recent papers suggested by reviewers, so that future readers may be able to appreciate the pros/cons and assess those models for this task.
Decision: Accept (poster)