Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation
Authors: Wenfang Yao
Chen Liu
Kejing Yin
William K. Cheung
Jing Qin
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal clinical data
latent diffusion model
chest X-ray image
eletronic health records.
Primary area: machine_learning_for_healthcare
TL;DR: We propose DDL-CXR to address the asynchronicity in clinical multimodal fusion by generating individualized up-to-date CXR images. Cross-modal interactions are captured by the generation process, leading to improved prediction performance.

Abstract: Integrating multi-modal clinical data, such as electronic health records (EHR) and chest X-ray images (CXR), is particularly beneficial for clinical prediction tasks. However, in a temporal setting, multi-modal data are often inherently asynchronous. EHR can be continuously collected but CXR is generally taken with a much longer interval due to its high cost and radiation dose. When clinical prediction is needed, the last available CXR image might have been outdated, leading to suboptimal predictions. To address this challenge, we propose DDL-CXR, a method that dynamically generates an up-to-date latent representation of the individualized CXR images. Our approach leverages latent diffusion models for patient-specific generation strategically conditioned on a previous CXR image and EHR time series, providing information regarding anatomical structures and disease progressions, respectively. In this way, the interaction across modalities could be better captured by the latent CXR generation process, ultimately improving the prediction performance. Experiments using MIMIC datasets show that the proposed model could effectively address asynchronicity in multimodal fusion and consistently outperform existing methods.

Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation
Wenfang Yao'; Chen Liu 13; Kejing Yin 2 William K. Cheung', Jing Qin' 'School of Nursing, The Hong Kong Polytechnic University "Department of Computer Science, Hong Kong Baptist University SSchool of Software Engineering, South China University of Technology
Abstract
Integrating multi-modal clinical data, such as electronic health records (EHR) and chest X-ray images (CXR), is particularly beneficial for clinical prediction tasks. However. in a temporal setting, multi-modal data are often inherently asynchronous. EHR can be continuously collected but CXR generally taken with a much longer interval due to its high cost and radiation dose. When clinical prediction needed, the last available CXR image might have been outdated, leading to suboptimal predictions. To address this challenge, we propose DDL- CXR, a method that dynamically generates an up-to-date latent representation of the individualized CXR images. Our approach leverages latent diffusion models for patient-specifer generation strategically conditioned on a previous CXR image and EHR time series, providing information regarding anatomical structures and disease progressions, respectively. this way, the interaction across modalities could be better captured by the latent CXR generation process, ultimately improving the prediction performance. Experiments using MIMIC datasets show that the proposed model could effectively address asynchronicity in multimodal fusion and consistently outperform existing methods.
Introduction
Clinical data in modern healthcare documented through various complementary modalities [1, 2]. Electronic health records (EHRs), for instance, systematically record the progression of diseases over time, including medical histories, laboratory test results, and treatment outcomes [3-6]. In parallel, medical imaging, such chest X-rays (CXRs), is valuable for providing visual insights into the patient's 's internal anatomy, organ functions, and potential abnormalities [7]. Recent studies have shown that strategic integration of multimodal clinical data could lead to improved performance for clinical predictions compared to relying solely on uni-modal data [8-13].
Despite the promising results obtained, the inherent asynchronicity of multimodal clinical data still hinders effective integration. Take the intensive care unit (ICU) setting as an example, patients are subject to continuous monitoring systems that capture vital signs, including heart rate, blood pressure, and oxygen saturation, with this information being routinely recorded in the EHR [14, 15). On the other hand, CXRs are captured only an as-needed basis and often as less as possible, due to limitations of radiation dose and resources [16]. However, patients admitted to ICU are in threatehrn conditions, which means their medical status is prone rapid changes and highly time-sensitive [17]. In the MIMIC-CXP dataset [18], is observed that among patients with positive discase findings in their CXR, over 70% of subsequent CXR images taken within median interval
'These authors contributed equally. "ooresppondecce to: Kejing Yin
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

0
(a) Initial Chest X-ray
(b) CXR taken after 34 hours
(c) Generated by DDL-CXR
Figure 1: A real ICU patient with rapid CXR changes. (a) Initial radiology findings: Low lung volumes but lungs are clear of consolidation or pulmonary vascular congestion. No acute car- diopulmonary process. (b) Radiology findings after 34 hours: Severe relatively symmetric bilateral pulmonary consolidation. (c) CXR generated by DDL-CXR given the initial CXR image shown in (a) and the EHR data within the 34 hours. Clear signs of bilateral pulmonary consolidation can be seen from the generated image. The visualization shows that DDL-CXR could generate updated CXR images that respect the anatomical structure of the patient and reflect the disease progression.
of less than 24 hours - exhibit changes in CXR findings. This implies that when a clinical prediction is needed, CXRs captured even only few hours ago could have become outdated, especially for ICU patients who commonly have respiratory, cardiac, infectious, and traumatic conditions [19]. Fig. 1 shows such an example of real patient the MIMIC dataset.
Motivation Existing works adopt the Searry-forward approach, i.e., using the last CXR image available for downstream prediction tasks [20, 10]. This strategy ignores the potential rapid changes between the prediction time and the time of the last CXR image taken and thus inevitably leads to suboptimal prediction performance. On the contrary, we hypothesize that generating an updated CXR image at the prediction time could mitigate the asynchronicity problem and enhance the prediction accuracy. Nevertheless, generating patient-specific CXR images presents unique challenges. While multimodal generation has been explored extensively in various fields, these methods are not readily adaptable for generating individualized CXR images. In domains such as text-to-audio [21] or tot--tt-mmgee generation [22]. the attributes that need to be controlled (e.g., painting style) can be explicitly defined input modalities (e.g., the text prompt). However, in the clinical context, explicit descriptions of a patient's 's anatomical structures, organ functions, and disease progression, which are highly specific to individual patients and critical for downstream prediction, are not directly available.
Contribution To tackle the aforementioned challenge, we propose Diffusion-based Dynamic Latent Chest X-ray Image Generation (DDL-CXR)², which utilizes tailored latent diffusion model (LDM) [22] to generate individualized CXR images for clinical prediction. Specifically, DDL-CXR learns to generate representations in a latent space encoded by a variational auto-encoder (VAE). To incorporate detailed information about the patient's anatomical structure and organ specifics, we use previous CXR image from the same patient as the reference image. To generate latent representations that align with the discase progression, we a Transformer model [23] to encode the inregular EHR data spanning from the reference CXR to the prediction time. To further capture the implicit interactions between EHR and CXR, we use the encoded EHR representation to predict the labels of abnormality finding of the target image. To force the LDM to capture the disease course in the EHR data, we explore contrastive learning approach for training the LDM. The generated up-to-date latent CXR is later fused with historical data for downstream clinical prediction.
We summarize our contributions follows:
To our knowledge, DDL-CXR is the first work to generate an updated individual CXR image to improve clinical multimodal fusion, thereby alleviating the asynchronicity between EHR and CXR. We propose a contrastive learning approach for the LDM training to enable the disease course in EHR to be captured and utilized by the LDM.
Experiments show that DDL CXR outperforms existing methods in both multi-modal clinical predic- tion and individual CXR generation.
The code is available :ttpps///iithb. www.cooo/mc.iieee http/Chhnliic htap/ChhnnlussvgDDD--OKo

---

2
Related Work
Clinical multi-modal fusion Integrating multi-modal clinical data has shown beneficial for various clinical prediction tasks [24], including COVID-19 prediction [25], pulmonary embolism diagnosis [8, 26), AD diagnosis [9] and X-ray image abnormality detection [27].
Different strategies have been proposed to facilitate the fusion of multi-modal clinical data [2. 28]. Hayat et al. [10] adopts feature-level fusion with an LSTM layer, while Zhang et al. [29] utilizes modality-correlatee encoder to capture long-range dependencies across modalities. Zhang al. [30] and Lee et al. [12] incorporate modality type embedding into the self-attention to capture the interaction. Despite the effort, existing methods for multi-modal fusion are driven only by downstream predictions. How to capture the more fundamental interaction between different data modalities remains an open challenge.
In the temporal setting, asynchronicity presents another major challenge. Unlike the settings of medical images and radiology reports [31], which are naturally aligned time, EHR and CXR are often highly asynchronous, bringing extra difficulties to information integration. "Carry- (Corrryforward" is a common strategy adopted, where the last available data from different modalities are used [10. 13). Lee et al. [12] and Zhang et al. [17] also adopt this approach while modeling the time information of the last available data.
Conditional latent diffusion models The diffusion model is one the state-of-the-ar art generative models [32, 33] that has found important applications in areas such image generation [34), sound generation [35]. joint audio and video generation [36], and tabular data generation [37]. To reduce the computational cost, LDM [22] proposes to train diffusion models on latent space encoded via pre-trained VAE, thus improving training and sampling efficiency as well as preserving generation quality. also incorporates an attention mechanism into its underlying neural backbone to allow more flexible conditioning.
Based on LDM, multi-modal generation models have been developed using priors obtained from large-scale contrastive pre-traing vre-training c.g., contrastive-image pairs for text-to-image generation [38] and contrastive language-audie pairs for text-to-audio generation [21]. However, it is infeasible to apply this method to clinical settings since many clinical data modalities, e.g., CXR and EHR, capture different aspects of patients and cannot be semantically aligned like the image and caption pairs as in
CLIP.
In clinical settings, LDM-based models are developed for brain MRI image generation, conditioned on age, sex, brain structure volumes [39], and a subset MRI slices [40]. For CXR image generation, Packhäuser et al. [41] adopts thoracic abnormality classifier-aides LDM to generate anonymous data generation. Weber et al. [42] utilizes pathology labels, radiological reports, and radiologists' annotations for synthesizing customized CXR images. Gu et al. [43] explores counterfactual generation for CXR using information from imaging reports. Generating individual CXR images that reflect disease courses in EHR and applying them medical predictions remains an open challenge.
CXR images for
DDL-CXR: The Proposed Method
In this work, we focus on improving multimodal clinical predictions by generating latent CXR images that are in line with patient conditions at prediction time. The generation process also works as a fusion module that captures the cross-modal interaction between EHR and CXR. The overview of DDL-CXR is depicted in Fig. 2. It consists of two stages: the LDM stage and the prediction stage. the LDM stage, we use the consecutive image pairs to train an LDM that generates representations within a latent space encoded by variational autoencoder (VAE). To generate patient- speciffc CXRs, an earlier CXR image of the same patient is used as a reference to capture the anatomical structure, and the EHR time series between the consecutive image pairs is used to capture the disease progression. In the prediction stage, conditioned on this composite information, DDL-CXR generates updated and informative CXR representations at prediction time, which are subsequently fused with available EHR data as well as the previous CXR image for downstream prediction tasks.
3

---

Zno
CXB
3
CXR prod
Loaus
y
beus
CXR
y
CXR
Duaa
EHR (5241)
EHH f cood
EBR X 4th
EEHR
contrastive
B Bitgh
Latent Diffusion
pLAT prod
Sampling
Training Noise added
Z11
AA
Z1
Frozen
Concatenation
(O.I)
Cross-atantian
time
LDM Stage
Prediction Stage
Figure 2: The overview of the proposed framework DDL- It consists of two stages. The LDM
stage learns generate individualized up-to-date latent CXR at time t1, Ze to address asyn- chronicity by conditioning on a previous CXR image taken at time to. XCXR, wwich provides the anatomical structure of the patient, as well as EHR data between In and l,, XEHR (fa,ty)" that provides information on disease progression. A contrastive loss and auxiliary loss are enforced for better EHR information integration. The generation module encapsulates cross-modal interactions assist in clinical prediction. The prediction stage fuses the generated latent CXR, the most recent CXR image, and the complete EHR time series for clinical predictions.
3.1 Notations and Preliminaries
EHR and CXR data Patient-wisely, we denote the EHR time series within the time interval between and ij as X
(t.,t,) X1, J,, where x, E B is the variables recorded time and K is the number of features. We denote the grayscale CXR images taken the time t, as XCXR E R W xH where W and H denote its width and height, respectively. For each CXR image,
extract the abnormality finding label, yt, from radiology reports using CheXpert [44].
Predictive latent space for CXR Using diffusion models semantic latent space, rather than high-dimensional data space, has shown substantial decrease in computational expenses with minimal impact on synthesis quality [22]. To obtain an informative and expressive latent space, first train VAE [45] consisting of encoder & and decoder D. The data used for training VAE are all available CXR images the training set with corresponding abnormality finding labels, (XCXR y, yxR)), The primary objective of the VAE is to reconstruct the original CXR image XCXR with D(E(X ((E(X,XR)). We follow the VAE training process in [22], incorporating pixel-wise reconstruction loss accompanied by a perceptual loss [46], an adversarial objective, and a lightly- penalized Kullback-Leible loss towards standard normal aiming constraining the latent spaces from excessively high variance. Besides, to improve the encoder's ' ability to predict, we also include a prediction loss regarding the abnormality label CXR We denote the encoded latent CXR by Z, = (X((RR E RCx W ", where C represents the channel the compressed representation and represents the compression ratio. We first pre-train the VAE model and then freeze it throughout the training and inference of DXR. More details on VAE training are presented in Appendix A.
3.2 LDM Stage: Dynamic Latent CXR Generation
As discussed previously, to generate up-to-date, patient-specific latent CXR, it is important to incorporate the unique anatomical details of the individual patient. Furthermore, the generated image must accurately reflect the evolving pathology as documented in the irregular EHR time series. To this end, we extract all sequential image pairs and the EHR time series between them. We denote each sample quadruplet: (XCXR, EHR XXRR yy CXR). CXR images are encoded (fo.t.)
using the pre-trained VAE as we aim to generate latent CXR images: Zr, E(XCXR Z., ((X(X)) We follow prior works diffusion models to learn LDM [22, 32). It comes down

---

to learning network that predicts the noise added to the noisy latent 7,17 at denoising step as ),,)) n.) Following prior works [22, 32], we parameterize eg by standard UNet [47]. Here Semul (-) is the encoder for the irregular EHR time series be detailed later. The detailed diffusion and denoising processes are presented in Appendix A.
6g
Z Zo L XEHR
(ta:t1)))
Neural backbone and conditioning mechanisms Due to the remarkable capability of UNet [47] in capturing the spatial structure of images, we follow prior works and use UNet as our neural backbone Eg. It predicts the noise added the diffuuson process, conditioned on the reference image and the EHR time series. To explicitly capture and utilize the anatomical structure of individual patients, first concatenate the reference latent CXR Zto and the step-n noisy latent Z,n0 To further integrate the disease course embedded in the EHR time series, we use the cross-attention mechanism capture the interaction between the modalities. Formally, the input the UNet layers is given by:
QK Vd
Attention Q K, V) softmax
V.
(1)
with We 10 (Zl, 1Z.)) K W
K
A (to.61)
BHR
V
Wy
f (X
(to.ts))s
where (.)) denotes the flattened intermediate representation of the UNet and denotes concatenation.
Capturing disease course via EHR time series To effectively capture useful information disease progression for future CXR generation, we adopt multi-takk Transformer-bar time series encoder [48] with the masked self-attention mechanism to handle the variable length of EHR time series [49]. The encoded representation EHR, (tto is given by
((t
cnnd
EHR (X
(to.11)) = Transformer ([bCLs, ((x.)..... (x+,)))
(2)
where o(x)) projects the original EHR time series into an embedding space and applies the positional encoding at time step hcss is the class token.
To further extract information that is relevant to CXR generation and facilitate modality fusion at the LDM stage, we incorporate auxiliary prediction task: using the class token from the encoded EHR to predict the abnormality findings y. associated with the CXR image CXR i.e., y.C = ghhrss) where y denotes the prediction function, e.g., MIP, which is trained by jointly minimize the loss function given by Laux M I,
CXR) log(1 (oXR), where M is the number of training samples for LDM and L is the number of classes of abnormality labels of CXR. The auxiliary task enables the EHR encoder extract CXR-rel related information, which further encourages the interaction between EHR and CXR to be captured in the subsequent generation.
1 EM mm
Ums log(? CXR ) (1
Enhancing semantic multimodal fusion via contrastive LDM learning The generation condi- tioning on EHR data is challenging because the EHR and CXR data are highly heterogeneous and the interactions are implicit. To force the LDM to utilize EHR information during generation, we propose contrastive way of learning the conditional LDM. Specifically, for each EHR time series, we obtain
perturbed version of its representation Exep,,)) (1 -3) ())((,..))) 36, where ô (O, is randomly drawn from a standard normal distribution, 3 is a hyperparameter controlling the strength of the noise. When the perturbed EHR is given as input, we expect the generated image be far away from the target image. This leads to the following training objective function:
LLUM =BBz, Zio XTuR ii x N-Ni..)) ||e- eo (Z!")) Zee JEHR EXR
m)
(3)
+ Xy max (1c- (2(11) Zeo E -12-11e (ee-ey (Z,5) Zin E II [l++,ao))
where a is hyperparameter controlling the tolerance of the noisy-conditional generation. A is coefficient controlling the strength the contrastive term. To ensure stability during training, we set the initial value of x to zero and linearly increase it to one during training.
5

---

3.3 Prediction Stage
In the prediction stage, we do not have access an up-to-date CXR image. Therefore, we generate an updated latent CXR Z at the prediction time 1 using the last available CXR image X as the reference image and the EHR time series in between X (to.41) To make predictions using available EHR, adopt another time series encoder, ford,, which has the same structure as fcom as in Eq. (2). Note that for prediction, the EHR data used, X-HRR
<48h covers all EHR time series with the observation time set as 48 hours, ensuring the available information is fully utilized. In other words, X
S XEER In clinical practice, clinicians make predictions not only based on the latest CXR, 448h
(fa.t))
but also on past CXR images as reference for disease basis. To this end, we employ all available data: CXR, EHR and the generated latent CXR Zi, to make the final clinical prediction:
28h
( CXR), y = Gv (Sreet (X
(X-HR gpred <48h),
pLAT (Zes))
(4)
Here Spred i {CXR, EHR, LAT) are encoders for CXR, EHR, and the generated latent CXR, accordingly. We parameterize fped and Speed using Transformer models, and pred using a ResNet model. The predicting model G6 with & denoting the model parameter, parameterized by self-attention layer. We learn it by minimizing the cross-entropy (CE) loss:
M L'
Lask
Ymm log(Ums) + (1 Umm) log(1 - Tma),
(5)
1=1
where T,' is the number of classes in the prediction task and M' is the number of training samples in the prediction stage.
4 Experiments
4.1 Experiment Settings
Datasets We empirically evaluate the clinical predictive performance of DDL-CXR using MIMIC-IV [50] and MIMIC-CXR [18]*. MIMIC-IV comprises de-identified critical care data from adult patients admitted either ICUs or the emergency department (EDs) of Beth Israel Deaconess Medical Center (BIDMC) between 2008 and 2019, and MIMIC-CXI contains chest X-rays and reports collected from BIDMC, with subset of patients matched with those in MIMIC-IV. For EHR data, follow preprocessing pipeline similar to that described in [10]. 17 clinical time series variables as well as age and gender are extracted, The details can be found in Appendix A.2.
Dataset construction and partition The inclusion criteria for this study involve ICU stays from the matched subset of MIMIC-IV and MIMIC-CN CXR that contain least one CXR image (with Anterior-Posteriae (AP) projection) during the ICU stay or within 24 hours before ICU admission. We exclude ICU stays with lengths shorter than 48 hours. The dataset is randomly split by the patient identifier with a ratio of 24:4:7 for training, validation, and testing, which avoids patient overlapping between subsets.
From the training patients, we further extract data for training the VAE, the LDM, and the prediction model. We extract all images from the training patients for training VAE and extract all CXR image pairs of the same patient taken at any interval greater than 12 hours for training the LDM, i.c.,
DLDM =
{(xO CXR XCRR X EHR (1..4))
V
CXR
(t1-ta)> 12h
where a single ICU stay may contain multiple data pairs for LDM training. This greatly enlarges the training subset for the LDM stage.
For the prediction stage, we extract the last available CXR image and the EHR time series in the first 48 hours and the label for the prediction task of each ICU stay, i.e., the triplet (XCXR X EHR <48h; Y Viss). Note that the EHR time series used in the prediction stage differs from that the LDM stage their time interval since they serve for different purposes.
Both open source under the PhysioNet Credentialed Health Data License 1.5.0 license.

---

Table 1: Overall performance for the phenotype classification and mortality prediction task as measured by AUPRC and AUROC scores. DDL-CXR outperforms all baselines in these metrics.
Phenolyping
Mortality
AUPRC
AUROC
AUPRC
AUROC
Uni-EHR [23]
0.434 +:0009
0.720 +0.006
0.498 +00007
0.815 +0.007
MMTM [52]
0.430 00.005
0.715 00.00
0.422 +0.014
0.785 +0.004
DAFT [9]
0,435 +00002
0.720 +.0003
0.448 +0.004
0.800 +0.003
MedFuse [10]
0,437 10.001
0.718 +0.002
0,443 #0.009
0.793 #0.003
DrFuse [13]
0.459 +0.003
0.729 +0.004
0.773 #0.008
GAN-based [53]
0.453 10.010
0.728 10.008
0.505 10.018
0.816 10.010
DDL-CXR (ours)
0.470 +0.003
0.740 +0.002
0.523
0.822 +0.009
We use the same approach to extract the validation subsets for hyperparameter tuning of VAE, LDM, and the prediction model. Note that the testing patients are held out for evaluating prediction performance only, and are not involved in the training and model selection of VAE and LDM.
Prediction tasks and evaluation metries We evaluate DDL-CXR with two clinical prediction tasks: in-hospital mortality prediction and phenotype classification using clinical data collected within the first 48 hours of ICU admissions. The phenotype classification is multi-label classification task, where the labels are defined by the 25 disease phenotypes, extracted following [51]. The details of the label prevalence and data cohort statisties can be found in Appendix A.2.
We evaluate the performance using two metries, the Area Under the Precision Reeaallon-Reeal Curve (AUPRC) and the Area Under the Receiver Operating Characteristics (AUROC). For the phenotyping task, we report avverr-d scores. We conduct each prediction experiment five times with distinct random seeds and reported the mean and standard deviation of the results.
Baseline Models We compare the following methods. (1) Uni-EHR, a single-modal classifier for EHR time series based on Transformer [23], (2) MMTM [52], multi-modal fusion method based on CNNs through squeeze and excitation operations, (3) DAFT [9] a general-purpose module for fusing tabular clinical information and image data by dynamically rescaling and shifting the feature maps of a convolutional layer, (4) Fuse [10], an LSTM-based multimodal fusion method developed for clinical prediction using EHR and CXR, (5) DrFuse [13], disentangled learning approach that handles modality missing and modal inconsistency in clinical multi-modal fusion, and (6) GAN N-based generation [53], model originally proposed to generate individual brain images conditioning on age and Alzheimer's ' Disease (AD) status via training a conditional GAN.
4.2 Prediction Performance
DDL-CXR obtains the best overall performance. We summarize the overall performance of the phenotype classification and in-hospital mortality prediction in Table 1, where DDL-CXR outperforms all baselines. This shows that generating an updated CXR during test time is beneficial for downstream tasks. On the contrary, DrFuse, MedFuse, DAFT, and MMTM use the last available CXR for prediction, which might have been outdated.
The performance gain of DDL-CXR in terms of AUPRC is particularly noteworthy as the AUPRC metric is especially relevant in the context as it underscores the effectiveness of our approach in identifying the positive class in imbalanced medical datasets. DDL-CXR achieves relative improve- ments of 2.4% and 3.56% over the best baselines in terms of AUPRC for phenotype classification and mortality prediction, respectively.
Mortality prediction with varying time interval We define the time interval (by hour) between the prediction time and the time of the last available CXR as 8 and compute the evaluation metrics in patient groups with different ranges of 5. The results are presented in Table 2. Since the label prevalence varies significantly between groups, making the comparison of AUPRC between groups less meaningful, we report AUROC in the paper and AUPRC in the appendix. DDL-CXR consistently outperforms the baseline models for most groups of 8 for the mortality prediction task. the 6 increases, the last CXR becomes more "outdated", and observe noticeable increase in the
7

---

Table The mean of AUROC score with standard deviation for mortality prediction for overall and different time gaps. 8 represents the time interval (by hour) between the prediction time and the time of the last available CXR. Numbers in bold indicate the best performance each column. DDL-CXR outperforms all baselines in most settings. The AUPRC scores can be found in Appendix B.1.
Overall
< 12
12 <
24 < 36
2
prevalence
14.7%
16.6%
19%
15.9%
9.26%
Uni-EHR [23]
0.815 =0.007
0.854 +0.010
0.799 +0.013
0.756 +0.019
0.796 +0.008
MMTM [52]
0.785 10.004
0.798 1..008
0.763 10.004
0.760 10.012
0.772 10.014
DAFT [9]
0.800 10.003
0.803 10.010
0.782 10.009
0.776 10.006
0.796 10.008
MedFuse [10]
0.793 10.003
0.812 10.004
0.762 10.007
0.760 1..009
0.800 10.010
DrFuse [13]
0.773 10.008
0,802 10.012
0.717 10.023
0.757 10.041
0.723 10.013
GAN-based [53]
0.816 10.010
0.846 100000
0.800 10.011
0.760 10.0266
0.806 10.016
DDL-CXR (ours)
0.822 10.009
0.867 10.015
0.800 10.008
0.753 10.015
0.830 10.011
Table 3: The AUPRC score of predicting each phenotype label. DDL-CXR obtains the highest average rank. Full names of phenotype labels and AUROC scores can be found in the Appendix.
Uni-EHR MMTM DAFT
MedFuse
DrFuse GAN-based DDL-CXR
Acute renal failure
0.573
0.568
0.572
0.565
0.564
0.563
0.588
Acute cerehrovascular disease
0.425
0.418
0.419
0.434
0.399
0.446
0,416
Acute myocardial infarction
0.185
0.192
0.187
0.219
0.209
0.171
0.206
Cardiac dysrhythmias
0.579
0.532
0.548
0.560
0.584
0.561
0.605
Chronic kidney disease
0.515
0.505
0.515
0.497
0.477
0,501
0.538
COPD and bronchiectasis
0.319
0.327
0.342
0.344
0.405
0.372
0.382
Surgical complications
0.370
0.379
0.385
0.381
0.377
0.344
0.388
Conduction disorders
0.276
0.287
0.298
0,286
0.632
0.609
0.633
CHF; nonhypertensive CAD
0.593
0.619
0.647
0.631
0.661
0,652
0.682
0.560
0.540
0.556
0.544
0.581
0.590
0.611
DM with complications
0.562
0.569
0.552
0.561
0.550
0.552
0.524
DM without complication
0.370
0.367
0.343
0.356
0.369
0.352
0,368
Disorders of lipid metabolism
0.594
0.576
0.570
0.566
0.584
0.587
0.601
Essential hypertension Fluid and electrolyte disorders
0.551
0.519
0.525
0.518
0.502
0.554
0.561
0.655
0.664
0.662
0.656
0.658
0.662
0.672
Gastrointestinal hemorrhage
0.180
0.142
0.162
0.192
0.191
0.151
0.180
Secondary hypertension
0.463
0.455
0.452
0.453
0.437
0.451
0.484
Other liver diseases
0.316
0.316
0.341
0.344
0.372
0,362
0.378
Other lower respiratory discase
0.219
0.209
0.206
0.223
0.255
0.236
0.242
Other upper respiratory discase
0.166
0.137
0.166
0.202
0.274
0.196
0.234
Pleurisy: pneumothorax
0.143
0.145
0.159
0.159
0.172
0.171
0.166
Pneumonia
0.412
0.437
0.429
0.419
0.406
0,415
0.428
Respiratory failure
0.655
0.686
0.674
0.671
0.692
0.663
0.669
Septicemia (except in labor)
0.585
0.573
0.580
0.565
0.562
0.573
0.603
Shock
0.590
0.584
0.582
0.592
0.572
0.587
0.586
Average Rank
4.4
4.64
4.4
4.24
3.88
4.16
2.28
performance gap between the best baseline and DDL-CXR in the group (5 2 36), from 0.806 to 0.830. This validates our hypothesis that the generation of a timely CXR, accounting for disease progression, can significantly enhance the performance of clinical predictions.
Phenotype classification The class-wise AUPRC scores for the phenotyping task are detailed in Table 3, where DDL-CXR demonstrates notable performance improvements, achieving the highest average rank across phenotype labels. Due to space limit, we report the standard deviations and the AUROC scores in Appendix B.2. The improvement over baseline multimodal fusion methods validates the effectiveness of facilitating fusion between EHR and CXR in the presence of asynchronicity.

---

Sample #1
Sample #2
Sample #3
Sample #4
Sample #5
Sample #6
Sample #7
X
X
X
Figure Examples of images generated by DDL-CXR, From top to bottom, the three rows are reference images XXXR ground-truth images X and generated images XXXR respectively. The tt
to generations show that DDL-CXR captures the anatomical information from x and the information of disease progression extracted from EHR is blended well towards generating X
4.3 Quality of Generated Chest X-ray Images
Quantitative Evaluation We evaluate the quality of generated CXRs using the test set of the LDM stage, where the ground-truth target CXR available. The Fréchet Inception Distance (FID) score [54] evaluates the similarity between the distributions of generated and ground-truth target CXRs by computing Fréchet distance on the representation obtained from a pre-trained Inception-v3 network.
Besides, we directly measure the Fréchet distance (FD) and Wasserstein distance (WD) the latent space of the
Table 4: Generation quality.
VAE between the generated and the target CXRs. Results are shown in Table 4. Results of "Last-CXR" are obtained between reference images Xn and target images X1, both are directly from the dataset without generation. Thus, this provides reference to the lower bound of the metrics used. DDL-CXR surpasses GAN-based methods across all metrics and obtains the lowest FD and WD. "w/o Z,," and "w/o (ta, " are obtained by removing the condition of the last available CXR and EHR data, respectively. Notably, excluding EHR data from the generation conditions resulted in lower FID scores, which is natural since the generation becomes less restrictive.
FID (L) FD (L) WD (L)
Last-CXR
16.50
2322.68 6353.06
w/o Lio
47.9T
3260.17 7491.17
w/o E
30.03
2412.80 7226.38
GAN-based
98.67
3651.27 7922.71
DDL-CXR
33.83
2316.08 7132.82
Qualitative Eyaluation To further visually examine the generated CXR images, we decode the latent CXR and visualize seven examples in Fig. 3. The first row shows the last CXR images used as reference, the second row displays the ground-truth CXR images, and the last row showcases the generated CXR images. The comparison between the first and third rows indicates that the generated CXR could well capture anatomical structure, while the comparison between the second and third rows demonstrates that the generated CXRs are in line with the latest imaging manifestations, implying that the disease progression embedded in EHR could be captured and utilized in the generation process. We further retrieve the radiology reports and the discharge summary of the corresponding patients from the database for case studies. Fig. 1 shows one example of the case study (Sample #4), where the patient rapidly turned from normal CXR to severe pulmonary consolidation. The discharge summary shows that the patient experienced transfusion-relater acute lung injury and sepsis. Evidently, the generated CXR could more accurately reflect the progressed condition of the patient. Due to space limits, we present more case studies in Appendix B.5.
4.4 Ablation Study
To better understand the factors contributing to the improved performance, we conducted an ablation study by removing the conditioning components in the LDM stage. The results are summarized

---

Table 5: Results of the ablation study.
Phenolyping
Mortality
AUPRC
AUROC
AUPRC
AUROC
Last-CXR
0.459 +0.012
0.730
+0.008
0.503 +0.010
0.817 +0.007
w/o Zio
0,448 =0.012
0.726
50.008
00.04
0.811 =0.008
w/o B(to..))
0.461 =0.002
0.723
#0.006
00.016
0.799 #0.015
w/o Contrastive
0.460 +000077
0.722 +0.011
0.483
+0.019
0.802 +0.011
w/o Laus
0.461 +0.003
0.718
+0.012
0.495
+0.026
0.811 +0.009
LaRt-CXR (w7o EHR)
0.376 20.009
0.661
10.007
0.243
0.664
DDL-CXR (w/o EHR)
0.385 +0.006
0.668
+0.007
0.269 +0.013
0.707 +0.008
DDL-CXR
0.470 10.003
0.740 E0.002
0.523 10.011
0.822 10.009
in Table 5. The variant "Last-CXR" has the same architecture as the classifier of DDL-CXR but removes the generated latent CXR Ze The improvement over Last-CXR shows that learning an LDM for generating updated latent CXR is more effective approach to multimodal fusion, and hence benefits downstream prediction, especially for the mortality prediction task. The variants "w/o Zio" and "w/o (tta.t)) remove the last available CXR and EHR data, respectively, from the condition during LDM training. "w/o Contrastive" removes the contrastive terms from the LDM objective function. "w/o Laux" removes the auxiliary loss which drives the EHR encoder to capture the CXR-related abnormality findings. The results show that adding each component brings slight improvement while incorporating the reference CXR, the EHR, the contrastive learning, and the auxiliary task achieves the best performance. We also remove the EHR data completely in the prediction stage and evaluate the performance using the last available CXR and the generated CXR, respectively. Results are shown as "Last-CXR (w/o EHR)" and "DDL-CXR (w/o EHR)" in Table 5. The results suggest that the generation of an updated CXR significantly benefits downstream clinical predictions. Additional experiment results on robustness against reduced training data size can be found in Appendix B.4.
5 Broader Impacts and Limitations
DDL-CXR holds promise for societal benefits, such more precise and timely medical interventions, and offers an alternative for patients with limited access to X-ray imaging. Nonetheless, the po- tential for generating fake profiles necessitates stringent safeguards, including expert validation of synthesized images, to prevent misuse and protect patient confidentiality, especially when applied to private datasets. Despite its promise, DDL-CXR has some limitations like the need for meticulous hyperparameter tuning and performance gap across different time intervals, as indicated in Table 2. Addressing such potential biases is priority for future research. Furthermore, while various metrics have been employed to assess generation quality, expert evaluation by radiologists would provide more insightful measure of the model's efficacy.
Conclusion
In this paper, we introduce DDL-CXR, which utilizes a powerful LDM to dynamically generate up- to-date latent chest X-rays to tackle the asynchronicity multi-modal clinical data for predictions. Our approach involves leveraging various conditions for patient-specific generation: the most recent available chest X-ray to incorporate detailed matient-specifer anatomical structure, as well as the EHR data with variable durations for disease progression information. To improve multi-modal fusion in the generation, we develop contrastive-learing: Saaativve-earnin--bbeee based LDM to capture and utilize disease courses in EHR. Through quantitative and qualitative validations, we demonstrate the superior performance of DDL-CXR in both image generation and enhancing multi-modal fusion via conditional generation for clinical prediction.
10

---

Acknowledgments and Disclosure of Funding
This work is partially supported by the General Research Fund of Hong Kong Research Grants Council (project no. 15218521), a grant under Theme-based Research Scheme of Hong Kong Research Grants Council (project no. (455-001/22-N), the General Research Fund RGC/HKBU1220262 from the Research Grant Council, the Research Matching Grant Scheme RMGS2021 8 06 from the Hong Kong Government, the National Natural Science Foundation of China (62302413), and the Health and Medical Research Fund (23220312).
References
[1] Rowa Aljondi and Salem Alghamdi. Diagnostic value of imaging modalities for COVID-19; scoping review. Journal of Medical Internet Research, 22(8); 2020.
[2] Farida Mohsen, Hazrat Ali, Nady El Hajj. and Zubair Shah. Artificial intelligence-basee based methods for fusion of electronic health records and imaging data. Scientific Reports, 12(1):17981, 2022.
[3] Kejing Yin, Dong Qian, and William K Cheung. PATNet: Propensity-adjusted temporal network for joint imputation and prediction using binary EHRS with observation bias. IEEE Transactions on Knowledge & Data Engineering, 26(06))2600-261,, 2024.
[4] Kejing Yin, William K Cheung. Benjamin CM Fung, and Jonathan Poon. Learning inter- modal correspon- dence and phenotypes from multi-modal electronic health records. IEEE Transactions on Knowledge & Data Engineering, 34(09))4328-4333 4,41, 2022.
[5] Kejing Yin, Ardavan Afshar, Joyce C Ho, William K Cheung, Chao Zhang, and Jimeng Sun. LogPar: Logistic PARAFAC2 factorization for temporal binary data with missing values. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1625-1635, 2020.
[6] Lihong Song, Chin Wang Cheong, Kejing Yin. William K Cheung, Benjamin CM Fung, and Jonathan Poon. Medical concept embedding with multiple ontological representations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4613 2019.
[7] Shih- Chih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Medicine, 3(1):136, 2020.
[8] Shih-Cheng Huang, Anuj Parcek, Roham Zamanian, Imon Banerjee, and Matthew P Lungren. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific Reports. 10(1):22147, 2020.
[9] Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. Combining 3D image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Computer Assisted InterventionMMCCAAI 2021: 24th International Conference, Strasbourg, France, September 27-October I, 2021, Proceedings, Part 24, pages 688-698. Springer, 2021.
[10] Nasir Hayat, Krzysztof J. Geras, and Farah E. Shamout. MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images. In Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 Proceedings of Machine Learning Research, pages 479-503. PMLR, 05-06 06 Aug 2022.
[11] Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren. Multimodal deep learning for biomedical data fusion: a review. Brieffiggs in Bioinformatics, 23(2):bbab569, 2022.
[12] Kwanhyung Lee, Soojeong Lee, Sangchul Hahn, Heejung Hyun, Edward Choi, Byungeun Ahn, and Joohyung Lee. Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention. In Proceedings of the Sth Machine Learning for Healthcure Conference, 2023.
[13] Wenfang Yao, Kejing Yin, William Cheung, Jia Liu, and Jing Qin. DrFuse: Learning disentangled repre- sentation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16416-16424, 2024.
[14] Julia Adler Milstein, Catherine M DesRoches, Peter Kralovec, Gregory Foster, Chantal Worzala, Dustin Charles, Talisha Searcy, and Ashish K Jha. Electronic health record adoption in us hospitals: progress continues, but challenges persist. Health Affairs, 34(12):2174 21102222174- 2015.
11

---

[15] Arom Choi, Kyungsoo Chung, Sung Phil Chung, Kwanhyung Lee, Heejung Hyun, and Ji Hoon Kim. Advantage of vital sign monitoring using wireless wearable device for predicting septic shock in febrile patients in the emergency department: A machine learning- analysis. Sensors, 22(18))7054, 2022.
[16] Claudia I Henschke, David F Yankelevitz, Austin Wand, Sheila D Davis, and Maria Shiau, Accuracy and efficacy of chest radiography in the intensive care unit. Radiologic Clinics of North America, 34(1):21 -31, 1996.
[17] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300-41313. PMLR, 2023.
[18] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz. Nathaniel R Greenbaum, Matthew P Lungren. Chih-ying Deng. Roger G Mark, and Steven Horng. MIMIC-CXR, de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019.
[19] Anusoumya Ganapathy, Neill KJ Adhikari, Jamie Spiegelman, and Damon C Scales. Routine chest X-rays in intensive care units: systematic review and meta- weta-analysis. Critical Care, 16(2):1-12, 2012.
[20] Declan Grant, Bartlomiej W Papicž, Guy Parsons, Lionel Tarassenko, and Adam Mahdi. Deep learning classification of cardiomegaly using combined imaging and non-imaging ICU data. Medical Image Understanding and Analysis: 25th Annual Conference, MIUA 2021, Oxford, United Kingdom, July 12-14, 2021, Proceedings 25, pages 547-558. Springer, 2021.
[21] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 21450-21474 21474. PMLR, 23-29 Jul 2023.
[22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages (0684-10695, 2022.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polasukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[24] Chaoqi Yang, M Brandon Westover, and Jimeng Sun. ManyDG: Many-domain generalization for healthcare applications. In The Eleventh International Conference on Learning Representations, 2023.
[25] Zhicheng Jiao, Ji Whae Choi, Kasey Halsey, Thi My Linh Tran, Ben Hsich, Dongcui Wang. Feyisope Eweje, Robin Wang, Ken Chang, Jing Wu, ct al. Prognostication of patients with COVID- 19 using artificial intelligence based chest X-rays and clinical data: retrospective study. The Lancet Digital Health, 3(5): e286 c294, 2021.
[26] Zhuo Zhi, Moe Elbadawi, Adam Daneshmend, Mine Orlu, Abdul Basit, Andreas Demosthenous, and Miguel Rodrigues. Multimodal diagnosis for pulmonary embolism from EHR data and CT images. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 2053-2052 IEEE, 2022.
[27] Chihcheng Hsieh, Isabel Blanco Nobre, Sandra Costa Sousa, Chun Ouyang, Margot Brereton, Jacinto Nascimento, Joaquim Jorge, and Catarina Morcira. MDF-Net for abnormality detection by fusing X-rays with clinical data. Scientific Reports, 13(1): 2023.
[28] Can Cui, Haichun Yang. Yaohong Wang. Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett Landman, and Yuankai Huo. Deep multi-modal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering, 2023.
[29] Yao Zhang, Nanjun Hc, Jiawci Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang Hc, and Yefeng Zheng. mmFormer: Multimodal medical Tansformer for incomplete multimodal learning of brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 107-117. Springer, 2022.
[30] Chaohe Zhang, Xu Chu, Liantao Ma. Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao. M3Care: Learning with missing modalities in multimodal healthcare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2418 2428, 2022.
12

---

[31] Shruthi Bannur, Stephanie Hyland. Qianchu Liu, Fernando Pérez-García, Maximilian llse. Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Adilya Nori, Javier Alvarez- Valle, and Ozan Oktay. Learning to exploit temporal structure for biomedical vision- languae processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15016 15027, June 2023.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding, Advances in Neural Information Processing Systems,
2022.
[35] Dongchao Yang, Jianwei Yu, Helin Wang. Wen Wang, Chao Weng. Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[36] Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo Hc, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. MM-Diffusion: Learning multi-modal diffusion models for joint audio and vidco generation. In Proceedings of the IEEE/CVÉ Conference Computer Vision and Pattern Recognition, pages 10228, 2023.
[37] Akim Kotelnikov. Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pages 17564-17579. PMLR, 2023.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical lext-conditiona image generation with CLIP latents, 2022. URL https://arxiv. org/abs/2204.06115, 7, 2022.
[39] Walter HL Pinaya, Petru- Daniel Tudosiu, Jessica Daffion, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models, pages 117-126. Springer, 2022.
[40] Wei Peng. Ehsan Adeli, Tomas Bosschieter, Sang Hyun Park. Qingyu Zhao, and Kilian M Pohl. Generating realistic brain MRIs via conditional diffusion probabilistic model. In International Conference on Medical Image Computing and Computer-
Intervention, pages 24. Springer, 2023.
[41] Kai Packhäuser. Lukas Folle, Florian Thamm. and Andreas Maier. Generation of anonymous chest radiographs using latent diffuusion models for training thoracic abnormality classification systems. In 2023
IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1-5, IEEE, 2023.
[42] Tobias Weber, Michael Ingrisch, Bernd Bischl, and David Rügamer. Cascaded latent diffusion models for high-resolution chest X-ray synthesis. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 180-191. Springer, 2023.
[43] Yu Gu, Jianwei Yang. Naoto Usuyama, Chunyuan Li, Sheng Zhang, Matthew Lungren, Jianfeng Gao, and Hoifung Poon. Biomedjourney: Counterfactual biomedical image generation by instruction -estuettoo-lecaming from multimodal patient journeys. arXiv preprint arXiv:2310. 10765, 2023.
[44] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. CheXpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590-597, 2019.
[45] Diederik Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
[46] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[47] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
EiterventionMMICAA
13

---

[48] George Zerveas. Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A Tansformer-based framework for multivariate time series representation learning. Proceedingy of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 2114-2124, 2021.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems. 30, 2017.
[50] Alistair EW Johnson, Lucas Bulgarelli, Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, 2023.
[51] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data, Scientific Data, 6(1):96, 2019.
[52] Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L luzzolino, and Kazuhito Koishida. MMTM: Multimodal transfer module for CNN fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Partern Recognition, pages 13289-13299, 2020.
[53] Tian Xia, Agisilaos Chartsias, Chengjia Wang, Sotirios A Tsaftaris, Alzheimer's Disease Neuroimaging Initiative, al. Learning to synthesise the ageing brain without longitudinal data. Medical Image Analysis, 73:102169, 2021.
[54] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash cquilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[55] Mohammad Amin Morid, Olivia Sheng, Joseph Dunbar. Time series prediction using deep learning methods healthcare, ACM Transactions on Management Information Systems, 14(1):1-29, 2023.
14

---

A Experiment Details
A.1 Details of Architectures and Training Procedures of DDL-CXR
The training and validation processes are executed on a server equipped with a RTX 4090-24GB GPU card and 16 vCPU Intel Xeon Processor. The method is implemented using Py PyToochh 1.9.1 and py PyTorch-Lighming 1.4.2. DDIM [33] sampling with 200 steps is employed to accelerate the sampling process, and AdamW W optimizer is used for all model training. Our implementation is partially based on the repository of the latent diffusion model [22]².
Variational autoencoder (VAE) The VAE training process, outlined in [22], includes pixel- wise reconstruction loss, a perceptual loss [46], an adversarial objective, and a Leibler loss toward standard normal to constrain latent spaces, given by:
VAVE
min max (ice X CXR DI ))) Lad (D(E + log D (x
(6)
I ); E;E,) Leu (B(E(((()))
where T is a classifier that predicts the CheXpert labels associated with the image and we parameterize it with an MLP. All CXRs in the training subset of the LDM stage are gathered for VAE training. A compression rate = 8 is adopted, and the training continues for a maximum of 50 epochs. The model with the minimum validation error, as measured using CXRs from the validation subset, is selected. The resulting latent representation has a dimension of 4 x 28 x 28 3136. To restrict the normal prior in the latent space and prioritize reconstruction quality, a KL-divergence weighting of 1 x 10 is set. We use the base learning rate of 4.5 x 10 which is scaled by the number of GPU cards and batch size.
Latent diffusion model (LDM) stage in DDL-CXR In the LDM stage of our DDL-CXR model, we employ the UNet architecture [47] as the neural backbone, denoted by Eg. Meanwhile, we utilize multivariate time series Transformer [48] for the EHR conditioning encoder f fiit The Transformer (FHR is designed with one layer, a model dimension set to 128, and a maximum EHR data length of 70. The UNet model En features an input channel of 8 and output channel of 4. The encoding section comprises three blocks, with model channels set at 224, 448, and 672, consisting of a ResBlock module followed by a spatial transformer. The bottleneck consists of two ResBlock modules with spatial transformer in between. The decoder mirrors the encoder architecture. As discussed in Section 3.2, we introduce the encoded EHR information through multi-head cross-altention to the spatial transformer module of Eg. The context dimension is set to 128, and the number of attention heads 8. The model is trained for 200 epochs with batch size of 32, and the model with the smallest composite loss on the validation set is selected for subsequent latent Chest X-ray (CXR) generation. We set the hyperparameters a to 0.2, and 33 to 0.5, empirically.
Latent CXR generation via LDM In the LDM stage, we aim to generate latent CXR images at time f1, conditioned on XCXR and XEHR VAE, given by:
We first encode the CXR images using the pre-trained (fo.f1)*
Z 8(X CXR
Zr CXC
Essentially, the latent CXR generation requires us to estimate the underlying data distribu-
tion qZz Ze., x EHR
(tost1)) The LDM approximates this distribution via a model distribution
(Z,) Z.,, x EEIR
(0)
(f.tt)) where ( represents the prior of CXR in the latent space encoded by
the VAE.
We follow prior work on diffusion models to learn our LDM, which involves two processes [32, 22]. In the diffusion process, we gradually add Gaussian noise to 2,") in N steps, producing a sequence of noisy representations f1 (222) ..... (NN) with the transition probability given by:
q( (Z (n -1) = N(2 (n) /1
3, 1 I,
(8)
q( ((Z (0)) = N(Z (1) 2, 110 (1 ān)e),
open source under MIT license.
15

---

where ~ (O, represents the added noise. The noise level is represented by ā :- =- (1-Ba).
where {Bn E (0, 1) is pre-defined variance schedule. At step N, ( ~ (O I) follows an isotropic Gaussian distribution. In the denoising process, we start with sample from the isotropic Gaussian distribution N (O, I) and gradually recreate the latent CXR Z conditioned on the reference latent CXR, Zug and the EHR data in between, XEHR The generation process given by:
(fu.f2)
N II pa
EHR Zio X
(1o.11)) p (Z!
Z
Z (Z! Zio X
EHR (1a.t1))
(9)
where
[ 11--1) [ (z,")) Zig. X
EHR (fort1) = N (z!" (m-1); D) " (Z n, Zoo x EHR
(to.f1) .0..t
(10)
and the mean Meg and variance o are parameterized by
p4 (z((), n, Zoo X (00.))))
Bn E0
(z((")
Zio: BBBR EUR (xbt))))))))
(11)
v /m
V1 ān
and
1 an-1 Bm.
(12)
On
where JEHR (-) is the encoder for the irregular EHR time series. 60 (Z (n) denotes the generation noise predicted by the neural backbone.
Zio ccomd
X EHR
(fo.4s)
));
n
Prediction stage in DDL-CXR In the prediction stage, the EHR data is encoded using a one-layer Transformer with a model dimension of 128. We set the dimension of the feedforward layers to 512. The context dimension also set 128, and the number of attention heads is We use another Transformer with the same architecture to encode the generated latent CXR Z1. We use ResNet-34 model encode the last available CXR image Xo. The encoded EHR, the encoded latent CXR Z as well as the encoded Xo are fed into self-attention layer for final prediction.
A.2 Details of Data Preprocessing
EHR data preprocess We follow a similar EHR data extraction and processing pipeline as [10] but change the sampling frequency from 2h to Ih and introduce two static variables, age, and gender. We extract 17 clinical time series variables (12 continuous and categorical) with discretization and standardization processes exactly the same as in [10]. In addition to the 17 clinical time series variables mentioned in the paper [10], e.g. five categorical (capillary refill rate, Glasgow Coma Scale eye-opening, Glasgow Coma Scale motor response, Glasgow Coma Scale verbal response, and Glasgow Coma Scale total) and 12 continuous (diastolic blood pressure, fraction of inspired oxygen, glucose, heart rate, height, mean blood pressure, oxygen saturation, respiratory rate, systolic blood pressure, temperature, weight, and pH), we introduce two crucial static variables (age and gender) to represent the demographic information of patient, patient demographic information is vital for achieving accurate predictions [55]. To construct our dataset, we sampled time series data at hourly intervals, followed by discretization and standardization processes. We adopt masks to handle missing values in time series to capture the missing pattern, acknowledging that the absence of medical data might be intentional and non-random, driven by caregivers [55].
Data Cohort and Potential Selection Bias We summarize the number of samples in the LDM stage and the prediction stage in Table 6. The label prevalences of the two prediction tasks summarized in Tables 7 and 8. We include the disease prevalence in Table and data cohort statistics in Table 9. Here we summarize few key points:
The statistics the clinical variables are close to each other, suggesting that patients our data cohort generally have similar distribution as that in the entire database.
The overall disease phenotype prevalence similar.
* For a few thorax-related diseases, our data cohort has slightly higher prevalence, suggesting that potential selection bias exists.
16

---

Table 6: Data statistics in training, validation, and testing sets for each stage.
Stage
Training
Validation
Test
LDM stage
8545
1392
2353
Prediction stage
5142
861
1483
Table 7: Label prevalence of in tospoosptall mortality prediction task.
Training
Validation
Test
Negative 4396
737
1264
Positive 746
124
219
Negative/Positive
5.89
5.94
5.77
Table Number of samples and prevalence disease phenotypes in training, validation, and testing sets during the prediction stage. The prevalence of disease phenotypes among all ICU stays from MIMIC-IV database having LoS 2 48h is given in the last column.
Discase Label
Training
Validation
Testing
Training Prevalence
Validution Prevalence
Testing Prevalence
MIMIC-IV Prevalence
Acute and unspecilfied renal failure
1932
342
561
0.38
0.40
0.38
0.34
Acute cerebrevascular discase
467
88
113
0.09
0.10
0.08
0.07
myocurdial
Cardiac dyarhhthhins
2049
361
611
0.40
(142
0.41
0.38
Chronic kidney diseuse
1258
216
399
0,24
0.25
0.27
0.23
obstractive palmonary disease
151
Complications of surgical/medica care
1218
219
372
0.24
0.24
0.25
0.25
Conduction disorders
92
Congestive heart failure; nontypetensive
1674
288
189
0.33
0.33
0,33
0,30
Coronary atheroselerosis and related
1605
257
491
0,31
0.30
0.33
0.33
complications
Diabetes mellitus without complication
1068
182
325
0.21
0.21
0.22
0.18
Disorders of lipid metaholism
2047
349
595
0.40
0.41
0.40
0.41
Essential hypertensoon
Fluid and electrolyte disorders
2685
455
749
0.52
0.53
0.51
0.45
Gasrronnessi hemorrhage
68
129
Hypertension with complications
1129
202
361
0.22
0.24
0.24
Other liver discases
899
147
279
0,17
0.17
0.19
0.15
Other respirstory
Other upper respiratory disease
376
67
96
0.07
0.08
0.06
0.07
Pleurisy; pocumothorax, pulmonary collapse
553
99
141
0,11
0.10
0.09
Pacumonia
187
0.22
0.22
Respiratory failure; insufficiency; arrest (adult) Septicemia (except labor)
1741
307
506
0.34
0.36
0.34
0.25
Sbock
1157
192
357
0.23
0.22
0.24
0.18
Table 9: Statistics of data cohort. We report meantstd for continuous variables and the mode for categorical variables.
DDL-CXR Cohort
MIMIC-IV Databuse
Variables
Sample size
5142
26330
Age
63.9216.6
64.3216.4
Gender (Men, %)
54.98
55.99
Diastolic blood pressure
62.2+10.1
62.8+10.5
Fraction inspired oxygen
0.4:0.2
0.4+0.2
Glucose
141.1=41.7
139439.6
Heart Rate
87.3+15,4
85.6+15
Height
170+1.
170=
Mean blood pressure
77.1=10
78.3+10.4
Oxygen saturation
96.742
96.5+2
Respirutory rute
19.8+
19,5+3,7
Systolic blood pressure
118.3+15.2
118.6=15.6
Temperature
36.9+0.4
Weight 7.240.3
80:80
19.5
pH
7.2+0.3
Capillary refill rate GCS eye opening
Normal
Normal
Spontancously
Spontancously
GCS motor response
Obeys Commands
Oheys Commands
GCS verbal response GCS total In-hospital mottality (%)
Oriented
Oriented
12
17

---

B Additional Results
B.1 AUPRC of Mortality Prediction
We summarize the AUPRC score for the mortality prediction task in Table 10.
Table 10: The AUPRC score for mortality prediction for overall and different time gaps. ô represents the time gap between the generation (or prediction) time and the occurrence time of the last available CXR. Numbers in bold indicate the best performance in each column. DDL-CXR outperforms all baselines in overall and 8 24h settings.
Overall 14.7%
8 12
12 å 24 19%
24 S 8 36 15.9%
2 36 9.26%
prevalence
16.6%
Uni-EHR [23]
0.498 #0.007
0.579
+0.009
0.541 +0.016
0.416 +0.018
0.411 =0.005
MMTM [52]
0.422 #0.014
0.430 00.010
0.458 +0.020
0.476 #0.021
0,374 =0.012
DAFT [9]
0.448 00.004
0.460
10.011
0.478 40.010
0.508 40.015
0.395 +0.007
MedFuse [10]
0.443 +00009
0.498 +00013
0.484 40.023
0.432 +0.015
0.355 =0.008
DrFuse [13]
0,460 +0.004
+0.033
40.033
0.382 +0.072
0.399 #0.019
GAN-based [53]
0.505 +0.018
0.567 +0.015
0.551 +0.018
0.414 +0.041
0.442 00.034
DDL-CXR (ours)
0.523 #0.011
0.610 =0.015
0.566 10.013
0.429 +0.009
0,440 =0.008
B.2 AUROC of Phenotype Prediction by Disease Label
We summarize the AUROC score for the phenotype prediction task by disease label in Table 11.
Table 11: The AUROC score by disease labels. Results show that DDL-CXR effectively tackles the
asynchronicity issue in clinical multi-modal fusion, achieving the highest average rank across all disease labels.
Uni-EHR [23)
MMTM (52)
DAFT [9]
MedPuse [10]
DrPuse [13)
GAN-based [53)
DDL-CXR (ours)
Acule renal failure
0.71810.003
0.706:10.004
0.712=0.005
0.706:20.04
0.702+0.003
0.710=0005
0.723=0.004
Acute cerehrwascular discase
0.880=0000
0.803+0:003
0.888++0.00
0.895++0.005
0.873++0.009
0.875=0001
0.870=00.08
myocardial infarction
0.717+0:005
0.722=0.009
0.724+0.009
0.730+0.015
0.735=0.009
Cardise dysrhythmnia
0.677.00005
0.64810.004
0.656:10.004
0.66710.010
0.68210.005
0.673
0.003
Chronic kidney disease
0.745=0008
0.735+0.008
0.732+0.004
0.720+40004
0.710+00011
0. 0.748=0.00 006
0.768=0.006
COPD and bronchiectasis
0.706:10.006
0.6891100144
0.707
0.686:10.006
0.733.10.011
0.738100003
0.740
Surgical complications
0.636.10.007
0.656110.008
0.648.10.066
0.662.10.005
0.64910.004
0.61010001
0.649
Conduction disorders
0.741:20006
0.717+00011
0.735=0.012
0.724=:0.005
0.84750.011
0.842=00006
CHF: nonhy perteusive
0.77110.001
0,71710.010
0.796.10.007
0.77910.008
0.797.1..00
0.79810.006
0.820
Coronary atherosclerosis & others
0.740:10.005
0.728:10.003
0.733-10.007
0.70111.008
0.742.10.006
0.753:00007
0.764:0000
DM with complications
0.85810.005
0.85310.006
0.851 10.000
0.845
0.84910.000
0.851 20.000
0.805
DM without complisation
0,716.10.005
0.70710.007
0.694
0.70410..003
0.712.10.005
0.707
0.717-00.009
Disorders lipid metabolism
0.704:10007
0.677:00.05
0.676:0.006
0.677
0.696:10.005
0.714=0000F
Esscutial hy peretcasion
0.662.0.007
10.091
0.625
0.632.10.006
0.612.10.03
0.65910.008
0.667 000008
Fluid and electrolyte disorders
0.671:20.02
0.671 0.67410.003
0.665-10.006
0.671
0.67400007
0.677
Gastrointestinal hementhage
0.664100.05
0.6351
0.65310.018
0.66410.005
0.67710.013
0.662100008
0.678_0.00
Secondary hypertension
0.725-10.07
0.72618.007
0.722:10.805
0.716:10.004
0.714:10.10
0.734100005
0.752
Other liver disenses
0.681-20.077
0.680+66005
0.693-00.016
0.690+00.044
0.697-40.09
0.700=0010
Other lower respiratory discase
0.596:10.044
0.60610.007
0.608.1..010
0.615
0.634110.05
0.61910019
0.614_0.011
Other upper respiratory discase
0.691=00.00
0.691 169109
0.711=00.066
0.726=0.010
0.741 +0.013
0.699=0011
O. 0.700-0.017
Pleurisy: pneumothorax
0.634=00.05
0.647+00.15
0.674=0.014
0.673+40.008
0.687+0.014
0.670=00027
0.666=0.010
Pneumonia
0.704:00006
0.728-10.004
0.723=00.007
0.714:20.055
0.715
0,707=00004
0.719=0.008
Respiratory failure
0.800=00.05
0.816+00.05
0.815=0..03
0.809=00.005
0.810+0.004
0.805=0
000.00
Septicemia (except in labor)
0.76420.005
0.757+0.005
0.760=00.077
0.741:200005
0. 0.74910.003
0.763=00004
0.774=00005
Shock
0.803-60.044
0.803++0.005
0.805-+0.006
0.507:33.005
0.800+00.008
1.801=0007
0,806+0.0077
Average Rank
4.56
4.76
4.32
4.64
3.88
3.84
2
18

---

B.3 AUPRC of Phenotype Prediction by Disease Label
We summarize the AUPRC score for the phenotype prediction task by disease label in Table 12.
Table 12: The AUPRC score by discase labels with detailed standard deviation.
Uni-EH1R (23] MMTM [52]
DAFT [9]
MedIFuse [10] DrFuse [13]
GAN-bused [53] DBL.CIR (ours)
Acule renal failure
0.573=00004
0.565:40.05 0.572=00.006 0.568:0.007
0.564=0.005
0.563=00.055
0.588+0.008
Acute cerebrovacular discase
0,425=0,010
0,41810.010
0.4191100077
0:43410.017
0.399
0.44611.0010
0.4161100018
Acute myocardial infarction
0.185=00088
0.192:10.001
0,187:0.009
0.219::00015
0.209=000133
0.171
0.2061100020
Cardinc dysrhythmias
0.579-0.006
0.532:400015
0.548=0:005
0.560=0.012
0.584-0.009
0.561
0.605++0.000
Chronic kidney diseuse
0.515=0.016
0.505=+0.011
0.515500.099
0.497=00009
0.477=0.016
0.501 =0.00
0.508+0.010
COPD bronchiectasis
0.319=00012
0.32710.0011
0.34210.011
0.344:0.000
0.405-0.0010
0.372-50.012
0.202
Surgical complications
0.370=00.009
0.379
0.38510.007
0,38110.006
0.377=200003
0.34410.000
0.38810.001
Conduction disorders
0.276=00006
0.287
0.298:00013
0.286=#0.016
0.632:0.010
0.609
0.633++0000
CHE; noehypertensive CAD
0.593=0.007
0.619+0.019
0.647=#0.017
0.631
0.661=00015
0.652=00.12
0.682+00009
0.560=0.007
0.54040.006
0,556::0.008
0.544:1.009
0.581=0.012
0.590=00014
0.611
DM with complications
0.562-0,013
0.56910.016
0.552:10.006
0.561
0.550=0.0115
0.552
0.52410.007
DM without complication
0.370=00005
0.367:0.013
0,343::0.014
0.356:20.006
0.369=0.010
0.352=00012
O. 0,36810..00
Disorders of Lipid metabolism
0.594_0.007
0.576.10.022
0.570110.009
0.566110003
0.584
0.587.00.008
0.60110.010
Essential hypertension
0.551=00006
0.519+0.0033
0.525::0.005
0.518:00.004
0,502:00.005
0.554=0.011
0.561
Fluid electrolyte disorders
0.655-0.005
0.664+0.0005
0.662:40.044
0.656-=0.00
0.658=000006
0.662=+0012
0.672+00.00
Gastrointestinal hemorrhage
0.180=0.015
0. 0.142=00.04
0.162::0.019
0.192=0.0114
0.191=0.014
0.151=00.008
0,180
Secondary hypertension
0.403=0.012
0.452110.007 0.455
0.453-10.013
0.437=0.012
0.451
0,484410.009
Other liver discases
0.316=00013
0.316:10.010
0. 0.341:10.007
0.344.10.008
0.372=0.014
0. 0.302220.018
0.378
Other lower respiratory discase
0.219-00005
0.209+000022
0.206=00000
0,223-+0.055
0.255-0.011
0.236=00.008
0,242+0.007
Other upper respiratorry disease
0.166=00015
0.137++0.009
0.166:#0.019
0,202=+00014
0.274=0.018
0.196=00019
0.234+0.089
Pleurisy: pneumothorax
0.143=00.005
0.145:10.008
0.1591:0.009
0.159:10.013
0.172:00.007
0.071
0.166:10.009
Pacumonia
0.412=000122
0.437.10.005
0.4291:0.008
0,419:10.088
0,406:2.0016
0.015
0.42810.013
Respiratory failure
0.655-0.009
O 1.66:00.011
0.674::0.004
0.671 +000009
0.692:00.005
0. 0.663==0.007
O 0.6691
Septicemia (except labor)
0.585-0.008
0.573+0.012
0.580=00:014
0.565++0.009
0.562=00.13
0.573+00.009
0.603+00010
Shock
0.590=0.002
0,584:10.005
0.582:40.099
0.592:10.008
0.572=#00021
0.587==0.011
0.586200003
Average Rank
4.4
4.64
4.4
4.24
3.88
4.16
2.28
B.4 Ablation Study for a Reduced Percentage of Data
To investigate the sensitivity the amount of training data of the proposed model and the main baselines, we conduct experiments with varying training sizes. The results are summarized in Table 13 and Table 14. Results show that DDL-CXR consistently outperforms baseline methods by large margin, demonstrating its robustness against training data size.
Table 13:
Performance of the phenotype classification task with different training sizes. Results are reported in meantstd.
AUPRC
AUROC
100%
75%
50%
100%
75%
50%
Uni-EHR [23]
0.434 +0.009
0.428
+0.011
0.419 +0.010
0.720
+0.006
0.711
+0.008
0.706
+:0.066
MMTM [S2]
0.430 1:0.005
0.421
1:0004
0.406 10:003
0.715 10.003
0.707 10.002
0.694 1:0.00
DAFT [9]
0.435 +00002
0.422
+0.004
0.407 +0.003
0.720 +0.003
0.709 +0.002
0.699 +00003
MedFuse [10]
0.437 +00001
0.420 +0.004
0.412 +0.002
0.718 +0.002
0.707 +0.003
0.700 +:00001
DrFuse [13]
0,459 +0.003
0.442 +0.005
0.431 +0.004
0.729
+0.004
0.717 10.005
0.709 10.004
DDL-CXR
0.470 10.003
0.457 10.003
0.433 10.011
0.740 10.002
0.734 1.0.02
0.715 100005
Table 14:
Performance of the mortality prediction task with different training sizes. Results are reported in meantstd.
AUPRC
AUROC
100%
75%
50%
100%
75%
50%
Uni-EHR [23]
0.498
+0.007
0.437 +00011
0.429
+00012
0.815
+00007
0.791
+00005
0.782
+-0.013
MMTM [52]
0.422 +0.014
0.405
+0.010
0.399
+0.012
0.785
+0.004
0.782 +0.002
0.775 +0.005
DAFT [9]
0.448 10.004
0.428
10.006
0.413 10.005
0.800
1..003
0,790
100004
0.778 1.0.007
McdFuse [10]
0.443 +00009
0.420
+0.015
0.411 +0.009
0.793 +0.003
0.784
+0.004
0.775 +0.005
Dri-use [13]
0.460 +0.004
0.430
+0.013
0.415 +0.030
0.773 +0.008
0.755 10.008
0.766 +0.030
CXL-CXR
0.523 1.0.111
0.474
10.009
0.466
10.012
0.822 10000
0.801
10.008
0.790 1:0.008
19

---

B.5
Additional Case Studies
We demonstrate additional case studies in Fig. 4 Fig. We retrieve findings from the radiology reports. The case studies show that DDL-CXR could generate CXR images that align with the disease progression of the individual patient.
(a)lnitial radiology findings: the lung vol-
(b) Radiology findings after hours: There interval progression stitial pulmonary edema and potential interval increase in bibasal consolida- tions.
(c) CXR image generated by DDL-CXR given the initial CXR image shown (a) and the EHR data within the hours.
chest demonstrates umes, which results in bronchovaseu- lar crowding. Bibasilar opacities may reflect atelectasis There is probable small left pleural effusion.
Figure 4: Case Study of Sample #1
Heart remains enlarged with left ventricular prominence. Interval ap- pearance of lincar opacity in the right mid lung which may reffect fluid locu- lated within the minor fissure possi- bly subsegmental atelectasis.
Radiology findings after 18 hours: There fluctuating patchy opacities at the right lung base suggestive of at- electasis. Low volumes with crowding of the pulmonary vasculature.
CXR image generated by DDL-CXR given the initial CXR image shown in (a) and the EHR data within the 18 hours.
Figure 5: Case Study of Sample #2
(a) The dense stelectatic streaks in the left mid zone has decreased. The bi- lateral chest tubes remain in place and there is evidence pneumothorax.
(b) Radiology findings after 13 hours: No left pacumothorax is uppreciated. The hemidiaphragms are now sharp be with only mild atelectatic changes the bases.
CXR image generated by DDL-CXR given the initial CXR image shown in
(a) the EHR data within the 13 hours.
Figure 6: Case Study of Sample #3
20

---

PORT HUR
(a) Initial radiology fiudings: The lung volumes are low. Mild candiomegaly without pulmonary odema. No pleural effusions.
(b) Radiology findings after 13 hours: There is mild bibasilar atelectasis. There pacumothorax large pleural effusion.
(e) CXR image generated by DDL-CXE given initial CXR image shown in the EHR data within the 13 hours.
Figure 7: Case Study of Sample #5
(u) Initial radiology findings: The lungs bilaterally demonstrate severe, extensive rounded nodular densities.
(b) Radiology findings after 31 hours: Bilateral nodular parenchymal opaci- unchanged this patient with known lymphoma. There are likely layering eftusions, left greater than right.
(c) CXR image generated by DDL-CXR given initial CXR image shown in and the EHR data within the 31 hours.
Figure 8: Case Study of Sample #6
PORTIBLE
(a)lnitial radiology findings: Heart size is normal. Mediastinal and hilar contours are within normal limits, Pul- monary vasculature normal.
Radiology findings after 29 hours: Small umount of right pleural effusion is present, more conspicuous than on prior study.
CXR image generated by DDL--CXR given the initial CXR image shown in (a) and the EHR data within the 29 hours.
Figure 9: Case Study of Sample
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflected the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations in Section 5. We will address these limitations in future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The related codes for reproducing the main experimental results are submitted in supplementary materials and the results are discussed in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit the codes as supplemental material to the paper submission. The codes will be made publicly available upon paper acceptance. The datasets used in the paper are open-source and publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Psllccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.oo. /twpps.ccpppblll///
SeaeidddeeoooSSubmimeeeeee ceesssooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of data splits and the hyperparameter searching space are illustrated in Section 4.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are reported by taking the average of five runs of model training along with the standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of computer resources for the experiments are given in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps//neurt: hwwrrpsuu...///
Answer: [Yes]
Justification:
We conform in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential societal impacts and negative societal impacts of the work performed in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited all code, data, and models used in the paper. The licenses are explicitly mentioned.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. The paper utilizes de-identified public datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study involves only public de-identified datasets and no IRB approvals needed.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "hmg0NB91ZS", "reviewer": "Reviewer_7ywn", "summary": "The paper proposes a Diffusion-based Dynamic Latent Chest X-ray Image Generation (DDL-CXR) to address the asynchronicity in multimodal clinical data fusion. The primary objective is to generate up-to-date latent representations of chest X-ray images based on existing X-ray data and EHR. The method leverages latent diffusion models conditioned on previous X-ray images and EHR time series, aiming to capture anatomical structures and disease progressions. The authors claim that this approach improves clinical prediction performance, as demonstrated through experiments using the MIMIC datasets (MIMIC-CXR and MIMIC-IV).", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper identifies and aims to tackle an interesting question in ICU settings: misalignment of CXR data and temporal EHR data, therefore proposing to dynamically generate up-to-date chest X-ray images. The use of latent diffusion models, conditioned on both previous X-rays and EHR data, is sound. The experiments on the MIMIC datasets are thorough and show promising results.", "weaknesses": "One weakness is the lack of clarity and detail in the methodology section. The descriptions of the latent diffusion model and the integration of EHR data could be more precise and structured to enhance understanding. For example, the demonstration of the Gψ is not clear; how are each modality combined? How is the attention mechanism learned? Please also discuss the computational cost as diffusion models are generally costly and require more data samples for better generation quality. In addition, please include more experiments regarding the generated CXR image's quality apart from the FID-related score, but from task-relevant prediction performance. For example, only include single CXR modality to justify the performance improvement over the initial ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and insights. We hope the following responses address your concerns.\n\n---\n**1. Description of latent diffusion model could be more precise.**   \nThe latent diffusion model (LDM) used in this work has the same architecture as the original LDM paper. Due to space limit, in the main text, we included the citation of LDM paper in the main text and focused on introducing the generation method conditioning on EHR data, i.e., $\\epsilon_\\theta$. The detailed diffusion and denoising processes are presented in Appendix A.1. Thank you for the comments and we will further clarify this in the final version.\n\n---\n**2. Integration of EHR.**  \nIn the LDM stage, EHR is integrated in the prediction of noise added in the denoising steps, $\\epsilon_\\theta(\\mathbf{Z}\\_{t_1}^{(n)},\\mathbf{Z}\\_{t_0}, f(\\mathbf{X}^{EHR}\\_{(t_0, t_1)}), n)$. As described in Section 3.2, this noise term $\\epsilon_\\theta$ is parameterized by a standard UNet. Again, due to space limit, we omitted the details of UNet and included reference in the main text. However, in Eq. (1), we highlight how EHR is integrated via constructing the input of the UNet layers by cross-attention. As suggested, we will further improve the clarity of this part in the final version.\n\n---\n**3. Demonstration of $\\mathcal{G}\\_\\psi$; how are modalities combined and how is attention learned.**  \nIn the prediction stage, EHR data is integrated via the prediction model $\\mathcal{G}\\_\\psi$, as shown in Eq. (4). It is parameterized by a self-attention layer. Concretely, the encoded inputs of $\\mathcal{G}\\_\\psi$ are firstly concatenated. For clarity, let $\\mathbf{h}=[f^{\\text{CXR}}\\_{\\text{pred}}(\\mathbf{X}^{\\text{CXR}}\\_{t_0}), \\ f^{\\text{EHR}}\\_{\\text{pred}}(\\mathbf{X}^{\\text{EHR}}\\_{\\leq \\text{48h}}), \\ f^{\\text{LAT}}\\_{\\text{pred}}(\\widehat{\\mathbf{Z}}\\_{t_1})].$ Then the attention layer can be written as $\\text{attention}(\\mathbf{h})=\\text{softmax}(\\frac{(\\mathbf{W}^\\prime_Q \\mathbf{h})\\cdot (\\mathbf{W}^\\prime_K \\mathbf{h})^\\top }{\\sqrt{d}})\\mathbf{W}^\\prime_V \\mathbf{h}$. The output of this attention layer will then be fed into a fully connected layer to produce the final predictions. The attention layer has three learnable parameters $\\mathbf{W}^\\prime_Q$, $\\mathbf{W}^\\prime_K$, and $\\mathbf{W}^\\prime_V$ and they are updated end-to-end with other parameters of the prediction stage. We will further clarify this in the final version.\n\n---\n**4. Computational cost of LDM**  \nThank you for pointing this out. We fully agree with the reviewer that diffusion models are generally more costly. Using one RTX 4090, the training of the LDM stage and the prediction stage take around 12.5 hours and 30 minutes, respectively. Additionally, we also trained the autoencoder from scratch, which takes around 24 hours. Although there are pretrained autoencoders available, we opted for training from scratch to avoid data leakage. The baseline models, on the other hand, requires less than four hours to train. Similar to other LDM-based applications and models, the higher computational cost is indeed one of the limitations of the proposed method. We will include this discussion in the limitation statement. \n\n---\n**5. Quality of Generated CXR images via prediction performance**  \nThank you for the suggestion. We added an ablation study of single CXR modality to evaluate the performance gain of DDL-CXR over the last available CXR only. The results are summarized as below and included as Table R1 in the pdf file attached in the global rebuttal. \n\n*Table: AUPRC and AUROC scores for phenotype classification with single CXR modality.*\n| Input Type       | AUPRC | AUROC |\n|-|-|-|\n| Last CXR (EHR removed)  | 0.376±0.009 | 0.661±0.007 |\n| DDL-CXR (EHR removed) |  0.385±0.006 | 0.668±0.007 |\n\n\n*Table: AUPRC and AUROC scores for mortality prediction with single CXR modality.*\n| Input Type       | AUPRC | AUROC |\n|-|-|-|\n| Last CXR (EHR removed)  | 0.243±0.002 | 0.664±0.006 |\n| DDL-CXR (EHR removed) | 0.269±0.013 | 0.707±0.008 |\n\nThe results suggest that the generation of the latest CXR significantly benefits clinical prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QvaS9lciQw", "reviewer": "Reviewer_1LZb", "summary": "The paper proposes a new method, DDL-CXR, to generate chest X-ray images that address asynchronicity in multi-modal clinical data for predictive tasks. The authors leverage latent diffusion models for patient-specific generation, conditioned on a previous CXR image and EHR time series. The experiments using MIMIC datasets show that the proposed model outperforms existing methods in phenotype and mortality prediction tasks.", "questions": "- Is the generated images created from Z_hat_t_1 using a VAE decoder? is the dimension of that the same as Z_t_1?\n- It would be clearer if the authors mentioned why combining CXR and time-series data improves mortality and phenotype predictions, possibly by citing observational studies in medical papers. As a non-medical expert, I find it confusing that acute renal failure and chronic kidney disease, which do not seem related to CXR, benefit from the proposed model combined with CXR data. According to the paper, CXR with radiology labels helps rather than distracts. What is the reason behind this?\n- I understand the usefulness of this approach for downstream predictive tasks. However, for qualitative analysis, it’s also crucial to compare different models to make sure whether the generated images are plausibly aligned with the patient’s medical records, instead of the sheer performances of the downstream tasks. Do authors think this model can be reliably used as a proxy for current chest x-ray image?\n- What is the procedure for choosing the hyperparameters of the model? How sensitive are they compared to other methods?\n- Was only one GPU with 24GB used for training and validating the method? How many hours did it take?", "limitations": "The limitations of the paper are well-stated.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The motivation for the paper is convincing, and it demonstrates effectiveness in several predictive tasks.\n- The method used in the approach is reasonable.", "weaknesses": "- As far as I understand, there have been works leveraging the temporal structure of images. For example, \"Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing\" tackles a related topic (although text-image multimodal context). Since the title of the paper is \"Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation,\" I am curious why the authors did not discuss this aspect. Also, in the paper, the authors claim \"DDL-CXR is the first work to improve multimodal fusion via generating up-to date individualized CXR images.\" A few baseline models are also based on combining CXR and time-series data.\n- Is there an ablation study for a reduced percentage of data? How sensitive is this method when the amount of training data decreases compared to other multimodal and time-series-only baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We hope the following responses address your concerns.\n\n---\n**Weakness 1: Related works leveraging the temporal structure of images and the claim of contribution.**\n\nThe mentioned paper (BioViL-T) is indeed relevant. BioViL-T learns spatiotemporal features from a sequence of images using a hybrid CNN-Transformer whereas we capture the temporal relationships using a generative approach. BioVil-T does not consider the asynchronicity between modalities, as the clinical reports and images are already temporally aligned. However, EHR and CXR are highly asynchronous, brining extra difficulties to the information integration. We will include the discussions in our final version.\n\nWe would like to clarify that this is the first work which *generates updated individual CXR images to improve clinical multimodal fusion, thereby tackling the asynchronicity issue*. None of the baseline methods take the asynchronicity between EHR and CXR into account. We will further clarify this in the final version to avoid confusion.\n\n---\n**Weakness 2: Ablation study for a reduced percentage of data.**  \n\nAs suggested, we add a set of ablation experiments to demonstrate the sensitivity with the amount of training data of the proposed model and the main baselines (due to time and resource limits, we are not able to include results of GAN-based method during the author response period). Due to space limit, we include the results in Tables R2-3 in the pdf file attached in the global rebuttal. Results show that DDL-CXR consistently outperforms baseline methods by a large margin, demonstrating its robustness against training data size. \n\n---\n**Q1: Is the generated images created from $\\widehat{\\mathbf{Z}}\\_{t_1}$ using a VAE decoder? Is the dimension of that the same as $\\mathbf{Z}\\_{t_1}$?**\n\nYes. The generated images $\\widehat{\\mathbf{X}}^{CXR}\\_{t_1} \\in  \\mathbb{R}^{3\\times 224 \\times 224}$ is created from $\\widehat{\\mathbf{Z}}\\_{t_1} \\in  \\mathbb{R}^{4\\times 28 \\times 28}$  using the VAE decoder. The dimension of $\\widehat{\\mathbf{Z}}\\_{t_1}$ is same as that of $\\mathbf{Z}\\_{t_1}$. We will further clarify this in the final version.\n\n---\n**Q2.1: Why combining CXR and time series improves prediction?**  \nIn practice, clinicians rely heavily on both EHR and CXR since they are complementary and contextual [1]. Clinical studies demonstrate that when interpreting CXR, radiologists also need EHR data like time series [2], and combining CXR and EHR improves predictive value [3].\n\n**Q2.2: Why acute renal failure and chronic kidney disease benefit from the proposed model combined with CXR?**  \nAlthough the two diseases may not be directly related to CXR, they often lead to complications that have clear radiological manifestations. Pulmonary complications are common for acute renal failure patients, and represent an important cause of death [4]. Cardiovascular complications are common among patients with chronic kidney disease [5]. These complications can be captured by CXR and thus these diseases could also indirectly benefit from CXR data. \n\n**Q2.3: Why CXR with radiology labels helps rather than distracts?**  \nOne key issue in effective multimodal fusion is to align information from different modalities. The radiology labels are used during the LDM stage to encourage better alignment between the temporal disease progression captured in EHR time series and the radiological manifestations. This tailored model makes the radiology labels helps rather than distracts.\n\n[1] Huang, Shih-Cheng, et al. \"Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines.\" *NPJ digital medicine* (2020).  \n[2] Boonn, William W., et al. \"Radiologist use of and perceived need for patient data access.\" *Journal of digital imaging* (2009).  \n[3] Jabbour, Sarah, et al. \"Combining chest X-rays and electronic health record (EHR) data using machine learning to diagnose acute respiratory failure.\" *Journal of the American Medical Informatics Association* (2022).  \n[4] Faubel, Sarah. \"Pulmonary complications after acute kidney injury.\" *Advances in chronic kidney disease* (2008).  \n[5]Di Lullo, Luca, et al. \"Chronic kidney disease and cardiovascular complications.\" *Heart failure reviews* (2015).  \n\n---\n**Q3: Alignment of generated images with the patient's medical records and reliably using the model as a proxy for CXR.**  \nWe include a visual example in Fig. 1, according to the ground-truth medical records retrieved from the MIMIC-IV database, the patient develops bilateral pulmonary consolidation in 34 hours, which is captured by our generated image. Due to page limit, we include more illustrative examples in Figs. 4-9 in Appendix B. Those examples suggest that generated images could align with patients' medical records. However, as discussed in the limitation statement, we believe clinical reliability of using the model as a proxy for real CXR images would require further clinical evaluations and expert validations.  \n \n---\n**Q4: What is the procedure for choosing the hyperparameters of the model? How sensitive are they compared to other methods?**  \nWe use *grid search* to perform hyperparameter tuning and the final used values are reported in Appendix A. As discussed in the limitation statement, the proposed method requires careful hyperparameter tuning. The most sensitive hyperparameters include $\\alpha$ (controls the tolerance of the noisy-conditional generation), $\\beta$ (controls the noise level in the contrastive LDM learning), and the hidden dimension of the Transformer model in the prediction stage.\n\n---\n**Q5: Was only one GPU with 24GB used for training and validating the method? How many hours did it take?**  \nYes, only one RTX 4090-24GB GPU is used. The training of LDM stage and prediction stage take around 12.5 hours and 30 minutes, respectively. Besides, we also trained the autoencoder from scratch, which takes around 24 hours.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uISuYNPI6F", "reviewer": "Reviewer_y1yv", "summary": "The paper aims to address the asynchronicity problem in multimodal clinical data, such as EHR and CXR, for predicting clinical tasks. The asynchronicity in multimodal clinical data comes from the CXR, which is generally taken with a much longer interval compared to the continually collected EHR. The authors propose a method based on the latent diffusion model to generate an up-to-date latent representation of the individualized CXR images.", "questions": "- The relative improvements are not 2.4% and 3.56% over the best baselines as indicated in the description. \n- How is the missing data problem handled in the EHR?", "limitations": "+ Limitations discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This is a technically solid paper. The experiments were thoroughly conducted, a list of data fusion works was compared, and the ablation study was included. The proposed method has shown consistent improvements. \n+ The method of generating an up-to-date CXR is interesting with the latent diffusion model.", "weaknesses": "- My primary concern is the clinical impact of the work. The method shows improvement when the time interval is longer than 36 hours. I believe the best clinical practice would be to take a new CXR in the ICU. The clinical usage of the generated CXR is limited other than making predictions such as mortality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and positive feedback on our paper. We hope the following responses address your concerns.\n\n---\n**Weaknesses: Clinical usage is limited. When the time interval is longer than 36 hours, the best clinical practice would be to take a new CXR.**  \nThank you for the comments. We fully agree that taking another CXR when necessary would be the best. However, sometimes it could be difficult or even infeasible to perform CXR scans for critically ill patients since transportation of patients outside ICUs is potentially harmful. Bedside CXR could also be potentially risky of accidental removal of devices and microbial dissemination, it is also widely recognized to have low accuracy and diagnostic values while posing higher radiation risks; therefore, there is an emerging trend calling for reducing routine CXR in ICUs [1]. We demonstrate that when a new CXR is not available, the proposed method could achieve enhanced prediction via generating an updated latent CXR, which we believe to be clinically impactful and useful.  \n\n[1] Ioos, Vincent, et al. \"An integrated approach for prescribing fewer chest X-rays in the ICU.\" Annals of intensive care 1 (2011): 1-9.\n\n---\n**Q1: The relative improvements are not 2.4% and 3.56% over the best baselines as indicated in the description.**  \nWe would like to clarify that the figures 2.4% and 3.56% refer to the relative improvements of *AUPRC score* for the phenotype classification and mortality prediction tasks. In phenotype classification, the best baseline is DrFuse with AUPRC of 0.459 and the relative improvement is given by (0.470-0.459)/0.459\\*100% = 2.4\\%. For mortality prediction, the best baseline model is GAN-based (0.505) and the relative improvement is (0.523-0.505)/0.505 *100\\% = 3.56\\%. \n\n---\n**Q2: How is the missing data problem handled in the EHR?**  \nWe follow existing publications to handle the missing data [2,3]. The EHR data are first converted to regular time series with a fixed time interval of one hour. The missing values are then imputed by previous values of the same feature. If there is no observations before the missing values, median and mean values are used for imputation for categorical and continuous variables, respectively. We will also test other data imputation methods in the future work.  \n\n[2] Hayat, Nasir, et al. \"MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images.\" ML4H. 2022.  \n[3] Yao, Wenfang, et al. \"DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency.\" AAAI. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xlRv5shit8", "reviewer": "Reviewer_Bxwq", "summary": "Authors introduce a method that overcomes the forward method to generate \"new and up to date latent representations\" of a chest X-ray using a diffusion based model. It addresses an important question in healthcare regarding differing time scales (EHR and CXRs) and how to leverage information from the EHR to inform a new representation that is aligned in longitudinal data. Though model suffers with longer time, results are generally strong and compared against many baselines. Few questions on why pre-trained models weren't used to represent EHR, but overall good paper.", "questions": "1. Why were pre-trained EHR foundation models not used to represent information from the EHR? For example ExBehrt (https://arxiv.org/abs/2303.12364), EHRshot (https://arxiv.org/abs/2307.02028)? Some clarification would be helpful. \n2. Curiosity question: why do you think Uni-EHR performs so well relative to the other multimodal \"fusion\" methods? This was interesting to see in Results Table 1?\n3. Can Authors provide a patient cohort table and description of the patients? Do we know what type of patients are in the MIMIC-CXR dataset? Are there case and controls or are we looking at patients only with specific conditions that necessitated a CXR? Won't this model introduce a bias if we're looking at special patients?", "limitations": "1. \"Furthermore, while various metrics have been employed to assess generation quality, expert evaluation by radiologists would provide a\nmore insightful measure of the model’s efficacy\". I think this is important comment made in the text.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Paper is compared against many baselines \n2. Paper is well structured and method is clearly motivated", "weaknesses": "1. **Not a weakness**: I think this paper introduced cross attention for multiple modalities in the EHR and should be cited in regards to this line in the paper \" . To further integrate the disease course embedded in the EHR time series, we use the cross-attention mechanism to capture the interaction between the two modalities\" in Section 3.2: (https://arxiv.org/abs/2402.00160v2)\n2. **Not a weakness**: Table 3 suddenly calls it PRAUC instead of AUPRC as mentioned in Section 4.1. I advise authors to keep it consistent.\n3. Model performance seems marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and positive feedback on our paper. As suggested, we will include relevant discussions and citations on cross attention for EHR and change all \"PRAUC\" to \"AUPRC\" for consistency in the final version. We hope the following responses address your concerns.\n\n---\n**Q1: Why not using pre-trained EHR foundation models.**  \nUsing pre-trained foundation models to represent information from EHR is certainly interesting and could be beneficial. Existing foundation models mostly take *discrete diagnosis, lab test and procedure codes* (e.g., ICD-9 codes) as input. In this work, we include continuous variables in the form of time series in the EHR as they are considered more predictive in ICU settings. Handling such continuous variables is challenging for foundation models and we will investigate the feasibility in future work.\n\n---\n**Q2: Why does Uni-EHR perform so well relative to multimodal \"fusion\" methods?**  \nPrimarily, this is related to the predictive tasks we considered in this work. In the phenotype classification task, classifying diseases like *diabetes*, *acute renal failure* and *shock* relies more on EHR data while the information from CXR is much less predictive. This observation is consistent with those reported in MedFuse [1] and DrFuse [2].\n\nHaving said so, classifying diseases like *CHF* and *conduction disorder* requires strategic fusion of both EHR and CXR data. Results in Table 3 show that multimodal methods, especially our proposed one, achieve significant improvements over the Uni-EHR model.\n\nSimilarly, for the mortality prediction, the EHR data is more predictive as time series data is routinely recorded by bedside monitoring devices and caregivers upon patient admission to the ICU. Thus it provides more comprehensive description towards patient risk. CXR, on the other hand, provides a snapshot of the radiological manifestations of patients.\n\n[1] Hayat, Nasir, et al. \"MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images.\" ML4H. 2022.  \n[2] Yao, Wenfang, et al. \"DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency.\" AAAI. 2024.\n\n---\n**Q3: Description of patients and potential selection bias.**  \nWe thank the reviewer for the constructive suggestion. We follow the `matched subset` setting in DrFuse to include patients, where only those with at least one CXR available are included.\n\nThe statistical description of the clinical variables for our data cohort can be found in the table below. We also include the statistics of patients with *length-of-stay (LoS) ≥ 48 hours* from the entire MIMIC-IV database regardless of presence of CXR image. A more readable version can be found in Tables R4 and R5 in the pdf file attached in the global rebuttal. Here we summarize a few key points:\n- The statistics of the clinical variables are close to each other, suggesting that patients in our data cohort generally have similar distribution as that in the entire database. \n- The overall disease phenotype prevelance are similar.\n- For a few thorax-related diseases, our data cohort has slightly higher prevelence, suggesting that potential selection bias exists.\n\nHaving said so, fair comparision is ensured by using the same data cohort for all models. We will explore and address the potential selection bias arising from the inclusion criteria in future work.\n\n|Type|Clinical Variable| DDL-CXR cohort | MIMIC-IV database |\n|-----|-----|-----| -----|\n| static| Sample size| 5142 |26330| \n| static| Age| 63.9±16.6|64.3±16.4| \n| static| Gender (Men, %)| 54.98 |55.99| \n| continuous| Diastolic blood pressure| 62.2±10.1|62.8±10.5| \n| continuous| Fraction inspired oxygen| 0.4±0.2|0.4±0.2| \n| continuous| Glucose| 141.1±41.7|139±39.6| \n| continuous| Heart Rate| 87.3±15.4|85.6±15| \n| continuous| Height| 170±1.1|170±1.5| \n| continuous| Mean blood pressure| 77.1±10|78.3±10.4| \n| continuous| Oxygen saturation| 96.7±2|96.5±2| \n| continuous| Respiratory rate| 19.8±3.7|19.5±3.7| \n| continuous| Systolic blood pressure| 118.3±15.2|118.6±15.6| \n| continuous| Temperature| 37±0.4|36.9±0.4| \n| continuous| Weight| 80.8±20|81.5±19.5| \n| continuous| pH| 7.2±0.3|7.2±0.3| \n| categorical| Capillary refill rate| Normal | Normal | \n| categorical| Glascow coma scale eye opening| Spontaneously|Spontaneously| \n| categorical| Glascow coma scale motor response| Obeys Commands|Obeys Commands| \n| categorical| Glascow coma scale total| 15|15| \n| categorical| Glascow coma scale verbal response| Oriented|Oriented| \n| phenotype| Acute renal failure| 0.38| 0.34| \n| phenotype| Acute cerebrovascular disease| 0.09| 0.07| \n| phenotype| Acute myocardial infarction| 0.09| 0.09| \n| phenotype| Cardiac dysrhythmias | 0.40| 0.38| \n| phenotype| Chronic kidney disease | 0.24| 0.23| \n| phenotype| COPD and bronchiectasis | 0.17| 0.16| \n| phenotype| Surgical complications | 0.24| 0.25| \n| phenotype| Conduction disorders| 0.11| 0.12| \n| phenotype| CHF; nonhypertensive| 0.33| 0.30| \n| phenotype| CAD | 0.31| 0.33| \n| phenotype| DM with complications| 0.13| 0.12| \n| phenotype| DM without complication| 0.21| 0.18| \n| phenotype| Disorders of lipid metabolism| 0.40| 0.41| \n| phenotype| Essential hypertension| 0.44| 0.42| \n| phenotype| Fluid and electrolyte disorders| 0.52| 0.45| \n| phenotype| Gastrointestinal hemorrhage| 0.07| 0.07| \n| phenotype| Secondary hypertension| 0.22| 0.24| \n| phenotype| Other liver diseases| 0.17| 0.15| \n| phenotype| Other lower respiratory disease| 0.14| 0.12| \n| phenotype| Other upper respiratory disease| 0.07| 0.07| \n| phenotype| Pleurisy; pneumothorax | 0.11| 0.09| \n| phenotype| Pneumonia| 0.22| 0.18| \n| phenotype| Respiratory failure| 0.34| 0.25| \n| phenotype| Septicemia (except in labor)| 0.27| 0.21| \n| phenotype| Shock| 0.23| 0.18| \n| | In-hospital mortality| 0.15| 0.12|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their valuable comments and constructive suggestions. We are glad that reviewers appreciated the new method for clinical multimodal fusion via generative models to address asynchronicity. We summarize the **strengths** from the reviewers as follows: 

1. Interesting question and convincing motivation identified in ICU settings.
2. Reasonable and technically sound approach.
3. Thorough experiments with promising results compared with multiple baselines.
4. Well structured paper with clear writing. 

We conduct extra ablation experiments and statistical analysis and report the results in the attached pdf file. We hope they can address concerns from reviewers. 

- **Table R1 Ablation study of single CXR modality for performance evaluation.** We conduct this ablation study to evaluate the quality of generated CXR in terms of prediction performance, as suggested by `Reviewer 7ywn`. It shows that the generated latent CXR are beneficial in prediction tasks. 

- **Tables R2 and R3: Predictive performance with varying training sizes.** We train our model with reduced amount of training data (75% and 50%), as suggested by `Reviewer 1LZb`. The results shows that our model DDL-CXR consistently outperforms baselines and is robust against reduced training data.

- **Tables R4 and R5: statistics of data cohort.** As suggested by `Reviewer Bxwq`, we conduct statistical analysis of the data we used for training DDL-CXR and those in MIMIC-IV database. The results show that the data statistics of input variable are similar while thorax-related disease prevalence of data for DDL-CXR is slightly higher. Having said so, fair comparison is ensured since all experiments are conducted using the same data cohort. 

Again, we thank all reviewers for their time and effort.

Meta-review: Congratulations on the acceptance of your paper. Secondly, I would like to thank both the reviewers and the authors who engaged in a very fruitful and comprehensive discussion over the discussion period. The authors provided clear responses -- within reason -- to the majority of the queries raised by the reviewers. The additional results presented and the clarifications provided have strengthened the outlook of the paper and led to reviewers revising their scores. In particular highlighting the benefit of using generated CXR images, which was not evidenced in the very beginning, provided the evidence base to support the usefulness of the proposed method.

I would like to invite the authors to add all the additional experiments and clarifications to the camera-ready version.

Again many congratulations!

AC
Decision: Accept (poster)