Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment
Authors: Chen Liu
Wenfang Yao
Kejing Yin
William K. Cheung
Jing Qin
Venue: NeurIPS 2025 spotlight ()
Keywords: Longitudinal Multimodal Fusion; Disease Progression Modeling; Clinical Prediction
Primary area: applications

Abstract: Longitudinal multimodal data, including electronic health records (EHR) and sequential chest X-rays (CXRs), is critical for modeling disease progression, yet remains underutilized due to two key challenges: (1) redundancy in consecutive CXR sequences, where static anatomical regions dominate over clinically-meaningful dynamics, and (2) temporal misalignment between sparse, irregular imaging and continuous EHR data. We introduce $\texttt{DiPro}$, a novel framework that addresses these challenges through region-aware disentanglement and multi-timescale alignment. First, we disentangle static (anatomy) and dynamic (pathology progression) features in sequential CXRs, prioritizing disease-relevant changes. Second, we hierarchically align these static and dynamic CXR features with asynchronous EHR data via local (pairwise interval-level) and global (full-sequence) synchronization to model coherent progression pathways. Extensive experiments on the MIMIC dataset demonstrate that $\texttt{DiPro}$ could effectively extract temporal clinical dynamics and achieve state-of-the-art performance on both disease progression identification and general ICU prediction tasks.

Multimodal Disease Progression Modeling via Spatio- temporal Disentanglement and Multiscale Alignment
Chen Liu', Wenfang Yao', Kejing Yin'; William K. Cheung', Jing Qin' 'School of Nursing, The Hong Kong Polytechnic University Department of Computer Science, Hong Kong Baptist University
Abstract
Longitudinal multimodal data, including electronic health records (EHR) and sequential chest X-rays (CXRs), critical for modeling disease progression, yet re- mains underutilized due to two key challenges: (1) redundancy in consecutive CXR sequences, where static anatomical regions dominate over clinically-meaning dynamies, and (2) temporal misalignment between sparse, irregular imaging and continuous EHR data. We introduce DiPro, a novel framework that addresses align-
these challenges through region- disentanglement and ment. First, we disentangle static (anatomy) and dynamic (pathology progression) features in sequential CXRs, prioritizing disease-relevan changes. Second, hierarchically align these static and dynamic CXR features with asynchronous EHR data via local (pairwise interval-level) and global (full-sequence) synchro- nization to model coherent progression pathways. Extensive experiments on the MIMIC dataset demonstrate that DiPro could effectively extract temporal clinical dynamics and achieve state-of-the-as performance on both disease progression identification and general ICU prediction tasks.
Introduction
Accurately modeling disease progression, i.e., the temporal evolution of a disease, is critical for personalized clinical care decision- makinn [1-3]. By capturing progression dynamics, predictive models enable early identification of deterioration, improve precise risk stratification, and inform individualized treatment planning [2, 4-6]. ICU settings, for instance, tracking sepsis progression through multimodal data (e.g., vitals, labs, and organ functions) is essential for identifying early deterioration and initiating lifesaving treatments [7. 8].
Modern healthcare increasingly relies on longitudinal clinical data to track disease progression. Sequential chest X-rays (CXRs) capture valuable visual evidence of anatomical and pathological changes over time, while electronic health records (EHRs) provide continuous physiological metrics and treatment responses [9, 10]. The complementary nature these modalities offers unique opportunity: a fusion of imaging and clinical time-series data could enable more holistic modeling of disease trajectories [11, 12]. Despite a growing number of studies that explored disease progres- sion modeling and multimodal fusion using longitudinal clinical data, two key challenges remain insufficiently addressed:
Redundancy in clinical image sequences. Static anatomical features (e.g., chronic cardiac largement or stable skeletal deformities) dominate sequential CXR scans, often obscuring subtle but clinically important pathological changes (e.g., new infiltrates evolving edema). Without explicit disentanglement, the signals of disease progression become diluted, reducing the utility
"Correpondencee to: Kejing Yin (cckyyin@comp.hhhu.cd.....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of sequential imaging analysis. Recent cxRe progression models (e.g.. [13, 14]) primarily focus on extracting symptom-related features and anatomical-disears co-occurrences, treating all imaging features uniformly. However, distinguishing long-term anatomical baselines from evolving pathological changes is the key to improving the precision of disease progression modeling, yet largely overlooked [15, 16].
Temporal misalignments across modalities. While EHRs provide continuous, high- high-frreuenccy measurements (c.g., hourly vitals lab tests), CXRs offer only sparse, irregular snapshots, creating an intrinsic misalignment in timescales. This discrepancy in temporal granularity complicates the alignment of cross- cross-modal trends and may obscure short-term clinical deteriorations between imaging intervals. Existing multimodal fusion approaches prioritized the latest CXR for simplicity [17, 18], neglecting the temporal CXR context, while longitudinal fusion methods [19, 20] rely on rigid imputation or fixed time embeddings, lacking adaptive mechanisms to synchronize fine-grained cross- modal patterns. Thus, strategically integrating these progression dynamics to model cross-modal clinical trajectories remains underexplored.
To address the aforementioned challenges, we propose Disease Progression-Awart Clinical Pre- diction, DiPro?, novel framework that disentangles time-variant and time-invariant information from longitudinal CXRs and integrates these representations with EHR data across multiple temporal granularities. Specifically, our framework contains three modules: (1) Spatiotemporal Disentangle- ment: We disentangle dynamic pathological changes from static anatomical structures in different spatial regions across consecutive CXRs. This separation helps reduce redundancy and prevents interference between features with different clinical semanties, allowing the model focus on meaningful progression signals for more effective temporal representation learning. (2) Progression- Aware Enhancement: To improve the model's sensitivity progression direction, reverse the order of CXR pairs and train the model to produce reversed dynamic features while keeping static features consistent. This strategy further separates the two types of features and emphasizes their distinct clinical semantics. (3) Multimodal Fusion via Multiscale Alignment: To fully synergize the multimodal features, we introduce a multi guati- fusion module that achieves alignment in multiple scales: the local fusion aligns consecutive CXR pairs with EHR data, while the global fusion aligns the semantics of the entire CXR and EHR sequences. This bridges fine ane-ggained EHR-CXL interactions with global disease progression patterns.
Our contributions are summarized follows:
We present a framework to disentangle dynamic and static information from longitudinal CXRs, with dedicated architectural constraints to capture representations in line with disease progressions.
We propose multiscale multimodal fusion approach that facilitates multi-grained interactions between temporally misaligned CXR dynamics and EHR time-series data.
Extensive experiments demonstrate that DiPro achieves state-of-the- art performance on both disease progression modeling and ICU-related prediction tasks. Quantitative evaluation shows that the model aligns well with existing clinical knowledge.
2 The DiPro Approach
To address the challenges of redundancy in longitudinal CXRs and temporal misalignment with EHR data, we propose DiPro, to systematically disentangle static and dynamic features from CXRs and align multimodal data across hierarchical timescales. The approach is motivated by two key observations: (1) disease progression in CXR sequences unfolds via region-localized pathological changes [21, 22], and (2) EHR and imaging data exhibit complementary dynamics at different temporal granularities. DiPro integrates three cohesive modules: (1) Spatiotemporal Disentanglement (STD) to isolate mathology-sensinggg
features from sequential CXRs, (2) Progression-
Aware Enhancement (PAE) to enforce temporal consistency in learned dynamics, and (3) Multiscale Multimodal Fusion (MMF) to bridge local EHR-CH CXR interactions with global progression trends Figure illustrate the DiPro framework. We next detail each module.
"The code is available https: :ttps://gitthbb www/Choommaaeeooo svg/DiPro..

---

Spatiolemporal Disemtanglement (STD)
Progression-Aue Enhancement (PA
Multtsrale Multimodal
C Clinical Tasks
Fusion (MMF)
Disus Prgpasiar
Hanataatin
D
Improved
Worsened M
Docal
fa
RE
Aara
fx
Fusion
iiaale Leggu
pall
TP
Local
D
D
S
D
poll
Dalobui
R
fa
STD
X
galobal
X
f
28
Global Fusion
S7-1
Reverse
X
Salobal
R
R
R
RG
Mansilyy
Figure Overview of the DiPro framework. The model comprises three main modules: (1) Spatiotemporal Disentanglement (STD) separates dynamic pathological features (DF) from static anatomical structures (S;) in region-level chest X-rays across time; (2) Progression-ANe
Enhance- ment (PAE) strengthens the model's understanding of progression direction by reversing CXR pair order and enforcing the reversed dynamic features D to predict the reversed progression, while maintaining consistency in static features; (3) Multiscale Multimodal Fusion (MMF) integrates CXR features with temporally misaligned EHR data via local (interval-level) and global (sequence-level) fusion, enabling accurate predictions across multiple clinical tasks, including disease progression identification, length-of-stte classification, and in n-hosppitll mortality prediction.
2.1 Notations and Preliminaries
For each patient, let XXXR (XCXR)T li=l denote a set T CXR images during an ICU stay, where each X & xHxWxx is the CXR image taken at time 4. Each CXR image contains R anatomical regions (R, ) Each consecutive image pair (X ti+1 is associated with label set YCXR = {y," r.k ) rr,== disease l: in the r-th region has worsened, remained stable (no change), or improved. The whole EHR time serie recorded with M timestamps is EER [Xtg X x.y..... 2t with x, RN being the N-dim variables recorded at time l. We use XEHR to denote all available EHR time series. Patient's demographic attributes are denoted XDEM RR where
R.K where y, E 0, 1} indicates whether the progression status of
the number of attributes. Our objective is to learn a mapping fe (XCXR, EHR DDEM + y, that maps the integrated CXR, EHR time series, and demographic information the final clinical outcome prediction y.
Spatiotemporal Disentanglement (STD)
We propose novel method disentangle region- based time-varian (dynamic) and time-invariant (static) information from consecutive CXR image pairs. The disentanglement is motivated by their distinct clinical roles: dynamic features reflect discase progression, while static features capture patient-specific anatomical structures. Inspired by prior works demonstrating the effectiveness of feature disentanglement in representation learning [23, 24], method targets more efficient and structured latent representations.
Feature extraction. For each anatomical region R we first use shared pretrained ResNet- 50 [25] to decode it and obtain the region feature F7, E IRd Then we concatenate the two consecutive feature vectors and then pass them through two separate projection heads, fa and fa for static and dynamic information extraction, respectively.:
S = f. ( FE...))
= fa Fi.,,.
where || denotes channel-wise concatenation. Here, Sy and D represent the static and dynamic features, respectively, for region at time step pair (t,, 4441).
Orthogonal disentanglement loss. encourage effective disentanglement between static and dynamic representations, apply an orthogonal constraint [26]. Specifically, we minimize the

---

squared cosine similarity between static and dynamic features within each region and time pair:
T
R
Corh (
Σ
(sim (ST.D))
(2)
where sim(:.-) denotes the cosine similarity. This term ensures that the dynamic and static features are unrelated as possible in the latent space, thereby promoting better disentanglement between time-invariant and time-varying patterns.
Temporal consistency for static features. Anatomical structures are expected to remain stable over time in sequential CXR images [27, 28). To enforce temporal consistency of static features, introduce mean squared error (MSE) loss that encourages static features from consecutive time pairs remain close:
T-2 ΣΣΣ
R
Limmp
s - 1
N
where N (T 2) x R is the number of consecutive static feature pairs considered. This constraint ensures that the learned static features remain stable and coherent over time.
2.3 Progression-Aware Enhancement (PAE)
Intuitively, reversing the order of CXR pair should invert the progression direction while preserving static anatomical information. Hence, dynamic features should reflect this reversal, providing a more robust and interpretable representation of temporal change. To make dynamic features more sensitive to the direction of disease progression, we introduce a progression-aware enhancement.
By reversing the input order of the region feature pair (F Fi ). we obtain the dynamic and static features as:
D fa(FF FE.)) S fo (F [F].))
We then feed both the original and reversed dynamic features into K disease-specific progression classification heads { where each head fa corresponds to a disease and predicts its progression status y/*** for the given region r. Let Di** be the predicted label from the original direction, and D. be the prediction from the reversed input, i.e., D. f& (D,,, yr = (D,)) expect that reversing the input order should generate contrary prediction. Thus, we convert the label into -y;" indicate reversed progression direction as the ground truth label.
Training objective. For dynamic features, we supervise predictions using cross-entropy (CE) loss for both original and reversed progression prediction; For static features, we leverage MSE to encourage their consistency over the reversal.
R
K
R CE * CE(T. -,),))));))))))) + Astatic Σ
CPAE
S
(5)
where Astatic is hyperparameter. This objective encourages the model to encode progression- aware dynamic information and time-invariant anatomical information, improving the reliability of progression modeling across time.
2.4 Multiscale Multimodal Fusion (MMF)
To effectively integrate temporally disaligned patient data, we propose a multiscale fusion framework that combines visual and temporal features from longitudinal chest X-rays (CXRs) and electronic health records (EHRs). EHR signals are composed of dynamic time-series measurements and static demographics, while CXRs encode both time-varying imaging biomarkers and static anatomical traits. Our fusion proceeds in three stages: (1) local CXR-EHR fusion within each interval, (2) global hierarchical fusion, and (3) static feature integration and prediction.

---

Local CXR-EHR fusion within temporal intervals. We first encode the whole EHR time series X using a global Transformer encoder Φ [29] get the global representation of EHR: Eglobal ( Given time interval [4, [++1] from consecutive image pair (Xt, x obtain EHR representations focused on the interval, we first define the time embedding for each EHR timestamp i, relative to the CXR time interval as: Tí; = STE. (14 - 44, 4+11 - Lj, a(l1, (111-1441 (4)((4+) 14))), where a(-) is the sigmoid function approx- imate the indicator of whether L E [4,,441], These time embeddings are then stacked as T, [TR......, 1- we then apply cross-attention [29] between the time embeddings and global EHR features with learnable parameters WQ, W and Wv using center-focused attention mask:
QK
El = softmax
+
AttnMaskk
V, = WoT,, K WR Eglohal V
Wy WV Eslobal
vd
The attention mask is defined as:
(-110 1j 4 -00,
if E [ta, 4+11
AttnMaskij =
otherwise.
We then fuse dynamic CXR features Diocal {D 1 and local EHR representations El via a cross- cross-attention layer. The keys and values are constructed by concatenating modality-speciffc projections of both inputs.
D
(oyyerNorm(Crossstttnn/
[E/
[D
(8)
Global hierarchical fusion We collect all locally fused features Delnhal (Disse 1 intervals. We refine the global EHR representation by attending over Dilobul
T-1 11=1
across CXR
Halobal LayerNorm CrossAttn (EEeoll
(Ealobal Delobul))
We include an additional self-attention layer [29] further enhance global interactions between the two modalities. The resulting enriched sequence then serves as the query in the final cross-attention mechanism with the static features for prediction.
Final static fusion and prediction. We first embed demographic information DEM via an MLP, then concatenate it with static CXR features Salobal
(S
H statiic
|Sslobal
MLM(X
(10)
fusion module 4, composed of cross-attention layer followed by prediction heads, integrates dynamic and static features to generate predictions:
y $ (Da
Hllobul
Haltic)
(11)
The final training loss combines cross cross-entropy with all auxiliary objectives:
C= CE(ý.y) + Aouth Soth + Aump + APAE
(12)
where Apred,, Aurth Aemp, and λ hyperparameters. Details of the model architecture, hyperpa- rameters, and training procedure are provided in Appendix A.2.1.
Experiments
3.1 Experiment Setting
Datasets. We evaluated DiPro on the large-scale, public dataset, MIMIC [30], which contains de-identified health data of intensive care unit (ICU) admissions. Our study leveraged three derived datasets from the MIMIC ecosystem: (I) MIMIC-IV [30] provides electronic health records (EHR) including demographic information and time-series physiological measurements per ICU stay; (2) MIMIC-CXR [31] contains sequential chest radiographs during ICU hospitalizations; and (3) Chest ImaGenome [32] augments imaging with fine-gme fine-grained annotations: bounding boxes for anatomical regions and localized change labels (improved, worsened, or no change) between consecutive CXRs. To facilitate longitudinal analysis, we selected ICU stays with 2 2 CXRs to track disease progression. From MIMIC-IV, we extracted EHR data consisting of 7 demographic variables and 38 physiological time-series variables, including vital signs and laboratory results.
5

---

Clinical tasks and evaluation metries. We evaluate the performance of DiPro on two types of clinical tasks to demonstrate the advantages of our multimodal framework:
(T)Disease progression identification. Given two consecutive CXRs, the task is to predict the disease progression status (improved, worsened, no change) for seven common thoracic conditions: atelectasis, enlarged cardiac silhouette, consolidation, pulmonary edema, lung opacity, pleural effusion, and pneumonia. Following Karwande et al. [14], we derive the progression label of each disease for a CXR pair from the progression labels of annotated regions in the Chest ImaGenome dataset. Notably, our work is the first to integrate EHR data for this task. Specifically, we extract the EHR data recorded within the time interval of two CXRs and integrate them for a richer context for progression. We report macro- macro-precision precision, macro-recall, macro-Fi score, AUPRC and AUROC following prior works [13, 14. 33].
(2)Geerall ICU prediction. We consider two clinically vital tasks: In-hospital mortality prediction and ICU length of stay prediction. Both tasks focus on forecasting patient outcomes leveraging multimodal data: EHR time series and sequential CXRs collected during the first 48 hours after ICU admission. The In-hospital mortality prediction task is a binary classification problem: it predicts patient mortality prior to hospital discharge. We evaluate performance using AUROC and AUPRC, following [19, 17J. Length of stay prediction task aims to estimate patient ICU stay duration. We frame this as multi- teati---cssss classification problem by discretizing stay duration into four intervals: [2. 3), [3, 4), [4, 6), and 2 6 days. The counting starts from 2 days as we are using ICU stays longer than 48 hours. Following [11, 34, 35], model performance is evaluated using Cohen's 's kappa and accuracy.
All experiments are conducted with three random seeds, with results reported as the mean + standard deviation across independent runs. Details on cohort selection, label prevalence, data statistics and data processing procedures for each task are provided in Appendix A.I.
Baselines.
We compare DiPro with the following three types of baselines:
(1) Sequential CXR disease progression specialists (unimodal): CheXRelNer [14] combines local and global visual features with anatomical dependencies to model longitudinal disease changes. CheXRelFormer [33] adopts hierarchical Siamese Transformer to capture multi-level feature discrep- ancies across CXR images. SDPL [13] learns symptom-aware embeddings to extract and compare condition-specific features from two radiographs.
(2) Longitudinal multimodal specialists: UTDE [19] models asynchronous longitudinal data via gated attention based imputation framework. UMSE [20] uses riplet-structung set embeddings and modality aware attention mechanism to handle missing data and fuse multiple modalities.
(3) Clinical multimodal fusion specialists: MedFuse [17] introduces an ASTM--based module for both uni-modal and multimodal input. DrFuse [18] disentangles modality-shared and modality- specific features, and utilizes disease- wise attention for effective fusion. Both models are designed to take the last available CXR for modality input, we extend them to the setting of multiple CXRs with minimal architectural modification. Details of all the baseline models are provided in Appendix A.2.2.
3.2 Prediction Performance
DiPro excels in modeling disease progression in sequential CXRs. Table reports the mean performance across seven disease progression identification tasks (per-disease results in Table 11). Compared to CXR-based progression models using only unimodal sequential CXRs, DiPro achieves relative improvements of 15.3% in Fl and 12.2% in AUPRC over the state-of-the-ar SDPL [13]. This suggests that explicitly disentangling disease dynamics from static anatomical structures across CXR pairs reduces redundancy and enables more effective progression modeling. Compared to Che Net [14), which models region-disease co-occurrence via graphs, DiPro enhances the disentangled regional progression dynamics using a tailored PAE module, offering a more targeted mechanism to capture disease-region progression. A broader baseline comparison with large vision-languaae models is presented in Table 17.
DiPro excels in longitudinal multimodal fusion. As shown the Multimodal Methods block of Table 1, adding EHR to unimodal DiPro improves performance (relative increase of 2.9% in F1 and 2.1% in AUPRC). This confirms DiPro's ability to effectively leverage complementary EHR features for disease progression prediction. Furthermore, DiPro outperforms all basclines, whether

---

Table Performance Comparison on Disease Progression Identification Tasks. This table reports the macro- averr-e performance (# standard deviation) of various unimodal and multimodal methods across seven disease progression identification tasks. DiPro achieves the best results in both unimodal and multimodal settings, indicating its effectiveness in modeling disease progression from sequential CXRs and its strength in longitudinal multimodal fusion. Detailed results are provided in Table 10. Per-disease results are provided in Table 11. (Numbers in bold indicate the best performance in each
column, and those underlined represent the best- best-performing baseline.)
Method
Precision
Recall
F1
AUPRC
AUROC
Unimodal Methods (CXR)
CheXRReNNet [14]
0.395-000ISS
0.392210.000
0.389110.110
0.394100000
0.37410.011
CheXRelFormer [33]
0.389=0.044
0.379=+0033
0.354++0.032
0.372=00003
0.551
SDPL [13]
0.408
0.406=+0.020 0.452++.0011
0.393++0.010
0.417=0.032
0.609-0.003
DiPro (ours)
1.05500004
0.453-10.009
0.003
0.651
Multimodal Methods
UTDE:
0.48T-000777
0.4627-0.002
01:449--0:005
0.312200011
0.659F000T
UMSE [20]
0.353=0.011
0.361++0.009
0.352++0.013
0.066
0.544+00.004
MedFuse [17]
0.423-0.049
0.413++0.045
0.409:+-00022
0.422+0.040
0.530+00.000
DrFuse [18]
0.442-0.009
0.461
0.429:10.010
0.408-10.003
0.62810.002
DiPro (ours)
0.084
0.471
0.466110.018
0.4781100118
0.664
Table 2: Performance Comparison General ICU Prediction Tasks. This table presents the average performance (+= standard deviation) on two general ICU prediction tasks: mortality (AUPRC, AUROC) and length of stay (Kappa, ACC). Results are reported for both "Last" and "Long." CXR settings. The "Long." setting incorporates both longitudinal CXR and EHR data, whereas the "Last" setting uses only the most recent CXR together with EHR data. DiPro demonstrates superior performance across both settings, highlighting its effectiveness in longitudinal multimodal fusion. Note that in the "no-CXR" setting (second-to -o-last row), DiPro effectively reduces Transformer- based [29] EHR encoder. More results are provided in Table 12.
CXR Used
Mortality
Length of Stay
Method
Last
Long.
AUPRC
AUROC
Kappa
ACC
UTDE [19]
0.717
0.887110004
0.160:10.16
0.381
0.710=0.019
0.88710.012
0.195$0.031
0.400::.002
UMSE [20]
0.722-1.039
0.896:10.012
0,21710.013
0.419:00010
0.712=0.028
0.89110.011
0,20410.019
0.410:0.013
0.686+00018
0.869++0011
0,213+0.012
0.413++0.004
MedFuse [17]
0.71610.018
0,88110,005
0,21011.039
0.412100027
0.70910.012
0.86510.014
0.114110.048
0.338110.041
DrFuse [18]
0.68410.008
0.85410.017
0.142.10.014
0.36011.001
0.712+00.009
0.885+00003
0.226
0.427++0.114
DiPro (Ours)
0.702+1.003
0.897=00002
0.248+0.008
0.440-E0.007
using the last available CXR or longitudinal CXR in multimodal settings, or using unimodal EHR, across all tasks, including disease progression identification (Table 1) and general ICU prediction (Table 2). Specifically. it achieves relative gains of 3.8% in F1 score (disease progression), 2.8% in AUPRC (mortality prediction), and 3.0% in accuracy (length of stay) compared the respective best- performing baselines. Notably, DiPro consistently outperforms longitudinal multimodal specialists like UTDE [19] and UMSE [20]. These results highlight the superiority of DiPro's 's multiscale fusion strategy in addressing temporal misalignment compared to unimodal imputation or unified time-embedding methods.
DiPro alleviates redundancy and misalignment in sequential CXRs. Table 2 presents com- parison multimodal models that integrate EHR data with either the most recent CXR (Last) or longitudinal CXRs (Long.). For UMSE [20), using longitudinal CXRs do not consistently improve general ICU prediction performance compared to using only the latest image, suggesting that unified time embeddings and bottleneck attention inadequately address temporal redundancy in sequential
7

---

I.. Jung
Normalized Attention Seure
Mediastinum
lung
0.4
Moetslity
Langth Say
Cardiac
L. lower lung
silhoucfie
A1
0.2
6:15
R. costoplarenic
angle
R. lower lung zono
0.1
0.00
0.12 U.L.S
8.09
0.63
611
0.
0.08
8.60
L. costophrenic angle
L. hilar structures
0.0
lower lung aureee
hing
R. lowver long aedte L. huler ansetores hike sbuctutes
comupluenic angle
R. cosoplaentc angle
Candiac silfouutte
Mediantion
R. bilar struciures
Attlactais
Palmonary
Lug opacily
Anstomical Regions
Unlorged candinc silbouette
(a) Disease Progression Prediction
(b) Mortality and Length-of-stay Prediction
Figure 2: Averaged attention weights of CXR regions in different downstream tasks. The radial axis in (a) is log-scaled to enhance distribution visibility. Mean attention weights across CXR regions reveal DiPro's clinical alignment: (a) overlapping distributions for pneumonia, lung opacity, and pleural effusion reflect shared pathologies, while (b) ICU tasks show divergent patterns: higher weights for right-sided regions in mortality (linked to higher risk) versus diffuse attention in length-of-stay (reflecting multifactorial ICU conditions).
imaging data. Similarly, DrFuse [18] experiences a slight performance drop mortality prediction when naively concatenating sequential CXR features, highlighting the limitations of direct aggrega- tion without temporal alignment. Alternatively, DiPro explicitly disentangles disease progression dynamics from time-invariant CXR features, reducing redundancy. It aligns multimodal trajectories by fusing progression-awatt aware features with EHR data at both interval and sequence levels, enabling more efficient use of longitudinal data. This approach yields superior performance across tasks.
DiPro echoes with clinical knowledge. To analyze the anatomical basis of DiPro's 's decision- making, visualize the normalized attention weight of CXR regions of each disease in the disease progression task. As shown in Figure 2a, DiPro demonstrates high attention scores on the cardiac silhouette when identifying cardiomegaly, which aligns with the radiographic diagnostic criteria where cardiothoracic ratio (CTR) > 0.5 on posteroanterior chest radiographs indicates cardiac enlargement [36, 37). Similarly, the model highlights hilar structures for pulmonary edema detection, corroborating the pathophysiologial mechanism that pulmonary venous congestion in left ventricular failure manifests perihilar vascular redistribution and interstitial edema [38]. Notably, the radar plot reveals overlapping attention weight distributions for pneumonia, lung opacity, and pleural effusion, suggesting shared radiographic features due to common pathways [39-41]. This suggests that DiPro captures clinically meaningful correlations in radiographic patterns.
In Figure 2b, further analyze DiPro's normalized regional attention weights for two ICU pre- diction tasks. For in-hospital mortality prediction, DiPro assigns notably high attention to the right costophrenic angle, a region clinically associated with pleural effusions and lower lobe pathologies, both of which are common in critically ill patients and have been linked to increased mortality risk [42, 43). Furthermore, the model consistently prioritizes right-sided anatomical structures over left ones (e.g., right lung: 0.13 vs. left: 0.06; right hilar region: 0.10 vs. left: 0.03). This pattern aligns with the predominance of right-sided pulmonary complications (c.g., aspiration pneumonia and plcural effusion)) linked to increased risk of mortality [44-46]. For length-of-stay prediction, however, the model exhibits more distributed attention pattern across multiple thoracic regions, including bilateral lungs, hilar structures, and mediastinum. This scattered pattern suggests that predicting the length-of-stay requires a broader view of radiographic features, which is consistent with the understanding that hospital stay duration aggregates diverse and multifactorial conditions, such as pulmonary congestion, atelectasis, and cardiomegaly [47, 48).
3.3 Ablation Study
To better each component's contribution in DiPro, we ablate key modules (results in Table 3). The variant "A1" replaces MMF with a simple fusion strategy that concatenates CXR and EHR features, followed by multi-head self-attention layer. The variant "A2" removes the PAE module. The
8

---

Table 3: Results of the ablation study. This table presents the results of ablating major modules to assess their contribution to overall performance. Variants "A1"-"A4" correspond to variants of DiPro with progressively removed modules, while "DiPro-" denotes the variant using automated bounding boxes generated by the RGRG model [49] instead of Chest ImaGenome annotations.
Components
Disease Progression
Mortality
Length of stay
ID
STD
PAE
MMF
F1
AUPRC
ACC
DiPro DiPro-
0.466
0.742=1.003
0.440
0.407
0.736+0.021
0.430=50.006
AI
V
X
0.460
0.724++0011
0.416++0.027
A2
V
X
Y
0.403
0.730=100029
0.432:10.018
A3
V
X
X
0.439 0.694
0.40411.014
A4
X
X
X
0.362
0.721
0.425+10.031
variant "A3" removes both MMF and PAE, leaving only a basic aggregation using self-attention. The "A4" variant removes the STD module and disables MMF and PAF, reducing DiPro to plain enccde-conncatnatteattennee
baseline.
As shown in Table 3, removing any component of DiPro results in performance drop, underscoring the necessity of each module. Notably, contributions vary across tasks. Different components contribute variably across tasks. For disease progression identification, the STD module is critical, yielding relatively Fl improvements ("A3" vs. "A4"" by 21.3% through disentangling progression dynamics from static anatomical features. The PAE module further enhances performance (7.6% relative F1 gain, DiPro vs. "A2"), highlighting the benefit of progression-awe aware feature enhancement. However, the MMF module contributes less the same task. possibly due to the constraint of using only EHRs linked to CXR pairs, restricting the multiscale fusion capability. Conversely, for general ICU prediction, MMF notably improves performance, highlighting the value of a well-designed fusion strategy for handling longitudinal multimodal data. Interessttss incorporating the STD module into a simple mmcode-concateente-attiee. suggesting that disentanglement alone, without dedicated dynamic/static modeling and multiscale fusion, is insufficient for complex prediction tasks. More metrics can be found Tables 13 and 14.
baseline results in performance degradation ("A3" vs. "A4"),
Robustness with Automated Bounding Boxes To assess DiPro's ' robustness to automated region annotations, conducted ablation study replacing Chest ImaGenome bounding boxes with those generated by the automated region detection model from RGRG [49],denoted as "DiPro-" in Table 3. While using automated bounding boxes results in slight performance drop compared to curated annotations, DiPro consistently outperforms all baselines across disease progression +..499+0.000 in F1), mortality
0.039 in AUPRC), and length-of-stay (0.427+00.014 in ACC) tasks. This demonstrates that DiPro remains effective even in the absence of manually curated labels, supporting its applicability to datasets lacking fine-gra fine-grained anatomical annotations. While more accurate anatomical annotations can improve prediction performance, DiPro still achieves strong and generalizable results even with fully automated region proposals.
The ablation study on loss penalties is presented in Table 15, the final selected penalty weights for each prediction task are summarized in Table 16, and the robustness analysis under missing EHR data is reported in Tables 18 and 19.
4 Related Work
Modeling disease progression in sequential CXRs. Recent years have seen growing interest in leveraging longitudinal CXRs for clinical prediction, as they are routinely used to monitor disease progression and naturally provide sequential imaging data [13, 14, 33, 50-54]. Most methods focus on capturing temporal differences using deep learning architectures. For instance, Karwande et al. [14] used a graph attention network to model region-level temporal changes, while Eshraghi Dehaghani et al. [54] adopted a Transformer-baser detection model for localized progression signals. Wang et al. [50] introduced time-aware causal attention, and Mbakwe al. [33] proposed hierarchical Transformer for multi-scale comparison. Other approaches enhance clinical relevance via auxiliary

---

tasks, such as symptom prediction [13] or spatiotemporal contrastive learning with radiology reports [55]. Despite these efforts, effectively addressing redundancy in sequential CXR modeling remains a fundamental challenge.
Leveraging multimodal data for elinical prediction. Multimodal data offers rich temporal and semantic information for clinical prediction tasks [53, 56-61]. Several methods combine the latest CXR with EHR data to improve performance using sophisticated fusion strategies [17, 18, 62, 63). More recent efforts target the challenges of heterogeneous, misaligned longitudinal data. For example, Li al. [58] applied ICA to extract latent EHR signals and aligned them with CT seans using time- aware Transformers. Susman et al. [57] proposed ensemble models to integrate multimodal sequences and highlight salient features for dementia prediction. Others address temporal irregularity directly: Lee et al. [20] introduced unified time embeddings and modality
attention, while Zhang et al. [19] imputed sparse clinical notes with temporal attention and fused them with multivariate time series. Yet. the core challenge of capturing disease progression across misaligned modalities remains underexplored [53, 63, 64], limiting our ability to fully leverage cross- -ross-modal synergy in clinical contexts.
5 Impacts and Limitations
DiPro advances multimodal disease progression modeling through its efficient integration of regional progression-awee aware feature disentanglement and multi multi-timescale alignment. The approach demon- strates significant potential for generalization to asynchronous clinical workflows, such as Alzheimer's s disease monitoring using longitudinal MRI/PET scans with cognitive test records, or heart failure pro- gression tracking using periodic echocardiograms with continuous vital signs. However, the current implementation relies on anatomical annotations (bounding boxes) to localize features. While effective, this requirement may limit scalability in practice. Future work could reduce dependency on manual labels by adopting emerging segmentation tools, such as medical SAMs or weakly supervised localization methods [65], which can better align the framework with real-world clinical workflows. Meanwhile, Our study excludes visits with only a single CXR. This selection may introduce sampling bias and reduce the overall cohort size. However, it is necessary for modeling disease progression between consecutive CXRs, as the longitudinal task inherently requires at least two images per patient. Developing methods that can handle single-CXR visits remains an important direction for future work.
pnngression-specifne
Conclusion
In this paper, we propose DiPro, novel framework that tackles critical challenges in fusing lon- gitudinal multimodal data for clinical tasks: redundancy in sequential CXRs and temporal mis- alignment across modalities. By explicitly disentangling disease progression dynamics from static anatomical features via dedicated constraints, DiPro extracts clinical meaningful and discriminative dynamic/static patterns. To further enhance temporal alignment, we propose multiscale multimodal fusion strategy that bridges CXR-derived progression features with EHR time-series data through interval-wist and full full-sequence-evee interactions. Extensive experiments demonstrate that DiPro achieves state-of-the-art performance on both disease progression identification and general ICU prediction tasks, while providing interpretability consistent with clinical understanding.
Acknowledgments and Disclosure of Funding
This work is partially supported by Innovation and Technology Fund of Hong Kong Innovation and Technology Commission (project no. ITS/202/23), a Collaborative Research Fund of Hong Kong Research Grants Council (project no. C5055-24G), the National Natural Science Foundation of China (62302413), the Health and Medical Research Fund (23220312). the General Research Fund BUTHHHBBU1200266 from the Research Grant Council, and the Research Matching Grant Scheme RMGS2001 8 8_06 from the Hong Kong Government.
10

---

References
[1] Diane R Mould, Nicholas HG Holford, and Carl C Peck. Disease progress models. In Atkinson's 's Principles of Clinical Pharmacology, pages 389-403, Elsevier, 2022.
Michael Moor, Bastian Rieck, Max Horn, Catherine R Jutzeler, and Karsten Borgwardt. Early prediction of sepsis in the ICU using machine learning: a systematic review. Frontiers in Medicine, 8:607952, 2021.
[3] Jianyong Zhong, Hai-Chun Yang, and Agnes Fogo. perspective on chronic kidncy disease progression. American Journal of Physiology- Renal Physiology, (12(3)F3355FF38) 2017.
[4] Mostafa Mchdipour Ghazi, Mads Nielsen, Akshay Pai, M Jorge Cardoso, Mare Modat, Sébastien Ourselin, Lauge Sørensen, Alzheimer's Disease Neuroimaging Initiative, et al. Training recurrent neural networks robust to incomplete data: Application to Alzheimer's disease progression modeling. Medical Image Analysis, 53:39 46, 2019.
[5] Xiang Wang. David Sontag, and Fei Wang. Unsupervised learning of disease progression models. In Proceedings the 20th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 85-94, 2014.
[6] Monica A Konerman, Lauren A Beste, Tony Van, Boang Liu, Xuefei Zhang, Ji Zhu, Sameer D Saini, Grace L Su, Brahmajee K Nallamothu, George N loannou, et al. Machine learning models to predict discase progression among veterans with hepatitis C virus. PloS One, ()):)11221111 2019.
[7] Christopher W Seymour, Foster Gesten, Hallie C Prescott, Marcus E Friedrich, Theodore J Iwashyna. Gary S Phillips, Stanley Lemeshow, Tiffany Osborn, Kathleen M Terry, and Mitchell M Levy. Time to treatment and mortality during mandated emergency for sepsis. New England Journal of Medicine, 376(23):2235-2244, 2017.
[8] Mervyn Singer, Clifford Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon Bernard, Jean- Chiche, Craig Coopersmith, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA, 315 -10, 2016.
[9] Laura Swinckels, Frank C Bennis, Kirsten A Ziesemer, Janneke FM Scheerman, Harmen Bijwaard, Ander de Keijzer, and Josef Jan Bruers. The use of deep learning and machine learning on longitudinal electronic health records for the early detection and prevention of diseases: scoping review. Journal of Medicul Internet Research, 26:e48320. 2024.
[10] Anna Cascarano, Jordi Mur-Petit, Jeronimo Hernandez-Goovzlez,, Marina Camacho, Nina de Toro Fadie, Polyxeni Gkontra, Mare Chadeau Hyam, Jordi Vitria, and Karim Lekadir. Machine and deep learning for longitudinal biomedical data: a review of methods and applications. Artificial Intelligence Review, 56 (Suppl 2):1711-177 2023.
[11] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019.
[12] Huajun Zhou, Fengtao Zhou, Chenyu Zhao, Yingxue Xu, Luyang Luo, and Hao Chen. Multimodal data integration for precision oncology: Challenges and future directions. arXiv preprint 196611. 2024.
[13] Ye Zhu, Jingwen Xu, Fei Lyu, and Pong C Yuen. Symptom disentanglement in chest. X-ray images for fine-grained progression learning. International Conference on Medical Image Computing and Computer- Intervention, pages -078 Springer, 2024.
[14] Gaurang Karwande, Amarachi B Mbakwe, Joy Wu, Leo A Celi, Mchdi Moradi, and Ismini Lourentzou. CheXRelNet: An anatomy-awart aware model for tracking longitudinal relationships between chest X X-rays. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 581-591. Springer, 2022.
[15] Ellen Dicks, Lisa Vermunt, Wiesje M der Flier, Pieter Jelle Visser, Frederik Barkhof, Philip Scheltens, Betty M Tijms, Alzheimer's Disease Neuroimaging Initiative, al. Modeling grey matter atrophy as function of time, aging or cognitive decline show different anatomical patterns in Alzheimer's 's discase. Newrolmage: Clinical, 22: 101786, 2019.
[16] Linda K McEvoy, Dominic Holland, Donald J Hagler Jr. Christine Fennema Noteaaaie, James B Brewer, and Anders M Dale. Mild cognitive impairment: baseline and longitudinal structural MR imaging measures improve predictive prognosis. Radiology, 259(3):834 843, 2011.

---

[17] Nasir Hayat, Krzysztof Geras, and Farah E Shamout. MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images. In Machine Learning for Healthcare Conference, pages 479-503. PMLR, 2022.
[18] Wenfang Yao, Kejing Yin. William Cheung, Jia Liu, and Jing Qin. DrFuse: Learning disentangled repre- sentation for clinical multi- modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages (6416-16426 2024.
[19] Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference Machine Learning, pages 111300-41315 PMLR, 2023.
[20] Kwanhyung Lee, Soojeong Lee, Sangehul Hahn, Heejung Hyun, Edward Choi, Byungeun Ahn, and Joohyung Lec. Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention. In Machine Learning for Healthcare Conference, pages 423-442. PMI.R, R,, 2023.
[21] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest X-ray database and benchmarks weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2097-2106, 2017.
[22] Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Petra Jiraskova, Rickmer Braren, and Nassir Navab. Xplainer: From X-ray observations explainable zero-shot diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 420-429. Springer, 2023.
[23] Paul Pu Liang. Yun Cheng. Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36:27351- 27393, 2023.
[24] Xinqi Zhu, Chang Xu, and Dacheng Tao. Where and what? Examining interpretable disentangled representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5861-5870, 2021.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[26] Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping. Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 1671-7671, 2023.
[27] Mengwei Ren. Neel Dey. Martin Styner, Kelly Botteron, and Guido Gerig. Local spatiotemporal repre- sentation learning for longitudinally-consie
neuroimage analysis. Advances in Neural Information Processing Systems, 35: 11113351-13555 2022.
[28] Ziyu Zhou, Haozhe Luo, Jiaxuan Pang, Xiaowei Ding, Michael Gotway, and Jianming Liang. Learning anatomically consistent embedding for chest radiography. In BMVC: proceedings of the British Machine Vision Conference. British Machine Vision Conference, volume 2023, page 617, 2023.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[30] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. MIMIC-IV, freely accessible electronic health record dataset. Scientific Data, 10(1).1, 2023.
[31] Alistair EW Johnson, Tom J Pollard, Seth Berkowitz. Nathaniel R Greenbaum, Matthew P Lungren. Chih-ying Deng, Roger G Mark. and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019.
[32] Joy Wu, Nkechinyere Agu, Ismini Lourentzou, Arjun Sharma, Joseph Paguio, Jasper Seth Yao, Ed- ward Christopher Dee, William Mitchell, Satyananda Kashyap. Andrea Giovannini, et al. Chest ImaGenome dataset (version 1.0.0). Physio Net, 2021.
12

---

[33] Amarachi B Mbakwe. Lyuyang Wang. Mehdi Moradi, and Ismini Lourentzou. Hierarchical vision Transformers for disease progression detection in chest X-ray images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 685-695. Springer, 2023.
[34] Yanbo Xu, Siddharth Biswal, Shriprasad Deshpande, Kevin Maher, and Jimeng Sun. RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2565-22622 2573, 2018.
[35] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias, Attend and diagnose: Clinical time series analysis using attention models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[36] Tarun Agrawal and Prakash Choudhary. Segmentation and classification on chest radiography: a systematic survey. The Visual Computer, 39(3):875-113, 2023.
[37] Erdem Yanar and Firat Hardalaç. Clinical decision support for early diagnosis of cardiomegaly by using deep learning techniques chest X-rays. In 2023 Computing in Cardiology (CinC), volume 50, pages IEEE, 2023.
[38] Robert Wilkins, James R Dexter, and Philip Gold. Respiratory disease: case study approach to patient care. FA Davis, 2006.
[39] Catia Cillóniz, Celia Cardozo, and Carolina García-Vide Epidemiology. pathophysiology. and microbiol- ogy of community--aquiinn pneumonia. Annals of research hospitals, 2(1), 2018
[40] M Elmukhtar, Habas Ala, Said Abdusslam, Rayani Amnna, Farfar Kalifa, Habas Eshrak, Alfitori Gamal, Eraaes Almehdi, Habas Aml, Naser Abdel, al. Diagnostic approach to pleural effusion based on pathogenesis and radiological findings: A narrative review. Yemen Journal of Medicine. 3(2):102- 113, 2024.
[41] Karen Marcdante and Robert M Kliegman. Nelson essentials of pediatrics E-book. Sciences. 2014.
Elsevier Health
[42] BTS Guideline. Non-invasive ventilation in acute respiratory failure. Thorax, 57(3):122-211, 2002.
[43] Jean-Louis Vincent, Serdar Akça, Arnaldo De Mendonça. Philip Haji- Mici-llccael, Charles Sprung, Rui Moreno, Massimo Antonelli, Peter M Suter, SOFA Working Group, et al. The epidemiology of acute respiratory failure critically patients. Chest, 16665)1662-1660, 2002.
[44] Rahul Lohan. Imaging of ICU patients. Thoracic Imaging: Basic to Advanced, pages 173-194, 2019.
[45] Adel Salah Bediwy, Mohammed Al-Biltagi. Nermin Kamal Saced, Hosameldin A Bediwy, and Reem Elbeltagi. Pleural effusion in critically ill patients and intensive care setting. World Journal of Clinical Cases, 11(5):989, 2023.
[46] David M Maslove, Benson Tze-Ming Chen, Helena Wang, and Ware G Kuschner. The diagnosis and management of pleural effusions the ICU, Journal of Intensive Care Medicine, 28(1):24-36, 2013.
[47] SP Wright, D Verouhis, G Gamble, K Swedberg, N Sharpe, and RN Doughty. Factors influencing the length of hospital stay of patients with heart failure. European Journal of Heart Failure, 5(2):201-209 2003.
[48] David A Gruenberg, Wayne Shelton, Susannah L Rose, Ann E Rutter, Sophia Socaris, and Glenn McGee. Factors influencing length of stay in the intensive care unit. American Journal of Critical Care, 15(5): 502-509, 2006.
[49] Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable region- guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7433-7442, 2023.
[50] Fuying Wang, Shenghui Du, and Lequan Yu. HERGen: Elevating radiology report generation with longitudinal data. European Conference on Computer Vision, pages 183-200. Springer, 2024.
[51] Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Garzia, Maximilian Ilse, Daniel C Castro. Benedikt Boecking. Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision- language processing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15016-15027, 2023.
13

---

[52]
Francesco Dalla Serra, Chaoyang Wang, Fani Deligianni. Jeff Dalton, and Alison Q O'Nell Controllable chest X-ray report generation from longitudinal representations. In The 2023 Conference on Empirical Methods in Natural Language Procexsing, 2023.
[53] Luoting Zhuang. Stephen H Park, Steven Skates, Ashley E Prosper, Denise Aberle, and William Hsu. Advancing precision oncology through modeling of longitudinal and multimodal data. arXiv preprint arXiv:2502.07836 2025.
[54] Mehrdad Eshraghi Dehaghani, Amirhosscin Sabour, Amarachi B Madu, Ismini Lourentzou, and Mehdi Moradi. Representation learning with a Transformer-basee detection model for localized chest X-ray dis- ease and progression detection. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 578-587. Springer, 2024.
[55] Kang Liu, Zhuoqi Ma, Xiaolu Kang, Yunan Li, Kun Xie, Zhicheng Jiao, and Qiguang Miao. Enhanced contrastive learning with multi-view longitudinal data for chest X-ray report generation. arXiv preprint urXiv:2502.20056. 2025.
[56] Virgilio Kmetzsch, Emmanuelle Becker, Dario Saracino, Daisy Rinaldi, Agnes Camuzat, Isabelle Le Ber, Olivier Colliot, PREV-DEMALS study group, et al. Disease progression score estimation from multimodal imaging and microRNA data using supervised variational autoencoders. IEEE Journal of Biomedical and Health Informatics, 26(12):6024 60356, 2022.
[57] Aviad Susman, Rupak Krishnamurthy, Yan Chak Li, Mohammad Olaimat, Serdar Bozdag, Bino Varghese. Nasim Sheikh-Bahaei, and Gaurav Pandey. Longitudinal ensemble integration for sequential classification with multimodal data, arXiv preprint 100iv:2441.05990 2024.
[58] Thomas Z Li, John M Still, Kaiwen Xu, Ho Hin Lec, Leon Y Cai, Aravind R Krishnan, Riqiang Gao, Mirza Khan. Sanja Antic, Michael Kammer, et al. Longitudinal multimodal Transformer integrating imaging and latent clinical signatures from routine EHRs for pulmonary nodule classification. In Inter- national Conference on Medical Image Computing and Computer-A Intervention, pages 649-659. Springer, 2023.
[59] Elisa Warner, Joonsang Lee, William Hsu, Tanveer Syeda-Mahmood, Charles E Kahn Jr. Olivier Gevaert, and Arvind Rao. Multimodal machine learning in amage-based and clinical biomedicine: Survey and prospects. International Journal of Computer Vision, 132(9)3353-3369, 2024.
[60] Yidan Feng, Bohan Zhang, Sen Deng, Zhanli Hu, and Jing Qin. Asynchronous multi-medal learning for dynamic risk monitoring of acute respiratory distress syndrome in intensive care units. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14 23. Springer. 2025.
[61] Chenyu Lian, Hong- Yu Zhou, Dongyun Liang, Jing Qin, and Liansheng Wang. Efficient medical vision- language alignment through adapting masked vision models. IEEE Transactions on Medical Imaging, 2025.
[62] Yidan Feng, Bingchen Gao, Sen Deng, Anqi Qiu, and Jing Qin. Unified multi-modal learning for any modality combinations in Alzheimer's 's discase diagnosis. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 487-497. Springer, 2024.
[63] Wenfang Yao, Chen Liu, Kejing Yin, William Cheung, and Jing Qin. Addressing asynchromicity in clinical multimodal fusion via individualized chest X-ray generation. Advances in Neural Information Processing Systems, 37:29001-29028, 2024.
[64] Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Longitudinal correlation analysis for decoding multi-modal brain development. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 400-409. Springer, 2021.
[65] Jay N. Paranjape, Shameema Sikder, S. Swaroop Vedula, and Vishal M. Patel. S-SAM: SVD-based fune-tuning of segment anything model for medical image segmentation. In International Conference on Medical Image Computing and Computer- Assistee Intervention, pages 720-730. Springer, 2024.
[66] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard. Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arx: 19786, 2025.
[67] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309.16600 16609, 2023.
14

---

[68] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka. Eduardo Pontes Reis, and Pranav Rajpurkar. Med-Flamingo: multimodal medical few-shot learner. In Machine Learning for Health (MLAlf). pages 353-367. PMLR, 2023.
[69] Hu Wang. Yuanhong Chen, Congbo Ma. Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi-modal learning with missing modality via shared-specte specific feature modelling. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 15878-15887, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly describe the two core chal- lenges-temporal redundancy in CXRs and multimodal misalignment-andd correspond directly to the proposed solutions: disentanglement of dynamicc/tatic features and hierarchi- cal multimodal fusion. These claims are substantiated through method design and validated by comprehensive experiments.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations in Section 5, We will address these limitations in future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer[[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide the implementation settings used to obtain the main experimental results in the appendix and supplementary materials. Detailed results and analyses can be found in Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The source code is submitted supplementary material alongside the paper and will be made publicly available upon acceptance. All datasets used this study are open-source and publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. /nips.cc/ppll wwt.c//pwwccg.uueeeeeeeoo http////guideee/eeeubbbiiiiiiiii wtw.eesuuumsssoooooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details of training and testing, including data splits, preprocessing steps, and hyperparameter search space, are provided in Appendix A.2.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are reported by taking the average of three runs of model training along with the standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed information on the compute resources used, including GPU types, memory, and architecture for all experiments in the Appendix A to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eerrpp...eee Ceaublic/EthcsSuiddee
Answer: [Yes]
The research strictly follows the NeurIPS Code of Ethics by ensuring data privacy, avoiding harm to participants, adhering to institutional protocols, and considering potential societal impacts such faimess,, bias, and security throughout the study.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the potential impacts of the proposed work in Section 5.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of high -igh models or scraped datasets that could pose misuse risks, and therefore no additional safeguards were required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets such as MIMIC, which is cited properly along with its license terms (PhysioNet Credentialed Health Data License). Additionally, all external codebases and models referenced are appropriately credited with their original citations and licenses mentioned where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments or direct research with human subjects. All experiments conducted publicly available, de-identified dataset, so participant instructions or compensation details are not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
21

---

The study uses publicly available, de-identified data (e.g., MIMIC), which does not involve direct interaction with human participants.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is only used for editing and formatting purpose of this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htaps://nnene //teur/eespp//eeeeee /c//onnereecco/22255///////////e for what should or should be described.
22

---

A Experiment Details
A.1 Details of Data Preprocessing
EHR Data Preprocessing We adapted the EHR data processing pipeline from [17], modifying the sampling frequeney from 2-hour 1-hour intervals. We incorporate vital signs, laboratory measurements, and clinical scores, resulting in a total of 38 clinical time-series variables, includ- ing alanine aminotransferase, albuminn, alkaline phosphate, anion gap, asparate aminotransferase. bicarbonate, bilirubin, blood urea nitrogen, chloride, creatinine, diastolic blood pressure, fraction inspired oxygen, Glasgow coma scale (eye opening, motor response, and verbal response), glucose, heart rate, height, hematocrit, hemoglobin, magnesium, mean blood pressure, oxygen saturation, partial pressure of carbon dioxide, partial thromboplastin time, platelets, positive end-expiratory pressure, potassium, prothrombin time, respiratory rate, sodium, systolic blood pressure, temperature, troponin-T, urine output, daily weight, white blood cell count, and pH. Following [17], we applied identical discretization and standardization procedures. Demographic data including age, height, admission weight, gender, race, language, and marital status were also incorporated.
CXR Preprocessing CXR studies were temporally aligned with corresponding ICU stays to construct longitudinal imaging sequences. We retained only AP view images, matched them to ICU stays based on study timestamps, and excluded outlier admissions using length-of-stuu filtering. Each matched CXR was restricted to those with available bounding-bop annotations in Chest ImaGinnmme. Meanwhile, in this study, we do not perform explicit longitudinal registration of CXR images. To mitigate potential positional misalignment over time, we follow the preprocessing strategy of CheXRelNet [14]: we apply cropping based on anatomical bounding boxes from Chest ImaGenome dataset. This preprocessing allows us to focus on capturing the semantic-level disease progression within consistent anatomical structure, rather than relying on pixel-level alignment. As result, our model is designed to be more robust to positional variability across different time points.
Data Splitting and Statistics Our analysis focused on ICU stays containing at least two CXRs. The dataset was partitioned into training (70%), validation (10%), and test sets (20%) at the subject level to prevent data leakage. Table 4 summarizes the sample counts for each task, while Table 5 and Table 6 detail the label distributions for discase progression identification and general ICU prediction tasks, respectively. The distribution of CXR examinations per patient is presented in Table 7.
Table 4: Data statistics in training, validation, and testing sets for each task.
Task
Training
Validation
Test
Discase Progression Identification
3982
560
1137
General ICU Prediction
1889
285
546
Table 5: Label distribution for the Disease Progression Identification task.
Enlarged Cardiac
Pulmonary Edema
Lung Opacity
Pleural Effusion
Category
Atelectasis
Consolidation
Pneumonia
Silhouette
Improved 328 (15.3%)
83 (4.6%)
124 (14.2%)
572 (31,9%) 784 (19.2%)
317 (13.1%) 98 (12.7%)
Worsened 674 (31.4%)
141 (7.8%)
285 (32.7%)
597 (33.3%)
1273 (31.2%)
702 (2.9%)
387 (50.1 1%)
No Change
1143 (53.3%)
1575 (87.5%)
463 (53.1%)
624 (34.8%)
2025 (49.6%)
1407 (58.0%)
287 (37.2%)
Total
2145
1799
872
1793
4082
2426
772
Table 6: Label distribution for the General ICU Prediction Tasks.
Mortality
Length of Stay
1
(2,3)
[3, 4)
(4,6)
>> 6
Count
2255
465
793
641 (23.6%)
687
599 (22.0%)
(%)
(82.9%)
(17.1%)
(29.2%)
(25.3%)
23

---

Table Data statistics of Numbers of CXR in training, validation, and testing sets for the General ICU Prediction Task.
Split
Training
1367
465
56
Validation
199
76
10
0
Test
409
120
16
1
Total
1975
661
82
2
A.2
Implementation Details
A.2.1 Details of Architectures and Training Procedures DiPro
The training and validation processes are executed on a server equipped with a RTX 3090-24GB GPU card and 14 vCPU Intel(R) Xeon(R) Gold 6330 CPU. The method is implemented using PyTorch 1.9.1 and PyTorch-Lighunnn 1.4.2 with CUDA 11.1 environment. AdamW W optimizer and CosineAnnealingL.. learning schedular are used for training.
Model Architecture and Hyperparameters. DiPro consists of four major components: (1) a CXR processing module, which employs shared ResNet backbone extract regional visual features, followed by a multi-layer perceptron (MLP) for feature adjustment and two parallel MLP-basee projection heads that encode static and dynamic representations; (2) EHR processing module, which includes one-layer multivariate transformer encoder for global temporal modeling, local multi-heal attention layer for capturing short-term dependencies. time-embedding MLP for relative temporal encoding. and a separate MLP for demographic features: (3) a multimodal fusion module. which integrates CXR and EHR representations through local and global attention layers, followed by lightweight transformer block and group-based static feature fusion mechanism; and (4) prediction head, implemented as a task-specific MLP for downstream classification regression. The detailed hyperparameter settings and implementation specifics are provided in the released code repository.
Training Configuration The model was trained with base batch sizes of 8 (for disease progression identification) and (for general ICU prediction), using 4-step gradient accumulation to achieve an effective batch size of 32 or 16. Training proceeded for maximum of 100 epochs with early stopping triggered after 10 epochs without validation improvement. Task-specific selection metrics were employed: macro-F1 for disease identification, accuracy for length-of-stay classification, and AUPRC for mortality prediction. The hyperparameter search spaces for each task are documented in Table 8.
Table 8: Hyperparameter search space used for model tuning.
Hyperparameter
Search Grid
Learning rate
8 10-5 10 x 10 9.5 2,5x 10-5 0.1, 0.2, 0.3
Dropout rate
Hidden dimension
64, 128, 256
Asemp
0.01, 0.001, 0.1,
Ageed
2, 6, 10
APAE
0.01, 0.1, 2,
Aorth
0.001, 0.01, 0.1. 10
A.2.2 Implementation of Baselines
Since none of the baselines can handle both disease identification and general ICU prediction tasks within a unified framework like ours, we introduce minimal modifications to adapt existing approaches. For sequential CXR disease progression specialists (e.g.. CheXRelNet [14], CheXRelFormer [33], and SDPL [13]), we concatenate EHR time series and demographie data, then integrate the same EHR encoder and attention fusion layer as model for fair comparison. For multimodal fusion
24

---

baselines, we adapt UTDE [19], originally designed for longitudinal clinical notes and EHR, by replacing its text encoder with our image encoder to process longitudinal CXRs. Similarly, we modify MedFuse [17] by concatenating CXR representations at the sequence level before fusion, and extend DrFuse [18] by first disentangling CXR-EHR pairs and then concatenating the CXR features for fusion.
Hyperparameter Search. We conducted unified hyperparameter search for all baseline models using the following grid: learning rates {1 x 10-s x 10 x 10-6, 1 x 10-7), dropout rates {0.1, 0.2, 0.3], and hidden dimensions {64, 128, 256, 320}- All other hyperparameters were kept consistent with those specified in the original implementations provided by the official source code of each baseline.
Computational Efficiency and Inference Cost Comparison We summarize the comparison of computational efficiency and inference cost between DiPro and baseline models in Table evaluated under the multimodal input setting (sequential CXR EHR) for the disease progression identification task. Compared the CheXRelNet baseline, which also an anatomical region-basse based model, DiPro reduces FLOPs by 9.8% and latency by 22.0%, while achieving relative gain of 16.5% in FI for disease progression.
Table 9: Computational efficiency and inference cost comparison between DiPro and baseline models. (F1 scores are extracted from Table 10.)
Params
FLOPPs
MACs
Latency Mean (ms)
Throughput (samples/s)
Model
(M)
(G)
(G)
Score
DiPro (Ours)
31.06
82.75
41.37
23.05
43.38
0.466.10.018
CheXRelNet [14]
49.15
90.90
45.45
27.34
36.58
0.382:+0.016
CheMRelFormer [33]
49.13
19.95
9.98
14.86
67.28
0.352:+0.021
SDPL [13]
34.24
9.45
4.73
13.35
74.90
0.393+0.0110
UTDE [19]
6.69
0.80
5.24
190.90
0.449110.005
UMSE [20]
23.99
8.27
4.14
9.65
103.62
0.352.10.013
MedFuse [17]
27.21
8.28
4.14
9.78
102.23
0.409
DrFuse [18]
56.40
16.45
8.23
19.93
50.18
0.429=1.001
B Additional Results
Wright Ablation Study for Loss Penalties We conducted ablation experiments in which each loss component, Aorth (orthogonal disentanglement loss), Atemp (temporal consistency loss for static features), and λPAE (Progression-Awaie Enhancement loss), individually disabled by setting its corresponding weight to zero. The results are summarized in Table 15. Key observations include:
Orthogonal disentanglement loss (North): Removal caused notable decreases in disease progression and length-of-stay predictions, emphasizing the benefit of disentangling dynamic pathology from static anatomy.
Progression-Aware Enhancement loss (ΛPΛg): Disabling this loss reduced disease pro- gression prediction performance, indicating its role in enhancing sensitivity to progression direction.
Temporal consistency loss (Atemp): Its removal led to significant performance drops across ICU tasks, highlighting the importance of maintaining longitudinal consistency of static anatomical information.
These results demonstrate that all three loss components contribute meaningfully and in distinct ways to the model's overall performance across different tasks. We report the final selected penalty weights for each prediction task Table 16, which reflect the relative contribution of different regularization terms overall model performance.
25

---

Table 10: Detailed Performance Comparison on Disease Progression Identification Tasks. Sequential CXR disease progression specialists (e.g., CheXRelNet [14], CheXRelFormer [33], and SDPL [13],, originally designed for unimodal sequential CXR inputs, are extended to multimodal integration by incorporating transformer-ben based EHR encoder and applying cross-attention fusion. Methods that can naturally process uni-CXR inputs (e.g., UMSE [20] and MedFuse [17]) are included for comparison under the unimodal CXR setting. Numbers in bold indicate the best overall
performance, while underlined values denote the top-performing baseline.
Method
Precision
Rocall
F1
AUPRC
AUROC
ACC
Unimodal Methods (CXR)
CheXRelNet [14]
0.395++0.015
0.392++0.010
0.389++.010
0.394+0.010
0.574++0.011
0.508+0.013
CheXRellFrmeer [33]
0.38911.044
0.309.10.033
0.35410.032
0.372-10.023
0.551110.041
0.446
SDPL [13]
0.408.10.006
0.406110.020
0.393110.110
0,4171100322
0.6091100031
0,538
UMSE [20]
0.337=0.004
0.337::0008
0.329100008
0.347=0.004
0.513+10.006
0.476=0.004
McdFuse [17]
0.439==0.006
0.440:t0.009
0.433+0.010
0.453=0.011
0.643+00.006
0.543=0.024
DiPro (ours)
0.475=00.004
0.452
0.453:00.009
0,468:+0.013
0.651+0.0116
0.567=0.007
Unimodal Methods (EHR)
Transforner [29)
0.412.10.011
0.35810.010
0.327-10.033
0.38410.001
0.569110.008
0.5091100009
Multimodal Methods
CheXReINet [14]
0.391=0.020
0.387+0.019
0.382=0.016
0.390=2.019
0,572+00.088
0.504=0.024
CheXRelFormer [33]
0.359=0.019
0.362+00026
0.352#0.022
0.371
0.544++0.022
0.474 0.009
SDPL [13]
0.409+0.0099
0.398+00.066
0.393++0.003
0.401=0.008
0.582++00008
0.529+0.014
UTDB [19]
0.481++0.017
0.402++0.00
0.449-10.000
0.472+100014
0.659++0.011
0.527=+00016
UMSE [20]
0.353110.011
0.3611:0.099
0.352-10.013
0.36410.006
0.544.10.004
0,48410.011
MedFuse [17]
0.423:10.049
0.413::0.045
0.409110.042
0.422=1.0000
0.60911.055 0
0.530=0.000
DrFuse [18]
0.442=000099
0.461=000007
0.429:10.010
0.438=0.003
0.628::0.002
0.475=000021
DiPro (ours)
0,484.10.008
0.471.10.024
0.466.10.018
0.478:10.018
0.664.10.013
0.565
Table 11: Performance Comparison on Disease Progression Identification Tasks across Different
Diseases. This table reports the F1 performance of various unimodal and multimodal methods across seven disease progression identification tasks.
Enlarged
Pulmonary Edema
Lung Opacity
Pleural Effusion
Method
Atelectasis
Cardiac Silhouette
Consolidation
Pneumonia
Unimodal Methods (CXR)
CheXReiNei [i4]
0.425
0.340
0.381
0.408
0.398
0.366
0.407
CheXRelFormer [33]
0.359
0.318
0.319
0.390
0.354
0.380
0.357
SDPL
0.396
0.362
0.350
0.439
0.431
0.443
0.331
DiPro (Ours)
0.436
0.338
0.388
0.527
0.523
0.509
0.452
Multimodal Methods
UTTE [19]
0.445
0.338
0.402
0.470
0.503
0.478
0.510
UMSE [20]
0.352
0.313
0.343
0.346
0.384
0.354
0.368
MedFuse [17]
0.422
0.340
0.392
0.455
0.447
0.433
0,363
DrFuse [18]
0.434
0.310
0.354
0.499
0.464
0.447
0.498
DiPro (Ours)
0.453
0.362
0.399
0.530
0.509
0.500
0.510
Broader Baseline Comparison with Large Vision-Language Models. To provide a comprehen- sive evaluation, we include recent large-scale vision-language models (VLMs) capable of processing multi-image inputs (e.g., Gemma3 [66], QWen2.5VL. [67], and Med-F Sed-ilamingo [68]) as baselines for disease progression identification. We input consecutive CXR pairs and use a few-shot prompting approach (following [68]) to have the VLMs predict progression status for thoracic conditions (results in Table 17). DiPro achieves notably higher accuracy with lower computational cost, highlight- ing the advantages of our design: spatialtemporal disentanglement (STD) and progression-awee -ware enhancement (PAE) for clinical prediction.
Robustness to Missing EHR Data We conducted additional experiments on the general ICU tasks, where EHR data were randomly dropped during training at rates of 25%, 50%, and 75% (following (69]), while testing was performed on the complete dataset. Notably, although model performance naturally decreases as the EHR missing rate increases, DiPro consistently outperforms
26

---

Table 12: Detailed Performance Comparison on General ICU Prediction Tasks. The "Input Modalities" section specifies the data sources used for each method. "Last" and "Long." indicate CXR- only input settings, where "L.ast" denotes the use of the last available CXR and "Long." represents the use of longitudinal CXRs. The inclusion of EHR data is indicated by the "EHR" column. "Sequential CXR Disease Progression Specialists" are extended to multimodal integration by incorporating a transformer- EHR encoder and applying cross-attention fusion. For "Clinical Multimodal Fusion Specialists", which were originally designed for single-CXR inputs, we modify MedFuse [17] by concatenating CXR representations the sequence level before fusion, and extend DrFuse [18]
by first disentangling CXR-EHR pairs before concatenating CXR features for multimodal fusion.
Input Modalities
Montlity
Length of Stay
Method
Last
Long
EHR
AUPRC
AUROC
Kappa
ACC
F1
AUPRC
Unimodal Methods (EHR)
Transformer (29)
0.712=0.009
0,885=0.003
0.226+0.019
0,227#0014
0.360::0.024
0,386=0.014
Sequential CXR Disease Progression Specialists
ChexReINet [14]
0.007:000000
0.291 =00050
0.876=0.015
0.624=0.036
0.166+20.044
0.039:0.0200
0,380+0.028
0.11::00014
0.355=+0.009
0.238:100110
0:358=0.015
0.275=0.004
0.218-50011
0.5223-0.119
0.005:50.322
026710.643
6.212-40.01
0255-66:11
ChexRelFormer (33)
0.522-10.041
0.766.10.011
0.103
0.333:10.005
0.306.10.005
0.005_00007
100006
0.60820.035
0.01140.007
0.267:0.010
0.151:10:02
0.261=0.005
SDPL [13)
0.717.10.024
0.87810.019
4,23140.009
0.30.00000
0.385.10.011
0.404_.0.012
Longitudinal Multimodal Specialists
UTDE [19]
0.717=0.019
0.887=0.004
0.160=0.016
0.381+00013
0.324:00005
0.361=0.112
0.710:20.019
0.887=0.012 00396100.022
0.19510.031
0,40010.221
0.346:10.039
0.365_0.028
0. 10039
0.217.50.DD
001910010
0.550.00:026
03566-6.018
UMSE [20]
0.712200028
0.91=00.011
0,204:0.019
0410:40011
0.342:00.211
0.357=0.018
Clinical Multimudul Fuxion Specialists
MedRuse [17]
0.6861000188
0.716-+0.088
0.86910.011
0.881+00.005
0,213:0.012
0,210+000099
041240.027
0.413:1.004
0.362:1.025
0.350:4.006
0.012
0.019-0.11
D.7000500012"
00862-0.014
0,T14-500448
1033862-0644
6.555:50066
0.316-0.024
DrFuse (18)
0.68120.008
0.85120.017
0.14220.014
0.36010.011
0.348.100006
0.329=0.004
DiPro (Ours)
0.319=00:18
0.637=0.015
0.029:0.0177
0.284+0.010
0:227::0.044
0.278=0.004
0.742=00.03
0.897:00.002
0.248:00.08
0.440+000007
0.384++0.18
0.409-0.010
Table 13: Results of the ablation study. (Disease Progression Identification Task)
Precision
Recall
Fl
AUPRC
AUROC
ACC
DiPro
0.484=0.008
0.471=0.024
0.466:1.0018
0,478:100188
0.664:10.013 0.565=00.033
w/o MMF
0.481=10.099
0.460=0.013 0.460+10.014 0.472:10.008 0.654:10.017 0.580=100010
w/o PAE
0.443+100024
0.446=0.023
0.433+10.017 0.461=0.015 0.646:10.018 0.552+-0.024
w/o STD
0.372:10.014
0.371=0.015
0.362+000016 0.377=00007 0.556:10.006 0.4917
w/o
(PAE+MMF)
0.455=100017
0.452=0.015 0.439:100007 0.461=0.008 0.647:10.003 0.536=0.016
Table 14: Results of the ablation study, (Length of Stay Classification)
kappa
Precision
Recall
F1
AUPRC
AUROC
ACC
DiPro
0.248.10.008
0.431
0.433.10.005
0.38411.018
0.40911.0100
0.688.0.000
0.440-10.007
w7o M
0.214
0.3%610.033 0.408.10.001 0.32210:015 0.588100000 0.0
00.673160008
0.416-E07027
w/o PAR
0.235++0.022
0.002
0.423+00.016 0.386=+0.007
0.408:+0.018
0.688+0.000
0.432++00118
w/ STD
0.225+0.043
0.375=0.082
0.415#00031
0.372=0.044
0.880=0.025
0.667+0.025
0.425#0.031
wo (PAE+MMF)
0.194.10.014
0.367110.03 0.390110.010 0.357-10.009 0.380110.009 0.663.10.006 0.4041100014
Table 15: Ablation study of loss components by setting individual loss weights to zero. '-' indicates
that disease progression prediction could not be computed due to Liemp requiring at least three CXRs.
Components
Disease Progression
Mortality
Length of stay
ID
North
XPAE
Atemp
Fl
AUPRC
ACC
DiPro
0.466 + 0.018
0.742 + 0.003
0,440 + 0.007
B1
x
/
0.448 1 0.004
0.749 1 0.028
0.415 + 0.008
B2
0.441 1 0.013
0.708 1 0.038
0.432 0.005
B3
0.737 1 0.008
0.415 0.025
27

---

Table 16: Final selected penalty weights for each prediction task.
Prediction Task
Apred
Aorth
Atemp
APAE
In-hospital Mortality
6
0.1
1
0.1
Length of Stay
10
0.001
0.1
0.1
Disease Progression
2
Table 17: Performance Comparison on Disease Progression Identification Task between VLMs and DiPro.
Model
Precision (P)
Recall (R)
Fl Score
Throughput (samples/s)
DiPro (Uni-CXR) (Ours)
0.475
0.452
0.453
43,38
Gemma3 [66]
0.329
0.328
0.279
0.088
QWen2.5VI. [67]
0.304
0.330
0.251
0.097
Med-Flamingo [68]
0.355
0.345
0.301
0.234
all established baselines across all missing-datte -atan scenarios, demonstrating strong robustness in handling incomplete multimodal inputs (see Tables 18 and 19).
Table 18: Length of Stay Classification (Accuracy) under different EHR missing rates.
Method
75% Missing
50% Missing
25% Missing
0% Missing
DiPro (Ours)
0.399 + 0.011
0.391 + 0.021
0.415 + 0.005
0,440 + 0.007
UMSE [20]
0.339 0.029
0.376 + 0.005
0.396 = 0.009
0.410 0.013
UTDE [19]
0.377 0.015
0
0.384 1 0.034
0.400 0.021
DrFuse [18]
0.340 0.009
0.360 0.010
0.363 + 0.011
0.360 0.011
McdFuse [17]
0.332 + 0.039
0.360 + 0.033
0.360 + 0.025
0,412 + 0.027
Table 19: Mortality Prediction (AUPRC) under different EHR missing rates.
Method
75% Missing
50% Missing
25% Missing
0% Missing
DiPro (Ours)
0.696 0.011
0.718 1 0.012
0.751 1 0.006
0.742 0.003
UMSE [20]
0.648 1 0.009
0,685 1 0.010
0.686 1 0.051
0.712 +: 0.028
UTDE [19]
0.603 0.052
0.673 1 0.020
0.697 1 0.021
0.710 0.019
DrFuse [18]
0.602 0.024
0.663 0.024
0.674 0.077
0.684 0.008
MedFuse [17]
0.609 0.057
0.641 + 0.026
0,705 0.016
0.716 = 0.018
28


Reviews:
[{"review_id": "ZLnDNe2eis", "reviewer": "Reviewer_D1kC", "summary": "The paper presents DiPro, a framework for clinical prediction that addresses two key challenges in longitudinal chest X-ray analysis: a) redundant static anatomy that obscures disease progression, and b) temporal misalignment between sparse CXR scans and dense EHR time-series. The baselines show this framework surpasses previous works in multiple downstream tasks, demonstrating a significant improvement.", "questions": "Questions above ^", "limitations": "No limitations", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper explores a real problem in longitudinal healthcare data. It provides a novel method with multiple features:\n\nDisentangled Feature Learning: Separates static anatomy from dynamic pathology in CXRs, reducing redundancy and improving progression modeling.\n\nMultiscale Multimodal Fusion: Aligns temporally misaligned CXR and EHR data at both local and global levels for richer integration.\n\nProgression-Aware Training: Uses reversed CXR pairs to enforce directionality in progression modeling.\n\nAdditionally, compared with the baselines provided, this method shows strong performance as it achieves state-of-the-art results across disease progression, mortality, and length-of-stay prediction tasks. Confidence intervals are additionally provided.\n\nWeaknesses:\nClarification of Component Choices\n While the proposed framework demonstrates strong performance, the model involves several architectural components (e.g., disentanglement module, progression-aware enhancement, multiscale fusion). Although some ablations are included, it's not fully clear why these particular components were selected over other design choices. Could the authors please elaborate on the rationale behind each step, and how it compares to alternative strategies (e.g., simpler fusion mechanisms or temporal encodings)?\n\n\nBroader Baseline Comparison\n I recommend including additional baselines using recent vision-language models that support multi-image inputs. This would help contextualize DiPro's performance relative to large-scale pretrained models that are increasingly being applied to multimodal medical tasks.\n\n\n\nEfficiency and Generalizability\nSome discussion on computational efficiency and inference cost would be helpful, especially in clinical settings. Additionally, it would be valuable to explore how DiPro generalizes to datasets beyond MIMIC-CXR, or to other clinical imaging modalities (e.g. MRI) and additional datasets.", "was_revised": true, "final_justification": "The authors have addressed all my concerns. The method introduced is highly competitive against SOTA models in the space, providing a valuable contribution to the community.", "rebuttal": "We thank the reviewer for the thoughtful and constructive comment, and we appreciate the positive feedback of our work. Below, we address each point in detail:\n\n\n### **W1:  Clarification of Components Choices**\n\nOur model is to address two core issues in longitudinal multimodal clinical prediction:  \n1. *Redundancy in sequential CXRs* due to dominant static anatomy  \n2. *Temporal misalignment* between irregular CXR imaging and continuous EHR measurements\n\nBelow, we explain the motivation and empirical support for each main component, and how they compare to simpler alternatives.\n\n**(1) Spatiotemporal Disentanglement (STD)**\n\n- *Motivation:*  \nIn sequential chest X-rays, static anatomical features (like chronic cardiac enlargement or stable skeletal deformities) dominate sequential CXR scans. These static features can drown out subtle but meaningful pathological changes, such as new infiltrates or evolving edema. Without explicit disentanglement, disease progression signals are diluted, reducing the value of sequential imaging analysis [1,2].\n\n- *How STD Works:*  \nSTD disentangles dynamic pathological changes from static anatomical regions across consecutive CXRs. This separation reduces redundancy and prevents interference between features with different clinical sematics. As a result, the model can focus on true progression signals, leading to more effective temporal representation learning.\n\n- *Empirical Support:*  \nAdding STD to a baseline (which directly concatenates regional features) improves F1 for disease progression identification by 21.3% (Table 3 in the paper).\n\n[1] Schiavone, William A., and David S. Majdalany. \"Chest Radiography Pearls in Select Adult Congenital Heart Disease.\" Journal of Personalized Medicine 14.4 (2024): 397. \n\n[2] Zhu, Ye, et al. \"Symptom disentanglement in chest x-ray images for fine-grained progression learning.\" MICCAI-2024.\n\n\n**(2) Progression-Aware Enhancement (PAE)**\n\n- *Motivation:* \nICU patients often have reversible or recurring conditions. In practice, if a pair of CXR images is labeled as showing 'improved' progression, then reversing the order of the images results in the opposite label 'worsened' as expected. The PAE module is designed to make the model more sensitive to the direction of disease progression by leveraging this clinical insight.\n\n- *How PAE Works:*\nWe reverse the order of CXR pairs during training, requiring the model to generate dynamic features that reflect the opposite progression direction, while keeping static features unchanged. Since reversing the pair inverts progression but preserves anatomy, this approach further separates dynamic and static representations and highlights their distinct clinical roles.\n\n- *Empirical Support:* \nCompared to state-of-the-art models that use only sequential CXRs, DiPro achieves a 15.3% relative improvement in F1 and a 12.2% gain in AUPRC for disease progression identification (Table 2 in the paper).\n\n**(3) Multi-Scale Multimodal Fusion (MMF)**\n\n- *Motivation:*\nEHR and CXR data are temporally misaligned: EHRs offer continuous, high-frequency information, while CXRs are obtained infrequently and at irregular intervals. Moreover, different granularities in EHR data provide distinct and complementary clinical insights: local patterns can indicate acute infections, while global patterns reflect overall progression trends [3].\n\n- *How MMF Works:* \nMMF aligns CXR and EHR features at both local (pairwise) and global (sequence-wide) EHR scales.  The local fusion aligns consecutive CXR pairs with interval EHR data, while the global fusion aligns the semantics of the entire CXR and EHR sequences. \n\n- *Empirical Support:*\nCompared to simple fusion (concatenation + multihead attention), MMF achieves relative improvements of 2.5% in AUPRC for mortality prediction and 5.8% in accuracy for length-of-stay prediction (Table 3 in the paper).\n\n[3] Hanauer, D. A., et al. \"Opportunities and challenges in association and episode discovery from electronic health records.\" IEEE Intelligent Systems-2011.\n\n**(4) Comparison to Alternative Strategies**\n\nDiPro also consistently outperforms alternative multimodal fusion methods, including UTDE (uniform temporal encoding) and MedFuse (simple LSTM fusion with concatenation), as shown in Table 2 in the paper. Also, DiPro excels at modeling disease progression in sequential CXRs by achieving a 16.5% relative improvement in F1 for disease identification over CheXRelNet (which models region-disease co-occurrence in a CXR pair using graphs; see Table 1 in the pape).\n \n---\n\n### **W2:  Broader Baseline Comparison with Large Vision-Language Models**\n\nTo provide a comprehensive evaluation, we include recent large-scale vision-language models (VLMs) which are capable of processing multi-image inputs(e.g., Gemma3, QWen2.5VL, and Med-Flamingo [4]) as baselines for disease progression identification. We input consecutive CXR pairs and use a few-shot prompting approach (following [4]) to have the VLMs predict progression status for thoracic conditions (results in the following Table A). DiPro achieves notably higher accuracy with lower computational cost, highlighting the advantages of our design: spatialtemporal disentanglement (STD) and progression-aware enhancement (PAE) for clinical prediction.\n\n**Table A:** Performance Comparison on Disease Progression Identification Task between VLMs and DiPro.\n| Model        | Precision (P) | Recall (R) | F1 Score | Throughput (samples/s) |\n|--------------|---------------|------------|----------|-------------------------|\n| DiPro(Uni-CXR) (Ours)       | **0.475**         | **0.452**     | **0.453**    | **43.38**                   |\n| Gemma3       | 0.329         | 0.328      | 0.279    | 0.088                   |\n| QWen2.5VL    | 0.304         | 0.330      | 0.251    | 0.097                   |\n| Med-Flamingo | 0.355         | 0.345      | 0.301    | 0.234                   |\n\n\n[4] Moor, Michael, et al. \"Med-Flamingo: a multimodal medical few-shot learner.\" ML4H-2023.\n\n---\n\n### **W3: Efficiency, Inference Cost, and Generalizability to Other Modalities and Datasets**\n\nWe summarize the computational efficiency and inference cost comparison between DiPro and baseline models in the following table:\n\n**Table B:** Computational efficiency and inference cost comparison between DiPro and baseline models.\n| Model           | Params (M) | FLOPs (G) | MACs (G) | Latency Mean (ms) | Throughput (samples/s) | Performance of Disease Progression (F1)|\n|-----------------|------------|-----------|----------|-------------------|-------------------------|-------------------------|\n| **DiPro**(Ours)           | 31.06      | 82.75     | 41.37    | 23.05             | 43.38                   |0.466±0.018|\n| **CheXRelNet**      | 49.15      | 90.90     | 45.45    | 27.34             | 36.58                   |0.382±0.016|\n| CheXRelFormer   | 49.13      | 19.95     | 9.98     | 14.86             | 67.28                   |0.352±0.021|\n| SDPL            | 34.24      | 9.45      | 4.73     | 13.35             | 74.90                   |0.393±0.010 |\n| UTDE            | 6.69       | 1.61      | 0.80     | 5.24              | 190.90                  |0.352±0.013|\n| UMSE            | 23.99      | 8.27      | 4.14     | 9.65              | 103.62                  |0.449±0.005|\n| MedFuse         | 27.21      | 8.28      | 4.14     | 9.78              | 102.23                  |0.409±0.042|\n| DrFuse          | 56.40      | 16.45     | 8.23     | 19.93             | 50.18                   |0.429±0.010|\n\n\nAs shown in Table B above, compared to the ChexRelNet baseline which is also an anotomical region-based model, DiPro reduces FLOPs by 8.9% and latency by 15.6% while achieving relative gain of 16.5% in F1 for disease progression.\n\nWe appreciate the reviewer's thoughtful comment regarding generalizability. We fully agree that broader validation is important and would greatly enhance the clinical applicability of our approach. Currently, MIMIC-CXR and MIMIC-IV remain the only large-scale public datasets that include both longitudinal chest X-rays and paired EHR data. Nevertheless, we are actively working to broaden our evaluation by (1) partnering with clinicians to curate new multimodal longitudinal datasets, and (2) adapting DiPro’s approach to other imaging modalities, such as CT and MRI, in our ongoing and future research."}, {"review_id": "aQMvWDQnIX", "reviewer": "Reviewer_qVGM", "summary": "The authors proposed a novel framework called DiPro that provides comprehensive diagnosis over temporal sequence of data. The primary contributions are region-aware disentanglement and multi-timescale alignment. The author did extensive experiments which proved that DiPro achieves state-of-the-art performance on both disease progression modeling and ICU-related prediction tasks.", "questions": "1. I wonder whether the authors did some kind of registrations across temporal dimension. Even for the same patient, the exact position when X-ray images are taken may not be aligned, and difference in the position may cause unintended affect on the feature embeddings. \n\n2. For the current ternary disease progression prediction, it looks like the model only need one previous sample to make the prediction, and the whole longitudinal isn’t very useful? I think the paper will benefit from new experiments that encourage long range analysis, i.e. referring to a particular data point a few samples back.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The task (Longitudinal disease analysis progression) is an important task. The patient history play a big rolein disease analysis, especially for chronic diseases. And patient history are highly multimodal, as they combine clinical images with time series data like ECG. \n- The progression-aware enhancement is an interesting way to augment the training data. Reversing the input order while reversing the label allows the model to be trained on twice the amount of data available from the training dataset.\n\nWeakness:\n- While the author have done extensive experiments on CXR tasks, disease progression is a much larger topic that spans across multiple domains. It’s a bit too broad to claim “Multimodal Disease Progression Modeling” in the title while the paper focus solely on CXR and EHR. \n- Similarly, the disease progression task is limited as it only involves ternary prediction of worsened, stable and improved. A more detailed analysis (i.e. the progression of the size of the abnormal area, whether new abnormalities emerge) would be more interesting.", "was_revised": true, "final_justification": "Thanks authors for the clarification. While the authors acknowledged the weakness 1/2 mentioned above, they noted that these couldn't be addressed under the current setting and left them as future works. Therefore, I'm keeping my current score.", "rebuttal": "We sincerely appreciate the reviewer's positive feedback regarding both the importance of longitudinal disease progression analysis and the strengths of our approach, as well as the insightful suggestions. We address each point below:\n\n\n### **W1: Overclaim on \"Multimodal Disease Progression Modeling\" with EHR and CXR only**\n\nWe appreciate the reviewer's feedback and the opportunity to clarify our use of the term *\"Multimodal Disease Progression Modeling\"*. Our study addresses two types of clinical tasks: (1) general ICU prediction, which follows a well-established multimodal setup in line with prior works such as DrFuse [1] and MedFuse [2]; and (2) disease progression identification, a task first introduced by CheXRelNet [3], which focused exclusively on the CXR modality. To our knowledge, our work is the first to integrate EHR data into this progression identification task, which motivates our use of the term \"multimodal\" in this specific context. \n\nWhile our current study focuses on CXR and EHR, we fully agree with the reviewer broader multimodal integration,including modalities such as ECG, CT, or MRI, is important and worthwhile. We are actively exploring these extensions as part of our ongoing and future work. Thank you again for the constructive and insightful comment.\n\n[1] Yao, Wenfang, et al. \"DrFuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency.\" AAAI-2024.\n\n[2] Hayat, Nasir, Krzysztof J. Geras, and Farah E. Shamout. \"MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images.\" ML4H-2022.\n\n[3] Karwande, Gaurang, et al. \"CheXRelNet: An anatomy-aware model for tracking longitudinal relationships between chest x-rays.\" MICCAI-2022.\n\n---\n\n### **W2: A more detailed analysis on disease progression**\n\nWe fully agree that a more fine-grained analysis of disease progression would offer valuable clinical insights and represents an important direction for future research. At present, the MIMIC-CXR dataset does not provide detailed annotations like abnormality size changes, which limits the granularity of our current study. That said, we recognize the importance of this limitation and are actively seeking collaborations with clinicians to curate richer annotations, with the goal of enabling more detailed progression modeling in future work.\n\n---\n\n### **Q1: Clarification on CXR Registration in Temporal Dimension**\n\nIn this study, we do not perform explicit longitudinal registration of CXR images. To mitigate potential positional misalignment over time, we follow the preprocessing strategy of CheXRelNet: we include only AP-view CXRs and apply cropping based on anatomical bounding boxes from Chest ImaGenome dataset. This preprocessing allows us to focus on capturing the semantic-level disease progression within consistent anatomical structure, rather than relying on pixel-level alignment. As a result, our model is designed to be more robust to positional variability across different time points.\n\n---\n\n### **Q2: Long-Range Analysis for Disease Progression Modeling**\n\nWe thank the reviewer for raising this important point. For the disease progression identification task, we follow established evaluation protocols from prior works (CheXRelNet and SDPL), which focus on consecutive CXR pairs. However, we would like to clarify that our framework is not inherently limited to two-image analysis. In the ICU prediction task, our model utilizes all available CXRs (>= 2 images) within the first 48 hours of each patient's ICU stay. This setup enables the model to capture longer-range temporal patterns and  and richer clinical trajectories. As shown in Table 2 in the paper, incorporating these long-range longitudinal cues yields performance improvements: relative gains of 3.8\\% in F1 (disease progression), 2.8\\% in AUPRC (mortality prediction), and 3.0\\% in accuracy (length-of-stay prediction) compared to the respective best-performing baselines."}, {"review_id": "QnVmgB1zVd", "reviewer": "Reviewer_Darn", "summary": "This paper introduces DiPro, a novel framework designed to model disease progression using longitudinal chest X-rays (CXRs) and electronic health records (EHRs). The authors address two main challenges in CXR-based disease progression modeling: redundancy in sequential CXRs due to static anatomical features, and temporal misalignment between sparse imaging and dense EHR data. In particular, DiPro first disentangles static and dynamic features in CXRs, and enhances temporal modeling with progression-aware feature reversal. Then this model aligns multimodal data through local and global fusion mechanisms. Evaluated on MIMIC datasets, DiPro achieves better performance in disease progression detection and general ICU prediction tasks (mortality and length-of-stay), outperforming prior unimodal and multimodal baselines.", "questions": "- Your method relies on region-level annotations from Chest ImaGenome for spatiotemporal disentanglement. How would DiPro perform in the absence of such fine-grained labels? Can it generalize to datasets lacking anatomical bounding boxes? A suggestion is to conduct an ablation study with automated region (e.g., via segmentation models or weak supervision) to evaluate performance in a more realistic setting without curated annotations.\n- How does DiPro handle missing or sparse data, especially when sequential CXRs are limited or when EHR time series are incomplete? The authors can provide experiments or analysis on robustness to missing modalities or irregular sampling (e.g., random drop of CXRs or EHR segments), possibly comparing to models that address this explicitly.", "limitations": "- This work overlooks the use of clinical and radiology notes, which are critical sources of information for disease progression modeling. Prior CXR-based progression studies frequently incorporate free-text reports to provide rich semantic context and support supervision or contrastive learning. The exclusion of textual data in this framework limits its ability to capture nuanced clinical insights and may reduce its applicability in real-world multimodal clinical settings. Incorporating or at least comparing against text-based models would strengthen the study’s completeness and relevance.\n- The spatiotemporal disentanglement module requires region-level bounding box annotations (from Chest ImaGenome), which are costly and may not be available in typical clinical datasets, limiting scalability and generalizability.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a principled approach to disentangle static anatomical structures from dynamic pathological changes in sequential CXRs. This is essential to isolate clinically meaningful disease progression signals from irrelevant, redundant information.\n- Code is provided for reproducibility purposes.\n\nWeaknesses:\n\n- This works focuses only on the MIMIC dataset with no validation on external or multi-institutional cohorts, raising concerns about the model’s generalizability across different diseases or healthcare systems.\n- Real-world clinical data is often incomplete, but the paper does not explore robustness to missing CXRs or EHR segments, despite prior baselines (e.g., UMSE, DrFuse) explicitly addressing this issue.\n- The assumption that static and dynamic features can be cleanly orthogonalized may be too idealistic. In real-world imaging, pathological and anatomical features often interact in complex ways, possibly undermining the strict separation imposed by the orthogonality constraint.", "was_revised": true, "final_justification": "The authors conducted additional experiments on missing data and automatic segmentation. I appreciate the effort to strengthen the paper. However, I continue to believe that clinical and radiology notes are important resources for modeling disease progression, and the justification for excluding them remains unconvincing. That said, this is a quite solid paper overall, and I find that the strengths outweigh the weaknesses. Accordingly, I decide to maintain my positive score.", "rebuttal": "We thank the reviewer for the constructive feedback and the recognition of our principled approach and code release. We address each point in detail below:\n\n---\n\n### **W1: Generalizability to External or Multi-institutional Cohorts**\n\nWe thank the reviewer for the insightful comment. We fully agree that external validation across diverse cohorts is valuable for robust clinical application. Currently, MIMIC-CXR and MIMIC-IV remain the only large-scale public datasets providing both longitudinal CXRs and paired EHR data. We are actively collaborating with clinicians for external and cross-cohort validation as part of our ongoing and future work.\n\n---\n\n### **W2&Q2: Robustness to Missing Data**\n\nWe sincerely appreciate the reviewer’s constructive suggestion. In response, we conducted additional experiments where EHR data was randomly dropped during training at rates of 25%, 50%, and 75%  (following [1]), while testing was performed on the complete dataset. We will include these results in the revised manuscript. Notably, while there is a natural decrease in performance as the EHR missing rate increases, DiPro consistently outperforms all established baselines across all missing-data scenarios, demonstrating its robustness in handling incomplete inputs (see Tables 1a and 1b below).\n\n**Table 1a:** Length of Stay Classification (Accuracy)  \n| **Method** | **75% Missing** | **50% Missing** | **25% Missing** | **0% Missing** |  \n|------------|-----------------|-----------------|-----------------|----------------|  \n| **DiPro**(Ours)  | **0.399 ± 0.011** | **0.391 ± 0.021** | **0.415 ± 0.005** | **0.440 ± 0.007** |  \n| MLHC       | 0.339 ± 0.029   | 0.376 ± 0.005   | 0.396 ± 0.009   | 0.410 ± 0.013   |\n| ICML       | 0.377 ± 0.015   | 0.386 ± 0.012   | 0.384 ± 0.034   | 0.400 ± 0.021   |\n| DrFuse     | 0.340 ± 0.009   | 0.360 ± 0.010   | 0.363 ± 0.011   | 0.360 ± 0.011   |\n| MedFuse    | 0.332 ± 0.039   | 0.360 ± 0.033   | 0.360 ± 0.025   | 0.412 ± 0.027   |  \n\n**Table 1b:** Mortality Prediction (AUPRC)  \n| **Method** | **75% Missing** | **50% Missing** | **25% Missing** | **0% Missing** |  \n|------------|-----------------|-----------------|-----------------|----------------|  \n| **DiPro**(Ours)  | **0.696 ± 0.011** | **0.718 ± 0.012** | **0.751 ± 0.006** | **0.742 ± 0.003** |\n| MLHC       | 0.648 ± 0.009   | 0.685 ± 0.010   | 0.686 ± 0.051   | 0.712 ± 0.028   |\n| ICML       | 0.603 ± 0.052   | 0.673 ± 0.020   | 0.697 ± 0.021   | 0.710 ± 0.019   |\n| DrFuse     | 0.602 ± 0.024   | 0.663 ± 0.024   | 0.674 ± 0.077   | 0.684 ± 0.008   |\n| MedFuse    | 0.609 ± 0.057   | 0.641 ± 0.026   | 0.705 ± 0.016   | 0.716 ± 0.018   |  \n\n\n[1] Wang, Hu, et al. \"Multi-modal learning with missing modality via shared-specific feature modelling.\" CVPR-2023. \n\n---  \n\n### **W3&L2: Concerns of Assumption of Orthogonality Between Pathological and Anatomical Features**\n\nWe thank the reviewer for this insightful comment.\n\nWe agree that in clinical practice, pathological and anatomical features can be interdependent and may not always be cleanly separable. However, in our framework, the orthogonality loss $\\mathcal{L} _{\\text{orth}}$ acts as a **soft regularizer**, not a hard constraint. The full objective is:\n\n$$\n\\mathcal{L} = \\lambda _{\\text{pred}} \\cdot \\text{CE}(\\hat{\\mathbf{y}}, \\mathbf{y}) + \\lambda _{\\text{orth}} \\mathcal{L} _{\\text{orth}} + \\lambda _{\\text{temp}} \\mathcal{L} _{\\text{temp}} + \\lambda _{\\text{PAE}} \\mathcal{L} _{\\text{PAE}}\n$$\n\nWhile $\\mathcal{L} _{\\text{orth}}$ encourages disentanglement between static (anatomical) and dynamic (pathological) features, the cross-entropy (CE) loss ensures that predictive accuracy remains the primary learning driver(e.g., $\\lambda _{\\text{pred}}=2, \\lambda _{\\text{orth}}=1$ in the disease progression task). If strong coupling between the two are necessary for accurate prediction, the model learns to **relax the disentanglement** to prioritize clinical relevance.\n\n---\n\n### **Q1&L2: Robustness with Automated Bounding Boxes**\n\nWe sincerely appreciate the reviewer's valuable suggestion. To assess DiPro’s robustness to automated region annotations, we conducted an ablation study replacing Chest ImaGenome bounding boxes with those generated by the automated region detection model from RGRG [2].  We will add these results in the ablation study of revised manuscript. As shown in Table 2 (see below), while using automated bounding boxes results in a slight performance drop compared to curated annotations, DiPro consistently outperforms all baselines across disease progression (0.453±0.009 in F1), mortality (0.722±0.039 in AUPRC), and length-of-stay (0.427±0.014 in ACC) tasks. This demonstrates that DiPro remains effective even in the absence of manually curated labels, supporting its applicability to datasets lacking fine-grained anatomical annotations. \n\n**Table 2:** Performance comparison of DiPro with different bounding box sources.  \n| **Bbox Source**       | **Disease Progression (F1)** | **Mortality (AUPRC)** | **Length of Stay (ACC)** |\n|-----------------------|-----------------------------|-----------------------|--------------------------|\n| Chest ImaGenome       | 0.466 ± 0.018               | 0.742 ± 0.003         | 0.440 ± 0.007            |\n| Automated (RGRG [2])  | 0.457 ± 0.010               | 0.736 ± 0.021         | 0.430 ± 0.006            |\n\n\nThese results highlight that while more accurate anatomical annotations can improve prediction performance, DiPro still achieves strong and generalizable results even with fully automated region proposals.\n\n[2] Tanida, Tim, et al. \"Interactive and explainable region-guided radiology report generation.\" CVPR-2023.\n\n---  \n\n### **L1: On the Exclusion of Clinical and Radiology Notes** \n\nWe thank the reviewer for pointing out the importance of clinical and radiology notes in disease progression modeling. We agree that free-text information can provide rich semantic context and enhance model performance. In this work, we followed the modality setting and problem definition established in prior studies such as CheXRelNet, MedFuse, and DrFuse, focusing on CXR images and structured EHR data to enable fair and direct comparison. This design choice also reflects a practical motivation: enabling timely predictions from raw, automatically collected data without depending on the availability of clinical notes, which may be delayed in urgent care settings. We acknowledge the value of integrating clinical notes and plan to explore this direction in future work to further improve clinical relevance and applicability."}, {"review_id": "WCzTA5Trsa", "reviewer": "Reviewer_FTML", "summary": "DiPro proposes a multimodal EHR + imaging model that can better handle longitudinal images through tracking changes in images over time. It tries to separate the signal from the images into features that are constant and features that change over time.", "questions": "How were hyperparameters selected for the baselines?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Novel method for solving an important problem (longitudinal multimodal prediction)\n- Good selection of baselines\n\nWeaknesses:\n- The primary weakness of this paper is the discarding of all MIMIC visits with only one CXR. This likely causes a number of negative side effects, including weird sampling artifacts and an artificially limited training set size. Especially when most of the baselines were originally designed with much larger training sets.\n- There appear to be a lot of missing details for how many of the models were setup / parameters were explored. For example, for the EHR transformer, how many layers were used, what was the hidden size, etc, etc.\n- The standard deviations reported in table 3 are quite high, especially for mortality and length of stay. I'm a bit worried some of the analysis of that table might be reading into noise, more than signal.\n- In table 2, you are reporting a length of stay accuracy of 28.4% for DrFuse, which seems implausibly low since a majority class model that returns [2,3) should have an accuracy around 29.2% according to table 12? What steps did you take to debug and make sure this baseline was running correctly? What hyperparameter search did you do for it?\n- I don't see a hyperparameter search grid for the baselines?\n- The hyperparameter grid searches are missing 0 weights for lambda_orth, lambda_temp and lambda_PAE. If 0 was included, that would be a better test of whether those additional losses are useful.\n- It would be good to also report the final selected hyperparameters for some of the tasks to give a sense of how strong those other penalties need to be", "was_revised": true, "final_justification": "The authors have addressed all of my concerns. This is solid paper that improves performance on sequential CXR prediction, with no major concerns.", "rebuttal": "We thank the reviewer for acknowledging the novelty of our method and the importance of longitudinal multimodal prediction problem, as well as for the thoughtful feedback on our implementation and analysis. Below we address each point in detail:\n\n### **W1: Concerns about Single-CXR Visit Exclusion**\n\nWe thank the reviewer for highlighting this important point. We agree that excluding single-CXR visits (which account for ~30% of visits with CXR) may introduce sampling bias, and we will explicitly discuss this limitation in the revised manuscript. Consistent with CheXRelNet, our study models disease progression between consecutive CXRs, requiring at least two CXRs per patient for longitudinal analysis.\n\nWhile this reduces the overall cohort size and may affect comparisons with models trained on larger cohorts, it ensures fair evaluation across baselines within the defined progression task. We appreciate the reviewer’s suggestion and acknowledge that developing progression modeling approaches from a single CXR is a promising direction, which we plan to explore in future work.\n\n---\n\n### **W2: Details on Model Architecture and Hyperparameters**\n\nWe appreciate the opportunity to clarify our model setup and hyperparameters. Table A provides a detailed breakdown of DiPro’s architecture by module, which will be included in the revised manuscript for improved transparency.\n\n**Table A**: Architecture and Hyperparameters of DiPro.  \n| Module                | Structure / Type             | Key Parameters / Details                                |\n|-----------------------|------------------------------|--------------------------------------------------------|\n| **CXR Processing**    | Shared ResNet-50            | Pretrained on ImageNet; extracts 2048-dim features     |\n|                       | MLP (feature adjustment)    | Reduces 2048 → 256 (task-specific hidden size)         |\n|                       | Projection heads ($f_s$, $f_d$) | MLPs for static/dynamic features; hidden size 256      |\n| **EHR Processing**    | Global EHR Encoder          | 1-layer transformer; feedforward dim 1024, hidden size 256  |\n|                       | Time Embedding (MLP)        | Encodes EHR timesteps relative to CXR interval, 256    |\n|                       | Local EHR Encoder           | MultiheadAttention (1 layer, 8 heads, hidden 256)      |\n|                       | Demographics Encoder        | MLP, hidden size 256                                   |\n| **Multimodal Fusion** | Local Fusion                | MultiheadAttention (1 layer, 8 heads, hidden 256)      |\n|                       | Global Fusion               | Cross-attention (hidden 256), 1-layer transformer      |\n|                       | Static Features Fusion      | Group embedding (group_num = 8)                       |\n| **Prediction Head**   | MLP                         | Fully connected, task-specific output size             |\n\n---\n\n### **W3: Concerns About High Variance in Table 3 Results**\n\nWe thank the reviewer for highlighting the variability in our ablation study results. Given three experimental runs (random seeds), we report effect sizes (Cohen’s d) to quantify differences between DiPro and its ablation variants, as they provide a robust and interpretable measure of significance in small-sample scenarios [1]. Effect sizes are defined as: **d ≥ 1.3** (very large), **d ≥ 0.8** (large), **d ≥ 0.5** (medium), and **d < 0.2** (small).  \n\nTable B shows that key comparisons consistently exhibit **d > 0.8** (large effects), with most exceeding *1.5*, indicating the differences are meaningful rather than statistical noise. Additionally, variation in effect sizes highlights the relative contributions of modules; e.g., for Disease Progression, the PAE module is notably more critical than the MMF module.\n\n**Table B**: Ablation study results with performance metrics (mean±std) and effect sizes (Cohen’s d, in parentheses) comparing each variant to the full DiPro model.  \n**Note:** d ≥ 1.3 indicates a very large difference; d ≥ 0.8, large; d ≥ 0.5, medium; d < 0.2, small [1].\n| ID    | STD | PAE | MMF | Disease Progression (F1)       | Mortality (AUPRC)           | Length of Stay (ACC)       |\n|--------|--------|--------|--------|--------|--------|--------|\n| Dipro | ✓   | ✓   | ✓   | **0.466±0.018**               | **0.742±0.003**            | **0.440±0.007**            |\n| A1    | ✓   | ✓   | ✗   | 0.460±0.014 (0.37)          | 0.724±0.015 (1.66)       | 0.416±0.027 (1.22)       |\n| A2    | ✓   | ✗   | ✓   | 0.433±0.017 (1.88)          | 0.730±0.029 (0.58)       | 0.432±0.018 (0.59)       |\n| A3    | ✓   | ✗   | ✗   | 0.439±0.007 (1.98)          | 0.694±0.016 (4.17)       | 0.404±0.014 (3.25)       |\n| A4    | ✗   | ✗   | ✗   | 0.362±0.016 (6.11)          | 0.721±0.036 (0.82)       | 0.425±0.031 (0.67)       |\n\n[1] Sullivan, Gail M., and Richard Feinn. \"Using effect size—or why the P value is not enough.\" Journal of graduate medical education 4.3 (2012): 279-282.\n\n---\n\n### **W4: Correction of Reported DrFuse Result in Table 2**\n\nWe thank the reviewer for carefully pointing out this issue. There was a typo in Table 2: DrFuse's length-of-stay accuracy was reported as 28.4% but should be 36.0%, as correctly shown in Appendix Table 6. We apologize for this oversight and have conducted a thorough review of all numerical results to ensure accuracy in the revised version.\n\n### **W5&Q1: Hyperparameter Search Grid for the Baselines**\n\nWe appreciate the opportunity to clarify the setting. We conducted a unified hyperparameter search for all baseline models using the following grid:\n- **Learning rates:** {1e-5, 4e-7, 1e-6, 1e-7}\n- **Dropout rates:** {0.1, 0.2, 0.3}\n- **Hidden dimensions:** {64, 128, 256, 320}\n\nAll other hyperparameters were retained specified in the original implementations provided by the source code of each baseline. We will explicitly include these details in the revised manuscript to improve clarity.\n\n---\n\n### **W6: Including Zero-Weight Ablation Study for Loss Penalties**\n\n\nWe thank the reviewer for suggesting ablation studies by setting each loss weight ($\\lambda _{\\text{orth}}$, $\\lambda _{\\text{temp}}$, $\\lambda _{\\text{PAE}}$) to zero. We conducted these experiments and report results in Table C. Key findings are:\n\n- **Temporal consistency loss ($\\lambda _{\\text{temp}}$):**  \n  Removing this loss significantly reduced performance on ICU tasks, showing that maintaining consistency of static anatomical features benefits prediction when patient states change rapidly. The ability to distinguish static and dynamic components is important for accurate predictions.\n\n- **Progression-Aware Enhancement loss ($\\lambda _{\\text{PAE}}$):**  \n  Setting this weight to zero resulted in a notable decrease in performance on the disease progression task. Since PAE is designed to enhance the model’s sensitivity to disease progression direction (by reversing CXR pairs), its absence impairs the model’s ability to capture progression-related features.\n\n- **Orthogonal disentanglement loss ($\\lambda _{\\text{orth}}$):**  \n  While its removal had minimal impact on mortality prediction, it caused substantial performance drops for disease progression and length-of-stay predictions. This highlights the benefit of disentangling dynamic pathological changes from static anatomical regions across consecutive CXRs.\n\n\n**Table C**: Ablation study of loss components by setting individual loss weights to zero.\n| ID   | $λ_{orth}$ |$λ_{PAE}$| $λ_{temp}$  | Disease Progression (F1) | Mortality (AUPRC) | Length of stay (ACC) |\n|------|------|------|------|------|------|------|\n| dipro| ✓      | ✓     | ✓      |  0.466±0.018          | 0.742±0.003   | 0.440±0.007     |\n| B1   |✗      | ✓      | ✓     |  0.448±0.004              | 0.749±0.028       | 0.415±0.008          |\n| B2   | ✓      | ✗     | ✓      |  0.441±0.013              | 0.708±0.038       | 0.432±0.005          |\n| B3   |✓      |  ✓     | ✗      |  -                        | 0.737±0.008       | 0.415±0.025          |\n  \n*Note*: '-' indicates that disease progression prediction couldn't compute $\\mathcal{L}_{\\text{temp}}$ as it requires at least 3 CXR images.\n\nThese results demonstrate that all three loss components contribute meaningfully and in distinct ways to the model’s overall performance across different tasks. We will include the table in the revised manuscript.\n\n---\n\n### **W7: Final Selected Penalty Weights for Each Prediction task**\n\nWe thank the reviewer for the opportunity to clarify this point. We report the final selected penalty weights for each prediction task in Table D below, which reveals the relative importance of different penalty components for each task. We will include to the revised manuscript to better illustrate the varying importance of each penalty on model performance.\n\n**Table D**: Final selected Penalty weights for each prediction task.\n\n| Prediction Task      | $\\lambda _{pred}$ | $\\lambda _{orth}$ | $\\lambda _{temp}$ | $\\lambda _{PAE}$ |\n|----------------------|------------------|------------------|------------------|--------------------|\n| In-hospital Mortality| 6                | 0.1              | 1                | 0.1             |\n| Length of Stay       | 10               | 0.001            | 0.1              | 0.1             |\n| Disease Progression  | 2                | 1                | -               | 2               |\n\n*Note*: '-' indicates that $\\mathcal{L} _{\\text{temp}}$ is not applicable to disease progression task as it requires at least 3 CXR images.\n\n---\n\n**In summary, we will update the manuscript and supplementary material to provide:**\n- Complete model and baseline implementation details,\n- Explicit hyperparameter search grids and ablation protocols (including zero-weight ablations), and\n- Effect size analysis to complement standard deviations.\n\nWe are grateful for the reviewer’s valuable suggestions, which will contribute to enhancing the clarity and quality of our work."}, {"review_id": "MqmngXJb4i", "reviewer": "Reviewer_Sm5w", "summary": "Longitudinal multimodal data, signified by chest X-Rays (CXR) and paired EHR, provides rich information on disease progression, but are rarely utilized. The authors identified and addressed two challenges, namely (1) the redundancy in CXR sequences, where the static anatomy overshadows the meaningful dynamics, and (2) the temporal misalignment between sparse CXR and dense EHR. The proposed method, DiPro, uses region-aware disentanglement and multi-timescale alignment to model coherent progression pathways. The authors conducted experiments on the MIMIC dataset, showing that DiPro has risen to a new SOTA on both disease progression identification and ICU prediction.", "questions": "1. This is not a concrete suggestion or action item, but rather an open-ended question to the authors. Do you think DiPro is applicable to diseases where the progression is known to be monotonic (for example, the situation only worsens but never gets better)? In those cases, the carefully designed label on progression status would likely lose its information. Additionally, the progression status label (worsen, stable, improved) does not seem to account for different severity/extent of the disease progression – how much you think DiPro is able to account for that?\n2. For actionable items, please refer to Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. I agree with the challenges the authors identify for modeling disease progression using multimodal longitudinal data.\n2. Among the three core modules, disentanglement of static and dynamic features when modeling disease progression is very intuitive and reasonable.\n3. The description of the method is straightforward and easy to follow. Figure 1 is nicely made.\n4. Quantitative results look fairly convincing. The ablation study clears my concern on whether the progressive-aware enhancement is overengineering.\n\nWeaknesses:\n1. In Table 2, it can be observed that for some baseline methods, using the longitudinal CXR cannot even outperform using the last available CXR, which is echoing the authors’ claim that in CXR sequences the static anatomical information may overshadow the dynamics. With that in mind, it would be better to show the result of the proposed DiPro when using the last available CXR and see if DiPro is able to better utilize the longitudinal data. \n2. In Table 2, since the authors showed Unimodal EHR results for DiPro, it would be nice to have both Unimodal CXR and CXR+EHR results. Also, it is a bit unclear whether “Longitudinal” means CXR or CXR+EHR as it is not explicitly labeled. Table 1 conveys this information better.\n3. For figure 2(a), it is really hard to see the “overlapping distributions” since most values are small. Consider log-scaling (or some less aggressive rescaling) the axis for better display.", "was_revised": true, "final_justification": "I appreciate the comprehensive results shown by the latest rebuttal efforts.\n\nAlthough this work lacks the excitement, I would like to follow the suggestion given by a senior AC: \"as long as the paper presents technically solid, incremental progress to the field that would benefit some researchers, we should accept the paper\". I have increased my rating to 5.", "rebuttal": "We thank the reviewer for their positive comments of our work, as well as for the constructive feedback and insightful questions. Please find our detailed responses below:\n\n---\n\n### **W1: On Evaluating DiPro’s Ability to Leverage Longitudinal Data Versus Last Available CXR**\n\nWe appreciate the reviewer’s agreement that longitudinal CXRs do not consistently improve general ICU prediction performance compared to using only the latest image, as the static anatomical information in CXR sequences may overshadow dynamic changes. We also thank the reviewer for suggesting that it would be valuable to report DiPro’s performance using only the last available CXR, in order to directly assess its ability to leverage longitudinal information. We fully agree that exploring disease progression modeling with only a single available CXR is an interesting and valuable future direction, and we plan to investigate this in future work.\n\nDiPro is specifically designed to model disease progression through temporal comparisons between two consecutive CXRs, following established settings in prior works like CheXRelNet and SDPL. Thus DiPro is not applicable in a single-image setting. Nevertheless, DiPro consistently outperforms all best-performing *multimodal longitudinal baselines* on general ICU tasks, achieving relative gains of 3.6% in AUPRC for mortality prediction and 6.4% in accuracy for length-of-stay prediction (see Table 2 in the paper). These results strongly demonstrate DiPro’s effectiveness in utilizing longitudinal CXR data.\n\n---\n\n### **W2: Clarification of \"Longitudinal\" in Table 2 and Unimodal CXR Results of DiPro**\n\nWe thank the reviewer for highlighting the potential confusion caused by the presentation in Table 2. We would like to clarify that in Table 2, \"Longitudinal\" specifically refers to *Longitudinal CXR + EHR*, while \"Last available\" refers to *Last available CXR + EHR*. We will make this explicit in the revised manuscript to improve clarity.\n\nAdditionally, DiPro's performance using unimodal CXR for general ICU prediction tasks is reported in Tables 6 and 7 of the appendix. These tables show that DiPro’s multimodal longitudinal results outperform both unimodal EHR and unimodal CXR, with relative performance gains of 4.2% in AUPRC for the mortality prediction task and 3.0% in accuracy for length-of-stay classification, compared to the best of the two unimodal models. These results demonstrate that DiPro can effectively integrate heterogeneous modalities and leverage their complementary information to improve clinical prediction.\n\n---\n\n### **Q1: DiPro's Applicability to Monotonic Disease Progression**\n\nWe sincely thank the reviewer for raising this interesting and practical question. We believe that DiPro remains highly valuable for modeling monotonic disease progression. While the PAE module (which reverses input order of CXR pairs to make the model more sensitive to progression direction) in DiPro may be less relevant in strictly monotonic cases, the core architectures of DiPro remain well-suited for capturing progression trends over time. \n\nNotably, our results show that even without PAE module, DiPro outperforms state-of-the-art sequential CXR disease progression specialists (e.g., SDPL), achieving a relative improvement of 10.2% in F1 score for the disease progression task (see Tables 1 and 3 in the paper). These findings suggest that DiPro’s spatiotemporal disentanglement and multimodal integration can effectively model progression dynamics, including those that follow a monotonic pattern.\n\n---\n\n### **Q1: DiPro's Ability to Capture Progression Severity/Extent**\n\nWe appreciate the reviewer’s insightful question regarding DiPro’s ability to capture progression severity and extent. We fully agree that a more fine-grained analysis of disease progression would offer valuable clinical insights and represents an important direction for future research. At present, the MIMIC-CXR dataset does not provide detailed annotations like abnormality size changes, which limits the granularity of our current study. That said, we recognize the importance of this limitation and are actively seeking collaborations with clinicians to curate richer annotations, with the goal of enabling more detailed progression modeling in future work.\n\nNevertheless, as ICU mortality and length of stay are highly correlated with disease progression severity [1], we believe that DiPro’s strong performance on these general ICU tasks, with relative gains of 2.8% in AUPRC for mortality prediction and 3.0% in accuracy for length-of-stay classification compared to the best-performing baselines, suggests that the model is able to capture underlying disease severity semantically.\n\n[1]Higgins, Thomas L., et al. \"Early indicators of prolonged intensive care unit stay: Impact of illness severity, physician staffing, and pre–intensive care unit length of stay.\" Critical care medicine 31.1 (2003): 45-51.\n\n---\n\n### **W3: Figure 2(a) Visualization Improvement**\n\nWe would like to greatly thanks the review's suggestion. Applying a log-scale or alternative axis rescaling will improve the visibility of overlapping distributions in Figure 2(a). We will update the figure accordingly in the revised manustcript.\n\n\n\nWe appreciate the reviewer’s thoughtful questions and suggestions, which will enhance the clarity and quality of our work. We will revise the manuscript to incorporate these clarifications and improvements."}]

Author rebuttal:

Meta-review: This paper proposes DiPro which aims at solving multimodal disease progression modeling task. Their approach relies on region-aware disentanglement and multi-timescale alignment. They disentangle static (anatomy) and dynamic (pathology progression) features in sequential chest X-rays (CXRs), focusing on disease-relevant changes. Next, they hierarchically align these static and dynamic CXR features with asynchronous EHR data via local (pairwise interval-level) and global (full-sequence) synchronization to model coherent progression pathways.

The reviewers have argued that this paper presents a meaningful improvement on the baseline accuracy and the problem is very practical. The use of techniques such as Disentangled Feature Learning, Multiscale Multimodal Fusion, and Progression-Aware Training is appreciated by the reviewers. The authors' response has addressed the reviewers concerns,
Decision: Accept (spotlight)