CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis
Authors: Fuying Wang, Feng Wu, Yihan Tang, Lequan Yu
Accepted at: acl25

CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis

Fuying Wang1*, Feng Wu1*, Yihan Tang1, Lequan Yu1

1Department of Statistics and Actuarial Science,
School of Computing and Data Science,
The University of Hong Kong
{fuyingw,fengwu96,hank-tang}@connect.hku.hk, lqyu@hku.hk
Correspondence: lqyu@hku.hk

Abstract
Integrating multimodal Electronic Health Records (EHR) data—such as numerical time series and free-text clinical reports—has great potential in predicting clinical outcomes. However, prior work has primarily focused on capturing temporal interactions within individual samples and fusing multimodal information, overlooking critical temporal patterns across patients. These patterns, such as trends in vital signs like abnormal heart rate or blood pressure, can indicate deteriorating health or an impending critical event. Similarly, clinical notes often contain textual descriptions that reflect these patterns. Identifying corresponding temporal patterns across different modalities is crucial for improving the accuracy of clinical outcome predictions, yet it remains a challenging task. To address this gap, we introduce a Cross-Modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data. Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings. To ensure rich cross-modal temporal semantics in the learned patterns, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss for cross-modal alignment, along with two reconstruction losses to retain core information of each modality. Evaluations on two clinically critical tasks—48-hour in-hospital mortality and 24-hour phenotype classification—using the MIMIC-III database demonstrate the superiority of our method over existing approaches. The code is anonymously available at https://github.com/HKU-MedAI/CTPD.

1 Introduction
The increasing availability of Electronic Health Records (EHR) presents significant opportunities for advancing predictive modeling in healthcare (Acosta et al., 2022; Wang et al., 2024). EHR data is inherently multimodal and time-aware, encompassing structured data like vital signs, laboratory results, and medications, as well as unstructured data such as free-text clinical reports (Kim et al., 2023). Integrating these diverse data types is crucial for comprehensive patient monitoring and accurate prediction of clinical outcomes (Hayat et al., 2022; Wang et al., 2023; Zhang et al., 2023b). However, the irregularity and heterogeneity of multimodal data present significant challenges for precise outcome prediction.

Existing approaches primarily address either the irregularity of data (Shukla and Marlin, 2019; Horn et al., 2020; Zhang et al., 2021, 2023b) or the fusion of multiple modalities (Huang et al., 2020; Zhang et al., 2020b; Xu et al., 2021; Kline et al., 2022), but they often neglect broader temporal trends that span across patient cases. These cross-modal temporal patterns, present in both structured and unstructured data, can provide high-level semantic insights into a patient’s health trajectory and potential risks (Conrad et al., 2018). As illustrated in Fig. 1, high-level semantic patterns related to medical conditions can emerge across multiple modalities. Capturing these patterns in a cross-modal, temporal manner is essential for improving predictive performance. Furthermore, critical temporal patterns in EHR data unfold at different time scales (Zhang et al., 2023a; Luo et al., 2020; Ma et al., 2020; Ye et al., 2020), yet existing methods struggle to capture variations across these granularities. For instance, sudden changes in vital signs—such as a sharp drop in oxygen saturation or a rapid heart rate increase—may indicate an acute health crisis. In contrast, longer-term trends, such as persistently high blood pressure or a gradual decline in respiratory function, may signal deteriorating health or an impending critical event. Effectively analyzing patient states across multiple time scales is crucial for comprehensive EHR modeling, yet remains an open challenge in current methodologies.
"while in the ew pt needed to be put on a arm when sats dropped into the low 80 's ;", "pt finds it difficult to take a deep breath ;",
"when pt removes the nrb mask she desats into the low 80 's ;", "best manager placed naso cannula on pt and humed mask at 90 % and sats have remained in the upper 90 's to 100 's ;", "pt tolerating the high o2 but will begin to wean down as possible ;", "pt has a sputum ordered and has not been collected as of this writing"

"pt remains alert and oriented times three ;", "pt has had increased cough noted and requested cough medicine ;", "given dextro-guifensin 10cc ;", "pt was requesting cough med with codeine which still needs to be ordered by the team ;",
"O2 saturation remains in the 90's ;", "dox decreased to 750mg iv bid after she was given her first dose ;", "pt c/o sob and still becomes doe ;", "she requested for me to increase the nc flow so she is now on 4l nc with good sat ;", "things are diminished throughout ;"

"pt c/o cough , non-productive , dry ;", "tesseon and robitussin w/o effect ;", "robittusin and codeine ordered and given w/ good effect ;", "r mid-to lower base very diminished ; md aware ;", "md aware w/ rest ;", "cough relieved w/ rest ;", "o2 weaned to 2l nc ; tolerating well ;", "pt verbalized feeling better , breathing better ;", "pt w/ gram + cocci ;", "bid ex x2 drawn this am"

![A line graph showing oxygen saturation and respiratory rate over time, with annotations at key points labeled Note 1, Note 2, and Note 3.](page_186_624_1077_384.png)

Figure 1: Motivation of our proposed CTPD: we visualized the time-series EHR with corresponding clinical notes in one ICU stay of the MIMIC-III dataset, and observed the temporal patterns across two modalities: Blue text highlights respiratory status. Oxygen requirements gradually decreased from 8L to 4L, and then to 2L nasal cannula, indicating steady respiratory improvement. Note that this pattern is also reflected from the time series. Green text captures cough progression and medication effects. Symptom relief was observed after administering Robitussin with codeine, demonstrating a delayed but positive response to treatment. Yellow text represents infection monitoring. The detection of Gram-positive cocci prompted blood culture collection (bld cx) for further evaluation, indicating active infection surveillance.

To address these limitations, we propose the Cross-modal Temporal Pattern Discovery (CTPD) framework, designed to extract meaningful temporal patterns from multimodal EHR data to improve the accuracy of clinical outcome predictions. The core innovation of our approach is a novel temporal pattern discovery module, which identifies corresponding temporal patterns (i.e., temporal prototypes) with meaningful semantics across both modalities throughout the dataset. This approach ensures that the model captures essential temporal semantics relevant to patient outcomes, providing a more comprehensive understanding of the data. To further enhance the quality of the learned temporal patterns, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss for aligning pattern embeddings across modalities, along with auxiliary reconstruction losses to ensure that the patterns retain core information of the data. Moreover, our framework incorporates a transformer (Vaswani et al., 2017)-based fusion mechanism to effectively fuse the discovered temporal patterns with timestamp-level representations from both modalities, leading to more accurate predictions. We evaluate CTPD on two critical clinical prediction tasks: 48-hour in-hospital mortality prediction and 24-hour phenotype classification, using the MIMIC-III database. The results demonstrate the effectiveness of our approach, which significantly outperforms existing methods, and suggest a promising direction for improving multimodal EHR analysis for clinical prediction.

2 Related Works

2.1 EHR Time-Series Data Analysis

EHR data is critical for clinical tasks such as disease diagnosis, mortality prediction, and treatment planning (Harutyunyan et al., 2019; Zhang et al., 2023b). However, its high dimensionality and irregular nature pose challenges for traditional predictive models (Rani and Sikka, 2012; Lee et al., 2017). Deep learning models, such as RNNs and LSTMs, are often used to capture temporal dependencies in EHR data (Hayat et al., 2022; Deldari et al., 2023), but they struggle with irregular time intervals due to their reliance on fixed-length sequences (Xie et al., 2021). To address this, some methods update patient representations at each time step using graph neural networks (Zhang et al., 2021), while others employ time-aware embeddings to incorpo-
rate temporal information (Qian et al., 2023; Zhang et al., 2023b). Despite these advancements, existing approaches still struggle to model high-level temporal patterns essential for accurate clinical outcome prediction.

2.2 Prototype-based Pattern Learning

Prototype-based learning identifies representative instances (prototypes) and optimizes their distance from input data in latent space for tasks like classification (Li et al., 2023a; Ye et al., 2024). This approach has been widely applied in tasks such as anomaly detection, unsupervised learning, and few-shot learning (Tanwisuth et al., 2021; Li et al., 2023b). Recently, it has been extended to time-series data (Ghosal and Abbasi-Asl, 2021; Li et al., 2023a; Yu et al., 2024), demonstrating its potential for detecting complex temporal patterns. Additionally, prototype-based learning offers interpretable predictions, which is essential for healthcare applications (Zhang et al., 2024). However, learning efficient cross-modal temporal prototypes for multimodal EHR data remains an unexplored problem, as irregular time series and multi-scale patterns present significant challenges for existing methods.

2.3 Multi-modal Learning in Healthcare

In healthcare, patient data is typically collected in various forms—such as vital signs, laboratory results, medications, medical images, and clinical notes—to provide a comprehensive view of a patient’s health. Integrating these diverse modalities significantly enhances the performance of clinical tasks (Hayat et al., 2022; Zhang et al., 2023b; Yao et al., 2024). However, fusing multimodal data remains challenging due to the heterogeneity and complexity of the sources. Earlier research on multimodal learning (Trong et al., 2020; Ding et al., 2022; Hayat et al., 2022) often rely on late fusion strategies, where unimodal representations are combined via concatenation or Kronecker products. While straightforward, these approaches often fail to capture complex inter-modal interactions, leading to suboptimal representations. Recent works have introduced transformer-based models that focus on cross-modal token interactions (Zhang et al., 2023b; Theodorou et al., 2024; Yao et al., 2024). While these models are effective at capturing inter-modal relationships, they often struggle to extract high-level temporal semantics from multimodal data, limiting the ability to achieve a comprehensive understanding of a patient’s health conditions.

3 Methodology

3.1 Problem Formulation

In practice, multimodal EHR datasets contain multiple data types, specifically Multivariate Irregular Time Series (MITS) and free-text clinical notes. We represent the multimodal EHR data for the \( i \)-th admission as \( \{(\mathbf{x}_{(i)}, t^{TS}_{(i)}), (n_{(i)}, t^{Text}_{(i)}), y_{(i)}\}_{i=1}^N \). Here \( \mathbf{x}_{(i)} \) represents the multivariate time series observations, with \( t^{TS}_{(i)} \) indicating their corresponding time points. The sequence of clinical notes is represented by \( n_{(i)} \), and \( t^{Text}_{(i)} \) denotes the time points of these notes. The variable \( y_i \) denotes the clinical outcomes to predict. For simplicity, we omit the admission index \( i \) in subsequent sections. The MITS \( \mathbf{x} \) comprises \( d_m \) variables, where each variable \( j = 1, ..., d_m \) has \( l^{TS}_{(j)} \) observations, with the rest missing. Similarly, each clinical note sequence \( n \) includes \( l^{Text} \) notes. Early-stage medical prediction tasks aim to forecast an outcome \( y \) for the admission \( i \) using their multimodal EHR data \( \{(\mathbf{x}, t^{TS}), (n, t^{Text})\} \), specifically before a certain time point (e.g., 48 hours) after admission. Here, both \( t^{TS} \) and \( t^{Text} \) are constrained within a fixed temporal window following admission (e.g., 48 or 24 hours). Apart from this shared temporal boundary, the two sets of timestamps are recorded independently and are not inherently correlated.

3.2 Encoding MITS and Clinical Notes

Here, we introduce our time series encoder \( E_{TS} \) and text encoder \( E_{Text} \), which separately encode MITS \( (\mathbf{x}, t^{TS}) \) and clinical notes \( (n, t^{Text}) \) into their respective embeddings \( z^{TS} \) and \( z^{Text} \in \mathbb{R}^{T \times D} \). Here \( T \) denotes the number of regular time points, and \( D \) denotes the embedding dimension.

For MITS, we utilize a gating mechanism that dynamically integrates both irregular time series embeddings \( e^{TS}_{imp} \) and imputed regular time series embeddings \( e^{TS}_{mTAND} \) following the approach in (Zhang et al., 2023b). Formally, the MITS embedding \( z^{TS} \) is computed as:

\[
z^{TS} = g \odot e^{TS}_{imp} + (1 - g) \odot e^{TS}_{mTAND}
\]

where \( g = f(e^{TS}_{imp} \oplus e^{TS}_{mTAND}) \in \mathbb{R}^{T \times D} \), \( f(\cdot) \) is a gating function implemented via an MLP, \( \oplus \) denotes the concatenation, and \( \odot \) denotes point-wise multiplication.

The regular time series \( e^{TS}_{imp} \) embedding is derived by applying a 1D convolution layer to the imputed time series. At each reference time point
Figure 2: CTPD overview: the input Multivariate Irregular Time Series (MITS) and clinical note sequences are first encoded into regular embeddings. We then introduce the Cross-Modal Temporal Pattern Discovery (CTPD) module to extract meaningful temporal semantics. The extracted temporal patterns, along with the timestamp-level embeddings from both modalities, are fused to generate the final predictions.

\( \alpha = 1, ..., T \), the imputed values are sourced from the nearest preceding values or replaced with a standard normal value if no prior data is available. Concurrently, mTAND (multi-time attention) (Shukla and Marlin, 2021) generates an alternative set of time series representations \( e_{mTAND}^{TS} \) with the same reference time points r with irregular time representations. Specifically, we leverage V different Time2Vec (Kazemi et al., 2019) functions \( \{ \theta_v(\cdot) \}_{v=1}^V \) to produce interpolation embeddings at each time point \( \alpha \), which are then concatenated and linearly projected to form \( e_{mTAND}^{TS}(\alpha) \in \mathbb{R}^D \).

For clinical notes, each of them is first encoded using Bert-tiny (a text encoder), where we extract the [CLS] token to represent this as a sequence of note-level embeddings \( e^{\text{Text}} \in \mathbb{R}^{T \times D} \). To tackle irregularity, we sort \( e^{\text{Text}} \) according to their timestamps \( T^{\text{Text}} \) and treat the pair \( (e^{\text{Text}}, T^{\text{Text}}) \) as a multimodal time series. Following the approach for time-series data, we leverage V different Time2Vec features \( \{ \theta_v \}_{v=1}^V \) to produce interpolation embeddings at each time point \( \alpha \):

\[
z^{\text{Text}} = \text{mTAND}(\alpha, T^{\text{Text}}, e^{\text{Text}})
\]

This yields temporally aligned text embeddings at each time point \( \alpha \), enabling effective interaction with time series data. Note that our choice of BERT-Tiny is based on prior work (Park et al., 2022) in multimodal EHR analysis, where its lightweight architecture has proven effective. Nonetheless, our framework can be seamlessly integrated with other text encoders.

3.3 Discover Cross-modal Temporal Patterns from Multimodal EHR

High-level temporal patterns in multimodal EHR data often encode rich medical condition-related semantics that are crucial for predicting clinical outcomes. However, previous works primarily focus on timestamp-level embeddings, frequently overlooking these important temporal patterns (Bahadori and Lipton, 2019; Xiao et al., 2023; Sun et al., 2024). Drawing inspiration from object-centric learning in the computer vision domain (Locatello et al., 2020; Li et al., 2021), we propose a novel temporal pattern discovery module to capture complex patterns within longitudinal data.

Considering the hierarchical nature (Yue et al., 2022; Cai et al., 2024) of time series data, the critical temporal patterns for EHR may manifest across multiple time scales. Consequently, our approach performs temporal pattern discovery on multi-scale time series embeddings.

Extracting Cross-modal Temporal Patterns. Owing to the correspondence within multi-modal data, our cross-modal temporal pattern discovery module focuses on extracting corresponding temporal patterns across both modalities for a better understanding of multimodal EHR. Starting with the time series embeddings \( z^{TS} \) in Eq. 1, we generate multi-scale embeddings (each scale can be divided by T) \( \{ z^{TS}_{(1)}, z^{TS}_{(2)}, z^{TS}_{(3)} \} \) using three convolutional blocks followed by mean pooling with a stride of 2 along the time dimension. Each resulting embedding corresponds to a different temporal scale, where the sequence lengths are fractions of the original length \( T \) (e.g., \( T, T/2, \) and \( T/4 \)). The concate-
nated multi-scale embedding \( \mathbf{z}_{\mathrm{MS}}^{\mathrm{TS}} \in \mathbb{R}^{T' \times D} \) serves as the diverse temporal representation, where \( T' = T + T/2 + T/4 = 1.75T \). Since \( T \) is set to either 48 or 24 in our experiments, \( T' \) remains an integer. We then enhance these embeddings by applying position encoding: \( \hat{\mathbf{z}}_{\mathrm{MS}}^{\mathrm{TS}} = \mathbf{z}_{\mathrm{MS}}^{\mathrm{TS}} + \mathrm{PE}(\mathbf{z}_{\mathrm{MS}}^{\mathrm{TS}}) \), \( \hat{\mathbf{z}}_{\mathrm{MS}}^{\mathrm{Text}} \in \mathbb{R}^{T' \times D} \), where \( \mathrm{PE}(\cdot) \) denotes the position embeddings in (Vaswani et al., 2017). Furthermore, to capture potential temporal patterns, we define a group of \( K \) learnable vectors as temporal prototypes, \( \mathbf{P}^{\mathrm{Shared}} \in \mathbb{R}^{K \times D} \), initially sampled from a normal distribution \( \mathcal{N}(\mu, \mathrm{diag}(\sigma)) \in \mathbb{R}^{K \times D} \) and refined during training. The shared prototype embeddings are designed to capture semantic-corresponding temporal patterns across modalities, respectively, with \( \mu \) and \( \sigma \) randomly initialized and subsequently optimized.

To extract temporal patterns, we first calculate the assignment weights \( \mathbf{W} \) between prototype embeddings and modality embeddings using a dot-product attention mechanism:

\[
\mathbf{W}^{\mathrm{TS}} = \mathrm{Attention}(\mathbf{P}^{\mathrm{Shared}}, \hat{\mathbf{z}}_{\mathrm{MS}}^{\mathrm{TS}}),
\]
\[
\mathbf{W}^{\mathrm{Text}} = \mathrm{Attention}(\mathbf{P}^{\mathrm{Shared}}, \hat{\mathbf{z}}_{\mathrm{Text}})
\] (3)

The Attention mechanism is defined as:

\[
\mathrm{Attention}(\mathbf{q}, \mathbf{k})_{i,j} = \frac{e^{M_{i,j}}}{\sum_{l} e^{M_{i,l}}},
\] (4)

where \( \mathbf{M} = \frac{1}{\sqrt{D}} g_q(\mathbf{q}) \cdot g_k(\mathbf{k})^T \) and \( g_q(\cdot) \) and \( g_k(\cdot) \) are two learnable matrices. Next, we aggregate the input values to their assigned prototypes using a weighted mean to obtain updated embeddings \( \mathbf{z}_{\mathrm{updated}}^{\mathrm{TS}} \) and \( \mathbf{z}_{\mathrm{updated}}^{\mathrm{Text}} \):

\[
\mathbf{z}_{\mathrm{updated}}^{\mathrm{TS}} = \mathbf{W}^{\mathrm{TS}} \cdot v(\hat{\mathbf{z}}_{\mathrm{MS}}^{\mathrm{TS}}) \in \mathbb{R}^{K \times D},
\]
\[
\mathbf{z}_{\mathrm{updated}}^{\mathrm{Text}} = \mathbf{W}^{\mathrm{Text}} \cdot v(\hat{\mathbf{z}}_{\mathrm{Text}}) \in \mathbb{R}^{K \times D},
\] (5)

where \( v(\cdot) \) is a learnable matrix.

Finally, the prototype embeddings \( \mathbf{P}^{\mathrm{TS}} \) and \( \mathbf{P}^{\mathrm{Text}} \) are refined using the corresponding updated embeddings via a learned recurrent function:

\[
\mathbf{P}^{\mathrm{TS}} = f(\mathrm{GRU}(\mathbf{P}^{\mathrm{TS}}_{(0)})) \in \mathbb{R}^{K \times D}
\]
\[
\mathbf{P}^{\mathrm{Text}} = f(\mathrm{GRU}(\mathbf{P}^{\mathrm{Text}}_{(0)})) \in \mathbb{R}^{K \times D}
\] (6)

where \( \mathbf{P}^{\mathrm{TS}}_{(0)} \) and \( \mathbf{P}^{\mathrm{Text}}_{(0)} \) are the prototype embeddings from the previous step , GRU(\cdot) is Gated Recurrent Unit (Cho et al., 2014), and \( f(\cdot) \) denotes MLP. The above process is repeated for 3 iterations per step. Those refined embeddings denote the discovered temporal patterns for each modality. Note that our design of attention and GRU is inspired by Slot Attention (Locatello et al., 2020), which has been shown to be effective in learning object-centric representations from images.

TP-NCE Constraint. To ensure consistent semantics across modalities, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss, inspired by InfoNCE (Oord et al., 2018), to enforce the similarity of multimodal prototype embeddings for the same ICU stay while increasing the distance between prototype embeddings from different ICU stays. For a minibatch of \( B \) samples, the TP-NCE loss from MITS to notes is defined as:

\[
\mathcal{L}_{\mathrm{TPNCE}}^{\mathrm{TS} \rightarrow \mathrm{Text}} = - \sum_{i=1}^{B} (\log \frac{\exp(\mathrm{sim}(i, i)/\tau)}{\sum_{j=1}^{B} \exp(\mathrm{sim}(i, j)/\tau)})
\] (7)

where \( \tau \) is a temperature and, \( i, j \) (\( 1 \leq i, j \leq B \)) denote sample indices within the minibatch. The similarity function \( \mathrm{sim}(i, j) \) measures the similarity between the \( i \)-th \( \mathbf{P}^{\mathrm{TS}} \) and \( j \)-th \( \mathbf{P}^{\mathrm{Text}} \), and is defined as (for convenience, we omit the indices \( i \) and \( j \) in the equation below):

\[
\mathrm{sim}(\cdot) = \sum_{k=1}^{K} (\beta_k < \mathbf{P}^{\mathrm{TS}}(k), \mathbf{P}^{\mathrm{Text}}(k) >)
\] (8)

where \( \langle \cdot \rangle \) denotes cosine similarity, and \( k \) is the prototype index. The bidirectional TP-NCE loss is then given by: \( \mathcal{L}_{\mathrm{TPNCE}} = \frac{1}{2} (\mathcal{L}_{\mathrm{TPNCE}}^{\mathrm{TS} \rightarrow \mathrm{Text}} + \mathcal{L}_{\mathrm{TPNCE}}^{\mathrm{Text} \rightarrow \mathrm{TS}}) \). To account for varying prototype importance, an attention mechanism is used to generate weights \( \beta \) for the slots, based on global MITS and text embeddings: \( \beta = \mathrm{MLP}(\mathrm{concat}([\mathbf{g}_{\mathrm{MS}}^{\mathrm{TS}}, \mathbf{g}_{\mathrm{Text}}^{\mathrm{TS}}])) \), where \( \mathbf{g}_{\mathrm{MS}}^{\mathrm{TS}} \) and \( \mathbf{g}_{\mathrm{Text}}^{\mathrm{TS}} \in \mathbb{R}^{D} \) are global embeddings obtained by averaging \( \hat{\mathbf{z}}_{\mathrm{MS}}^{\mathrm{TS}} \) and \( \hat{\mathbf{z}}_{\mathrm{Text}} \) along the time dimension.

Auxiliary Reconstruction. To ensure that the learned prototype representations capture core information from multimodal EHR data, we introduce two reconstruction objectives aimed at reconstructing imputed regular time series and text embeddings from the learned prototypes. Specifically, we implement a time series decoder to reconstruct the imputed regular time series from \( \mathbf{P}^{\mathrm{TS}} \), and a text embedding decoder to reconstruct text embeddings from \( \mathbf{P}^{\mathrm{Text}} \). Both decoders are based on a transformer decoder architecture (Vaswani et al., 2017), and two mean squared error (MSE) losses denoted by \( \mathcal{L}_{\mathrm{TS-Recon}} \) and \( \mathcal{L}_{\mathrm{Text-Recon}} \) are used
as the objective function. Here, we define the overall reconstruction loss \( \mathcal{L}_{\text{Recon}} = \frac{1}{2}(\mathcal{L}_{\text{TS-Recon}} + \mathcal{L}_{\text{Text-Recon}}) \).

3.4 Multimodal Fusion

Since information from both modalities is crucial for predicting medical conditions, we propose a multimodal fusion mechanism to integrate these inputs. First, we apply a 2-layer transformer encoder (Vaswani et al., 2017) to capture interactions between timestamp-level and prototype embeddings across both modalities for each sample. We continue to use \( \mathbf{P}^{\text{TS}} \) and \( \mathbf{P}^{\text{Text}} \) to represent the resulted prototype embeddings for time-series data and clinical notes, respectively. Similarly, \( \hat{\mathbf{z}}_{\text{MS}}^{\text{TS}} \) and \( \hat{\mathbf{z}}_{\text{Text}}^{\text{Text}} \) denote the corresponding timestamp-level embeddings. Then, we aggregate \( K \) prototype embeddings and \( T \) timestamp-level embeddings of each modality using an attention-based pooling mechanism:

\[
\mathbf{F}^{\text{TS}} = \sum_{k=1}^{K} \gamma_k^{\text{TS}} \mathbf{P}^{\text{TS}}(k) + \sum_{t=1}^{T} \phi_t^{\text{TS}} \hat{\mathbf{z}}_{\text{MS}}^{\text{TS}}(t)
\]
\[
\mathbf{F}^{\text{Text}} = \sum_{k=1}^{K} \gamma_k^{\text{Text}} \mathbf{P}^{\text{Text}}(k) + \sum_{t=1}^{T} \phi_t^{\text{Text}} \hat{\mathbf{z}}_{\text{Text}}^{\text{Text}}(t)
\]

Here \( k \) and \( t \) refer to the indices of prototype embeddings and timestamp-level embeddings, respectively. Here \( \gamma^{\text{TS}}, \phi^{\text{TS}}, \gamma^{\text{Text}}, \phi^{\text{Text}} \) are learned attention weights by passing the corresponding embeddings through a shared MLP. The resulting global embeddings from both modalities are concatenated along the feature dimension to form the final global representation.

3.5 Overall Learning Objectives

To optimize our framework, we employ four loss functions jointly. The overall objective is a weighted sum of these loss functions:

\[
\mathcal{L} = \mathcal{L}_{\text{pred}} + \lambda_1 * \mathcal{L}_{\text{TPNCE}} + \lambda_2 * \mathcal{L}_{\text{Recon}}
\]

where \( \lambda_1, \lambda_2 \) are hyperparameters that control the weights of respective losses. Here \( \mathcal{L}_{\text{pred}} \), is a cross-entropy loss used for classification.

4 Experiment

4.1 Experimental Setup

Dataset. We assess our model’s efficacy using MIMIC-III v1.4\footnote{https://physionet.org/content/mimiciii/1.4/}, a comprehensive open-source multimodal clinical database (Johnson et al., 2016). We focus our evaluation on two critical tasks, 48-hour in-hospital mortality prediction (48-IHM) and 24-hour phenotype classification (24-PHE), as established in prior research (Zhang et al., 2023b; Hayat et al., 2022). We extracted raw, irregular time-series data (containing 17 clinical variables) from the MIMIC-III database and selected time series within the first 48-hour and 24-hour windows within each ICU stay for each respective task, as done in (Zhang et al., 2023b; Hayat et al., 2022; Harutyunyan et al., 2019). ICU stays shorter than 48 or 24 hours were excluded from our dataset, which explains the different sample size between these two tasks. Unlike (Harutyunyan et al., 2019), we did not apply imputation during preprocessing but instead retained the original irregular time-series structure, following (Zhang et al., 2023b). Following the dataset splitting by (Harutyunyan et al., 2019), we ensure that the model evaluation is robust by partitioning the data into 70% training, 10% validation, and 20% testing sets, based on unique subject IDs to prevent information leakage. For multimodal analysis, we paired numerical time-series data with corresponding clinical notes from each patient’s ICU stay, consistent with (Zhang et al., 2023b). Note that the pipeline of processing clinical notes we used follows the practice in (Khadanga et al., 2019). The dataset statistics, including sample counts before and after multimodal paring, are presented in Table 1. The additional details of experimental setup can be found in Appendix A.

Table 1: Number of samples for EHR and fully paired EHR-clinical notes across the training, validation, and test sets.

<table>
  <tr>
    <th></th>
    <th>Training</th>
    <th>Validation</th>
    <th>Test</th>
  </tr>
  <tr>
    <th colspan="4">Number of EHR.</th>
  </tr>
  <tr>
    <td>48-IHM</td>
    <td>16,093</td>
    <td>1,810</td>
    <td>3,236</td>
  </tr>
  <tr>
    <td>24-PHE</td>
    <td>26,891</td>
    <td>2,955</td>
    <td>5,282</td>
  </tr>
  <tr>
    <th colspan="4">Number of Paired EHR and Clinical Notes.</th>
  </tr>
  <tr>
    <td>48-IHM</td>
    <td>15,425</td>
    <td>1,727</td>
    <td>3,107</td>
  </tr>
  <tr>
    <td>24-PHE</td>
    <td>25,435</td>
    <td>2,807</td>
    <td>5,013</td>
  </tr>
</table>

Evaluation Metrics. The 48-hour In-Hospital Mortality (48-IHM) prediction is a binary classification with a marked label imbalance, indicated by a death-to-discharge ratio of approximately 1:6. Following previous work (Harutyunyan et al., 2019; Zhang et al., 2023b), we use AUROC, AUPR, and F1 score, for a comprehensive evaluation. The 24-
Table 2: Comparison of our method with baselines on 48-IHM and 24-PHE tasks using the MIMIC-III dataset. We report average performance on three random seeds, with standard deviation as the subscript. The Best and 2nd best methods under each setup are bold and underlined.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <th colspan="7">Methods on MITS.</th>
  </tr>
  <tr>
    <td>CNN (LeCun et al., 1998)</td>
    <td>85.80<sub>0.32</sub></td>
    <td>49.73<sub>0.65</sub></td>
    <td>46.37<sub>3.01</sub></td>
    <td>75.36<sub>0.18</sub></td>
    <td>38.10<sub>0.26</sub></td>
    <td>40.80<sub>0.37</sub></td>
  </tr>
  <tr>
    <td>RNN (Elman, 1990)</td>
    <td>84.75<sub>0.74</sub></td>
    <td>46.57<sub>1.18</sub></td>
    <td>45.60<sub>0.06</sub></td>
    <td>73.78<sub>0.10</sub></td>
    <td>36.76<sub>0.27</sub></td>
    <td>33.99<sub>0.48</sub></td>
  </tr>
  <tr>
    <td>LSTM (Hochreiter, 1997)</td>
    <td>85.22<sub>0.67</sub></td>
    <td>46.93<sub>1.28</sub></td>
    <td>45.72<sub>1.41</sub></td>
    <td>74.46<sub>0.23</sub></td>
    <td>36.80<sub>0.28</sub></td>
    <td>39.45<sub>0.49</sub></td>
  </tr>
  <tr>
    <td>Transformer (Vaswani et al., 2017)</td>
    <td>83.45<sub>0.97</sub></td>
    <td>43.03<sub>1.65</sub></td>
    <td>39.31<sub>3.83</sub></td>
    <td>74.98<sub>0.14</sub></td>
    <td>39.37<sub>0.26</sub></td>
    <td>36.13<sub>1.42</sub></td>
  </tr>
  <tr>
    <td>IP-Net (Shukla and Marlin, 2019)</td>
    <td>81.76<sub>0.38</sub></td>
    <td>39.50<sub>0.83</sub></td>
    <td>43.89<sub>1.07</sub></td>
    <td>73.98<sub>0.13</sub></td>
    <td>35.31<sub>0.29</sub></td>
    <td>39.38<sub>0.15</sub></td>
  </tr>
  <tr>
    <td>GRU-D (Che et al., 2018)</td>
    <td>49.21<sub>2.16</sub></td>
    <td>12.85<sub>2.24</sub></td>
    <td>19.63<sub>0.01</sub></td>
    <td>52.11<sub>0.42</sub></td>
    <td>17.99<sub>0.55</sub></td>
    <td>26.17<sub>0.23</sub></td>
  </tr>
  <tr>
    <td>DGM-O (Wu et al., 2021)</td>
    <td>71.99<sub>7.30</sub></td>
    <td>28.67<sub>7.34</sub></td>
    <td>31.72<sub>11.02</sub></td>
    <td>60.70<sub>0.82</sub></td>
    <td>22.56<sub>0.77</sub></td>
    <td>28.87<sub>0.64</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2021)</td>
    <td>85.27<sub>0.20</sub></td>
    <td>49.82<sub>0.97</sub></td>
    <td>48.02<sub>1.93</sub></td>
    <td>72.79<sub>0.09</sub></td>
    <td>35.95<sub>0.14</sub></td>
    <td>32.42<sub>0.72</sub></td>
  </tr>
  <tr>
    <td>SeFT (Horn et al., 2020)</td>
    <td>65.00<sub>0.84</sub></td>
    <td>22.93<sub>1.27</sub></td>
    <td>19.70<sub>15.59</sub></td>
    <td>60.50<sub>0.09</sub></td>
    <td>23.57<sub>0.07</sub></td>
    <td>21.26<sub>0.21</sub></td>
  </tr>
  <tr>
    <td>UTDE (Zhang et al., 2023b)</td>
    <td>86.14<sub>0.45</sub></td>
    <td>50.60<sub>0.28</sub></td>
    <td>49.29<sub>0.62</sub></td>
    <td>73.62<sub>0.57</sub></td>
    <td>36.80<sub>0.87</sub></td>
    <td>40.58<sub>0.64</sub></td>
  </tr>
  <tr>
    <th colspan="7">Methods on Clinical Notes.</th>
  </tr>
  <tr>
    <td>Flat (Deznabi et al., 2021)</td>
    <td>85.71<sub>0.49</sub></td>
    <td>50.96<sub>0.19</sub></td>
    <td>45.80<sub>0.61</sub></td>
    <td>81.77<sub>0.09</sub></td>
    <td>53.79<sub>0.21</sub></td>
    <td>52.13<sub>0.79</sub></td>
  </tr>
  <tr>
    <td>HierTrans (Pappagari et al., 2019)</td>
    <td>84.32<sub>0.34</sub></td>
    <td>47.92<sub>0.37</sub></td>
    <td>42.01<sub>0.58</sub></td>
    <td>80.79<sub>0.06</sub></td>
    <td>51.97<sub>0.08</sub></td>
    <td>50.85<sub>0.71</sub></td>
  </tr>
  <tr>
    <td>T-LSTM (Baytas et al., 2017)</td>
    <td>85.70<sub>0.21</sub></td>
    <td>45.29<sub>0.86</sub></td>
    <td>42.84<sub>5.53</sub></td>
    <td>81.15<sub>0.04</sub></td>
    <td>50.23<sub>0.07</sub></td>
    <td>49.77<sub>0.24</sub></td>
  </tr>
  <tr>
    <td>FT-LSTM (Zhang et al., 2020a)</td>
    <td>84.28<sub>0.95</sub></td>
    <td>43.93<sub>1.03</sub></td>
    <td>36.87<sub>6.34</sub></td>
    <td>81.66<sub>0.12</sub></td>
    <td>51.71<sub>0.24</sub></td>
    <td>50.21<sub>0.85</sub></td>
  </tr>
  <tr>
    <td>GRU-D (Che et al., 2018)</td>
    <td>72.58<sub>0.42</sub></td>
    <td>26.38<sub>0.46</sub></td>
    <td>31.64<sub>0.79</sub></td>
    <td>49.52<sub>0.65</sub></td>
    <td>16.90<sub>0.12</sub></td>
    <td>27.80<sub>0.08</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2021)</td>
    <td>85.40<sub>0.60</sub></td>
    <td>49.68<sub>1.26</sub></td>
    <td>35.76<sub>11.57</sub></td>
    <td>82.14<sub>0.07</sub></td>
    <td>54.57<sub>0.15</sub></td>
    <td>52.01<sub>0.93</sub></td>
  </tr>
  <tr>
    <th colspan="7">Methods on Multimodal EHR.</th>
  </tr>
  <tr>
    <td>MMTM (Joze et al., 2020)</td>
    <td>87.88<sub>0.07</sub></td>
    <td>53.58<sub>0.24</sub></td>
    <td>51.54<sub>1.58</sub></td>
    <td>81.46<sub>0.25</sub></td>
    <td>51.88<sub>0.12</sub></td>
    <td>51.59<sub>0.19</sub></td>
  </tr>
  <tr>
    <td>DAFT (Pilsterl et al., 2021)</td>
    <td>87.53<sub>0.22</sub></td>
    <td>52.40<sub>0.21</sub></td>
    <td>51.95<sub>0.64</sub></td>
    <td>81.18<sub>0.08</sub></td>
    <td>50.91<sub>0.31</sub></td>
    <td>50.72<sub>0.39</sub></td>
  </tr>
  <tr>
    <td>MedFuse (Hayat et al., 2022)</td>
    <td>86.02<sub>0.29</sub></td>
    <td>51.00<sub>0.22</sub></td>
    <td>49.29<sub>0.75</sub></td>
    <td>78.88<sub>0.14</sub></td>
    <td>45.99<sub>0.21</sub></td>
    <td>47.47<sub>0.23</sub></td>
  </tr>
  <tr>
    <td>DrFuse (Yao et al., 2024)</td>
    <td>85.97<sub>1.02</sub></td>
    <td>49.94<sub>1.91</sub></td>
    <td>49.75<sub>5.32</sub></td>
    <td>80.88<sub>0.18</sub></td>
    <td>49.62<sub>0.40</sub></td>
    <td>50.18<sub>0.23</sub></td>
  </tr>
  <tr>
    <td>CTPD (Ours)</td>
    <td><b>88.15<sub>0.28</sub></b></td>
    <td><b>53.86<sub>0.65</sub></b></td>
    <td><b>53.85<sub>0.16</sub></b></td>
    <td><b>83.34<sub>0.05</sub></b></td>
    <td><b>56.39<sub>0.17</sub></b></td>
    <td><b>53.83<sub>0.43</sub></b></td>
  </tr>
</table>

Table 3: Statistics analysis of CTPD on MIMIC-III dataset. p-values are computed from paired t-tests.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <td>SOTA</td>
    <td>87.88<sub>0.07</sub></td>
    <td>53.58<sub>0.24</sub></td>
    <td>51.95<sub>0.64</sub></td>
    <td>82.14<sub>0.07</sub></td>
    <td>54.57<sub>0.15</sub></td>
    <td>52.13<sub>0.79</sub></td>
  </tr>
  <tr>
    <td>CTPD (Ours)</td>
    <td>88.15<sub>0.28</sub></td>
    <td>53.86<sub>0.65</sub></td>
    <td>53.85<sub>0.16</sub></td>
    <td>83.34<sub>0.05</sub></td>
    <td>56.39<sub>0.17</sub></td>
    <td>53.83<sub>0.43</sub></td>
  </tr>
  <tr>
    <td>Gains</td>
    <td>+0.27</td>
    <td>+0.28</td>
    <td>+1.9</td>
    <td>+1.2</td>
    <td>+1.82</td>
    <td>+1.7</td>
  </tr>
  <tr>
    <td>p values</td>
    <td>0.016</td>
    <td>0.233</td>
    <td>7.74e−6</td>
    <td>7.89e−12</td>
    <td>1.10e−10</td>
    <td>2.08e−4</td>
  </tr>
</table>

hour Phenotype Classification (24-PHE) involves predicting the presence of 25 different medical conditions during an ICU stay, making it a multi-label classification task. For this task, we employ the AUROC, AUPR, and F1 score (Macro) for a thorough assessment of model efficacy. The F1 score threshold is determined by selecting the value that maximizes the F1 score on the validation set.

Implementation Details. We train the model with batch size of 128, learning rate of 4e-5, and Adam (Kingma and Ba, 2014) optimizer. We use a cosine annealing learning rate scheduler with a 0.2 warm-up proportion. To prevent overfitting, we implement early stopping when there is no increase in the AUROC on the validation set for 48-IHM or 24-PHENO over 5 consecutive epochs. All experiments are conducted on 1 RTX-3090 GPU card using about 1 hour per run. We clip the norm of gradient values with 0.5 for stable training. By default, we use Bert-tiny (Turc et al., 2019; Bhargava et al., 2021) as our text encoder.

Compared Methods. To ensure a comprehensive comparison, we compare our CTPD with three types baselines: MITS-only approaches, note-only approaches and multimodal approaches. For MITS-only setting, we compare CTPD with 4 baselines for imputed regular time series: RNN (Elman, 1990), LSTM (Hochreiter, 1997), CNN (LeCun et al., 1998) and Transformer (Vaswani et al., 2017), and 5 baselines for irregular time series, including IP-Net (Shukla and Marlin, 2019), GRU-D (Che et al., 2018), DGM-O (Wu et al., 2021), mTAND (Shukla and Marlin, 2021), SeFT (Horn et al., 2020), and UTDE (Zhang et al., 2023b). The imputation ap-
proach follows the MIMIC-III benchmark (Harutyunyan et al., 2019). For the note-only setting, we compare our model with 6 baselines: Flat (Deznabi et al., 2021), HierTrans (Pappagari et al., 2019), T-LSTM (Baytas et al., 2017), FT-LSTM (Zhang et al., 2020a), GRU-D (Che et al., 2018), and mTAND (Shukla and Marlin, 2021). In the multi-modal setting, we compare our model with 4 baselines: MMTM (Joze et al., 2020), DAFT (Pölsterl et al., 2021), MedFuse (Hayat et al., 2022), and DrFuse (Yao et al., 2024). To ensure a fair comparison, we implement Bert-tiny (Bhargava et al., 2021; Turc et al., 2019) as the text encoder across all baselines. Details of baselines can be found in the Appendix B.

4.2 Comparison with SOTA Baselines

Results on MIMIC-III. Table 2 presents a comparison of our proposed CTPD against 3 types of baselines: MITS-based methods, clinical notes-based methods, and multimodal EHR-based methods. Our CTPD, which incorporates cross-modal temporal pattern embeddings, consistently achieves the best performance across all 6 metrics. Specifically, CTPD shows a 1.89% improvement in F1 score on the 48-IHM task, and a 1.2% improvement in AUROC and 1.92% in AUPR on the more challenging 24-PHE task, compared to the second-best results. Additionally, we have conducted statistical analysis in Table 3 to assess statistical significance. CTPD demonstrated significant gains over SOTAs (p value < 0.05) in 5 out of 6 settings, indicating its effectiveness in analyzing multimodal EHR.

Evaluation on More Tasks and Datasets. To further evaluate the generalizability of our CTPD framework, we have extended it to another important admission-level task: 30-day readmission prediction on MIMIC-III. Additionally, we also conducted experiments on the additional MIMIC-IV dataset to assess our framework’s adaptability to different data sources. These results can be found in Appendix C.1 and Appendix C.2.

Discussion on Missing Modalities and Noisy Data Scenarios. Currently, CTPD framework is built on paired time-series and clinical notes. In practice, the dataset might have missing modalities, such as partial clinical notes or time-series data are missed. We acknowledge this is an interesting research direction. However, the study of missing modalities falls outside the scope of our work and will be explored in future research. Additionally, our evaluation is conducted on MIMIC-III (Johnson et al., 2016), a large-scale, de-identified real-world dataset containing patient records from critical care units at Beth Israel Deaconess Medical Center between 2001 and 2012. Our approach is designed to be adaptable and can be seamlessly applied to other real-world databases.

4.3 Model Analysis

Ablation Results on Different Components. We conduct ablation studies by removing the prototype embeddings, timestamp-level embeddings, and multi-scale feature extractor respectively, and analyze their impacts on two clinical prediction tasks, as shown in Table 4. Notably, prototype embeddings play the most significant role among the three components, with their removal resulting in a 1.96% AUROC decrease in 48-IHM and a 1.1% decrease in 24-PHE. The results also show that all three embeddings are important for capturing effective information for prediction.

Ablation Results on Learning Objectives. Table 5 presents the ablation results of the learning objectives. Combining both \( \mathcal{L}_{\text{TPNCE}} \) and \( \mathcal{L}_{\text{Recon}} \) leads to the best performance across 5 out of 6 settings. Our model remains robust under different loss function configurations, with only a 0.66% AUROC drop in the model’s performance in 48-IHM and a 0.85% drop in 24-PHE. Note that the first row of Table 5, i.e., CTPD without auxiliaries losses performs slightly worse than several SOTAs (Table 2) in 48-IHM task, while significantly outperforms compared approaches in 24-PHE task. We think The performance gap is mainly due to task complexity. The 48-IHM task is a simple binary classification problem with limited need for cross-modal reasoning, making our model more prone to

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th>Cont.</th>
    <th>AUCROC</th>
    <th>AUPR</th>
    <th>F1</th>
    <th>Cont.</th>
    <th>AUCROC</th>
    <th>AUPR</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Ours</td>
    <td>88.15<sub>±0.28</sub></td>
    <td>53.86<sub>±0.45</sub></td>
    <td>53.85<sub>±0.46</sub></td>
    <td>83.34<sub>±0.27</sub></td>
    <td>56.39<sub>±0.47</sub></td>
    <td>53.88<sub>±0.45</sub></td>
  </tr>
  <tr>
    <td>w/o prototype</td>
    <td>86.89<sub>±0.37</sub></td>
    <td>53.67<sub>±0.46</sub></td>
    <td>48.47<sub>±0.47</sub></td>
    <td>82.34<sub>±0.27</sub></td>
    <td>54.06<sub>±0.47</sub></td>
    <td>52.88<sub>±0.47</sub></td>
  </tr>
  <tr>
    <td>w/o timestamp embeddings</td>
    <td>87.18<sub>±0.36</sub></td>
    <td>54.32<sub>±0.46</sub></td>
    <td>45.85<sub>±0.46</sub></td>
    <td>82.41<sub>±0.27</sub></td>
    <td>54.30<sub>±0.47</sub></td>
    <td>53.34<sub>±0.46</sub></td>
  </tr>
  <tr>
    <td>w/o multi-scale embedding</td>
    <td>87.59<sub>±0.39</sub></td>
    <td>53.38<sub>±0.47</sub></td>
    <td>49.74<sub>±0.46</sub></td>
    <td>83.11<sub>±0.27</sub></td>
    <td>55.95<sub>±0.47</sub></td>
    <td>53.81<sub>±0.46</sub></td>
  </tr>
</table>

Table 4: Ablation results show the impact of removing different types of input embeddings.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th>Cont.</th>
    <th>AUCROC</th>
    <th>AUPR</th>
    <th>F1</th>
    <th>Cont.</th>
    <th>AUCROC</th>
    <th>AUPR</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>✓</td>
    <td>87.49<sub>±0.47</sub></td>
    <td>53.31<sub>±0.46</sub></td>
    <td>43.59<sub>±0.45</sub></td>
    <td>82.49<sub>±0.30</sub></td>
    <td>55.25<sub>±0.45</sub></td>
    <td>53.71<sub>±0.45</sub></td>
  </tr>
  <tr>
    <td>✓ ✓</td>
    <td>87.15<sub>±0.38</sub></td>
    <td>53.45<sub>±0.40</sub></td>
    <td>44.76<sub>±0.45</sub></td>
    <td>82.94<sub>±0.27</sub></td>
    <td>55.62<sub>±0.27</sub></td>
    <td>53.99<sub>±0.19</sub></td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>86.93<sub>±0.40</sub></td>
    <td>52.70<sub>±0.37</sub></td>
    <td>41.52<sub>±0.30</sub></td>
    <td>82.86<sub>±0.27</sub></td>
    <td>55.43<sub>±0.27</sub></td>
    <td>54.08<sub>±0.29</sub></td>
  </tr>
  <tr>
    <td>✓ ✓ ✓ ✓</td>
    <td><b>88.15<sub>±0.28</sub></b></td>
    <td><b>53.86<sub>±0.45</sub></b></td>
    <td><b>53.85<sub>±0.46</sub></b></td>
    <td><b>83.34<sub>±0.27</sub></b></td>
    <td><b>56.39<sub>±0.47</sub></b></td>
    <td><b>53.88<sub>±0.45</sub></b></td>
  </tr>
</table>

Table 5: Ablation study of loss functions \( \mathcal{L}_{\text{TPNCE}} \) and \( \mathcal{L}_{\text{Recon}} \). The Best results are highlighted in bold.
Table 6: Number of parameters of different modules in CTPD.

<table>
  <tr>
    <th>Module</th>
    <th>Multi-scale Convolution</th>
    <th>Shared Prototype</th>
    <th>Dual Slot Attention</th>
    <th>MITS Decoder</th>
    <th>Text Decoder</th>
    <th>Multimodal Fisom</th>
    <th>Text Encoder</th>
    <th>Others</th>
    <th>Total</th>
  </tr>
  <tr>
    <th># Trainable Parameters</th>
    <td>712.1K</td>
    <td>256</td>
    <td>364K</td>
    <td>1.9M</td>
    <td>1.9M</td>
    <td>1.23M</td>
    <td>4.4M</td>
    <td>0.294M</td>
    <td>9.4M</td>
  </tr>
</table>

Figure 3: Ablation study on the number of prototypes.

![Ablation study on the number of prototypes.](page_184_340_563_312.png)

overfitting due to its added components. In contrast, the 24-PHE task is a more complex multi-label classification problem that benefits from stronger temporal and cross-modal understanding, leading to larger performance gains over SOTAs.

Complexity of the Proposed Method To provide a more detailed analysis of the computational complexity of our proposed framework, we report the number of trainable parameters for each module in the Table 6. Our overall model only has 9.2 M parameters, which is lightweight for multimodal EHR analysis. In addition, we evaluated the inference speed of our CTPD and found that it requires only \( \mathbf{5.42\ ms} \) per time series-text pair on average. This indicates the framework’s potential for real-time deployment in clinical settings.

Ablation Results on Hyperparameters. We analyze the effects of the number of prototypes in Fig. 3. According to the results, we find that using 16 prototypes achieves the best results, though our model remains robust, with 8 prototypes yielding similar outcomes. Additional experimental results of hyperparameters and visualization are in Appendix C.3 and Appendix C.4.

5 Conclusion

In this paper, we present the Cross-Modal Temporal Pattern Discovery (CTPD) framework, which captures cross-modal temporal patterns and incorporates them with timestamp-level embeddings for more accurate clinical outcome predictions based on multimodal EHR data. To efficiently optimize the framework, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss to enhance cross-modal alignment, along with two reconstruction objectives to retain core information from each modality. Our experiments on two clinical prediction tasks using the MIMIC-III dataset demonstrate the effectiveness of CTPD in multi-modal EHR analysis.

6 Limitations

A limitation of our approach is its primary focus on extracting temporal semantics in the embedding space, which affects model interpretability. As a potential solution, we plan to explore retrieval-based methods (Patel et al., 2024) or pretrained generative models (Zhao et al., 2023) to enhance the interpretability of learned temporal pattern embeddings in future work. Besides, our work primarily focuses on exploring cross-modal temporal patterns to enhance representation learning, the scenario of missing modalities falls outside the current scope—since shared patterns across modalities cannot be fully captured in their absence. We plan to thoroughly investigate this direction in future work. Additionally, our framework is currently designed for specific clinical prediction tasks. In practice, there are various prediction tasks related to multimodal EHR analysis. Extending the proposed method into a more generalized or foundational model capable of handling multiple downstream tasks with minimal training annotations could be more practical and effective. Our future work will focus on resolving those aspects.

Potential Risks. The medical dataset used in our framework must be carefully reviewed to mitigate any potential identification risk. Additionally, our framework is developed solely for research purposes and is not intended for commercial use. Note that AI assistant was used only for polishing the writing of this paper.

7 Acknowledgement

This work was supported in part by the Research Grants Council of Hong Kong (27206123, C5055-24G, and T45-401/22-N), the Hong Kong Innovation and Technology Fund (ITS/273/22, ITS/274/22, and GHP/318/22GD), the National Natural Science Foundation of China (No. 62201483), and Guangdong Natural Science Fund (No. 2024A1515011875).
References

Julián N. Acosta, Guido J. Falcone1, Pranav Rajpurkar, and Eric J. Topol. 2022. Multimodal biomedical AI. Nature Medicine, 28:1773–1784.

Rasha Assaf and Rashid Jayousi. 2020. 30-day hospital readmission prediction using mimic data. In 2020 IEEE 14th International Conference on Application of Information and Communication Technologies (AICT), pages 1–6. IEEE.

Mohammad Taha Bahadori and Zachary Chase Lipton. 2019. Temporal-clustering invariance in irregular healthcare time series. arXiv preprint arXiv:1904.12206.

Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. 2017. Patient subtyping via time-aware lstm networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 65–74.

Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. Generalization in nli: Ways (not) to go beyond simple heuristics. Preprint, arXiv:2110.01518.

Wanlin Cai, Yuxuan Liang, Xianggen Liu, Jianshuai Feng, and Yuankai Wu. 2024. Msgnet: Learning multi-scale inter-series correlations for multivariate time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11141–11149.

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.

Nathalie Conrad, Andrew Judge, Jenny Tran, Hamid Mohseni, Deborah Hedgecott, Abel Perez Crespillo, Moira Allison, Harry Hemingway, John G Cleland, John JV McMurray, et al. 2018. Temporal trends and patterns in heart failure incidence: a population-based study of 4 million individuals. The Lancet, 391(10120):572–580.

Shohreh Deldari, Dimitris Spathis, Mohammad Malekzadeh, Fahim Kawzar, Flora Salim, and Akhil Mathur. 2023. Latent masking for multimodal self-supervised learning in health timeseries. arXiv preprint arXiv:2307.16847.

Iman Deznabi, Mohit Iyyer, and Madalina Fiterau. 2021. Predicting in-hospital mortality by combining clinical notes with time-series data. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 4026–4031.

Ning Ding, Sheng-wei Tian, and Long Yu. 2022. A multimodal fusion method for sarcasm detection based on late fusion. Multimedia Tools and Applications, 81(6):8597–8616.

Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211.

Gaurav R Ghosal and Reza Abbasi-Asl. 2021. Multimodal prototype learning for interpretable multivariable time series classification. arXiv preprint arXiv:2106.09636.

Hrayr Harutyunyan, Hrant Khachatryan, David C Kale, Greg Ver Steeg, and Aram Galstyan. 2019. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96.

Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. 2022. Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images. In Machine Learning for Healthcare Conference, pages 479–503. PMLR.

S Hochreiter. 1997. Long short-term memory. Neural Computation MIT-Press.

Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. 2020. Set functions for time series. In International Conference on Machine Learning, pages 4353–4363. PMLR.

Shih-Cheng Huang, Anuj Pareek, Roham Zamaniyan, Imon Banerjee, and Matthew P Lungren. 2020. Multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific reports, 10(1):22147.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-iii, a freely accessible critical care database. Scientific data, 3(1):1–9.

Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida. 2020. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13289–13299.

Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321.

Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. 2019. Using clinical notes with time series data for icu management. arXiv preprint arXiv:1909.09702.

Sein Kim, Namkyeong Lee, Junseok Lee, Dongmin Hyun, and Chanyoung Park. 2023. Heterogeneous graph learning for multi-modal medical data analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5141–5150.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. 2022. Multimodal machine learning in precision health: A scoping review. NPJ digital medicine, 5(1):171.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.

Chonho Lee, Zhaojing Luo, Kee Yuan Ngiam, Meihui Zhang, Kaiping Zheng, Gang Chen, Beng Chin Ooi, and Wei Luen James Yip. 2017. Big healthcare data analytics: Challenges and applications. Handbook of large-scale distributed computing in smart healthcare, pages 11–41.

Bin Li, Carsten Jentsch, and Emmanuel Müller. 2023a. Prototypes as explanation for time series anomaly detection. arXiv preprint arXiv:2307.01601.

Liangzhi Li, Bowen Wang, Manisha Verma, Yuta Nakashima, Ryo Kawasaki, and Hajime Nagahara. 2021. Scouter: Slot attention-based classifier for explainable image recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1046–1055.

Yuxin Li, Wenchao Chen, Bo Chen, Dongsheng Wang, Long Tian, and Mingyuan Zhou. 2023b. Prototype-oriented unsupervised anomaly detection for multivariate time series.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525–11538.

Junyu Luo, Muchao Ye, Cao Xiao, and Fenglong Ma. 2020. Hitnet: Hierarchical time-aware attention networks for risk prediction on electronic health records. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 647–656.

Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. 2020. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 825–832.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. 2019. Hierarchical transformers for long document classification. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pages 838–844. IEEE.

Sungjin Park, Seongsu Bae, Jiho Kim, Tackeun Kim, and Edward Choi. 2022. Graph-text multi-modal pre-training for medical representation learning. In Conference on Health, Inference, and Learning, pages 261–281. PMLR.

Ravi Patel, Angus Brayne, Rogier Hintzen, Daniel Jaroslawicz, Georgiana Neculae, and Dane Corneli. 2024. Retrieve to explain: Evidence-driven predictions with language models. arXiv preprint arXiv:2402.04068.

Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. 2021. Combining 3d image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 688–698. Springer.

Linglong Qian, Zina Ibrahim, Hugh Logan Ellis, Ao Zhang, Yuezhou Zhang, Tao Wang, and Richard Dobson. 2023. Knowledge enhanced conditional imputation for healthcare time-series. arXiv preprint arXiv:2312.16713.

Sangeeta Rani and Geeta Sikka. 2012. Recent techniques of clustering of time series data: a survey. International Journal of Computer Applications, 52(15).

Satya Narayan Shukla and Benjamin M Marlin. 2019. Interpolation-prediction networks for irregularly sampled time series. arXiv preprint arXiv:1909.07782.

Satya Narayan Shukla and Benjamin M Marlin. 2021. Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2101.10318.

Chenxi Sun, Hongyan Li, Moxian Song, Derun Cai, Baofeng Zhang, and Shenda Hong. 2024. Time pattern reconstruction for classification of irregularly sampled time series. Pattern Recognition, 147:110075.

Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. 2021. A prototype-oriented framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 34:17194–17208.

Brandon Theodorou, Lucas Glass, Cao Xiao, and Jimeng Sun. 2024. Framm: Fair ranking with missing modalities for clinical trial site selection. Patterns, 5(3).

Vo Hoang Trong, Yu Gwang-hyun, Dang Thanh Vu, and Kim Jin-young. 2020. Late fusion of multimodal deep neural networks for weeds classification. Computers and Electronics in Agriculture, 175:105506.
Iulia Turec, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin, Suhuan Cui, Yuan Zhong, Yaqing Wang, and Fenglong Ma. 2023. Hierarchical pretraining on multimodal electronic health records. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2023, page 2839. NIH Public Access.

Yuanlong Wang, Changchang Yin, and Ping Zhang. 2024. Multimodal risk prediction with physiological signals, medical images and clinical notes. Heliyon, 10(5).

Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan B Davidson. 2021. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 651–659.

Jingyun Xiao, Ran Liu, and Eva L Dyer. 2023. Gaformer: Enhancing timeseries transformers through group-aware embeddings. In The Twelfth International Conference on Learning Representations.

Feng Xie, Yuan Han, Ning Yilin, Marcus E. H. Ong, Feng Mengling, Wynne Hsu, Bibhas Chakraborty, and Nan Liu. 2021. Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies. J Biomed Inform., 126:103980.

Zhen Xu, David R So, and Andrew M Dai. 2021. Mu-fasa: Multimodal fusion architecture search for electronic health records. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10532–10540.

Haiyang Yang, Li Kuang, and FengQiang Xia. 2021. Multimodal temporal-clinical note network for mortality prediction. Journal of Biomedical Semantics, 12:1–14.

Wenfang Yao, Kejing Yin, William K Cheung, Jia Liu, and Jing Qin. 2024. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16416–16424.

Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dandan Guo, and Yi Chang. 2024. Ptarl: Prototype-based tabular representation learning via space calibration. arXiv preprint arXiv:2407.05364.

Muchao Ye, Junyu Luo, Cao Xiao, and Fenglong Ma. 2020. Lsan: Modeling long-term dependencies and short-term correlations with hierarchical attention for risk prediction. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 1753–1762.

Zhihao Yu, Xu Chu, Liantao Ma, Yasha Wang, and Wenwu Zhu. 2024. Imputation with inter-series information from prototypes for irregular sampled time series. arXiv preprint arXiv:2401.07249.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980–8987.

Dongyu Zhang, Jidapa Thadajarassiri, Cansu Sen, and Elke Rundensteiner. 2020a. Time-aware transformer-based network for clinical notes series prediction. In Machine learning for healthcare conference, pages 566–588. PMLR.

Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. 2023a. Warpformer: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3273–3285.

Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. 2021. Graph-guided network for irregularly sampled multivariate time series. arXiv preprint arXiv:2110.05357.

Xinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan, and Linda Ruth Petzold. 2023b. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300–41313. PMLR.

Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. 2024. Prototypical information bottlenecking and disentangling for multimodal cancer survival prediction. arXiv preprint arXiv:2401.01646.

Yu-Dong Zhang, Zhengchao Dong, Shui-Hua Wang, Xi-ang Yu, Xujing Yao, Qinghua Zhou, Hua Hu, Min Li, Carmen Jiménez-Mesa, Javier Ramirez, et al. 2020b. Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Information Fusion, 64:149–187.

Liming Zhao, Kecheng Zheng, Yun Zheng, Deli Zhao, and Jingren Zhou. 2023. Rleg: vision-language representation learning with diffusion-based embedding generation. In International Conference on Machine Learning, pages 42247–42258. PMLR.
Appendix

Table 7: Dataset description of 48-IHM and 24-PHE.

<table>
  <tr>
    <th></th>
    <th>Training</th>
    <th>Validation</th>
    <th>Test</th>
  </tr>
  <tr>
    <th>24-PHE</th>
    <td>5046</td>
    <td>573</td>
    <td>1369</td>
  </tr>
  <tr>
    <th>48-IHM</th>
    <td>4301</td>
    <td>466</td>
    <td>1183</td>
  </tr>
  <tr>
    <th>Positive</th>
    <td>644</td>
    <td>73</td>
    <td>184</td>
  </tr>
  <tr>
    <th>Negative</th>
    <td>3657</td>
    <td>393</td>
    <td>999</td>
  </tr>
</table>

A Experimental Setup Details

A.1 More details of Tasks.
We presented the class distributions for 48-IHM and 24-PHE in Table 7 and Table 8, respectively. 48-IHM is a binary classification task, whereas 24-PHE is a multi-label classification problem with 25 labels. Notably, our 24-PHE task differs from the phenotype classification problem in the MIMIC-III benchmark but follows the setup in (Zhang et al., 2023b). This setup focuses on acute care conditions that arise during ICU stays, where early prediction is critical for timely intervention. To enhance clinical relevance, we use only the first 24 hours of data for phenotype classification rather than the entire admission record. As highlighted in (Yang et al., 2021), early-stage diagnosis holds greater clinical significance.

The selection of 25 phenotype labels for the 24-PHE task follows established practices in the MIMIC-III benchmark (Harutyunyan et al., 2019), as also utilized in prior studies like UTDE (Zhang et al., 2023b) and MedFuse (Hayat et al., 2022). These labels cover conditions commonly observed in adult ICUs, including 12 critical and life-threatening conditions (e.g., respiratory failure, sepsis), 8 chronic conditions that are often considered comorbidities or risk factors (e.g., diabetes, metabolic disorders), and 5 ‘mixed’ conditions that exhibit characteristics of both chronic and acute conditions. Phenotype labels were determined using the MIMIC-III ICD-9 diagnosis table. To facilitate the translation and conversion of the above-mentioned conditions, we use the Health Cost and Utilization (HCUP) Clinical Classification Software (CCS)\footnote{https://hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp}. We first mapped each ICD-9 code to its corresponding HCUP CCS category, retaining only 25 categories. Diagnoses were then linked to ICU stays using the hospital admission identifier, as ICD-9 codes in MIMIC-III are associated with hospital visits rather than specific ICU stays. To reduce label ambiguity, we excluded hospital admissions involving multiple ICU stays, ensuring each diagnosis could be associated with a single ICU stay. Please find the class distribution of 25 phenotypes in Table 8.

A.2 Additional Information on Datasets
The 17 variables from the MIMIC-III dataset that we use include 5 categorical variables (capillary refill rate, Glasgow coma scale eye opening, Glasgow coma scale motor response, Glasgow coma scale total, and Glasgow coma scale verbal response) and 12 continuous measures (diastolic blood pressure, fraction of inspired oxygen, glucose, heart rate, height, mean blood pressure, oxygen saturation, respiratory rate, systolic blood pressure, temperature, weight, and pH).

B More Details on Baselines

B.1 Baselines only using MITS
• CNN (LeCun et al., 1998): CNN (Convolutional Neural Network) uses backpropagation to synthesize a complex decision surface that facilitates learning.
• RNN (Elman, 1990): RNN (Residual Neural Network) is trained to process data sequentially so as to model the time dimension of data.
• LSTM (Hochreiter, 1997): LSTM (Long short-term memory) is a variant of recurrent neural network. It excels at dealing with the vanishing gradient problem and is relatively insensitive to time gap length.
• Transformer (Vaswani et al., 2017): Transformer is a powerful deep learning architecture based on attention mechanism. It has great generalisability and has been adopted as foundation model in multiple research domains.
• IP-Net (Shukla and Marlin, 2019): IP-Net (Interpolation-Prediction Network) is a deep learning architecture for supervised learning focusing on processing sparse multivariate time series data that are sampled irregularly.
• GRU-D (Che et al., 2018): GRU-D is based on GRU (Gated Recurrent Unit). It incorporates features of missing data in EHR into
Table 8: Distribution of 25 phenotypes in 24-PHE task.

<table>
  <tr>
    <th>Phenotype</th>
    <th>Type</th>
    <th colspan="2">Training</th>
    <th colspan="2">Validation</th>
    <th colspan="2">Test</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Positive</th>
    <th>Negative</th>
  </tr>
  <tr>
    <td>Acute and unspecified renal failure</td>
    <td>acute</td>
    <td>6066</td>
    <td>20825</td>
    <td>2284</td>
    <td>2284</td>
    <td>1176</td>
    <td>4106</td>
  </tr>
  <tr>
    <td>Acute cerebrovascular disease</td>
    <td>acute</td>
    <td>2069</td>
    <td>24822</td>
    <td>2736</td>
    <td>2736</td>
    <td>363</td>
    <td>4919</td>
  </tr>
  <tr>
    <td>Acute myocardial infarction</td>
    <td>acute</td>
    <td>2870</td>
    <td>24021</td>
    <td>2632</td>
    <td>2632</td>
    <td>588</td>
    <td>4694</td>
  </tr>
  <tr>
    <td>Cardiac dysrhythmia</td>
    <td>mixed</td>
    <td>9011</td>
    <td>17880</td>
    <td>1975</td>
    <td>1975</td>
    <td>1785</td>
    <td>3497</td>
  </tr>
  <tr>
    <td>Chronic kidney disease</td>
    <td>chronic</td>
    <td>5663</td>
    <td>23175</td>
    <td>2564</td>
    <td>2564</td>
    <td>711</td>
    <td>4571</td>
  </tr>
  <tr>
    <td>Chronic obstructive pulmonary disease and bronchiectasis</td>
    <td>chronic</td>
    <td>5616</td>
    <td>23275</td>
    <td>2542</td>
    <td>2542</td>
    <td>685</td>
    <td>4597</td>
  </tr>
  <tr>
    <td>Complications of surgical procedure or medical care</td>
    <td>acute</td>
    <td>5880</td>
    <td>21011</td>
    <td>2293</td>
    <td>2293</td>
    <td>1202</td>
    <td>4080</td>
  </tr>
  <tr>
    <td>Conduction disorders</td>
    <td>mixed</td>
    <td>1971</td>
    <td>24920</td>
    <td>2744</td>
    <td>2744</td>
    <td>382</td>
    <td>4000</td>
  </tr>
  <tr>
    <td>Congestive heart failure; nonhypertensive</td>
    <td>mixed</td>
    <td>7623</td>
    <td>19268</td>
    <td>2150</td>
    <td>2150</td>
    <td>1486</td>
    <td>3796</td>
  </tr>
  <tr>
    <td>Coronary atherosclerosis and other heart disease</td>
    <td>chronic</td>
    <td>8840</td>
    <td>18051</td>
    <td>1964</td>
    <td>1964</td>
    <td>1787</td>
    <td>3495</td>
  </tr>
  <tr>
    <td>Diabetes mellitus with complications</td>
    <td>mixed</td>
    <td>2612</td>
    <td>24279</td>
    <td>2675</td>
    <td>2675</td>
    <td>503</td>
    <td>4779</td>
  </tr>
  <tr>
    <td>Diabetes mellitus without complication</td>
    <td>chronic</td>
    <td>5305</td>
    <td>21586</td>
    <td>2401</td>
    <td>2401</td>
    <td>1032</td>
    <td>4250</td>
  </tr>
  <tr>
    <td>Disorders of lipid metabolism</td>
    <td>chronic</td>
    <td>7841</td>
    <td>19050</td>
    <td>2081</td>
    <td>1541</td>
    <td>1541</td>
    <td>3741</td>
  </tr>
  <tr>
    <td>Essential hypertension</td>
    <td>chronic</td>
    <td>11340</td>
    <td>15551</td>
    <td>1689</td>
    <td>1689</td>
    <td>2238</td>
    <td>3044</td>
  </tr>
  <tr>
    <td>Fluid and electrolyte disorders</td>
    <td>acute</td>
    <td>7480</td>
    <td>19411</td>
    <td>2107</td>
    <td>1429</td>
    <td>3853</td>
    <td></td>
  </tr>
  <tr>
    <td>Gastrointestinal hemorrhage</td>
    <td>acute</td>
    <td>2001</td>
    <td>24890</td>
    <td>2746</td>
    <td>2746</td>
    <td>427</td>
    <td>4855</td>
  </tr>
  <tr>
    <td>Hypertension with complications and secondary hypertension</td>
    <td>chronic</td>
    <td>3646</td>
    <td>23245</td>
    <td>2583</td>
    <td>2583</td>
    <td>706</td>
    <td>4576</td>
  </tr>
  <tr>
    <td>Other liver diseases</td>
    <td>mixed</td>
    <td>2485</td>
    <td>24406</td>
    <td>2688</td>
    <td>2688</td>
    <td>492</td>
    <td>4790</td>
  </tr>
  <tr>
    <td>Other lower respiratory disease</td>
    <td>acute</td>
    <td>1414</td>
    <td>25140</td>
    <td>2810</td>
    <td>2810</td>
    <td>305</td>
    <td>4977</td>
  </tr>
  <tr>
    <td>Other upper respiratory disease</td>
    <td>acute</td>
    <td>1130</td>
    <td>35761</td>
    <td>2814</td>
    <td>2814</td>
    <td>238</td>
    <td>5044</td>
  </tr>
  <tr>
    <td>Pleurisy; pneumothorax; pulmonary collapse</td>
    <td>acute</td>
    <td>2516</td>
    <td>24375</td>
    <td>2671</td>
    <td>2671</td>
    <td>518</td>
    <td>4764</td>
  </tr>
  <tr>
    <td>Pneumonia (except that caused by tuberculosis or sexually transmitted disease)</td>
    <td>acute</td>
    <td>4121</td>
    <td>22770</td>
    <td>2502</td>
    <td>2502</td>
    <td>769</td>
    <td>4513</td>
  </tr>
  <tr>
    <td>Respiratory failure; insufficiency; arrest (adult)</td>
    <td>acute</td>
    <td>5382</td>
    <td>21509</td>
    <td>2365</td>
    <td>2365</td>
    <td>1025</td>
    <td>4257</td>
  </tr>
  <tr>
    <td>Septicemia (except in labor)</td>
    <td>acute</td>
    <td>4136</td>
    <td>22755</td>
    <td>2494</td>
    <td>2494</td>
    <td>793</td>
    <td>4489</td>
  </tr>
  <tr>
    <td>Shock</td>
    <td>acute</td>
    <td>2263</td>
    <td>24628</td>
    <td>2700</td>
    <td>2700</td>
    <td>456</td>
    <td>4826</td>
  </tr>
</table>

the model architecture to improve prediction results.

• DGM-O (Wu et al., 2021): DGM-O (Dynamic Gaussian Mixture based Deep Generative Model) is a generative model derived from a dynamic Gaussian mixture distribution. It makes predictions based on incomplete inputs. DGM-O is instantiated with multilayer perceptron (MLP).

• mTAND (Shukla and Marlin, 2021): mTAND (Multi-Time Attention network) is a deep learning model that learns representations of continuous time values and uses an attention mechanism to generate a consistent representation of a time series based on a varying amount of observations.

• SeFT (Horn et al., 2020): SeFT (Set Functions for Time Series) addresses irregularly-sampled time series. It is based on differentiable set function learning.

• UTDE (Zhang et al., 2023b): UTDE (Unified TDE module) is built upon TDE (Temporal discretization-based embedding). It models asynchronous time series data by combining imputation embeddings and learned interpolation embeddings through a gating mechanism. It also uses a time attention mechanism.

B.2 Baselines only using clinical notes

• Flat (Deznabi et al., 2021): Flat encodes clinical notes using a fine-tuned BERT model. It also utilizes an LSTM model that takes in patients’ vital signals so as to jointly model the two modalities. Furthermore, it addresses the temporal irregularity issue of modeling patients’ vital signals.

• HierTrans (Pappagari et al., 2019): HierTrans (Hierarchical Transformers) is built upon BERT model. It achieves an enhanced ability to take in long inputs by first partitioning the inputs into shorter sequences and processing them separately. Then, it propagates each output via a recurrent layer.

• T-LSTM (Baytas et al., 2017): T-LSTM (Time-Aware LSTM) deals with irregular time intervals in EHRs by learning decomposed cell memory which models elapsed time. The final patient subtyping model uses T-LSTM in an auto-encoder module before doing patient subtyping.

• FT-LSTM (Zhang et al., 2020a): FT-LSTM (Flexible Time-aware LSTM Transformer) models the multi-level structure in clinical notes. At the base level, it uses a pre-trained ClinicalBERT model. Then, it merges sequential information and content embedding into a new position-enhanced representation. Then,
Table 9: Comparison between our method with other baselines on 30-day Readmission task on MIMIC-III. We report average performance on three random seeds, with standard deviation as the subscript.

<table>
  <tr>
    <th>Model</th>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <th colspan="4">Methods on EHR.</th>
  </tr>
  <tr>
    <td>CNN (LeCun et al., 1998)</td>
    <td>0.757<sub>0.003</sub></td>
    <td>0.447<sub>0.006</sub></td>
    <td>0.449<sub>0.006</sub></td>
  </tr>
  <tr>
    <td>RNN (Elman, 1990)</td>
    <td>0.750<sub>0.001</sub></td>
    <td>0.449<sub>0.008</sub></td>
    <td>0.433<sub>0.003</sub></td>
  </tr>
  <tr>
    <td>LSTM (Hochreiter, 1997)</td>
    <td>0.757<sub>0.002</sub></td>
    <td>0.443<sub>0.003</sub></td>
    <td>0.430<sub>0.015</sub></td>
  </tr>
  <tr>
    <td>Transformer (Vaswani et al., 2017)</td>
    <td>0.749<sub>0.006</sub></td>
    <td>0.409<sub>0.012</sub></td>
    <td>0.433<sub>0.016</sub></td>
  </tr>
  <tr>
    <td>DGM-O (Wu et al., 2021)</td>
    <td>0.671<sub>0.017</sub></td>
    <td>0.325<sub>0.032</sub></td>
    <td>0.382<sub>0.020</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2019)</td>
    <td>0.743<sub>0.002</sub></td>
    <td>0.437<sub>0.001</sub></td>
    <td>0.441<sub>0.007</sub></td>
  </tr>
  <tr>
    <td>UTDE (Zhang et al., 2023b)</td>
    <td>0.758<sub>0.002</sub></td>
    <td>0.453<sub>0.004</sub></td>
    <td>0.445<sub>0.008</sub></td>
  </tr>
  <tr>
    <th colspan="4">Methods on Clinical Notes.</th>
  </tr>
  <tr>
    <td>Flat (Deznabi et al., 2021)</td>
    <td>0.755<sub>0.005</sub></td>
    <td>0.447<sub>0.018</sub></td>
    <td>0.437<sub>0.015</sub></td>
  </tr>
  <tr>
    <td>HierTrans (Pappagari et al., 2019)</td>
    <td>0.754<sub>0.001</sub></td>
    <td>0.425<sub>0.005</sub></td>
    <td>0.434<sub>0.005</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2019)</td>
    <td>0.757<sub>0.001</sub></td>
    <td>0.435<sub>0.001</sub></td>
    <td>0.440<sub>0.014</sub></td>
  </tr>
  <tr>
    <th colspan="4">Methods on Multiple modalities.</th>
  </tr>
  <tr>
    <td>MMTM (Joze et al., 2020)</td>
    <td>0.769<sub>0.005</sub></td>
    <td>0.469<sub>0.006</sub></td>
    <td>0.459<sub>0.007</sub></td>
  </tr>
  <tr>
    <td>DAFT (Pölsterl et al., 2021)</td>
    <td>0.765<sub>0.006</sub></td>
    <td>0.452<sub>0.018</sub></td>
    <td>0.458<sub>0.005</sub></td>
  </tr>
  <tr>
    <td>MedFuse (Hayat et al., 2022)</td>
    <td>0.763<sub>0.009</sub></td>
    <td>0.461<sub>0.012</sub></td>
    <td>0.454<sub>0.013</sub></td>
  </tr>
  <tr>
    <td>DrFuse (Yao et al., 2024)</td>
    <td>0.748<sub>0.002</sub></td>
    <td>0.443<sub>0.014</sub></td>
    <td>0.442<sub>0.017</sub></td>
  </tr>
  <tr>
    <td>CTPD (Ours)</td>
    <td><b>0.777<sub>0.006</sub></b></td>
    <td><b>0.474<sub>0.009</sub></b></td>
    <td><b>0.461<sub>0.019</sub></b></td>
  </tr>
</table>

it uses a time-aware layer that considers the irregularity of time intervals.

• GRU-D (Che et al., 2018): Please refer to the above subsection.

• mTAND (Shukla and Marlin, 2021): Please refer to the above subsection.

B.3 Baselines using multimodal EHR

• MMTM (Joze et al., 2020): MMTM (Multi-modal Transfer Module) is a neural network module that leverages knowledge from various modalities in CNN. It can recalibrate features in each CNN stream via excitation and squeeze operations.

• DAFT (Pölsterl et al., 2021): DAFT (Dynamic Affine Feature Map Transform) is a general-purpose CNN module that alters the feature maps of a convolutional layer with respect to a patient’s clinical data.

• MedFuse (Hayat et al., 2022): MedFuse is an LSTM-based fusion module capable of processing both uni-modal and multi-modal input. It treats multi-modal representations of data as a sequence of uni-modal representations. It handles inputs of various lengths via the recurrent inductive bias of LSTM.

• DrFuse (Yao et al., 2024): DrFuse is a fusion module that addresses the issue of missing modality by separating the unique features within each modality and the common ones across modalities. It also adds a disease-wise attention layer for each modality.

C More on Experimental Results

C.1 Results on 30-day Readmission

We further evaluated CTPD on the 30-day readmission prediction task (Assaf and Jayousi, 2020) using the MIMIC-III dataset. The results are presented in Table 9. This task involves predicting whether a patient will be readmitted based on data from their current ICU stay. Our results show that CTPD consistently outperforms baseline methods across all three evaluation metrics, further demonstrating its effectiveness.

C.2 Results on MIMIC-IV

To assess the adaptability of our framework to different data sources, we also conducted experiments on the MIMIC-IV dataset. Since MIMIC-IV lacks temporal clinical text, we evaluated our approach using tabular time-series data and chest radiographs as the two input modalities. The results are summarized in Table 10. CTPD achieved the best performance across all six evaluation settings. Given the relatively small standard deviations, our approach demonstrates statistically significant improvements over previous methods, further validating its generalizability.
Table 10: Comparison between our method with other baselines on 48-IHM and 24-PHE on MIMIC-IV. We report average performance on three random seeds, with standard deviation as the subscript. The Best and 2nd best methods under each setup are bold and underlined. “-” denotes the results are close to 0.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
    <th>AUROC (↑)</th>
    <th>AUPR (↑)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <th colspan="7">Methods on MITTS.</th>
  </tr>
  <tr>
    <td>CNN (LeCun et al., 1998)</td>
    <td>74.16<sub>0.79</sub></td>
    <td>36.32<sub>0.15</sub></td>
    <td>27.34<sub>8.89</sub></td>
    <td>67.50<sub>0.14</sub></td>
    <td>37.00<sub>0.02</sub></td>
    <td>20.90<sub>0.81</sub></td>
  </tr>
  <tr>
    <td>RNN (Elman, 1990)</td>
    <td>77.40<sub>1.10</sub></td>
    <td>38.87<sub>0.80</sub></td>
    <td>23.93<sub>9.2</sub></td>
    <td>66.89<sub>0.49</sub></td>
    <td>36.28<sub>0.70</sub></td>
    <td>18.72<sub>1.97</sub></td>
  </tr>
  <tr>
    <td>LSTM (Hochreiter, 1997)</td>
    <td>79.40<sub>0.50</sub></td>
    <td>41.95<sub>0.91</sub></td>
    <td>34.17<sub>1.97</sub></td>
    <td>67.59<sub>0.30</sub></td>
    <td>37.21<sub>0.49</sub></td>
    <td>20.87<sub>1.32</sub></td>
  </tr>
  <tr>
    <td>Transformer (Vaswani et al., 2017)</td>
    <td>75.73<sub>2.43</sub></td>
    <td>39.76<sub>2.64</sub></td>
    <td>21.45<sub>10.75</sub></td>
    <td>67.44<sub>0.21</sub></td>
    <td>37.33<sub>0.11</sub></td>
    <td>20.44<sub>0.40</sub></td>
  </tr>
  <tr>
    <td>IP-Net (Shukla and Marlin, 2019)</td>
    <td>77.58<sub>0.59</sub></td>
    <td>46.02<sub>0.73</sub></td>
    <td>36.80<sub>0.93</sub></td>
    <td>68.20<sub>0.21</sub></td>
    <td>38.40<sub>2.27</sub></td>
    <td>20.62<sub>0.58</sub></td>
  </tr>
  <tr>
    <td>GRU-D (Che et al., 2018)</td>
    <td>53.73<sub>2.11</sub></td>
    <td>17.54<sub>2.22</sub></td>
    <td>8.99<sub>15.56</sub></td>
    <td>50.53<sub>0.60</sub></td>
    <td>23.14<sub>0.36</sub></td>
    <td>9.42<sub>7.26</sub></td>
  </tr>
  <tr>
    <td>DGM-O (Wu et al., 2021)</td>
    <td>70.50<sub>0.43</sub></td>
    <td>33.79<sub>3.38</sub></td>
    <td>14.14<sub>12.42</sub></td>
    <td>59.74<sub>2.34</sub></td>
    <td>29.95<sub>2.05</sub></td>
    <td>4.33<sub>4.12</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2021)</td>
    <td>80.63<sub>0.33</sub></td>
    <td>45.17<sub>0.47</sub></td>
    <td>33.01<sub>3.47</sub></td>
    <td>66.87<sub>0.14</sub></td>
    <td>36.38<sub>0.18</sub></td>
    <td>19.07<sub>1.28</sub></td>
  </tr>
  <tr>
    <td>SeFT (Horn et al., 2020)</td>
    <td>61.97<sub>0.96</sub></td>
    <td>25.13<sub>4.42</sub></td>
    <td>-</td>
    <td>57.10<sub>0.04</sub></td>
    <td>26.98<sub>0.09</sub></td>
    <td>-</td>
  </tr>
  <tr>
    <td>UTDE (Zhang et al., 2023b)</td>
    <td>80.89<sub>0.10</sub></td>
    <td>45.08<sub>0.30</sub></td>
    <td>37.53<sub>2.03</sub></td>
    <td>67.68<sub>0.05</sub></td>
    <td>37.58<sub>0.19</sub></td>
    <td>17.63<sub>0.44</sub></td>
  </tr>
  <tr>
    <th colspan="7">Methods on CXR Image.</th>
  </tr>
  <tr>
    <td>Flat (Deznabi et al., 2021)</td>
    <td>62.37<sub>2.42</sub></td>
    <td>23.97<sub>3.35</sub></td>
    <td>23.38<sub>2.34</sub></td>
    <td>66.10<sub>0.37</sub></td>
    <td>36.04<sub>0.16</sub></td>
    <td>40.40<sub>0.54</sub></td>
  </tr>
  <tr>
    <td>HierTrans (Pappagari et al., 2019)</td>
    <td>61.51<sub>1.43</sub></td>
    <td>20.96<sub>1.62</sub></td>
    <td>17.95<sub>15.54</sub></td>
    <td>58.36<sub>0.28</sub></td>
    <td>27.84<sub>2.16</sub></td>
    <td>33.61<sub>5.74</sub></td>
  </tr>
  <tr>
    <td>T-LSTM (Baytas et al., 2017)</td>
    <td>53.62<sub>0.88</sub></td>
    <td>16.82<sub>0.38</sub></td>
    <td>21.40<sub>4.17</sub></td>
    <td>58.56<sub>1.45</sub></td>
    <td>28.12<sub>1.21</sub></td>
    <td>33.06<sub>0.51</sub></td>
  </tr>
  <tr>
    <td>FT-LSTM (Zhang et al., 2020a)</td>
    <td>48.63<sub>4.47</sub></td>
    <td>15.35<sub>9.90</sub></td>
    <td>11.85<sub>13.75</sub></td>
    <td>54.46<sub>2.15</sub></td>
    <td>25.29<sub>2.42</sub></td>
    <td>29.12<sub>7.55</sub></td>
  </tr>
  <tr>
    <td>GRU-D (Che et al., 2018)</td>
    <td>56.02<sub>1.05</sub></td>
    <td>18.63<sub>0.76</sub></td>
    <td>23.16<sub>6.34</sub></td>
    <td>57.94<sub>0.39</sub></td>
    <td>27.85<sub>0.47</sub></td>
    <td>27.13<sub>7.73</sub></td>
  </tr>
  <tr>
    <td>mTAND (Shukla and Marlin, 2021)</td>
    <td>62.80<sub>3.42</sub></td>
    <td>24.64<sub>2.22</sub></td>
    <td>24.82<sub>9.43</sub></td>
    <td>68.31<sub>0.68</sub></td>
    <td>38.53<sub>0.93</sub></td>
    <td>40.76<sub>1.14</sub></td>
  </tr>
  <tr>
    <th colspan="7">Methods on Multiple modalities.</th>
  </tr>
  <tr>
    <td>MMTM (Joze et al., 2020)</td>
    <td>80.65<sub>0.79</sub></td>
    <td>48.96<sub>0.50</sub></td>
    <td>47.40<sub>1.41</sub></td>
    <td>70.28<sub>0.44</sub></td>
    <td>39.61<sub>0.76</sub></td>
    <td>43.32<sub>0.29</sub></td>
  </tr>
  <tr>
    <td>DAFT (Pilsterl et al., 2021)</td>
    <td>81.87<sub>0.12</sub></td>
    <td>47.79<sub>0.88</sub></td>
    <td>48.91<sub>1.98</sub></td>
    <td>70.87<sub>0.24</sub></td>
    <td>40.22<sub>0.22</sub></td>
    <td>44.10<sub>0.26</sub></td>
  </tr>
  <tr>
    <td>MedFuse (Hayat et al., 2022)</td>
    <td>81.60<sub>0.28</sub></td>
    <td>48.35<sub>1.35</sub></td>
    <td>48.12<sub>0.75</sub></td>
    <td>70.82<sub>0.37</sub></td>
    <td>40.39<sub>0.31</sub></td>
    <td>44.03<sub>0.52</sub></td>
  </tr>
  <tr>
    <td>DrFuse (Yao et al., 2024)</td>
    <td>80.94<sub>0.44</sub></td>
    <td>45.64<sub>0.44</sub></td>
    <td>48.58<sub>1.35</sub></td>
    <td>70.27<sub>0.22</sub></td>
    <td>39.90<sub>0.23</sub></td>
    <td>43.43<sub>0.09</sub></td>
  </tr>
  <tr>
    <td>CTPD (Ours)</td>
    <td><b>83.53</b><sub>0.44</sub></td>
    <td><b>49.94</b><sub>0.23</sub></td>
    <td><b>49.53</b><sub>2.39</sub></td>
    <td><b>71.96</b><sub>0.40</sub></td>
    <td><b>42.46</b><sub>0.60</sub></td>
    <td><b>44.85</b><sub>0.61</sub></td>
  </tr>
</table>

Table 11: Ablation results on loss weights. The parameters \( \lambda_1 \) and \( \lambda_2 \) control the strength of the \( \mathcal{L}_{\text{TPNCE}} \) and \( \mathcal{L}_{\text{Recon}} \) losses, respectively.

<table>
  <tr>
    <th>\( \lambda_1 \)</th>
    <th>\( \lambda_2 \)</th>
    <th colspan="3">48-IHM</th>
    <th colspan="3">24-PHE</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>AUROC</th>
    <th>AUPR</th>
    <th>F1</th>
    <th>AUROC</th>
    <th>AUPR</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>0.1</td>
    <td>0.1</td>
    <td>87.21<sub>0.56</sub></td>
    <td>53.80<sub>2.85</sub></td>
    <td>47.41<sub>1.13</sub></td>
    <td>82.93<sub>0.55</sub></td>
    <td>55.62<sub>2.21</sub></td>
    <td>54.00<sub>0.01</sub></td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>0.5</td>
    <td><b>88.15</b><sub>0.28</sub></td>
    <td><b>53.86</b><sub>0.65</sub></td>
    <td><b>53.85</b><sub>0.16</sub></td>
    <td><b>83.34</b><sub>0.05</sub></td>
    <td><b>56.39</b><sub>1.17</sub></td>
    <td><b>53.83</b><sub>0.43</sub></td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.5</td>
    <td>87.20<sub>0.47</sub></td>
    <td>53.77<sub>0.47</sub></td>
    <td>42.44<sub>4.45</sub></td>
    <td>82.59<sub>0.09</sub></td>
    <td>55.22<sub>0.04</sub></td>
    <td>53.42<sub>0.04</sub></td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>0.5</td>
    <td>86.59<sub>0.19</sub></td>
    <td>51.93<sub>3.31</sub></td>
    <td>47.93<sub>1.13</sub></td>
    <td>82.44<sub>0.03</sub></td>
    <td>54.84<sub>1.18</sub></td>
    <td>53.29<sub>0.29</sub></td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>1.0</td>
    <td>86.64<sub>2.10</sub></td>
    <td>52.34<sub>7.9</sub></td>
    <td>50.09<sub>1.44</sub></td>
    <td>82.54<sub>0.48</sub></td>
    <td>55.03<sub>0.39</sub></td>
    <td>53.94<sub>1.31</sub></td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>2.0</td>
    <td>85.55<sub>1.19</sub></td>
    <td>50.66<sub>4.17</sub></td>
    <td>44.67<sub>3.88</sub></td>
    <td>76.88<sub>1.84</sub></td>
    <td>41.89<sub>1.61</sub></td>
    <td>44.83<sub>1.11</sub></td>
  </tr>
</table>

C.3 Ablation Results on Loss Weights

We analyze the effects of loss weights \( \lambda_1 \) and \( \lambda_2 \), shown in Table 11. For the loss weights, \( \lambda_1 = 0.1 \) and \( \lambda_2 = 0.5 \) consistently yield the best performance across all settings. In contrast, larger values (rows 4–6) lead to performance drops, likely due to the model overemphasizing alignment and reconstruction losses at the expense of prediction accuracy. Moderate changes to \( \lambda_1 \) and \( \lambda_2 \) (rows 1–3) yield stable results, particularly on the complex 24-PHE task, indicating robustness. However, the simpler 48-IHM task is more sensitive, possibly due to its limited temporal complexity and tendency to overfit. In practice, we adopt \( \lambda_1 = 0.1 \) and \( \lambda_2 = 0.5 \) to maintain a balanced loss scale without extensive tuning. Future work may explore finer-grained searches to better understand the trade-offs across tasks.

C.4 Visualization of Assignment Weights of Prototypes.

Fig. 4 presents the distribution of assignment weights across different time scales for a 48-IHM example. With time window sizes denoted by \( T \),

![Visualization of the learned prototypes in our CTPD framework. Here we select 5 representative clinical variables to visualize the time series. "CPR" denotes "Capillary Refill Rate", "DBP" denotes "Diastolic blood pressure", "HR" denotes "Heart Rate", "MBP" denotes "Mean blood pressure" and "OS" denotes "Oxygen saturation".](page_1042_1342_377_377.png)

Figure 4: Visualization of the learned prototypes in our CTPD framework. Here we select 5 representative clinical variables to visualize the time series. "CPR" denotes "Capillary Refill Rate", "DBP" denotes "Diastolic blood pressure", "HR" denotes "Heart Rate", "MBP" denotes "Mean blood pressure" and "OS" denotes "Oxygen saturation".
our model utilizes three time scales with 20, 10, and 5 prototypes respectively. The variation in assignment weights across these scales, as showcased in the figure, underlines our model’s proficiency in capturing and differentiating temporal patterns at varying scales. We will try to interpret these learned temporal pattern embeddings in our future work.

Reviewer paper summaries: The paper introduces the Cross-Modal Temporal Pattern Discovery (CTPD) component within the multimodal learning framework, designed to improve the analysis of multimodal Electronic Health Records (EHR) data, e.g., numerical time series and clinical notes.
Different from general attention components that deals with two modality embeddings independently, this paper employs shared structure between attention blocks of two modalities, aiming to capturing cross-modal temporal patterns.
Two other losses are also incorporated into the framework: 1). Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss, which is similar to InfoNCE loss and is designed to encourage alignment between paired time series and text embeddings; 2). MSE loss (reconstruction loss) between the raw time series and text embeddings and time series and text embeddings after the CTPD block.
The authors evaluate the whole framework on two clinical tasks—a 48-hour in-hospital mortality prediction task and a classification task —using the MIMIC-III database, demonstrating performance gains over existing methods.
Summary of strengths: (1) Employing the shared structure, specifically the query within the attention mechanism, between modalities to extract the temporal patterns is somewhat novel. The idea makes sense to me since attention mechanism cares about the intersections between input blocks, thus, the strong corelated temporal patterns in different modalities may be extracted with similar structure.
(2) A relatively comprehensive method evaluation. Specifically, Table 2 shows the performance comparison with various baselines. Section 4.3 demonstrates a detailed ablation study.
Summary of weaknesses: I have two major concerns:
(1) One of the main contributions of this work is the proposed CTPD component, and its potential usage of helping extract cross-modal temporal patterns. However, since this work introduces a complete framework with additional new components, such as TP-NCE loss and reconstruct loss, the pivotal component that contributes to final performance improvement is unclear to me. Especially (a). Table 5 demonstrates that the CTPD component alone cannot improve the performance compared to some baselines; (b). Table 10 demonstrates that tuning hyper-parameters, the weights of the two new losses $ λ1, λ2$, largely affect the final performance.  It makes me feel that: tuning effects may contribute to the performance improvement most.
(2) Another major concern is that: inaccurate mathematical descriptions. Please refer to “Comments section” blew.
Some minor concerns are listed below:
(3) The motivations for incorporating the TP-NCE loss and the reconstruction loss are not strongly justified. Specifically, the two losses are also the main components of the proposed method other than the CTPD component, however, (a). the idea of these two losses is not new: a variant of InfoNCE loss and MSE loss; (b). the incorporation is not strongly justified. 
(4) tuning hyper-parameters, $ λ1, λ2$, do not show any meaningful trade-offs but yield worse performance.
Comments, suggestions and typos: (1). it is unclear whether the symbol $g$ (line 220) denotes a scalar or a vector (normally, boldface indicates vectors, yet it appears to be a scalar here). 
(2). The dimensional notation $R^ {1.75T\times D}$ (line 282) raises questions about why a decimal factor is used in a dimension specification. 
(3). Where are the decoders mentioned in line 358 in Figure 2?
(4). it is unclear whether M at line 301 is a scalar or a matrix.
(5). How does $t^{TS}_{(i)}$ corelate with $t^{Text}_{(i)}$? Since they are both time points.
(5). the description of the attention mechanism within the CTPD lacks precise mathematical detail, such as indices T and K are not clearly connected to key variables.
(6). the encoding part in the figure 2 is confusing: the vector operation in the figure cannot correspond to the final matrix outputs; The similar confusion raises in the corresponding description in section 3.2.
Overall assessment scores: 2
Metareview score: 3.5
Metareview: The paper introduces CTPD (Cross-Modal Temporal Pattern Discovery), a framework for integrating multivariate time series and clinical text data in electronic health records. It captures high-level temporal patterns across modalities, improving predictions of in-hospital mortality and phenotype classification. CTPD aligns irregular time series and text into sequences and uses a shared prototype-based slot attention mechanism to extract K temporal pattern embeddings. The approach outperforms prior methods on MIMIC-III benchmarks, achieving state-of-the-art results.
Summary of reasons to publish: The paper is resubmitted to this cycle, the submission has been revised from last cycle.
Same as before, the strengthes include the problem investigated is important, the novelty at some extent, the writing is clear and easy to follow. 
The performance gain and so on.