Hierarchical Pretraining on Multimodal Electronic Health Records
Authors: Xiaochen Wang
Junyu Luo
Jiaqi Wang
Ziyi Yin
Suhan Cui
Yuan Zhong
Yaqing Wang
Fenglong Ma
Venue: EMNLP 2023 Main (2023)
Keywords: Clinical Text
Multimodal Learning
Pretraining
Electronic Health Records
TL;DR: A multimodal pretraining archtecture targeting at Electronic Health Records hierarchy.

Abstract: Pretraining has proven to be a powerful technique in natural language processing (NLP), exhibiting remarkable success in various NLP downstream tasks. However, in the medical domain, existing pretrained models on electronic health records (EHR) fail to capture the hierarchical nature of EHR data, limiting their generalization capability across diverse downstream tasks using a single pretrained model.
To tackle this challenge, this paper introduces a novel, general, and unified pretraining framework called MedHMP, specifically designed for hierarchically multimodal EHR data. The effectiveness of the proposed MedHMP is demonstrated through experimental results on eight downstream tasks spanning three levels. Comparisons against eighteen baselines further highlight the efficacy of our approach.

Hierarchical Pretraining on Multimodal Electronic Health Records
Xiaochen Wang', Junyu Jiaqi Wang', Ziyi Yin', Suhan Cui',
Yuan Zhong', Yaqing Wang". Fenglong Ma'
'The Pennsylvania State University, *Google Research xewang, junyu, jqwang, ziyiyin, suhan, yuanzhong, fenglong) @ @ssu.ddu Syagingwang@googe google.com
Abstract
Patient
Admission
Time
Pretraining has proven be a powerful tech- nique in natural language processing (NLP), exhibiting remarkable success in various NLP downstream tasks. However, in the medical domain, existing pretrained models on elec- tronic health records (EHR) fail to capture the hierarchical nature of EHR data, limiting their generalization capability across diverse down- stream tasks using a single pretrained model. To tackle this challenge, this paper introduces a novel, general, and unified pretraining frame- work called MEDHMP'. specifically designed for hicrarchically multimodal EHR data. The effectiveness of the proposed MEDHMP is demonstrated through experimental results on eight downstream tasks spanning three levels. Comparisons against eighteen baselines further highlight the efficacy of our approach.
Demographie information
ICD
Drug Codes
Stay-level Data
Clinical
Codes
Note
Clinical
Monitoring
Readings
hour
Figure An illustration of EHR hierarchy.
codes. These models pretrained on a single type of data are too specific, significantly limiting their transferability. Although some pretraining mod- els (Li al., 2022a, 2020; Meng et al., 2021) are proposed multimodal EHR data?, they ignore the heterogeneous and hierarchical characteristics of such data.
1 Introduction
The EHR data, as depicted in Figure 1, ex- hibit hicrarchical structure. At the patient level, the EHR systems record demographic information and capture multiple admissions/visie in time- ordered manner. Each admission represents spe- cific hospitalization period and contains multiple stay records, International Classification of Dis- eases (ICD) codes for billing, drug codes, and a cor- responding clinical note. Each stay record includes hourly clinical monitoring readings like heart rate, arterial blood pressure, and respiratory rate.
Pretraining is a widely adopted technique in nat- ural language processing (NLP). entails train- ing a model on large dataset using unsupervised learning before fine-tuning it on specific down- stream task using a smaller labeled dataset. Pre- trained models like BERT (Devlin et al., 2018) and GPT (Radford al., 2018) have demonstrated remarkable success across range of NLP tasks, contributing to significant advancements in various NLP benchmarks.
In the medical domain, with the increasing availability of electronic health records (EHR), researchers have attempted to pre-train domain- specific models to improve the performance of var- ious predictive tasks further (Qiu al., 2023). For instance, ClinicalBERT (Huang al., 2019) and ClinicalT5 (Lehman and Johnson, 2023) are pre- trained on clinical notes, and Med2Vee (Choi et al., 2016a) and MIME (Choi et al., 2018) on medical
In addition to the intricate hierarchy of EHR data, the prediction tasks vary across levels. As move from the top the bottom levels, the pre- diction tasks become more tine-sensitive Patient- level data are usually used to predict the risk of patient suffering from potential diseases after six months or one year, i.e., the health risk prediction
Most of the multimodal pretraining models use medical images and other modalities, such as (Hervella et al., 2021). However, it is impossible to link EHR data and medical images in practice due to data privacy issues.
'Surree codes are available at htttp:://githbb TLacchenWang-PSSUMeeeeee PSU/MedHMP.

---

task. Admission-level data are employed for rela- tively shorter-term predictions, such as readmission within 30 days. Stay-level data are typically uti- lized for hourly predictions, such as forecasting acute respiratory failure (ARF) within a few hours. Designing an ideal "one-in-all" medical pretrain- ing model that can effectively incorporate multi- modal, heterogeneous, and hierarchical EHR data as inputs, while performing self--upervised learn- ing across different levels, is a complex undertak- ing. This complexity arises due to the varying data types encountered at different levels. At the stay level, the data primarily consist of time-ordered numerical clinical variables. However, the ad- mission level, the data not only encompass sequen- tial numerical features from stays but also include sets of discrete ICD and drug codes, as well as un- structured clinical notes. As a result, it becomes challenging to devise appropriate pretraining tasks capable of effectively extracting knowledge from the intricate EHR data.
sion prediction at the admission level, and health risk prediction at the patient level. Through our experiments, we validate the effectiveness of the proposed MEDHMP by comparing it with state- of-the-art baselines. The results obtained clearly indicate the valuable contribution of MEDHMP in the medical domain and highlight its superior per- formance enhancements in these predictive down- stream tasks.
2 Methodology
As highlighted in Section 1, EHR data exhibit con- siderable complexity and heterogeneity. To tackle this issue, we introduce MEDHMP as solution that leverages pretraining strategies across multi- ple modalities and different levels within the EHR hierarchy achieve unification. In the following sections, we present the design details of the pro- posed MEDHMP.
2.1 Model Input
In this paper, we present a novel Hicrarchical Multimodal Pretraining framework (called MEDHMP) to tackle the aforementioned chal- lenges in the Medical domain. MEDHMP simultaneously incorporates five modalities as inputs, including patient demographics, temporal clinical features for stays, ICD codes, drug codes, and clinical notes. To effectively pretrain MEDHMP, we adopt a "bottom-to to-nn-ttoupp""o approach and introduce level-specific self-supervised learning tasks. At the stay level, we propose reconstructing the numerical time-ordered clinical features. We devise two pretraining strategies for the admission level. The first focuses modeling intra-modality relations by predicting set of masked ICD and drug codes. The second involves modeling inter-modality relations through modality-Ievel contrastive learning. To train the complete MEDHMP model, we utilize a two-stage training strategy from stay to admission levels?.
As shown in Figure 1, each patient data consist of multiple time-ordered hospital admissions, i.e., P = A2, AN], where A; (i E [I, is the i-th admission, and N is the number of ad- missions. Note that for different patients, N may be different. Each patient also has a set of de- mographic features denoted D. Each admis- sion A, consists of multiple time-ordered stay- level data denoted as S,, a set of ICD codes de- noted as C,, a piece of clinical notes denoted as Li and a set of drug codes G, i.c., A, {St, Ci, La, Gi). The stay-level data S; contains sequence of hourly-recorded monitoring stays, i.e., S, [S] S7 .... SM where S represents the feature matrix of the j-th stay, and Mc denotes the number of stays within each admission.
2.2 Stay-level Self-supervised Pretraining
We conduct the self-supervised pretraining in a bottom-to-top way and start from the stay level. When pretraining the stay-level data, we only use S; and D since the diagnosis codes C₁, drug codes G; and clinical notes L; are recorded at the end of the i-th admission. However, demographic in- formation is highly related to patient's clinical monitoring features in general. Due to the moni- toring features being recorded with numerical val- ues, we propose to use a reconstruction strategy as the stay-level pretraining task, as illustrated Figure 2.
We utilize two publicly available medical datasets for pretraining the proposed MEDHMP and evaluate its performance on three levels of downstream tasks. These tasks include ARF, shock and mortality predictions at the stay level, readmis-
'It is important to note that we have not incorporated patient-level pertaining in MEDHMP. This decision is based on the understanding that the relations among admissions in EHR data are not as strong us consecutive words in texts. Ar- hitrary modeling of such relations may impede the learning of stay and admission levels.

---

[m Cart marl
h
pressing the latent representation the unified di- mension size. We obtain the bimodal representa- tion by as follows:
LSTM
Embedding
Encuder
Reconstruction
W (h. W (h. d)
Time
Transformer
Demographic information
6 Softmax( by= Maxxooling(LaayenNnoa (h,, b()),
wi (h)
vd.
| tid
LSTM
Bimodal Representation
Decoder
where (-.-) means the operation of stacking, W h W
h W C &Rd, xd, are trainable parameters, and dp is the unified size of representation.
by
Time
Figure 2: Stay-level supeevueeee pretraining.
Using the fused representation by, MEDHMP then reconstructs the input clinical feature ma- trix S/. Since the clinical features are time-series data, we take by as the initial hidden state of the LSTM decoder LSTMaec to sequentially recon- struct the corresponding clinical feature m LSTMMcc (b²).
2.2.1 Stay-level Feature Encoding
Each stay S < S consists of a set of time- ordered hourly clinical features, i.e., S [m , -. m'rr), where m E Rd is the recorded feature vector at the t-th hour, T is the number of monitoring hours, and dy denotes the number of time-series clinical features. To model the temporal characteristic of S we directly apply long-short term memory (LSTM) network (Hochre- iter and Schmidhuber, 1997) and treat the output cell state by as the representation of the j-th stay, i.e.,
2.2.3 Stay-level Pretraining Loss
After obtaining the reconstructed clinical features [ . ... .r.. we then apply the mean squared error (MSE) as the pretraining loss to train the parameters in the stay-level as follows:
N
M
T
h/ = --LMM (mm), m
m mly
(1)
Estay
mm
m
M*T
where LSTMene the encoding LSTM network.
(3)
2.3 Admission-level Pretraining
2.2.2 Clinical Feature Reconstruction
A naive approach reconstructing the input stay- level feature S is simply applying an LSTM de- coder as (Srivastava al., 2015) does. However, this straightforward approach may not work for the clinical data. The reason is that the clinical feature vector m l C S extremely sparse due to the impossibility of monitoring all the vital signs and conducting all examinations for a patient. To ac- curately reconstruct such sparse matrix, need use the demographic information D as the guid- ance because some examinations are highly related to age or gender, which also makes us achieve the goal of multi-modal pretraining.
The stay-level pretraining allows MEDHMP ac- quire the sufficient capability of representing stays. laying the groundwork for the pretraining at the admission level. Next, we introduce the details of pretraining at this level.
2.3.1 Admission-leve Feature Encoding
As introduced in Section 2.1, each admission A; {S C,, Li, G.). To conduct the self-supervised pre- training, the first step is encode each input latent representation.
In Section 2.2, we can obtain the representation of each hourly feature by using Eq. (2). Thus, we can further have the stay-level overall representa- tion s; by aggregating all hourly representations of S; via a linear transformation as follows:
Specifically, we first embed the demographic in- formation D into a dense vector representation, i.e., d = MLPx(D), where MLP, denotes the multilayer perceptron activated by the ReLU func- tion. To fuse the demographic representation and the stay representation, we propose use trans- former block in which self-attention performed for modality fusion, followed by residual calcula- tion, normalization, and a pooling operation com-
s; W bb::b?..... :.b. b.:
(4)
where (::-) is the concatenation operation. W. RRd, Mdd and b, E IRdd, are parameters.
For ICD codes C, and drug codes Gi, they will be converted to binary vectors and then map them to

---

S;
Trandormar
S
a C
gi
1
Atklling Mask
Codk
G
1
B
MLP
MCP
au \gi
Trandfomer
C₁
S
C2
Ai
A
[Mar]]
+
S
LCL
Traasfermer [Noak]
a,
Adding &
Cheine Nark
C,
G
MLP
ag (C ag \BB ap lp-
B
1,
C
Intra-modality Mask Code Prediction (MCP)
Inter-modality Contrastive Learning (CI.)
Figure 3: Admission-level self-supervised pretraining.
latent representations via MLP layers, which is sim- ilar to the mapping of the demographic information, follows:
end, we first stack all the learned embeddings as follows:
f. = (s, ch. g/ 1,).
(6)
= MLP.(C), (G.)
(5)
Then another transformer encoder block is used to obtain the cross cross-modal admission representation as follows:
For the unstructured clinical notes Ci we directly use a pretrained domain-specific en- coder (Lehman and Johnson, 2023) to generate its representation I,
W W Vd,
à Softmax
W (7)
Using the learned representations, we can con- duct admission-level pretraining. Due to the unique characteristics of multimodal EHR data, we will focus two kinds of pretraining tasks: mask code prediction for indra-modalities and contrastive learning for moer-todalities, as shown in Figure
a, = MaxPooling(LayerNoofff +ā)),
where W w and W E Rd are trainable
parameters.
We can predict the masked codes using the learned admission representation a; using Eq. (7) as follows:
2.3.2 Intra-modality Mask Code Prediction
P = Sigmoid MML me (a:)),
the natural language processing (NLP) domain, mask language modeling (MLM) (Devlin et al., 2018) is prevalent pretraining task encouraging the model to capture correlations between tokens. However, the EHR data within an admission A, are significantly different from text data, where the ICD and drug codes are sets instead of sequences. Moreover, the codes are distinct. In other words, no identical codes appear C, and G,. Thus, essential to design a new loss function to predict the masked codes.
(8)
p Sigmoid MLP mg (AL (a:)),
where the predicted probability vectors pf E R|C and p E R|9
Finally, the MSE loss serves the objective function of the masked code prediction (MCP) task for the intra-modality modeling follows:
N
LMCP
P
c,
N
(9)
Let cº"h E |d([ and g." E RlGl denote the mask indicator vectors, where |C| and [9] denote the distinct number of ICD codes and drug codes, re- spectively. If the j-th ICD code is masked, then ciTh ci 1; otherwise, b] O. Let c' and g denote the embeddings learned for the remaining codes. To predict the masked codes, we need to obtain the admission representation. Toward this
1=1
1 g
112)
where is the element-wise multiplication.
2.3.3 Inter-modality Contrastive Learning The intra-modality modeling aims to learn fea- ture relations within a single modality using other modalities' information. On top of it, we also

---

consider mnter-modality relations. Intuitively, the four representations {s,, C, go, 1+} within A, share similar information. If a certain modality r; E {ss, C, gi; 1.} is masked, the similarity between r; and the aggregated representation a,'r; learned from the remaining ones should be still larger than that between r, and another admission's s represen- tation ay yry within the same batch, where A i.
3.1 Data Extraction
We utilize two publicly available multimodal EHR datasets - MIMIC-III (Johnson et al., 2016) and MIMIC-IV (Johnson et al., 2020) to pretrain the proposed MEDHMP. We adopt FIDDLE (Tang et al., 2020) to extract the pretraining data and use different levels' downstream tasks to evaluate the effectiveness of the proposed MEDHMP. For the stay-level evaluation, we predict whether the patient will suffer from acute respiratory failure (ARFvshuck/mortaiity within 48 hours by extract- ing data from the MIMIC-III dataset'. For the admission-level evaluation, we rely on the same pipeline for extracting data from the MIMIC-III dataset to prediet the 30-day readmission rate. For the patient-level evaluation, we conduct four health risk prediction tasks by extracting the heart fail- ure data from MIMIC-III following (Choi et al., 2016b) and the data of chronic obstructive pul- monary disease (COPD), amnesia, and heart failure from TriNetX². The details of data extraction and statistics can be found in Appendix A. The imple- mentation details of MEDHMP are in Appendix B.
Based on this intuition, we propose to use the noise contrastive estimation (NCE) loss as the inter- modality modeling objective as follows:
La = 3N
Σ E
Σ
u(r;);
(10)
u(r;)) log
B از=رير cSi((r,ay))))/7
where sim(:--) denotes the cosine similarity, B is the batch size, and the temperature hyperpa- rameter. a,) rr obtained using Eqs. (6) and (7) by removing the masked modality r,. Note that in our design, S; is a trained representation by optimizing the stay-level objective via Eq. (3). However, the other three modality representations are learned from scratch or the pretrained initialization. To avoid overfitting $,, we do not mask the stayllevel representation s; in Eq. (10).
3.2 Stay-level Evaluation
We conduct two experiments to validate the useful- ness of the proposed MEDHMP the stay level.
3.2.1 Stay-level Multimodal Evaluation In this experiment, we take two modalities, i.e., demographics and clinical features, as the model inputs. The bimodal representation by Iearned by Eq. (2) is then fed into a fully connected layer followed by the sigmoid activation function to cal- culate the prediction. We use the cross entropy the loss function finetune MEDHMP.
2.3.4 Admission-leve Pretraining Loss
The final loss function in the admission-level pre- training is represented as follows:
Lddmission = LMCP + XLCL:
(11)
where À is hyperparameter to balance the losses between the intra-modality mask code prediction
We use F-LSTM (Tang et al., 2020), F- CNN (Tang al., 2020), RAIM (Xu al., 2018), and DCMN (Feng et al., 2019) as the baselines. The details of each baseline can be found in Ap- pendix C. We utilize the Area Under the Receiver Operating Characteristic curve (AUROC) and the Area Under the Precision-Recal curve (AUPR) as evaluation metrics.
task and the mnter-modality contrastive learning.
2.4 Training of MEDHMP
We use two-stage training strategy train the proposed MEDHMP. In the first stage, we pre-train the stay-level task via Eq. (3) by convergence. In the second stage, we use the learned parameters in the first stage as initialization and then train the admission-level task via Eq. (11).
The experimental results presented in Ta- ble showcasing the superior performance of MEDHMP compared to the bimodal baselines in all three stay-level tasks. This indicates the profi- ciency of MEDHMP in effectively utilizing both
3 Experiments
In this section, we first introduce the data for pre- training and downstream tasks and then exhibit experimental results (mean values of five runs).
httpps///github com/MLD3/FFIOLE /tM/L/LDD//IIOLEexePPT iments/ tree/ mimaster//iimcceeepprinnnne 9https://trinetx.co com/

---

Task
ARF
Shock
Mortality
MedHMPa
w. MedHMI's
Original
Metric
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
58-
F-LSTM
89.67
10.57
70.28
23.09
81.55
48.62
F-CNN
69.61
10.68
69.27
23.51
80.71
42.29
1.TT
RAIM
59.38
20.02
77.17
DCMN
42.93
MEDHMP
71.66
14.34
71.04
24.19
82.17
47.52
S4-
6.0
60
7.2
Table Results (%) on stay-level tasks.
Transformet
St-
clinical and demographic features. Remarkably, MEDHMP demonstrates a particularly strong ad- vantage when handling tasks with smaller-sized datasets (See Table 8 for data scale). This observa- tion suggests that MEDHMP greatly benefits from effective pre-training procedure, enabling it deliver impressive performance, especially in low-resource conditions.
7.0
AUROC (%)
(UPR
Figure 4: Unimodal evaluation on the ARF task.
Model
AUROC
AUPR
BertLstm
63.35
7.24
LstmBert
60.67
6.84
BertCnn
63.07
7.19
Note that in the previous work (Yang and Wu, 2021), except for the demographics and clinical fea- tures, clinical notes are used to make predictions on the ARF task. We also conducted such experi- ments on the three tasks, and the results are listed in Appendix D. The experimental results still demon- strate the effectiveness of the proposed pretraining framework.
CanBert
61.59
7.04
BertStar
61.28
6.84
StarBert
61).67
6.84
BertEncoder
61.94
6.82
EncoderBert
60.57
7.00
MEDIIMP
67.77
9.34
Table 2: Results (%) on the readmission task.
mission level. In this task, the model will task all modalities as the input, including demograph- ics, clinical features, ICD codes, drug codes, and corresponding clinical note for admission. In this experiment, we first learn the representations, i.e., 8; using Eq. (4), c; and gi via Eq. (5), and I., to obtain the stacked embedding f.. We then ap- ply Eq. (7) to obtain the admission embedding a;- Finally, fully connected layer with the sigmoid function is used for prediction. We still use the cross-entropy loss the optimization function.
3.2.2 Stay-level Unimodal Evaluation
To validate the transferability of the proposed MEDHMP, we also conduct the following experi- ment by initializing the encoders of baselines using the pretrained MEDHMP. In this experiment, we only take the clinical features models' inputs. Two baselines used: LSTM (Hochreiter and Schmidhuber, 1997) and Transformer (Vaswani et al., 2017). We use the pretrained LSTM encoder LSTM in Scction 2.2.1 to replace the original linear encoders in LSTM and Transformer. Our en- coder will be finetuned with the training of LSTM and Transformer.
We follow the existing work (Yang and Wu, 2021) and use its eight multimodal approaches as baselines, which adopt modality-speciin encoders and perform modality aggregation via a gating mechanism. Different from the original model de- sign, we perform pooling operation on the latent representation of multiple clinical time series be- longing specific admission, such that baselines can also take advantage of multiple stays. Details of these models can be found Appendix D. We still use AUROC and AUPR as evaluation metrics.
The experimental results the ARF task are shown in Figure As mentioned in Sec- tion 2.4, we train the LSTM encoder LSTMenc twice. "w. MEDHMP." " means that the base- lines use a well-trained admission-leve LSTMene- "w. MEDHMP." " indicates that the baselines use a stay-level trained LSTM "Original" denotes the original baselines. We can observe that using partially- or well-trained encoders helps improve performance. These results also confirm the neces- sity of the proposed two-stage training strategy.
Admission-level results are listed in Table 2, and observe that the proposed MEDHMP out- performs all baselines. Compared the best base- line performance, the AUROC and AUPR scores of MEDHMP increase 7% and 29%, respectively. These results once again prove the effectiveness the proposed pretraining model.
3.3 Admission-level Evaluation
We also adopt the readmission prediction task within 30 days to evaluate MEDHMP at the ad-

---

Database
MIMIC-II
THNeeX
Task
Heart Failure
Heart Failure
COPD
Amnesia
Metric
AUPR
Fl
KAPPA
AUPR
F1
KAPPA
AUPR
F1
KAPPA
AUPR
F1
KAPPA
LSTM.
57.83
59,40
35,86
50.16
46.08
29.26
50.16
49.34
34,64
48.68
49.64
34,46
LSTM
57.83
56.70
48.20
44.44
26.64
49.52
47.76
33.44
47.92
48.80
32.98
Dipole,
59.71
60.50
37.68
47.70
41.86
25.52
48.92
41.06
28.30
48.74
45.78
30.78
Dipole
59.43
58.63
36.03
47.16
40.16
24.28
49.44
39.48
27.86
48.36
45.63
30.40
RETAIN,
68.71
66.20
47.12
58,16
52.18
35.64
57.62
50.66
38.36
62.70
56,50
43,90
RETAIN
67.76
65,56
45.63
57.50
50.88
34.52
57.40
49.85
37.36
62.52
56.32
43,66
AdaCure.
58.40
59.47
35.77
$7.63
47.98
32.03
54,06
47.10
34.70
62.62
52.56
41.54
AdaCare
59.40
57.58
35.84
55.43
45.13
31.43
56.63
46.60
34.53
61.62
50.54
39.22
HiTANet,
69.42
68.44
50.01
60.12
50.48
36.08
64.04
54.46
43.38
67.54
58.18
47.78
HiTANet
70.36
66.60
46,60
54.76
47.92
32.04
60.10
52.40
39.93
63.08
54.60
43.44
Table Performance (%) of baselines with/without pretraining for the health risk prediction task.
3.4 Patient-level Evaluation
INTANNea
Even though MEDHMP has not been pretrained on patient-level tasks, it is still capable of handling tasks at this level since its unimodal encoders ac- quire the ability to generate a high-que qualityaltty repre- sentation of each admission, thus become feasible to be utilized to boost existing time series-targe targeting models. Health risk prediction, which utilizes sequence of hospital admissions for illness fore- casting, is applied as the task the patient level.
AUPR (N)
50-
00
10
50
100
Propoction Finetuning Data (%)
Figure 5: Performance change with different training
data sizes using HiTANet on the TriNetX amnesia pre- diction task.
In this experiment, the model will take a se- quence of admission-level ICD codes as the input, which is still a unimodal evaluation. We use the fol- lowing approaches as baselines: LSTM (Hochreiter and Schmidhuber, 1997), Dipole (Ma et al., 2017),
0.6
LSTM
LSTMa
Score
0.34
RETAIN (Choi et al., 2016b), AdaCare (Ma et al.. 2020), and HiTANet (Luo et al., 2020). Details of these approaches
0.2
0.1
be found in Appendix E. Fol- lowing previous health risk prediction work (Chen al., 2021; Cui et al., 2022a), we use AUPR, F1, and Cohen's 's Kappa as the evaluation metrics.
Epoch Number Figure 6: Performance changes w.r.t. the number of epochs on the TriNetX heart failure prediction task.
3.4.1 Performance Comparison
low-resource settings such as zero-shot learning and few-shot learning. Inspired by these character- istics, we explore low-resource settings that sim- ulate common real-world health-related scenarios. We replicate the experiments introduced in the pre- vious section but vary the size of the training set from 1 % 100%.
The experimental results are shown in Table where the approach with the subscript "a" denotes the baseline using the pretrained MEDHMP to ini- tialize the ICD code embedding C; via Eq. (5). We can find that introducing the pretrained unimodal encoder from MEDHMP achieves stable improve- ment across most of the baselines and tasks. These results demonstrate the fiexibility and effectiveness of our proposed MEDHMP in diverse medical sce- narios. The knowledge from our pretrained model can be easily adapted to any sub-modia uobooodyy setting.
Figure 5 shows the experimental results using the HiTANet model. We can observe that using the pre- training initialization, HiTANet, always achieves better performance. Even with 10% training data, it achieve comparable performance with the plain HiTANet using 100% data. This promising result confirms that the proposed pretraining framework MEDHMP useful and meaningful for medical tasks, especially when the training data are insuffi- cient.
3.4.2 Influence of Training Size
Intuitively, pretraining could lead to improved ini- tialization performance compared to models trained from scratch, thereby enhancing its suitability in

---

Database
MIMIC-III
TriNetX
Task
Heart Failure
Heart Failure
COPD
Amnesia
Metric
AUPR
Fl
KAPPA
AUPR
F1
KAPPA
AUPR
F1
KAPPA
AUPR
FI
KAPPA
LSTM,+
57.83
59.40
35,86
50.16
46.08
29.26
50.16
49,34
34.64
48.68
49.64
34.46
LSTM,
57.57
58.27
35.67
49.88
44.86
28.58
49.90
47.65
33.77
48.48
48.70
33.52
LSTM
57.83
56.70
33.03
48.20
44.44
26.64
49.52
47.76
33.44
47.92
48.80
32.98
Table 4: Performance (%) of LSTM the health risk prediction task, which is initialized with parameters from MEDHMP, admission-level pertaining only, and without pretraining.
3.4.3 Convergence Analysis with Pretraining
Model
AUROC
AUPR
MEDHMP,
67.77
9.34
In this experiment, we aim to explore whether us- ing pretraining can speed up the convergence of model training. We use the basic LSTM model as the baseline and output the testing performance at each epoch. Figure 6 shows the results. We can observe that at each epoch, the Fl score of LSTM, is higher than that of LSTM, indicating the bene- fit of using pretraining. Besides, LSTM, achieves the best performance at the 5-th epoch, but the F1 score of the plain LSTM still vibrates. Thus, these results clearly demonstrate that using pretraining techniques can make the model converge faster with less time and achieve better performance.
MEDHMP,
65.75
9.08
MEDHMP,
64.87
MEDHMP
64.74
8.61
Table Results (%) the readmission task, where MEDHMP is initialized with bi-level pretraining, admission-level pretraining, stay-level pertaining, and without pretraining.
Mortality
Metric
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
MEDHMP...
71.66
14.34
71.04
24.19
82.17
47.52
MEDHMP,
64.65
10.59
67.94
22.50
79.67
42.66
MEDHMP
64.06
10,80
67.71
23.19
79.04
40.12
Table 6: Results (%) on the stay-level task, where
MEDHMP is initialized with bi-level pretraining, stay- level pertaining, and without pretraining.
4
A Ablation Study
4.1 Hierarchical Pretraining
Code Prediction (MCP) and Contrastive Learning (CL) losses. We compare MEDHMP pretrained with all loss terms, with MCP and stay-level loss. with CL and stay-level loss, and stay-level loss only, respectively. Results presented in Table 7 clearly demonstrate the efficacy of each proposed loss term as well as the designed pretraining strat- egy. Besides, lacking each of them results in per- formance reduction, highlighting that combining intra- and anter-modalityy modeling is indispensable for boosting the model comprehensively.
For the comprehensive analysis of the effect of stay- and admission-level pretraining, we perform abla- tion studies spanning downstream tasks at all three levels. Results of patient- admission-level and stay-level tasks are listed in Table 4, and respectively. The subscripts "a" (admission) and "s" (stay) in these tables indicate which pretrained model is used as the initialization of MEDHMP.
From the results of all three tables, can observe that the combination of both stay- and admission-level pretraining manifests superior per- formance, further underlining the necessity of adopting hierarchical pretraining strategies. Be- sides, compared with the model without any pre- training techniques, merely using part of the pro- posed pretraining strategy for initialization can im- prove the performance. These observations imply the correct rationale behind our design of hierarchi- cal pretraining strategies.
Related Work
Predictive modeling using EHR data has attracted significant attention in recent years (Cui et al., 2022b; Ma et al., 2021; Xiao et al., 2018; Wang et al., 2022). To enhance predictive performance, pretraining techniques have been explored. In this section, we provide concise overview of studies conducted on pretraining with both single-modal and multimodal EHR data.
4.2 Multimodal Modeling
5.1 Unimodal Pretraining with EHR Data
To investigate how intra- and inter-modality model- ing techniques benefit our admission-level pretrain- ing, we perform an ablation study on three tasks at the sevel to examine the effectiveness of Mask
Several pretrained models have been proposed by utilizing single-modal EHR data. Building upon the success of Large Language Models

---

Task
ARF
Shock
Morality
training but ignores the hierarchical nature of EHRs in their pretraining, resulting in the disadvantage
Merne
AUROC
AUPR
AUROC
ALPR
AUROC
AUPR
MEDHMPaa
24.19
MEDHMP MCCP-
64.91
12.35
68.61
25.29
81.32
47.50
that the pretrained models cannot tackle diverse
McSHMPOGEe
13.88
22.81
MrGHHMP,
64.65
10.59
67.94
22.50
79.67
42.66
downstream tasks at different levels.
Table 7: Ablation results (%) regarding MCP and CL
the readmission task.
Conclusion
In this puper, present a novel pretraining model called MEDHMP designed to address the hierarchi- cal nature of multimodal electronic health record (EHR) data. Our approach involves pretraining МEDHMP at two levels: the stay level and the admission level. At the stay level, MEDHMP uses reconstruction loss applied to the clinical fea- tures the objective. At the admission level, we propose two losses. The first loss aims to model intra-modality relations by predicting masked med- ical codes. The second loss focuses on captur- ing inter-modality relations through modality-leve contrastive learning. Through extensive multi- modal evaluation on diverse downstream tasks at different levels, we demonstrate the significant ef- fectiveness of MEDHMP. Furthermore, experimen- tal results on unimodal evaluation highlight its ap- plicability in low-resoure clinical settings and its ability accelerate convergence.
(LLMs) (Devlin al., 2018; Radford et al., 2018) in NLP, researchers have endeavored to train medical- language models using clinical notes (Li et al., 2022b; Lehman and Johnson, 2023; Alsentzer et al., 2019; Peng et al., 2019) and PubMed data (Luo et al., 2022; Lee et al., 2020; Yuan et al., 2022; Jin et al., 2019; Warikoo et al.. 2021). However, these models primarily rely on mask language modeling techniques for pretrain- ing, thereby overlooking the distinctive characteris- tics of medical data.
Given the time-ordered nature of admissions, medical codes can be treated as sequence. Some pertaining models have proposed to establish repre- sentations of medical codes (Rasmy et al., 2021; Li et al., 2020: Shang et ul., 2019; Choi et al., 2016a, 2018). Nevertheless, these studies still adhere to the commonly used pretraining techniques in the NLP domain. Another line of work (Tipirmeni and Reddy, 2022; Wickstrøm et al., 2022) is to con- duct supf-supe learning on clinical features. However, these pretrained models can only be used for the downstream tasks at the stay level, limiting their transferability in many clinical application scenarios.
Limitations
Despite the advantages outlined in the preceding sections, it important to note that MEDHMP does have its limitations. Owing to the adoption of a large batch size enhance contrastive learning (see Appendix B for more details), it becomes computa- tionally unfeasible to fine-tum the language model acting as the encoder for clinical notes during our admission-level pretraining. As a result, ClinicalT5 is held static to generate a fixed representation of the clinical note, which may circumscribe potential advancements. Additionally, as described in Ap- pendix A, we only select admissions with ICD-9 diagnosis codes while excluding those with ICD-10 to prevent conflicts arising from differing coding standards. This selection process, however, implies that MEDHMP currently lacks the capacity to be applied in clinical scenarios where ICD-10 is the standard for diagnosis code.
5.2 Multimodal Pretraining with EHR data
Most of the multimodal pretraining models in the medical domain are mainly using medical im- ages (Qiu et al., 2023) with other types of modal- ities, such as text (Hervella et al., 2021, 2022a.b; Khare et al., 2021) and tabular information (Hager al., 2023). Only a few studies focus on pretrain- ing on multimodal EHR data without leveraging medical images. The work (Li et al., 2022a, 2020) claims their success on multimodal pretraining uti- lizing numerical clinical features and diagnosis codes. In (Liu et al., 2022), the authors aim to model the interactions between clinical language and clinical codes. Besides, the authors in (Meng al., 2021) use ICD codes, demographics, and top- ies learned from text data as the input and utilize the mask language modeling technique to pretrain the model. However, all existing pretrained work on EHR data still follows the routine of NLP pre-
A Acknowledgement
This work is partially supported by the US National Science Foundation under Grants #2238275 and #2212323, and the US National Institutes of Health under Grant RO1 AG077017.

---

References
core memory network for patient outcome prediction with multimodal data. In 2019 IEEE International Conference Data Mining (ICDM), pages 200-2099 IEEE.
Emily Alsentzer, John R Murphy, Willie Boag, Wei- Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arxiv preprint wwxiv:1904.0333
Paul Hager, Martin J Menten, and Daniel Rueckert. 2023. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23924 23935..
Chacha Chen, Junjie Liang, Fenglong Ma, Lucas Glass, Jimeng Sun, and Cao Xiao. 2021. Unite: Uncertainty- based health risk prediction leveraging multi-sourced data. In Proceedings of the Web Conference 2021, pages 217-226.
Álvaro Hervella, José Rouco, Jorge Novo, and Marcos Ortega. 2021. Self-supervised multimodal reconstruction pre-training for retinal computer- aided diagnosis. Expert Syxtems with Applications, 185:11 15598.
Changyou Chen, Jianyi Zhang, Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng, and Trishul Chilimbi. 2022. Why do we need large batch sizes in contrastive learning? gradient-bias perspective.
Alvaro S Hervella, José Rouco, Jorge Novo. and Mar- cos Ortega. 2022a. Multimodal image encoding pre- training for diabetic retinopathy grading. Computers in Biology and Medicine, 143:105302.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR.
Álvaro S Hervella, José Rouco, Jorge Novo, and Marcos Ortega. 2022b. Retinal microaneurysms detection using adversarial pre-training with unlabeled multi- modal images. Information Fusion, 79:146-161 1611611
Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier rejedor-Sojo, and Jimeng Sun. 201 6a. Multi-layer representation learning for medical con- cepts. In proceedings of the 22nd ACM SIGKDD in- ternational conference on knowledge discovery and data mining, pages 1495-1504.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 973) 1780.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016b. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29.
Qiao Jin, Bhuwan Dhingra, William Cohen, and Xinghua Lu. 2019. Probing biomedical embeddings from language models. In Proceedings of the 3rd Workshop Evaluating Vector Space Representa- tions for NLP, pages 82-89.
Edward Choi, Cao Xiao, Jimeng Sun, and Walter F Stewart. 2018. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. Advances in Neural Information Processing Systems, 2018:4547-4557.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2020. Mimic-iv. PhysioNet. Available online at: htps::/physionet. orglconten//mmmiciv// accceiisse/..teeeeeeeee August 23, 2021).
Suhan Cui, Junyu Luo, Muchao Ye, Jiaqi Wang, Ting Wang, and Fenglong Ma. 2022a. Medskim: De- noised health risk prediction via skimming medical claims data. In 2022 IEEE International Conference on Data Mining (ICDM), pages 81-90. IEEE.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessi- ble critical care database. Scientific data, 3(1):1-9.
Suhan Cui, Jaqqi Wang, Xinning Gui, Ting Wang, and Fenglong Ma. 2022b. Automed: Automated medical risk predictive modeling on electronic health records. In 2022 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM), pages 948-953 LEEE.
Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, and CV Jawahar. 2021. Mmbert: Multimodal bert pretraining for improved medical vqa. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1033-1036 IEEE.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint 1rXiv:1810.0440
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformaticy, 36(4): 1234-12234-220..
Yujuan Feng, Zhenxing Xu, Lin Gan, Ning Chen, Bin Yu, Ting Chen, and Fei Wang. 2019. Demn: Double

---

Eric Lehman and Alistair Johnson. 2023. Clinical-t5:: Large language models built using mimic clinical text.
Yiwen Meng. William Speier, Michael K Ong. and Corey W Arnol.. 2021. Bidirectional representa- tion learning from transformers using multimodal electronic health record data to predict depression. JEEE journal of biomedical and health informatics. 25(8):3121-3122
Yikuan Li. Mohammad Mamouei, Gholamreza Salimi- Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewiez, and Kazem Rahimi. 2022a. Hi-behrt: Hierarchical transformer-bas model for accurate prediction of clinical events us- ing multimodal longitudinal electronic health records. IEEE Journal of Biomedical and Health Informatics.
Yifan Peng. Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning in biomedical natural language process- ing: an evaluation of bert and elmo on ten bench- marking datasets. arXiv preprint arXiv: 1906.05476.
Yixuan Qiu, Feng Lin, Weitong Chen, and Miao Xu. 2023. Pre-training in medical data: A survey. Ma- chine Intelligence Research, 20(2):147-179
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. 2020. Behrt: transformer for elec- tronic health records. Scientific reports, 10(1):1-12.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin longformer
Wang, and Yuan Luo. 2022b. and clinical-bigbird: Transformers for long clinical sequences. arXiv preprint arXiv:2201. 18838.
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi 2021. Med-bert: pretrained contextual- ized embeddings health records for disease prediction. NPJ digital medicine, 4(1):86.
large-scale structured electronic
Sicen Liu, Xiaolong Wang, Yongshuai Hou, Ge Li, Hui Wang, Hui Xu, Yang Xiang, and Buzhou Tang. 2022. Multimodal data matters: Language model pre-training over structured and unstructured elec- tronic health records. IEEE Journal of Biomedical and Health Informatics.
Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019. Pre-training of graph augmented transform- ers for medication recommendation. arXiv preprint arXiv:1906.00336
Junyu Luo, Muchao Ye, Cao Xiao, and Fenglong Ma. 2020. Hitanet: Hierarchical time-aware attention net- works for risk prediction on electronic health records. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery Data Mining, pages 647-656.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. 2015. Unsupervised learning of video representations using Istms. In International confer- ence on machine learning, pages 843-852. PMLR.
Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens. 2020. Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for struc- tured clinical data. Journal of the American Medical Informatics Association, 27(12):1221-193
Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang. Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformaties. 23(6).
Sindhu Tipimeni and Chandan K Reddy. 2022. Self- supervised transformer for sparse and irregularly sam- pled multivariate clinical time-series. ACM Transac- tions on Knowledge Discovery from Data (TKDD). 16(6):1-17
Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. 2017. Dipole: Diagnosis prediction in healthcare via attention brased bidirec- tional recurrent neural networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1903- 1911.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Fenglong Ma. Muchao Ye, Junyu Luo, Cao Xiao, and Jimeng Sun. 2021. Advances in mining heteroge- neous healthcare data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages
Jiaqi Wang, Cheng Qian, Suhan Cui, Lucas Glass, and Fenglong Ma. 2022. Towards federated covid-19 vaccine side effect prediction. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 437-453 337-452. Springer.
Liantao Ma. Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao. and Xinyu Ma. 2020. Adacare: Explainable clin- ical health status representation learning via scale- adaptive feature extraction and recalibration. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 825-832.
Neha Warikoo, Yung-Chun Chang, and Wen-Lian Hsu. 2021. Lhert: Lexically aware transformer-basee bidirectional encoder representation model for leam- ing universal bio-entity relations. Bioinformatics. 404

---

Kristoffer Wickstrøm. Michael Kampffmeyer. Karl Øyvind Mikalsen, and Robert Jenssen. 2022. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recognition Letters, 15.:54661.
is 7,686, and the number of unique drug codes [g] is 1.701. Finally, we get 99,000 admissions with 100,563 stays for pretraining MEDHMP.
The three datasets extracted from TriNetX are supervised by clinicians. We employ the extrac- tion method described in (Choi et al., 2016b) to identify case patients. Specifically, we identify the initial diagnosis date and utilize the patient's histor- ical data leading up to a six-month window, where the diagnosis date marks its end. This approach ensures that we prevent label leakage and success- fully accomplish the objective of early prediction. Three control cases are chosen for each positive case based on matching criteria such as gender, age, race, and underlying diseases. For control patients,
Cao Xiao, Edward Choi, and Jimeng Sun. 2018. Oppor- tunities and challenges in developing deep learning models using electronic health records data: sys- tematic review. Journal of the American Medical Informatics Association, 25(10):1419-142)
Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. 2018 Raim: Re- current attentive and intensive model multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD international conference Knowl- edge Discovery & Data Mining, pages 2565-2573
Bo Yang and Lijun Wu. 2021. How to leverage the multimodal EHR data for better medical prediction? In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
the last 50 visits in the database. The statistics of data used for both pretraining and downstream tasks can be found in Table 8.
B Implementation and Configuration
Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022. Biobart: pretraining and evaluation a biomedical generative language model. arXiv preprint arXiv:2204.03905
All models were implemented using PyTorch 2.0.0 and Python 3.9.12. Preprocessing and experiments were conducted in the Ubuntu 20.04 system with 376 GB of RAM and two NVIDIA A100 GPUs.
A Data Processing
Each experiment was repeated five times elim- inate randomness, and the mean of the evaluation metrics was reported in all experimental results.
We utilize two publicly available multimodal EHR datasets MIMIC-III (Johnson et al., 2016) and MIMIC-IV (Johnson et al., 2020) to pretrain the proposed MEDHMP. Considering that MIMIC-III uses ICD-9 codes while MIMIC-IV incorporates both ICD-9 and ICD-10 codes, we only select ad- missions with ICD-9 diagnosis codes to avoid po- tential conflicts between different coding standards. To prevent the label leakage issue during the testing stage, we remove all signals related to downstream tasks from the original data.
For unimodal evaluations, we used the same set of hyperparameters, regardless of whether a pre- trained encoder was used, to ensure fair compar- ison. For multimodal evaluations, we either used the hyperparameters reported by the authors of the baselines or suggested in their release codes. For detailed hyperparameters not provided by these au- thors, we used the same hyperparameters as in our model for a fair comparison.
We adopt the EHR-oriented preprocessing pipeline, FIDDLE (Tang al., 2020), for feature and label extraction at the stay level. We standard- ize the length of the clinical monitoring feature to T = 48 hours, which is the upper bound for clini- cal feature-related tasks mentioned in (Tang et al., 2020). We filter out the features with a frequency lower than 5% since extremely sparse features can significantly harm computing efficiency and be memory burdensome. After data preprocessing, each hourly clinical feature m is represented as a 1 --dimensional sparse vector, i.e., d, = 1,318 The demographics for each patient are represented as 73-dimensional sparse vector, i.c., the length of D is 73. The number of unique ICD codes |C|
dp was set to 256 for our pretraining and evalua- tion in downstream tasks. For stay-level pretrain- ing, our model was pretrained for 200 epochs with learning rate of 5e-4, a batch size of 128, and a weight decay of 1e-8. At the admission level. our model was pretrained for 300 epochs, with a learning rate of 2e-5 and a weight decay of le- 8. Following previous works (Chen et al., 2022, 2020), which emphasized the necessity of adopting large batch size in contrastive learning, we set the batch size 4096 to enhance our inter-modality modeling. 7 in contrastive learning loss was set 0.1. The hyperparameter À mentioned in Eq. (11) was set to 0.1 to balance the masked code predic- tion (MCP) and contrastive learning (CL) losses

---

Number of of stays
100,563
Pretraining
Number of admissions
99,000
Level
Dataset
Predictive Task
Total
Positive
Negative
ARF within 48 hours
5,038
402
4,636
Stay
MIMIC-III
Shock within 48 hours
7,182
693
6,489
Mortality within 48 hours
11,695
1,581
10,114
Downstream
Admission
MIMIC-III
Readmission within 30 days
33,179
1,444
31,735
MIMIC-III
Heart Failure after six months
7,522
2,820
4,702
COPD after six months
29,256
7,314
21,942
Patient
TriNetX
Amnesia after six months
11,928
2,982
8,946
Heart Failure after six months
12,320
3,080
9,240
Table Data statisties.
Model
Main Modality
Auxiliary Modalities
Clinical Feature Encoder
BertLstm
Clinical Notes
Clinical Features and Demographies
LSTM
LstmBert
Clinical Features
Clinical Notes and Demographics
LSTM
BertCan
Clinical Notes
Clinical Features and Demographics
CNN
CnnBert
Clinical Features
Clinical Notes and Demographics
CNN
BertStar
Clinical Notes
Clinical Features and Demographics
StarTransformer
StarBert
Clinical Features
Clinical Notes and Demographics
StarTransformer
BertEncoder
Clinical Notes
Clinical Features and Demographies
Transformer
EncoderBert
Clinical Features
Clinical Notes and Demographics
Transformer
Table 9: Baselines for the admission-leve task.
in the stay-level pretraining. The masking rate in the MCP task was set to 15%, following the de- sign of (Devlin et al., 2018). The optimizer used throughout the pretraining stage was AdamW.
for prediction. (3) Raim (Xu et al., 2018) is an attention- based model specially designed for ana- lyzing ICU monitoring data, which uses combina- tion of attention mechanisms and multimodal data integration. (4) DCMN (Feng et al., 2019) com- bines two separate memory networks, one for pro- cessing clinical time series and one for processing static tables. Its dual-attention mechanism design allows the model to aggregate features effectively.
For downstream tasks, we selected the hyper- parameters of our model using Grid Search. The batch size was chosen from the set [16, 32, 64), and the learning rate was searched in the range from 2e-5 to 5e-3. The maximum number of epochs was set to 30, and the patience for early stopping and weight decay were configured to 5 and 1c-2, respec- tively, to avoid overfitting. We found that the SGD optimizer performed better during the fine-tuning procedure.
D Stay-level Experiments with Clinical Notes
All the baselines utilized in the readmission predic- tion task are based on the previous work (Yang and Wu, 2021). In their study, the authors investigate various combinations of unimodal encoders and employ a gating mechanism for modality aggrega- tion. In this approach, one modality is considered the main modality, and the embeddings from the other modalities are added as auxiliary modalities. Specific details regarding the composition of these baselines, including how the unimodal encoders are combined, can be found in Table 9.
C Stay-level Experiments
Besides unimodal baselines mentioned in Section 3.2.2, the following approaches serving as base- lines in the multimodal evaluation at the stay level are listed below: (1) F-LSTM (Tang et al., 2020) is classic Long Short-Term Memory (LSTM) model taking concatenation of clinical features and demo- graphic information as input. (2) F-CNN (Tang et al., 2020) is typical Convolutional Neural Net- work (CNN) architecture using the concatenation of clinical features and demographic information
The experimental results are presented in Ta- ble 10. It is evident that relying solely on a single modality, such as clinical notes, inadequate for

---

ARF
Shock
Mortality
Modalities
Models
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
ClinicalTS (Lehman and Johnson, 2023)
50.06
5.92
56.76
13.01
72.18
26.55
Clinical Notes
ClinicaBERT (Huang et al., 2019)
51.75
7.60
44.28
9.89
58.17 17
16.42
BertLstm (Yang and Wu, 2021)
64.90
9.03
70.22
23.32
80.80
45,59
LstmBert (Yang
2021)
61.74
9.79
66.31
22.56
78.31
42.31
BertCan (Yang and Wu, 2021)
67.04
9.49
64.34
21.13
81.12
44.01
Demographics
ChnBert (Yang and Wu, 2021)
63.38
10.31
66.40
22.46
77.25
36.43
+ Clinical Features
BertStar (Yang Wu, 2021)
58.11
6.86
59.24
19.58
76.24
36.12
+ Clincial Notes
StarBert (Yang and Wu, 2021)
51.96
5.73
58.89
16.92
76.19
34.87
BertEncoder (Yang and Wu, 2021)
61.30
9.06
52.88
14.66
74.95
35.02
EncoderBert (Yang and Wu. 2021)
62.65
7.44
61.39
18.66
74.35
33.68
MEDHMP (ours)
71.67
11.05
70.57
24.30
82.06
42.18
Table 10: Comparison results (%) of stay-level tasks using clinical notes.
achieving accurate predictions when compared to multimodal baselines. Among all the multimodal models, proposed MEDHMP consistently out- performs the others in the majority of scenarios. These results highlight significance of integrating multimodal information in health predictive modeling tasks and (2) the effi- cacy of the proposed pretraining technique.
Task
Mortality
Metric
AUROC
AUPR
F-LSTM
82.30
45.01
F-CNN
76.37
35.11
key findings: (1) the
RAIM
83.64
46.40
DCMN
83.57
46.96
MEDHMP
84.43
49.00
Table 11: Results (%) on stay-level tasks.
E Patient-level Experiments
AUPRC (Area Under the Precision-Resd Curve) measures the area bencath the Precision-Recall curve, a plot of the precision against recall for different threshold values.
Baselines regarding the patient-level task are listed below. (1) LSTM(Hochreiterr and Schmidhuber, 1997) is a typical backbone model appearing in time series forecasting tasks. (2) HiTANNe((Lu et al., 2020) adopts the time-aware attention mech- anism design that enables itself to capture the dy- namic disease progression pattern. (3) (Ma al., 2017) relies the combination of bidirec- tional GRU and attention mechanism analyze se- quential visits of patient. (4) AdaCare(M) Ma et al., 2020) applies the Convolutional Neural Network for feature extraction, followed by a GRU block for prediction. (5) Retain (Choi et al., 2016b) utilizes the reverse time attention mechanism to capture dependency between various visits of patient.
F1 Score is the harmonic mean of precision and recall, offering balance between the two when their values diverge.
Cohen's Kappa is a statistic that measures inter-rater agreement for categorical items, ac- counting for the possibility of the agreement occurring by chance.
G Experiments EICU Database
To further validate the transferability of our pro- posed MEDHMP, we conduct experiments using data from additional medical databases, i.e., eICU Results can be found in Table 11. Our proposed MEDHMP shows superior performance consistent with experiments on the MIMIC-III database, im- plying excellent capability of learning general medical features.
F Evaluation Metrics
Evaluation metrics used in our experiments are listed below:
AUROC (Area Under the Receiver Operat- ing Characteristic Curve) represents the like- lihood that classifier will rank randomly chosen positive instance higher than ran- domly chosen negative instance. It provides an aggregate measure of performance across all possible classification thresholds.
Chtpp://cicn--rd.mittccuuu


Reviews:
[{"review_id": "O6mDjHpgbe", "reviewer": "Reviewer_TZeS", "summary": "", "questions": "- **Question A**: For most experiments, several simple but effective baselines (with the same size of parameters) could be included to show the efficacy of pre-training: 1) the fine-tuned model without any pretraining; 2) the fine-tuned model with only stay-level pretraining ($L_{stay}$); 3) the fine-tuned model with only admission-level pretraining ($L_{MCP} + L_{CL}$). Have you conducted these baselines for most experiments?\n- **Question B**: Do we really need two objectives in the admission-level pretraining? Have you conducted ablation studies for intra-/inter-modalities? For example, compare four types of experiments such as (1) $L_{stay} \\rightarrow L_{MCP} + L_{CL}$ vs. (2) $L_{stay} \\rightarrow L_{CL}$ vs. (3) $L_{stay} \\rightarrow L_{MCP} $ vs. (4) only $L_{stay}$\n- **Question C**: There are several possible admission-level tasks, but only one is presented here. Could you explain the reasoning behind this choice?\n- **Question D**: In the \"stay-level unimodal evaluation\" section, could you clarify how this experiment specifically contributes to validating the generalizability of MedHMP? (e.g., what aspects of the evaluation are designed to test generalizability, and how do the results support this claim?)\n- **Question E**: I am concerned about the generalization of this approach, as it is based on medical codes. Could this pre-trained model be applied to other medical databases such as eICU?\n- **Question F**: On Line 250, the term \"intra-modality\" may be causing confusion, especially in the context of the model's use of admission representation by fusing different modalities. This confusion was heightened for me when reading the context in Lines 291-293. Could you elaborate on the definitions of \"intra-modality\" and \"inter-modality\"?\n- **Question G**: What is the motivation of using other private clinical benchmark dataset (from TriNetX)?\n- **Question H**: Considering that this work is specific to the EHR domain, even though it tackles the multi-modal pre-training scheme, what would be the main benefits of this work to the general NLP community, and how might it also be beneficial for the clinical NLP community?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s invaluable comments and constructive suggestions, which significantly improve the quality of our paper. We hope the responses below can adequately address the reviewer’s concerns.\n\n`>>Question A:` __*Efficacy of pre-training at different levels.*__\n\n`>>Response:`Thanks for pointing this out. To validate the performance of the three approaches mentioned by the reviewer, we conduct the following three experiments at different levels.\n\nWe first compare the three approaches mentioned by the reviewer with the proposed MedHMP on the admission-level task - readmission prediction, since this level task can validate the three baselines simultaneously. We then use the tasks at the stay level to exhibit the performance of the models without pre-training and using stay-level pre-training only. Finally, we use the patient-level task - risk prediction - to show the performance of the models without pre-training and using the admission-level pre-training only. The results are shown in the following three tables:\n\n\n| Admission-level Task         |   Readmission   | | \n|----------|-------|-------| \n|           | AUROC | AUPR  |\n| Stay + MCP + CL (MedHMP) | __67.77__  | __9.34__ | \n| MCP + CL      | 65.75 | 9.08 | \n| Stay Only      | 64.87 | 8.60 | \n| Without Pretraining       | 64.74 | 8.61 | \t\n\n\n|Stay-level Tasks          |   ARF  ||     Shock     ||    Mortality   | |\n|----------|-------|-------|-------|-------|--------|-------|\n|          | AUROC | AUPR  | AUROC | AUPR  | AUROC  | AUPR  |\n| MedHMP | __71.66__ | __14.34__ | __71.04__ | __24.19__ | __82.17__  | __47.52__ |\n| Stay Only | 64.65  | 10.59  | 67.94  | 22.50  | 79.67   | 42.66  |\t\n| Without Pretraining | 64.06  | 10.80 | 67.71  | 23.19  | 79.04   | 40.12  |\n\n| Patient-level Task (LSTM) | |MIMIC             |        | |Heart Failure         |       | | COPD          |     |   | Amnesia |               |\n|--------------------------|-----------|--------|--------|---------------|--------|--------|---------|--------|--------|---------|--------|--------|\n|                          | AUPR      | F1     | KAPPA  | AUPR          | F1     | KAPPA  | AUPR    | F1     | KAPPA  | AUPR    | F1     | KAPPA  |\n| Stay + Admission (MedHMP)| __57.83__ | __59.40__ |__35.86__ | __50.16__ | __46.08__ | __29.26__ | __50.16__ | __49.34__ | __34.64__ | __48.68__ | __49.64__ | __34.46__ |\n| MCP + CL                 | 57.57     | 58.27   | 35.67   | 49.88       | 44.86   | 28.58   | 49.90    | 47.65   | 33.77   | 48.48    | 48.70   | 33.52   |\n| Without Pretraining      | __57.83__ | 56.70   | 33.03   | 48.20       | 44.44   | 26.64   | 49.52    | 47.76   | 33.44   | 47.92    | 48.80   | 32.98   |\n\n\nFrom the results of all the three tables, we can observe that the proposed MedHMP outperforms those baselines consistently, which demonstrates the efficacy of the proposed pre-training strategy. Besides, using a part of the proposed pre-training strategy can improve the performance, compared with the model without any pre-training techniques. We will add these results to the final version.\n\n`>>Question B:` __*Lack of ablation study on different components of admission-level loss.*__\n\n`>>Response:`Thanks for the comment. To validate the necessity of these two losses in our admission-level pre-training, we conduct the following experiment as the reviewer suggested. We take the three tasks at the stay-level as an example. The experimental results are shown in the following table. We can observe that using either loss term proposed at the admission level pre-training on top of the stay-level pre-training, the performance is better than that only using the stay-level pre-training. Besides, using them together can further enhance the performance. These results clearly demonstrate the efficacy of the proposed loss terms and the designed pre-training strategy. We will add these results to the final version.\n\n|          |   ARF  ||     Shock     ||    Mortality   ||\n|----------|-------|-------|-------|-------|--------|-------|\n|          | AUROC | AUPR  | AUROC | AUPR  | AUROC  | AUPR  |\n| Stay + MCP + CL (MedHMP) | __71.66__ | __14.34__ | __71.04__ | __24.19__ | __82.17__  | __47.52__ |\n| Stay + MCP      | 64.91 | 12.35 | 68.61 | 25.29 | 81.32  | 47.50 |\n| Stay + CL       | 62.99 | 13.88 | 70.05 | 22.81 | 80.58  | 44.40 |\n| Stay Only | 64.65  | 10.59  | 67.94  | 22.50  | 79.67   | 42.66  |\n\n`>>Question C:` __*Absence of more admission-level tasks*__\n\n`>>Response:`Thanks for pointing this out. In our experiments, we pre-train the proposed model on the MIMIC datasets. For the admission-level tasks on the MIMIC datasets, hospital readmission prediction is a representative task and has been widely explored by researchers. Another commonly-used task is diagnosis prediction (Yang and Wu, 2021), where ICD codes are used as labels. However, the mask code prediction strategy adopted in our pretraining procedure also takes ICD codes as input. To avoid the issue of label leakage, we do not list this task in our experiments. We will explore other admission-level downstream tasks in our future work.\n\n`>>Question D:` __*Ambiguity of generalizability*__\n\n`>>Response:`Thanks for the comment. The experiments of the stay-level unimodal evaluation aim to demonstrate the efficacy of the pre-trained encoder, which can be used to further enhance the performance of  original baselines. The word “generalizability” is confusing, and we will rephrase the sentence in lines 397-400.\n\n`>>Question E:` __*Concern about generalization on other medical databases*__\n\n`>>Response:`Please note that medical codes are a specific modality in the pre-training. From the results listed in Table 1 in our original paper and Questions A and B, we can observe that even for the tasks without using medical codes, the designed pre-training strategy still performs better than baselines, since MedHMP can pre-train on the stay-level data. To validate the generalization of our model, we extract a new dataset from eICU to predict mortality after 48 hours. The experimental results are shown in the following table. The results still demonstrate the efficacy of the proposed MedHMP on this new dataset. We will add these results to the final version.\n\n|          |    Mortality   ||\n|----------|--------|-------|\n|          | AUROC  | AUPR  |\n| F-LSTM | 82.30  | 45.01  |  \n| F-CNN | 76.37   | 35.11  |\n| DCMN       | 83.64   | 46.40  |\n| RAIM       |  83.57  | 46.96  |\n| MedHMP | __84.43__   | __49.00__  |\n\n`>>Question F:` __*Unclear definitions of intra- and inter-modality*__\n\n `>>Response:` Thanks for pointing this out. Intra-modality modeling aims to capture the feature relations within a specifc modality, such as ICD codes and medication codes, with the proposed masked code prediction. For example, assume that the total number of ICD codes is 5 in a dataset. For an admission, its ICD code binary vector is [1, 0, 1, 1, 0]. Assume that we randomly mask one code, which results in a masked ICD code vector [1, 0, 1, MASK, 0]. Our goal is to predict whether the masked position is 1 or 0.\n\nInter-modality interaction tries to model the relation cross modalities. For instance, a list of ICD codes should have a strong connection with clinical notes that belong to the same admission, while clinical notes for other admission may not be highly irrelevant. By adopting our proposed contrastive learning loss, the model acquires the capability of capturing correspondence between different modalities, hence better estimate the data distribution across modalities.  \n\nWe will clearly elaborate these two terms in the final version to avoid any confusion.\n\n`>>Question G:` __*Motivation of adopting private datasets*__\n\n `>>Response:` Even though we extract a heart failure dataset from the MIMIC-III database in our experiment, the recorded data in the MIMIC-III database are from ICU patients, which is too specific to reflect the characteristics of EHR data. Besides, the average number of admissions for each patient is very small, which is only 2.61. Finally, all the tasks used in the experiments at the stay and admission levels are extracted from the MIMIC databases. In other words, the pre-training and downstream tasks are validated using the data with the same distribution.\n\nTo validate the proposed MedHMP in a more general scenario, we extract three new datasets from the TriNetX database. The EHR data in these datasets record any types of admissions, not only limited to the ICU ones, and the average number of admissions per patient is more than 30. We believe validation on these private datasets can confirm the generalization ability and efficacy of the proposed model. \n\n`>>Question H:` __*Inquiry about broad impact on (Clinical) NLP community*__\n\n`>>Response:` Thank you for posing the insightful question. We posit that our emphasis on integrating the inherent hierarchical nature of data within the pretraining process has broad applicability across the NLP spectrum. Consider semi-structured data like web content, which inherently exhibits a hierarchical structure often intertwined with multiple modalities. Traditional NLP models pretrained on generic corpora might struggle with the noisy text without utilizing additional information. Our pretraining approach, which is capable of leveraging both hierarchical and multimodal attributes, can serves as a beacon for NLP researchers aiming to pretraining their model similarly for a decent adaptability.\n\nFor the clinical NLP community, our work provides a direction that has not been fully explored by previous studies, which fills a research gap where EHR hierarchy has not been considered while pretraining clinical NLP models. We believe that clinical NLP researchers will be inspired by our current exploration, and practitioners are enabled to acquire more effective pretrained language models for their application demands. \n\n__Ethical Concern:__ \n\n`>>Response:` The data in the TriNetX database are de-identified and do not contain any sensitive patient information. We have obtained the access to this database within our institute. Thus, we do not think there are potential ethical concerns when using this database.\n\n__Typos Grammar Style and Presentation Improvement:__\n\n`>>1.` __Demographic features:__ \n\n`>>Response:` Following the previous work (Tang et al., 2020), we extract and utilize the following demographic features: gender, admission type, language, age, wardid (identifier of ward), admission location, marital status, religion, care unit, ethnicity, and insurance type. An example is listed below:\n\n|      Feature          | Value                       |\n|----------------|-----------------------------|\n| Gender         | F                           |\n| Admission type | EMERGENCY                     |\n| Language       | SPAN                        |\n| Age            | 59                          |\n| Wardid | WARDID=50                     |\n| Admission location | CLINIC REFERRAL/PREMATURE     |\n| Marital status | MARRIED                     |\n| Religion | CATHOLIC                    |\n| Care Unit| MICU                      |\n| Ethnicity      | HISPANIC/LATINO - DOMINICAN |\n| Insurance| Medicare |\n\n`>>2.` __Type of notes:__ \n\n`>>Response:` Still following the previous work (Yang and Wu, 2021), we leverage a wide range of clinical notes including not only discharge summary, but also pharmacy notes, nursing notes, physician notes, radiology reports, ECG reports, echo reports, respiratory notes, nutrition notes, rehabilitation notes, social work notes and case management notes in the pretraining and downstream tasks. The inclusion of diverse types of clinical notes ensures that a fair comparison can be performed between our method and baselines mentioned in this work. \n\n\n`>>3.` __Subscript of M:__ \n\n`>>Response:` Thanks for detecting the omission. We will make the modification in our camera-ready version.\n\n`>>4.` __How the model handles various lengths of M:__ \n\n`>>Response:` We adopt the padding operation to ensure the consistency of the number of stays across different admissions. According to our statistics, only less than 2% of admissions possess more than 3 stays. Therefore, we set the maximum number of stays to 3. \n\n\n`>>5.` __Setting of maximum sequence length:__ \n\n`>>Response:` Similar to most Transformer-based language models, the maximum sequence length of Clinical-T5 is set to 512 by default. Moreover, we adopt the preprocessing step suggested by the provider of ClinicalT5 (https://physionet.org/content/clinical-t5/1.0.0/) to reduce the length and noise of clinical text. \n\n\n`>>6.` __Dimension of $f_i$:__ \n\n`>>Response:` The operation that we take to aggregate representations of different modalities is vertical stacking rather than horizontal concatenation. Therefore, the dimension of $f_i$ is still $d_r$, while the representation size of each data sample changes from 1*$d_r$ to 4*$d_r$ after the aggregation. \n\n`>>7.` __About standard derivation:__ \n\n`>>Response:` Thanks for proposing the addition. We will make modifications accordingly in our final version of this study.\n\n\n`>>8.` __Bold error in Table 3:__ \n\n`>>Response:` Thanks for pointing this out. We will correct this error. \n\n`>>9.` __Smaller-sized datasets:__ \n\n`>>Response:` The size of datasets are described in Table 4, based on which we claim that ARF and Shock datasets are smaller in size. We will add the reference of Table 4 to Line 383 to guide readers for a better sense of the data we use. \n\n`>>10.` __Definition of AUROC and AUPRC:__ \n\n`>>Response:` AUROC measures the overall ability of a model to discriminate between the positive class and the negative class across different decision thresholds. AUPR measures the model's precision and recall across different decision thresholds, especially in the context of imbalanced datasets. We will add their clear definitions to the final version of our paper as well as the definitions of other evaluation metrics (F1, KAPPA) we use in our experiments.\n\n\n`>>11.` __Context of Table 6:__ \n\n`>>Response:` Previous work (Yang and Wu, 2021) has their experiments conducted in a different setting, where each admission only corresponds to one stay. In this setting, the admission- and stay-level tasks are mixed up with each other. We argue that this setting ignores the hierarchical nature of EHRs, therefore, we only leverage stay-level information in our experiments regarding ARF, shock and mortality predictions as we treat them as stay-level tasks, and choose readmission prediction as our admission-level tasks, where multiple stays serve as input for each admission. \n\nThe reason we conduct experiments in Table 6 is that we would like to avoid any omission of possible baselines, even if their setting is different from ours. Hence, we include Table 6 in the appendix, which further corroborates the efficacy of MedHMP in contrast to baselines. Moreover, the comparison between Table 6 and Table 2 indicates that our model does not manifest better performance given additional input, validating our assumption that mixing up stay- and admission-level tasks may cause problems due to the mismatching information in different granularities.\n\n\n`>>12.` __Captions improvement:__ \n\n`>>Response:` Thanks for proposing the problem. We will add more detailed captions in our final version to ensure the self-containedness of our work.\n\nWe sincerely appreciate the typos and presentation problems the reviewer has pointed out, and we would like to make modifications on our manuscript accordingly. We will add the clarification above to our paper to improve both rigor and readiness of this work based on your valuable suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- MedHMP is a novel framework for multimodal EHRs, addressing gaps with hierarchical and heterogeneous data.\n- It spans patient, admission, and stay levels, for broad applicability in predictive healthcare.\n- Source code inclusion enhances transparency and reproducibility.\n- Focus on hierarchical EHR data may inspire development in domain-specific pretraining models beyond healthcare.", "reasons_to_reject": "- Insufficient experimental designs/results (Please refer to the \"Questions For The Authors\" section.)\n- Insufficient justification for pre-training decisions (Please refer to the \"Questions For The Authors\" section.)\n- Not sure for generalizability (Please refer to the \"Questions For The Authors\" section.)\n- Potential ethical or privacy considerations (Please refer to the \"Justification For Ethical Concerns\" section.)", "questions_to_authors": "- **Question A**: For most experiments, several simple but effective baselines (with the same size of parameters) could be included to show the efficacy of pre-training: 1) the fine-tuned model without any pretraining; 2) the fine-tuned model with only stay-level pretraining ($L_{stay}$); 3) the fine-tuned model with only admission-level pretraining ($L_{MCP} + L_{CL}$). Have you conducted these baselines for most experiments?\n- **Question B**: Do we really need two objectives in the admission-level pretraining? Have you conducted ablation studies for intra-/inter-modalities? For example, compare four types of experiments such as (1) $L_{stay} \\rightarrow L_{MCP} + L_{CL}$ vs. (2) $L_{stay} \\rightarrow L_{CL}$ vs. (3) $L_{stay} \\rightarrow L_{MCP} $ vs. (4) only $L_{stay}$\n- **Question C**: There are several possible admission-level tasks, but only one is presented here. Could you explain the reasoning behind this choice?\n- **Question D**: In the \"stay-level unimodal evaluation\" section, could you clarify how this experiment specifically contributes to validating the generalizability of MedHMP? (e.g., what aspects of the evaluation are designed to test generalizability, and how do the results support this claim?)\n- **Question E**: I am concerned about the generalization of this approach, as it is based on medical codes. Could this pre-trained model be applied to other medical databases such as eICU?\n- **Question F**: On Line 250, the term \"intra-modality\" may be causing confusion, especially in the context of the model's use of admission representation by fusing different modalities. This confusion was heightened for me when reading the context in Lines 291-293. Could you elaborate on the definitions of \"intra-modality\" and \"inter-modality\"?\n- **Question G**: What is the motivation of using other private clinical benchmark dataset (from TriNetX)?\n- **Question H**: Considering that this work is specific to the EHR domain, even though it tackles the multi-modal pre-training scheme, what would be the main benefits of this work to the general NLP community, and how might it also be beneficial for the clinical NLP community?", "paper_topic_and_main_contributions": "This paper addresses the challenge of combining the hierarchical (i.e., patient-admission-stay) and heterogeneous (i.e., multimodal or varied in data type) nature of EHR data into one pretraining model. This model aims to handle diverse medical predictive tasks, covering eight downstream tasks over three levels.\n\nThe main contributions include:\n- A new two-stage (\"bottom-to-up\") hierarchical multimodal pretraining framework specifically designed for EHR data.\n- The development of level-specific self-supervised learning tasks, including (1) stay-level reconstruction, (2) admission-level intra-modality modeling, and (3) admission-level inter-modality modeling.\n- Comprehensive experiments on diverse clinical benchmarks across two databases, demonstrating the superior performance of MedHMP across various tasks and outperforming eighteen baseline models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L134: Please specify the demographic features used in the model and provide an example.\n- L138: Please specify the types of notes considered (e.g., discharge summary).\n- L143: Please consider the revision of notation from $M$ to $M_i$ for dependency on admission $i$.\n- L231: Please add the context how the model handles varying lengths of $M$ (e.g., padding).\n- L240-242: Please specify clinical notes types and explain text processing if too lengthy (e.g., maximum sequence length).\n- L272: Please clarify dimensions of $f_i$; if 4*$d_r$, state explicitly and correct the manuscript.\n- L333: Please include standard deviations in results for full statistical context.\n- Table 3: Please correct one bold formatting error.\n- L383: Please define \"smaller-sized datasets.\"\n- L472: Please provide information on AUROC as well as AURPC.\n- L982: Please check context of \"Table 6\" in admission-level task section; clarify if needed.\n- Figures 1/2/3: Please ensure captions are self-explanatory.", "Justification_for_Ethical_Concerns": "This paper employs a dataset from TriNetX, which is not open-sourced, for the validation of certain experiments. This raises several potential ethical concerns that warrant a separate review:\n- The absence of an explicit section or statement addressing ethical considerations in the paper indicates a lack of reflection on these crucial aspects. \n- The use of a non-open-sourced dataset may involve patient data or other sensitive information. The lack of transparency and details about the data collection, consent procedures, and privacy safeguards can raise questions about the ethical handling of potentially sensitive information.\n- https://trinetx.com/real-world-resources/publications/trinetx-publication-guidelines/ refers to the publication guidelines of TriNetX. A review should be conducted to ensure that the researchers have complied with these guidelines, including any requirements for data use, acknowledgment, or disclosure."}}, {"review_id": "IXmhUjWVBO", "reviewer": "Reviewer_P82s", "summary": "", "questions": "* Regarding the source code: I wonder if the authors will make public the code, including the pre-processing, training, and evaluation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The insights and expertise displayed by the reviewer in their feedback are truly invaluable to us. We summarize the concerns that the reviewer raise as follows:\n\n`>>Question 1:` __*Absent ablation study on different pretraining tasks*__\n\n `>>Response:` Thanks for proposing the concern. We have conducted experiments to investigate the usefulness of different pretraining tasks in MedHMP, and results can be found below:\n\n|          |   ARF  ||     Shock     ||    Mortality   ||\n|----------|-------|-------|-------|-------|--------|-------|\n|          | AUROC | AUPR  | AUROC | AUPR  | AUROC  | AUPR  |\n| MedHMP | __71.66__ | __14.34__ | __71.04__ | __24.19__ | __82.17__  | __47.52__ |\n| Stay + MCP      | 64.91 | 12.35 | 68.61 | 25.29 | 81.32  | 47.50 |\n| Stay + CL       | 62.99 | 13.88 | 70.05 | 22.81 | 80.58  | 44.40 |\n| Stay Only | 64.65  | 10.59  | 67.94  | 22.50  | 79.67   | 42.66  |\n\nAccording to our experiments, the combination of them manifest best performance, thus each pretraining task has been proven to be indispensable. As for the two objectives that we jointly use in admission-level pretraining, MCP appears to be slightly more effective. However, it cannot independently help the model achieve the best performance. Therefore, we would like to claim that each pretraining task can be regarded as key to the downstream performance.  We will add these experimental results to our camera-ready version of this study that serves as a more comprehensive analysis on our pretraining strategy.\n\n`>>Question 2` __*Question about open-source option*__\n\n `>>Response:` We have provided our source codes including training and evaluation in  the supplemental materials, as well as guidance on how to adapt existing pre-processing pipeline. We will prepare and release a more detailed version of these resources after the acceptance of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written. I enjoyed reading it and also learned a lot from it.\n* From the perspective of representation learning, the proposed approach is designed for different data types in an appropriate way. Specifically, the authors use reconstruction for numerical data, classification for the discrete set, and contrastive learning from cross-modal data.\n* The evaluation is comprehensive, across patient-level, admission-level, and stay-level evaluation. Besides, the proposed representation learning works well in all the evaluations.\n\nOverall, I think this is an excellent paper and I will recommend the acceptance.", "reasons_to_reject": "One thing I am curious about except for the downstream evaluation is the analysis of the pre-training analysis. To be specific, there are too many variables in the representation learning process and I wonder about the difficulty of different pre-training tasks. For example, some tasks might be easy since the model finds the shortcut and some might be key to the downstream performance.", "questions_to_authors": "* Regarding the source code: I wonder if the authors will make public the code, including the pre-processing, training, and evaluation.", "paper_topic_and_main_contributions": "Paper Topic: Electronic Health Records, Multi-modal Representation Learning\n\nMain Contribution:\n\nThis paper aims to perform representation learning for Electronic Health Records, which normally contain three levels: patient, admission-level, and stay-level.\n* For stay-level pre-training, it adopts Feature Reconstruction as its pre-training task with the consideration of demographic information.\n* For admission-level pre-training, there are two types of objectives: one is to predict the masked codes and the other is to perform contrastive learning between original features and noisy features.\nAfter such reasonable representation learning, the trained model can achieve promising results in different levels of evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "FM0VnBBF0o", "reviewer": "Reviewer_qKr6", "summary": "", "questions": "How is a stay defined? Would be nice to clarify this more precisely in the paper. I see that you state \"Each stay record includes hourly clinical monitoring readings like heart rate...\". Does this mean that a stay is defined as one of the consecutive hours during an admission? Thus, the super script in S^j refers to the jth hour during an admission? Most of the information is there but would be nice to state these things a bit more precisely. \n\nRelated to the first question, on page 3, line 159, it is stated that \"each stay... consists of a set of time-ordered hourly clinical features. However, I thought a stay referred to an hour during an admission? Also a subscript k is used here, but it is stated that this value k refers to the t-th hour? It may be clearer to use t instead of k here. Now I am wondering if a stay is an overnight stay? Therefore, a single admission can consist of multiple overnights?  ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the time and effort the reviewer dedicated to understanding and commenting on our work. We summarize the concerns the reviewer holds as follows:\n\n`>>Question 1:` __*The description of “stay” is obscure*__\n\n\n `>>Response:` We are more than willing to provide a more detailed explanation of the definition of “stay” to help readers get a better sense of the complex data structure we are aiming to tackle in this work. A simple description of the relation between admission and stay is that, admission may consist of multiple stays as the patient may be under monitored for multiple periods during the admission, while each stay includes multiple hours. In Section 2.1, we introduce the definition of S, where $S_i^j$ is the j-th stay of the patient’s i-th admission. The hourly recorded feature is defined in Section 2.2.1 in the form of $m_{i, k}^j$, which subordinates to the stay $S_i^j$. Hope the above explanation can facilitate the understanding. We do appreciate the suggestion of using t instead of k in terms of denoting the t-th hour. We will adopt this idea to further improve the readability of this study. Thanks for pointing this out!\n\n`>>Question 2:` __*Gradient boosting methods that should have been able to serve as baselines are absent*__\n\n `>>Response:` Gradient boosting is effective for tabular data. However, fusing modalities poses challenges if we use boosting for our demographic table and DNN for other modalities. This is because traditional boosting doesn't produce representations suitable for fusion. We could consider applying boosting to all modalities. Yet, for time series and clinical texts, boosting might not capture sequential nuances effectively. Given these considerations, we feel omitting gradient boosting in a multimodal context is justifiable. Its absence only marginally impacts the study's solidness and rigor. Nonetheless, we're grateful to the reviewer for initiating this insightful discussion on a potential direction.\n\n__Typos Grammar Style and Presentation Improvement:__\n\n`>>1.` __*Absence of clear definition of “well-trained”*__ \n\n `>>Response:` Here, the word “well-trained” refers that we have performed two-stage pretraining on the LSTM encoder, as the counterpart of the one merely trained at stay level. We thank the reviewer for proposing the doubt and we would like to add more descriptions about this part in our camera-ready version of paper. \n\nWe truly value the author's observations and suggestions regarding Figure 3, and we plan to make the adjustments accordingly to the final version of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is clear that the authors put lots of thought into devising this hierarchical pretraining strategy and substantial work into the execution of their ideas. \n\nIt is great that the authors present a comprehensive set of experiments and tasks on various time scales. \n\nThe authors describe how the hierarchical and multi-modal nature of electronic health records can be leveraged during pretraining, which is an important contribution.\n\nThe authors make good use of published baselines that demonstrate the superiority of their method\n\nThe authors make us of compelling metrics, including AUROC, AUPR, F1 and KAPPA.", "reasons_to_reject": "The writing and notation are a bit hard to follow at points. See the \"questions for the authors\" section.\n\nThere is not a comparison to a gradient boosting method. This would serve as a simple baseline that a broader audience would be familiar with. With tabular data, this method can do surprisingly well so it is important to include this. \n\n", "questions_to_authors": "How is a stay defined? Would be nice to clarify this more precisely in the paper. I see that you state \"Each stay record includes hourly clinical monitoring readings like heart rate...\". Does this mean that a stay is defined as one of the consecutive hours during an admission? Thus, the super script in S^j refers to the jth hour during an admission? Most of the information is there but would be nice to state these things a bit more precisely. \n\nRelated to the first question, on page 3, line 159, it is stated that \"each stay... consists of a set of time-ordered hourly clinical features. However, I thought a stay referred to an hour during an admission? Also a subscript k is used here, but it is stated that this value k refers to the t-th hour? It may be clearer to use t instead of k here. Now I am wondering if a stay is an overnight stay? Therefore, a single admission can consist of multiple overnights?  ", "paper_topic_and_main_contributions": "This paper aims to leverage the hierarchical nature of electronic health records to do multi-stage pretraining. They use a reconstruction pretraining strategy to pretrain their model on the most granular, stay level. Then, they use the values that they compute in the first stage to initialize further pretraining using an intra-modality mask code prediction loss and an inter-modality contrastive learning loss. They present superior performance on a range of downstream tasks with varying time-scales. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be helpful to add more information to the caption of Figure 3, so that everything that is being depicted in the figure is clarified. \n\nOn page 6, line 412, what does well-trained refer to?\n\nWould be nice to add a bit more detail to Table 3's caption. \n"}}, {"review_id": "xu3NUkNUbP", "reviewer": "Reviewer_B8GM", "summary": "", "questions": "Why use an LSTM rather than a transformer for the stay-level feature encoding?\n\nCan you ablate for admission-level training only (see above)?\n\nCan you explain section 2.3.2? I am unclear how equation 9 works.\n\nCan you explain section 3.2.2? It is confusing how you modify the Transformer baseline with LSTM_enc because you can't warm-start the Transformer from a pre-trained LSTM.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The comments provided by the reviewer  have been greatly beneficial and we truly appreciate it. We are more than willing to address each question respectively for the clarification.\n\n`>>Question 1:` __*Lack of explanation of using LSTM rather than Transformer*__\n\n`>>Response:` Thanks for noticing this detail. We had considered using Transformer instead of LSTM when designing our architecture. However, the experimental results listed in Table 2 and Table 6 show that baselines using transformer as the encoder of stay-level features usually manifest less competitive performance compared to those using LSTM, which implies that LSTM may be a more appropriate choice as the encoder. Given that the amount of data is relatively limited in the clinical field, it is understandable that sophisticated architecture may not outperform simple models, which explains why LSTM outperforms Transformer as the stay-level encoder. Based on the observation and reasoning, we choose LSTM instead of Transformer as the stay-level encoder in MedHMP.    \t\n\n`>>Question 2:` __*Potential ablation study concerning stay-level pretraining*__\n\n`>>Response:` Thanks for proposing the idea of ablation study. We conduct ablating experiments regarding both patient-level and admission-level tasks, and the results are listed below. \n\n\n| Admission-level Task         |   Readmission   |\n|----------|-------|          \n|           | AUROC | AUPR  |\n| Stay + MCP + CL (MedHMP) |__67.77__ | __9.34__ | \n| MCP + CL      | 65.75 | 9.08 | \n| Stay Only      | 64.87 | 8.60 | \n| Without Pretraining       | 64.74 | 8.61 | \t\n\n\n| Patient-level Task (LSTM) | |MIMIC             |        | |Heart Failure         |       | | COPD          |     |   | Amnesia |               |\n|--------------------------|-----------|--------|--------|---------------|--------|--------|---------|--------|--------|---------|--------|--------|\n|                          | AUPR      | F1     | KAPPA  | AUPR          | F1     | KAPPA  | AUPR    | F1     | KAPPA  | AUPR    | F1     | KAPPA  |\n| Stay + Admission (MedHMP)| __57.83__ | __59.40__ |__35.86__ | __50.16__ | __46.08__ | __29.26__ | __50.16__ | __49.34__ | __34.64__ | __48.68__ | __49.64__ | __34.46__ |\n| MCP + CL                 | 57.57     | 58.27   | 35.67   | 49.88       | 44.86   | 28.58   | 49.90    | 47.65   | 33.77   | 48.48    | 48.70   | 33.52   |\n| Without Pretraining      | __57.83__ | 56.70   | 33.03   | 48.20       | 44.44   | 26.64   | 49.52    | 47.76   | 33.44   | 47.92    | 48.80   | 32.98   |\n\nAccording to our experimental results, even though merely performing admission-level pretraining has already contributed to decent performance promotion, the addition of stay-level pretraing can further boost the gain and enhance model’s capability of handling downstream tasks at different levels. \n\nBesides, the ablating experiment about stay-level input during inference indicates that stay-level input is not only beneficial during training, but also indispensable during the inference, which corroborates the necessity of simultaneously taking the hierarchical input into consideration while handling EHRs.\n\n\n`>>Question 3:` __*Insufficient explanation of Masked Code Prediction (MCP)*__\n\n`>>Response:` It is our pleasure to provide more descriptions about these details. Given an admission, we randomly mask some of its ICD and drug codes (Equation 6), then fuse multimodalities via Transformer (Equation 7), these two steps allow our model to obtain the representation of admission whose part of codes are masked. Based on the representation ($a_i$), our model is expected to calculate the probability vectors ($p^c_i$, $p^g_i$). \n\nThe Equation 9 is designed to urge the model to predict masked codes. We would like to sincerely thank the reviewer for pointing out the notable typo in Equation 9, which significantly introduces confusion. It should get rid of multiplication between probability and masking indicator vector (i.e. $p^c_i \\odot c^m_i$ and $p^g_i \\odot g^m_i$). Instead, the loss is calculated between the original output probability and masking indicator vector, which is equivalent to the representation of masked codes. The correction version of Equation 9 should be:\n\n$L_{MCP}=\\frac{1}{N} \\sum_{i=1}^N (||p^c_i  -  c^m_i ||_2^2 + ||p^g_i  -  g^m_i ||_2^2)$\n\nWe will make the correction in our revision.\n\n\n`>>Question 4:` __*Insufficient explanation of unimodal evaluation experiments*__\n\n`>>Response:` In our setting, we add our pretrained LSTM encoder onto the top of sequential models. Basically, we combine LSTM and Transformer with the pretrained encoder in a cascade pattern rather than a transfer learning pattern where the pretrained model passes parameters to scratch models.\n\n\n`>>Question 5:` __*Absence of patient-level pretraining*__\n\n`>>Response:` As what we claimed on Page 2 footnote 3, currently we take a cautious attitude towards modeling relations between multiple admissions at the patient level. However, we admit that the interaction between admissions can be potentially modeled and we would like to continue work on that in the future. \n\n`>>Question 6:` __*Lack of novelty*__\n\n`>>Response:` Thanks for acknowledging our effort on taking the step of pretraining on fine-grained data. Admittedly, a number of colleagues have contributed a lot to the pretraining in the healthcare domain, however, hierarchical nature of EHR is relatively overlooked in previous explorations and we have taken the action of tackling this research gap with our original pretraining strategy. Besides, we have proposed in-depth analysis on the EHR hierarchy, which might be inspiring to colleagues previously omitting this research problem. We sincerely hope that our work can benefit the whole research community and we would like to continue exploring understudied areas in this direction.\n\n\n`>>Question 7:` __*Ambiguous definition of smaller-sized datasets*__\n\n`>>Response:` Thanks for pointing this out. We have provided statistics of datasets in Table 4. We will add the reference of this table in Line 383 to guide readers for a better sense of the data we use. \n\n`>>Question 8:` __*Lack of explanation of choosing two-stage pretraining strategy*__\n\n`>>Response:` Thanks for proposing the question. Reconstruction is a relatively hard objective compared to the ones we use in admission-level pretraining. While ICD codes, drug codes and other data in admission- and patient-level are relatively clean, clinical time series are highly sparse and noisy, which makes the reconstruction much more difficult than masked code prediction and contrastive learning. Based on this characteristic, we would like to separate the reconstruction from the joint pretraining for the facilitation.\n\n__Typos Grammar Style and Presentation Improvement:__\nWe wholeheartedly thank the author for highlighting the typos and presentation shortcomings, and we are keen to update our manuscript in response. We would give our sincerest appreciation on the reviewer’s assistance on helping us polishing our mathematical formulas.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Great figures, mostly very clear and well-written.\n2. Almost all of the reasonable experiments were done, and the results across multiple datasets and tasks are convincing. It is especially nice to note that the model performs well on smaller datasets, indicating it might generally be better for low-resource settings.\n3. The hierarchical pre-training is a valuable tool for the many tasks that could benefit from data at the stay level as well as clinical notes and ICD codes. If open-sourced, this model would be broadly helpful to anyone doing machine learning with electronic health records.\n", "reasons_to_reject": "1. They acknowledge that there is no patient-level pre-training objective, only stay and admission level ones. The authors point out that this might impede the other training objectives, but seeing as so many health-related tasks (e.g. diagnosis, readmission prediction) could benefit from learning interactions across multiple admissions, this seems like a missing piece.\n2. Some sections are unclear (2.3.2, equation 9 especially and section 3.2.2).\n3. Slightly limited novelty: There has been a fair amount of work on pertaining and even some hierarchical pretraining. However, they do a great job citing related works, and their inclusion of the most fine-grained data, the stay-level data, is novel in multimodal pretraining.\n\nMore Minor:\n\n4. Unclear why they use an LSTM rather than a transformer for the stay-level feature encoding. h_i^j could easily also be the CLS token of a transformer, especially given that some experiments were done with a transformer (figure 4) and seemed to do better than the LSTM. I may be misunderstanding this though.\n5. Lacking an ablation for admission-level only training (i.e. only using the clinical notes and ICD codes and not the stay-level clinical data). What does the stay-level clinical data bring to the table for admission and patient-level tasks both at inference and during training?", "questions_to_authors": "Why use an LSTM rather than a transformer for the stay-level feature encoding?\n\nCan you ablate for admission-level training only (see above)?\n\nCan you explain section 2.3.2? I am unclear how equation 9 works.\n\nCan you explain section 3.2.2? It is confusing how you modify the Transformer baseline with LSTM_enc because you can't warm-start the Transformer from a pre-trained LSTM.", "paper_topic_and_main_contributions": "This paper develops an approach to pretraining model and method tailored specifically to the hierarchy of electronic health records. This paper includes 3 different levels of modalities: patient-level demographics, admission-level modalities (discharge summaries, ICD and Drug codes) and stay-level clinical monitoring readings. They use pretraining at the two lower levels of the hierarchy (admission- and stay-level), using an intra-modality reconstruction loss at the stay-level, an intra-modality masked code prediction loss at the admission level, and a inter-modality contrastive loss at the admission level (by pooling the stay-level features into one admission-level feature vector).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 329: I think it should be equation 11 instead of 6.\n\nTable 2 caption: of -> on\n\nLine 383, what are the sizes of the datasets?\n\nCan you clarify why you chose a two-stage training rather than mixing all the objectives? I assume this might be because the stay-level objective of reconstruction is “harder” and the admission-level one might mess it up, but clarification would be much appreciated.\n\nSection 3.2.2 is confusing, LSTM_enc replaces the linear encoders in LSTM and Transformer?\n\nEquation 7: typo, f needs an i\n\nEquation 9: I think there is a typo because the optimal solution here is that p is always 1.\n\nIt would be great to have a visualization of what multimodal training does. Maybe some tsne figures when ablating over the training objectives colored by classes? How do the admission-level or stay-level representations change based on if the training is multimodal or not?"}}]

Author rebuttal:

Meta-review: The paper presents a two-stage hierarchical multimodal pretraining framework, named MedHMP, which is designed for Electronic Health Records (EHR) data. This framework aims to address the challenges of managing the hierarchical and heterogeneous nature of EHR data. By performing extensive experiments across diverse clinical benchmarks, the paper convincingly demonstrates that MedHMP outperforms existing baseline models. The hierarchical multimodal approach is well-suited for EHR data, offering a promising direction for predictive healthcare. The reviewers largely agree that the study provides strong support for its claims. Given that the majority of the reviewers found the work both sound and exciting, recommending a strong acceptance seems justified. However, the authors should pay close attention to the detailed questions and concerns raised by the reviewers in future revisions.
Decision: Accept-Main