Multimodal Patient Representation Learning with Missing Modalities and Labels
Authors: Zhenbang Wu
Anant Dadu
Nicholas Tustison
Brian Avants
Mike Nalls
Jimeng Sun
Faraz Faghri
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal learning
missing modalities
missing labels
clinical predictive modeling
patient representation learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose MUSE to effectively learn representations for patients with missing modalities and labels.

Abstract: Multimodal patient representation learning aims to integrate information from multiple modalities and generate comprehensive patient representations for subsequent clinical predictive tasks. However, many existing approaches either presuppose the availability of all modalities and labels for each patient or only deal with missing modalities. In reality, patient data often comes with both missing modalities and labels for various reasons (i.e., the missing modality and label issue). Moreover, multimodal models might over-rely on certain modalities, causing sub-optimal performance when these modalities are absent (i.e., the modality collapse issue). To address these issues, we introduce MUSE: a mutual-consistent graph contrastive learning method. MUSE uses a flexible bipartite graph to represent the patient-modality relationship, which can adapt to various missing modality patterns. To tackle the modality collapse issue, MUSE learns to focus on modality-general and label-decisive features via a mutual-consistent contrastive learning loss. Notably, the unsupervised component of the contrastive objective only requires self-supervision signals, thereby broadening the training scope to incorporate patients with missing labels. We evaluate MUSE on three publicly available datasets: MIMIC-IV, eICU, and ADNI. Results show that MUSE outperforms all baselines, and MUSE+ further elevates the absolute improvement to ~4% by extending the training scope to patients with absent labels.

Published as a conference paper at ICLR 2024
MULTIMODAL PATIENT REPRESENTATION LEARNING WITH MISSING MODALITIES AND LABELS
Zhenbang Wu Anant Dadu Nicholas Tustison', Brian Avants,
Mike Nalls Jimeng Sun', Faraz Faghri
I University of Illinois SibanaaChamppla Natiooall Institutes of Health
Data Tecnica International, *University of Virginia, *University of Pennsylvania ww12aillinen edu
ABSTRACT
Multimodal patient representation learning aims to integrate information from multiple modalities and generate comprehensive patient representations for subse- quent clinical predictive tasks. However, many existing approaches either presup- pose the availability of all modalities and labels for each patient or only deal with missing modalities. In reality, patient data often comes with both missing modal- ities and labels for various reasons (i.c., the missing modality and label issue). Moreover, multimodal models might over-rely on certain modalities, causing sub- optimal performance when these modalities are absent (i.e., the modality collapse issue). To address these issues, we introduce MUSE: mutual-consisten graph contrastive learning method. MUSE uses a flexible bipartite graph to represent the patient-modality relationship. which can adapt to various missing modality patterns. To tackle the modality collapse issue, MUSE learns to focus on modality- general and label-decisive features via a cutual-consiseent contrastive learning loss. Notably, the unsupervised component of the contrastive objective only re- quires self self-supervision signals, thereby broadening the training scope to incorpo- rate patients with missing labels. We evaluate MUSE on three publicly available datasets: MIMIC-IV, eICU, and ADNI. Results show that MUSE outperforms all baselines, and MUSE+ + further elevates the absolute improvement to 44% by ex- tending the training scope to patients with absent labels.
INTRODUCTION
Patient data spans wide range of modalities, including images (Johnson et al. 2019 Jack et al., 2008), text (Johnson et al.] (2023), physiological signals (Jing et al.) 2018 Kemp et al. 2000), and demographics (Pollard al. 2018 Kahn 1994). Physicians often jointly consider information from several modalities to make informed decisions about a patient's diagnosis, treatment, and ongoing care. For instance, tracking the progression of Alzheimer's 's Disease requires diverse data such as clinical scores, genetic profiles, neuroimaging scans, and biomarker readings (Khachaturian [985] Veitch et al. 2019). Multimodal patient representation learning aims to integrate information from these diverse modalities and generate comprehensive patient representation for downstream clinical predictive tasks like disease progression, mortality prediction, and readmission prediction (Kline et al. 2022).
Conventional multimodal learning typically presumes that all modalities are available for every patient (Ramachandram & Taylor 2017] Xu et al., 2023). However, in reality, patient data might be missing certain modalities. For example, some Alzheimer's patients may not have genetic or neuroimaging data due to accessibility issues or cost concerns (Schott & Bartlett 2012). Further, fragmented clinical records are common among patients switching healthcare providers. In such cases, conventional multimodal approaches often disregard these incomplete data, which largely reduces the number of training samples and narrows their application to only patients with complete modalities. To address this, multimodal learning with missing modalities (Ma et al. 2001 Chen &
Zhang 2020 Zhang et al. 2022) is proposed. Among them, modality imputation (Tran et al. 2017 "Work done during an intersship National Institutes of Health and Data Teenica International.

---

Published as a conference paper at ICLR 2024
Modalities
Labels
Modalities
Labels
Modalities
Labels
1
112
mg
m
M12
mg
y
1ft
m2
fg
Pi
V
Pt
V
M
P1
NA
v
Patients
P2
v
v
P2
NA
Ps
NA
NA
P3
NA
NA
Conventional Multimodal Learning
Missing Modalities
Missing Modalities and Labels (Ours)
Complete modalities
Incomplete modalities Complete labels
Incomplete modalities
Complete labels
Incomplete labels
Existing multimodal learning works either presume the availability of all modalities for each patient or only deal with missing modalities. However, in reality, patient data usually comes with missing modalities and labels due to variations in clinical practice, linkage issues, legal and privacy concerns, or data corruption.
Figure 1:
Shang et al., 2017) approaches leverage generative models to reconstruct the absent modalities using available ones, while direct prediction (T'ai et al.] 2019) Kim et al.. (02T) approaches use specialized network designs to perform the downstream task in the presence missing modalities.
However, besides modalities, patient data can also miss label information. Alzheimer's patients, for example, might not have the disease progression label due to inconsistent follow-ups or patient dropout (Schott & Bartlett 2012). We term this problem multimodal learning with missing modalities and labels. Fig. I provides a comparison among these three settings. To solve this problem, identify the following challenges:
Missing Modalities and Labels. Modality imputation approaches often rely on strong assump- tions about data distribution and fail to fully use inter-patient relationships. On the other hand, direct prediction approaches either oversimplify the patient-modalicy relationship or suffer from scalability issues when adapting to varied modality absence patterns (You al. 2020). Further- more. these methods only deal with the missing modality issue and usually require supervision signals from labels. As result, existing methods cannot leverage the information from the pa- tients without labels.
Modality Collapse. Multimodal models tend to over-rely on a subset of modalities during model training and ignore potentially valuable information from other modalities (Javaloy et al. 2022). As a result, existing models often do not generalize well to samples missing those key modalities.
To overcome these challenges, we introduce MUSE, a mutual-consistent graph contrastive learning framework. In essence, MUSE incorporates the following components:
Bipartite Patient-Modality Graph for Relationship Modeling. To address the challenge of missing modalities, we construct a bipartite graph of two distinct node sets: patients and modali- ties. The modality features serve as links between patient and modality nodes. This dynamic graph structure supports patients with diverse modalities and facilitates efficient information transfer be- tween modalities and patients.
Mutual-Consister Contrastive Loss for Representation Learning. Addressing the modality collapse problem and the challenge of missing labels, we introduce mutuall-onsistent contrastive loss. The unsupervised contrastive objective encourages similar representations for the same pa- tient with different modalities, thereby learning modality-agnostic features. Meanwhile, the super- vised contrastive objective further emphasizes the similarities among patients with the same label, thus learning label-decisive features. Notably, the unsupervised contrastive objective is trained with edge dropout augmentation and can seamlessly extend the training scope to patients with missing labels.
To assess the effectiveness of MUSE, we conduct experiments on three publicly available patient datasets: MIMIC-IV (Johnson et al., 2023), eICU (Pollard et al.) 20188. and ADNI (Jack et al.) 2008). Our findings show that MUSE outperforms all baselines when trained solely on patients with labels. Furthermore, MUSE+ enhances the absolute AUC-ROC score improvement by 44% by expanding the training to include patients without labels. We also conduct detailed analyses and ablation studies investigate the factors contributing the performance gain achieved by MUSE.
2

---

Published as a conference paper at ICLR 2024
Bipartite Modality Graph
Patient Node Representations Z
Construct
GNN
MLPP
Lee
Same Labe!
Modaittis
Labels
Lesend
Pationt Node
Wis
my
Loup
Pa
V
Augmanted with Edge Drapout
Stomese
Mocality
Patients
NA
Same Patient
Unouperniied
NA
P!
Uneup
Aruas
Mutual Condestent Contrastive Loss
Supervisd
Construct
Augmented Graph
Augmented Representations
LCE
Gassitictioon
Figure 2: MUSE models the patient-modalin relationship as a bipartite graph G, where patients and modalities constitute two separate node sets, and modality features act edges. An augmented ver- sion G' is then derived via edge dropout, signifying the same patients but with variations in modality inputs. The two graphs are encoded with Siamese GNN to get the patient node representations. The unsupervised contrastive objective aims to maintain consistent patient representations across varied modalities, thereby learning modality-
features. Meanwhile, the supervised contrastive ob- jective promotes similarities between patients with identical labels, thereby learning label-decisive features. Note that during inference time, the augmentation branch is omitted.
2 PRELIMINARIES
Multimodal patient representation learning jointly models the interaction among multimodal obser- vations for patient, and learns comprehensive representations for downstream clinical predictive tasks. Formally, we use (r) (x)" x(?" x to denote the multimodal observation data (e.g., medical images, clinical reports, and biosignals) for the patient p, where M is the number of modalities. And we use (()) to denote the label for the predictive tasks (e.g., hospital readmission,
patient mortality, and discase diagnosis).
Conventional multimodal learning typically trains a multimodal model fe(.) with parameter θ by minimizing a loss function C(-) on training set of N patients Der ((x{") ,...,X ((p) (P)) } p= as in Eq.
arg min (e, (x(fe(x)
(P)),
with the assumption that a complete set of modalities and labels accessible to all patients. How- ever, in real-world settings, patients miss both modalities and labels due to various reasons. Formally, we represent the missingness in modalities as M E {0, 10 X where Alp m] = the patient p has the modality m. And we use L E {0, 1 to denote the missingness in labels, where L(p) = 1 the label is available for the patient This paper then investigates: How can we effectively learn representations for patients with missing modalities and labels?
3 THE MUSE APPROACH
MUSE is driven by the following insights learn multimodal representations for patients with miss-
ing data:
Insight To handle patients with different modalities, the model must be flexible enough to ac- commodate different missing patterns and effective enough to model the patient-modal relationship. Inspired by recent works that utilize graphs to model the cross-data relationships (You et al. 2020) Chen & Zhang 2020 Zhang et al. 2022), we represent the multimodal patient dataset an undi- rected bipartite patient-moodeee modalliy graph G. This graph represents patients and modalities as two types of nodes. An edge is drawn between patient and modality node if the patient possesses data for that modality. The feature of the modality functions as the edge's 's attribute. The flexibility of graph structure allows us effectively and efficiently model the natient-modal relationship.
3

---

Published as a conference paper at ICLR 2024
Insight 2: Intuitively, to address the modality collapse issue, the model needs to focus features that are general across modalities and decisive for labeling. In this way, the model can still infer the label from modality- are missing. Based on this intuition, we design a mutual--onsistent contrastive loss. Specifically, first obtain augmented version of the bipartite graph 9 via edge dropout, denoted as g". The unsupervised contrastive goal is to align the same patient nodes in G and g'. Mcanwhile, supervised contrastive loss is applied to enhance similarities between patients sharing the same label. Collectively, these objectives encourage the model to learn modal- features, thereby enhancing its generalizability. Moreover, the unsupervised contrastive loss only requires self-supervision signals, broadening the training scope to patients with missing labels.
features learned from existing modalities, even if certain modalities
and label-decisive
Fig. 2 illustrates the MUSE framework. We will next describe the two modules in detail and introduce the training and inference strategy.
3.1 MULTIMODAL DATA AS BIPARTITE GRAPH
Multimodal learning with graphs has attracted much attention recently due to their flexibility in
modeling the interactions among modalities and samples (Ektefaie et al., 2023 However, existing approaches either only construct a graph for each modality (Zhang et al. 2022). which ignores
cross-modal relationship, build hypergraph whose complexity increase the combination of modalities (Chen & Zhang 2020), which suffers from the scalable issue. In contrast, MUSE utilizes a bipartite patient-modality graph maintain good balance between effectiveness and scalability.
Bipartite Patient-Modalitt Graph Construction. MUSE transforms the multimodal patient fea- tures (X(p)) J-m and modality missingness matrix M to an undirected bipartite patient-modality graph = E). Here, the node set Vp U VM consists of patient nodes V, (u.,..., u,y and modality nodes Va = (ui,.... , PMr }. The edge set & is determined using the modality missing- ness matrix M the adjacency matrix. Specifically, an edge Cup"m. Fm. is drawn if the patient p has the data for modality m.
To initialize edge embeddings, utilize the associated modality features. Specifically, for the edge Cugum. we initialize its embedding by encoding the raw data xm of modality m pertaining to patient p, as in Eq.
(0)
Encoder, (x...) (P)),
where Encoder,,, (.) is the backbone feature encoder for modality m, and Cupum E Rd is the encoded modality embedding with dimension d. For node embeddings, we initialize modality nodes as one-hot encodings of dimension M, i.e., h = OneHot(m).. Patient nodes, on the other hand, are initialized with constant one vectors with the same dimension, i.c., h. 1.
Message Passing over the Bipartite Graph. Given the constructed bipartite graph, we utilize multi-layer Graph Neural Network (GNN) to integrate information from both modalities and pa- tients, and learn consolidated patient representations, as in Eq. 3
= =NN(()),
(3)
where Z E RR is the learned patient representations. Essentially, GNN propagates messages in the graph, where each node aggregates information from its neighboring nodes. Due to the multihop aggregation capability, GNNs can model long-term relations among modalities and patients. Specif- ically, in the /-th propagation layer, node h, aggregates information from its neighboring nodes (i) via the connecting edges, as in Eq. 4
h Aggregate (h (h Message ( (h) (t-1) h(-"), of (t1)) (4l N())))
(4)
where Message (1) extracts message information from the source node representation h (1-1) the target node representation h (1-1) and the edge representation ex (f-1), Meanwhile, Aggregate( compiles the neighboring messages via mean, sum, max, or other pooling operators.
4

---

Published as a conference paper at ICLR 2024
3.2 MUTUAL- CONSISTENT CONTRASTIVE LEARNING
Multimodal features can be broadly categorized into two dichotomies: label-decisive versus non- label-decisive, and modality-agnostic versus modallty-dependent (Xue et al. 2023). However, prior research on absent modalities does not differentiate between these features You et al. 2020). As result, a predictive model might over-rely on modality modaliyy-dependent features, and yield compromised performance when the corresponding modalities are absent. For a model to consistently perform well across varying modalities, it is crucial to separate modality-agnostic and modality-dependee label-decisive features and focus on the modality-agnostic ones.
Based on this intuition, we introduce a simple yet effective nuutual-conssseen contrastive loss. Specifically, we first obtain an augmented version of the bipartite graph G. through random edge dropout, which is denoted as G'. This essentially mirrors the original set of patients but showcases different modality missing patterns. Subsequently, the same GNN encoder, yielding Z' GNN(G'), where E IR Nxd is the augmented patient representations. Building this, MUSE introduces three loss objectives.
pass the augmented graph G' through
Contrastive Loss. Patient nodes from graphs & and represent the same set of patients but with different modalities as inputs. Naturally, we want to encourage the similarities between nodes from the same patient in different graphs extract modality-agnostic features. To do this, MUSE deploys an unsupervised contrastive loss, as in Eq.
Unsup (Z, 2')
log
(5)
p=1 -
where Zp and 2/ are the nodes from the same patient in graph 9 and G', respectivvel.. and s(..)) :) is the cosine similarity function with 7 as a temperature hyperparameter. Notably, this loss function does not require label information to supervise the training. Thus, it can seamlessly incorporate patients with absent labels and extend the training scope.
While extracting agnnsicy-agnostic features is crucial, it is not the sole consideration. Patient data often encompass abundant auxiliary information irrelevant the downstream task, such as admin- istrative details. Thus, it is imperative to prioritize label-decisive features as well. In light of this, MUSE encourages patient nodes with the same label to predict each other. The corresponding loss denoted as Ssup (2, Y. L), which is shown in Eq. 6
ΣΣ
[(Ll)) = 1)1(y, = yy) log
(6)
ΣN 1 (L(r) 1) yr))
where 1(-) the indicator function. Here, 1 [(g 1) and 1 [L[rr] 1) only select patients with label information. And zp and zn denote patients with matching labels, whereas zp and z, signify patients with distinct labels.
Classification Loss. To further optimize for downstream tasks, we incorporate supervised cross- entropy loss. Specifically, we first obtain the predicted label through non-linear network, as Y MLP(Z), where MLP(-) is multi-layer perceptron. Subsequently, we optimize the supervised classification loss, LCr Y Y) Σx ((L(p) 1)CE vp), where [[[pp) 1) selects patients with labels, and CE(.,.) is the cross-entropy loss between predicted label ýp and true label Yp for patient p.
3.3 TRAINING AND INFERENCE
To train MUSE, for each batch, we transform the batch data a bipartite graph. Then, we employ an edge dropout of 0.15 probability to derive the augmented graph. Next, we input both the original and the augmented graph through the Siamese GNN. We leverage in-batch negatives for the contrastive losses in Eq. 5 and Eq. 6 Notably, similarity calculations are performed only once between each pair and are stored for reuse. The trio of loss objectives are added together to obtain the final loss. To infer MUSE, only feed the original graph through the GNN and obtain the predicted label via the MLP classifier. The pseudocode of MUSE is available in Appx. A More implementation details can be found in Sec. 4.1 and Appx.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Baselines. We compare MUSE to three categories of baselines. (1) The first category is modality
imputation methods, including CM-AE (Ngiam al., 2011 and SMIL (Ma et al. 202Tp. (2) The
second category is direct prediction approaches. including MT (Ma et al. (022). (3) The last
category graph-based methods, including GRAPE YYuu et al. 2020p, HGMF (Chen & Zhang 2020), and M3Care (Zhang et al. [022). Note that these methods depend on a supervision signal from task labels, and consequently, they only utilize labeled patients during training. To ensure fair comparison, we implement our method in two distinct settings. When trained solely with labeled patients, it is referred to as MUSE; and when trained with both labeled and unlabeled patients, is designated as MUSE+. More details can be found in appendix
B
Implementations. MUSE uses the GraphSage (Hamilton et al. 2017) with edge attributes version in You et al. (2020) the GNN. The edge dropout rate is set 15%. We split the dataset into 70%, 10%, 20% training, validation, and test sets. We train all models for 100 epochs on the training set, and select the best model by monitoring the performance the validation set. The final results reported on the test set. For each metric, we report the average scores and standard deviation by performing bootstrapping (i.e., sampling with replacement) 1000 times. Additionally, we conduct independent two-sample t-tests to assess whether MUSE achieves a significant improvement over the baseline methods. More implementation details are provided in Appx. [ The code of MUSE is publicly available
4.2 RESULTS ON ICU DATA.
Datasets. We evaluate MUSE on two real-world ICU datasets. MIMIC-IV (Johnson al.) 2023) covers 431K visits for 180K patients admitted to the ICU in the Beth Israel Deaconess Medical Center. We use demographics (age, gender, and ethnicity), diagnosis, procedure, medication, lab values, and clinical notes as input modalities. eICU (Pollard et al. 2018) covers K visits for 180K patients admitted to the ICU in the Beth Israel Deaconess Medical Center. We use demographics, diagnosis, procedure, medication, lab values, and vital signals as input modalities.
Tasks & Metrics. We focus on two common clinical predictive tasks. Readmission prediction aims to predict whether patient will be readmitted within the next 15 days following current discharge. Mortality prediction predicts whether a patient will pass away upon discharge in the cICU setting, or within 90 days after discharge in the MIMIC-IV setting. Both tasks are binary classifications. To evaluate the performance of the models, we calculate the Area Under the Precision- Recall Curve (AUC-PR) and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC) scores.
Backbone Encoders. For demographic (age, gender, ethnicity) and sequential medical coding data (diagnosis, procedure, and medication), use the Transformer (Vaswani et al. 2017) architecture as the backbone encoder. For times series data (lab values and vital signals), we use the Recurrent Neural Network (RNN). For text data (clinical notes), we use the TmnBBERT (Jiao et al., 2019) encoder. We add project layer to map the modality embeddings to the same latent space.
More details on datasets, statistics, tasks, metries, and backbone encoders can be found in Appx. D
Results. Table I presents the mortality and readmission prediction results on the MIMIC-IV and eICU datasets. Firstly, we observe that modality imputation methods CM-AE (Ngiam et al. 2011) and SMIL (Ma et al., 2021) perform the worst. This is reasonable as they rely on strong assumptions to learn the mapping from lower-dimensional latent space to the higher-dimensional original input space. Direct prediction-based method, MT (Ma et al. 2022), offers slightly improved results by utilizing a specialized Transformer (Vaswani et al. 2017 architecture to model relationships among various modalities. However, it overlooks the valuable relation information among different patients. Next, graph-based approaches achieve the best performance among all baselines. These approaches model both inter- and intra- retatosslaioosships between patients and modalities via GNNs. Within this category, HGMF (Chen & Zhang, 2020) performs the worst, likely due to its complex network de- sign that might be challenging to optimize and prone to overfitting. Conversely, GRAPE (You al.)
https://github. wtt..///wwww//sss///..

---

Published as a conference paper at ICLR 2024
Table 1: Results the MIMIC-IV and cICU datasets. A dagger (1) indicates the standard deviation is greater than 0.02. An asterisk (*) indicates that MUSE achieves significant improvement over the best baseline method, with p-value smaller than 0.05. MUSE achieves the best performance across all baselines, and MUSE+ + can further improve the performance by incorporating patients with missing labels into the unsupervised contrastive loss.
MIMIC-IV
eCCU
Method
Mortality
Readmission
Mortality
Readmission
AUC-ROC AUC-PRC AUC-ROC AUC-PRC AUC-ROC AUC-PRC AUC-ROC AUC-PRC
CM-AE
0.8530
0.4351
0.6817
0.4324
0.8624
0.3902
0.7462t
0.4338f
SMIL
0.8607
0,4438
0.6894%
0.4368
0,8711
0,4066
0.7506
0,4447
MT
0.8739
0.4452
0.6901
0.4375
0.8882
0,4109
0.7635
0.4500
GRAPE
0.8837
0.4584f
0.7085
0.4551
0.8903
0.4137
0.7663
0.4501
HGMF
0,8710
0,4433
0.7005
0,4421
0.8878
0.410M
0.7604
0,4496f
M3Care
0.88967
0.4603$
0.7067
0.4532
0.8964
0,4155
0.7598f
0.4430
MUSE
0.9004
0.4735
0.71524
0.46707
0.9017
0.4216*
0,7709
0.4631
MUSE+
0.9201
0.4883#
0.7351
0.4985
0.9332
0.4387
0.8003
0.4844
2020 and M3Care (Zhang et al., 2022 employ simpler graph designs and achieve better perfor- mance. Lastly, we observe that both MUSE and MUSE+ outperform all other baselines. Specifically, MUSE, trained on the same patient set as other baselines, achieves ~2% absolute improvement in AUC-ROC score. Further amplifying this lead, MUSE+ clevates the improvement to 44% by incor- porating additional patients with missing labels into the unsupervised contrastive loss.
4.3 RESULTS ON ALZHEIMER'S 'S DISEASE DATA.
Datasets. The Alzheimer's Disease Neuroimaging Initiative (ADNI) (Jack et al. 2008) database provides longitudinal neuroimaging data, cognitive test scores, biomarker profiles, and genetic in- formation for over 2K Alzheimer's 's discase, mild cognitive impairment, and normal. We use the processed data from the Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) challenge (Marinescu et al. 2019). This challenge offers pre-processed features for 1737 patients spanning 12741 visits. Specifically, we utilize features from magnetic resonance imaging (MRI), positron emission tomography (PET), and diffusion tensor imaging (DTI).
Tasks & Metries. The TADPOLE challenge involves classifying patient visits into three categories: normal control, mild cognitive impairment, and Alzheimer's 's disease. We adopt the official metries: the balanced accuracy score and the one-vs-one macro AUC-ROC score.
Backbone Encoders. Given that the TADPOLE challenge supplies extracted features for each modality like the volumes, thicknesses, and surface areas of region-offinteres (ROI) in MRI scans, or the mean, axial, and radial diffusivity in DTI ROIs equipped with ReLU activation and batch normalization as the backbone encoder.
implement multi-layer perceptron
More details on datasets, statistics, tasks, metries, and backbone encoders can be found in Appx.
Results. Table 2 showcases the outcomes for predicting Alzheimer's discase progression using the ADNI dataset. We note that the standard deviations of the results are considerably larger than those in the ICU setting, with many having stan- dard deviations > 0.02 (highlighted by the dagger t). This could likely be attributed the smaller size of the dataset, resulting in more volatile training process. However, the overall perfor- mance trend is similar the ICU settings. The imputation based methods, CM-AE (Ngiam et al., (011) and SMIL (Ma et al. 2021), achieve the worst performance. Prediction-basee method MT (Ma et al.) 2022) achieves some improvement, while graph- based methods GRAPE (You et al., 2020) and M3Care (Zhang et al. 2022) achieve the best performance among all baselines. Particularly, we observe that HGMF (Chen Zhang 2020 's smaller dataset. Lastly, MUSE and MUSE+ give the best performance among all models, achieving ~3% absolute improvements.
Table Alzheimer's disease progression prediction results on the ADNI dataset. The dagger (+) and asterisk (*) have the same meanings as Tab
Method AUC-ROC Accuracy
basedd
CM-AB
0.8722
0.7305
SMIL
0.8761
0.7338+
MT
0.8935
0.7604
GRAPE
0.9031
0.7820+
HGMF
0.8845+
0.7463t
M3Care
0.9101
0.7822+
MUSE
0.9158
0.7973
performance further deteriorates
MUSE+
0.9309*
0,8291
7

---

Published as a conference paper at ICLR 2024
AUC-ROC
AUC-ROC
0.85
0.85
GRAPE
MUSE
0.75
0.75
0.70
0.1
0.2
D.3
0.5
0.7
0.0
0.1
0.2
0.3
0.4
Modality Missing Ratio
Label Missing Ratio
Figure 3: Experiment with different modality and label missing ratios.
4.4 ANALYSIS OF DIFFERENT DATA MISSING RATES.
Next, we analyze the impact of the data missing rate. Specifically, we randomly mask out modal- ities with probabilities {0.1, 0.2, 0.3, 0.5, 0.7}. and labels with probabílities f0, 0.1, 0.2, 0.3, 0.4). We compare MUSE with the two top- perfformnni baselines: GRAPE (You et al. 2020) and M3Care [Zhang et al. 2022) for the mortality prediction task on the MIMIC-IV dataset. The results are shown in Fig. 3 Results highlight that MUSE and MUSE+ consistently outperform the two base- lines across all configurations. Moreover, we observe that as the missing ratio increases, the gap between MUSE+ and other baselines also increases. This demonstrates MUSE+'s robustness against missing data by incorporating complementary information from patients with missing labels.
4.5 ANALYSIS OF THE LEARNED REPRESENTATIONS.
Further, we study the quantitative evidence behind MUSE's success. Specifically, we measure the cosine sim- ilarity between representations from the same patient us- ing the original modalities and those with 30% randomly masked modalities. Fig. 4 show the results calculated on the test set of the MIMIC-IV dataset for mortality pre- diction. We observe that the modality imputation-based methods (CM-AE (Ngiam
Cosine Similarities
al., 201T) and SMIL. (Ma et al.] 202T)). as well as the direct prediction-baser method (MT (Ma et al.) (022)) have much lower cosine similarities compared to the graph-based methods. This explains their performance drop as the patient representa- tions vary lot once certain modalities contrast, MUSE and MUSE+ achieve the highest cosine similarity score among all baselines and also the best pre- diction performance. These findings reaffirm our hypothesis regarding the efficacy of learning aonality-agnostie representations.
CM-AE SMIL
MT
GRAPE
HGMF
MaCare MUSE
MUSE+
Figure 4: Cosine similarity of the repre- sentations from the same patients with different modalities.
missing. In
4.6 ABLATION STUDY.
This section analyzes the performance lift brought by different components of MUSE. Specifically, A1 replaces edge dropout with feature dropout; A2 removes the mutual-consistent contrastive loss and only uses the cross-entropy loss; A3 removes the supervised contrastive loss; and A4 removes the unsupervised contrastive loss. The results can be found in Tab. 3 We observe that Al suffers from the most significant performance drop, indicating that simple feature dropout cannot simulate the missing modality scenario. A2-4 remove different contrastive objectives, and all suffer from certain levels of performance drop. Specifically, A2 can be seen as special version of GRAPE (You ct al. 2020) with additional input augmentation. It does not explicitly capture the modal-agnostic features and thus deleterious the most. A3 and A4 perform similarly, indicating that supervised and unsupervised mutual-consistent contrastive loss contributes differently to the overall objectives. This matches our intuition that the unsupervised contrastive loss forces the model to learn modal-agnostic features while the supervised version forces the model learn label-decisive features.
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study on the influence of each individual component. The dagger (f) and asterisk have the same meanings as Tab.
MIMIC-IV
eCCU
ADNI
Method
Mortality Readmission Mortality Readmission AD Progression
A1 MUSE w/o Edge Dropout
0,8830
0.6911
0.8655
0.7593
0,8811
A2
MUSE w/o Contrastive Loss
0,8846
0.7055
0.8895
0.7611
0,9040
A3 MUSE w/o Supervised Contrastive Loss
0,8934
0.7088
0.8916
0.7699
0,9086
A4 MDSE, w/ Unsupervised Contrastive I Loss
0.8901
0.7103
0.8911
0.7706
0.9003
MUSE
0.9004
0.7152
0.9017
0.7709
0.9158
MUSE+
0.9201
0.7351
0.9332
0.8003
0.9309
4.7 RUNTIME ANALYSIS.
This section compares the prediction performance ver- sus training time relationship for all methods. We report the per-epoch training time and AUC-ROC score for the MIMIC-IV mortality prediction task on single NVIDIA A100 GPU. The results can be found in Fig. 5 MUSE achieves the best prediction performance with relatively small computation loads. MOSE- further improves the pre- diction results with reasonable computation overhead (due to the incorporation of patients with missing label). This demonstrates that the design of MUSE keeps a good balance between effectiveness and efficiency. Furthermore, the sim- ple definition mutual-consistent contrastive loss allows re-using computations across various loss functions. These factors make MUSE a lightweight yet powerful framework.
+ MUSE+
6.41
AUC-ROC
0.00
MUSE
M3Care
GRAPE
83 MT
SMIL
CM-AE
0.85
Training Time (Secand/Epoch)
Figure 5: Prediction performance
V.S. training time comparison on the MIMIC-IV mortality prediction task.
5 RELATED WORK
Multimodal Learning with Missing Modalities, Conventional multimodal approaches typically assume the availability of all modalities (Ramachandram & Taylor (017). which may not be satis- fied in real-world scenarios. To address this, multimodal learning with missing modalities is pro- posed. CM-AE (Ngiam et al. 201T) leverages a cross- modslity auto-encoder to recover the missed modalities based on existing ones. SMIL (Ma et al.) 202I further incorporates Bayesian meta- learning technique. However, these imputation- based methods tend to make strong assumptions about data distribution and may introduce additional noise. Furthermore, when imputing miss- ing modality for a certain patient, they fail to make full use of this modality from other patients. late-fusion Transformer layer model the interaction
In contrast, MT (Ma al. 2022) among modalities. MulT (T'sai et al., 2019) uses a pairwise Transformer layer to fuse the modal- ity embeddings. And ViLT (Kim et al.) 2021) and UMSE (Lee al., 2023) directly enable the interaction among raw inputs from different modalities. These models can directly perform down- stream tasks with the existence of missing modality due to specific network design. More recently, baseh approaches have been proposed. GRAPE (You et al.) (020)) utilizes similar bipartite graph as MUSE, but does not enforce the similarity of the same patient with different modalities. HGMF (Chen & Zhang 2020 uses heterogeneous graph to model different modality missing patterns. M3Care (Zhang et al. 2022) learns patient similarity for each modality. However, these methods either oversimplify the relationship, utilize overcomplex network that suffers from training and over-fitting issues. Furthermore, they require labels as supervisions and fail to utilize patients with missing labels in the training process.
CONCLUSION
Existing multimodal clinical predictive models either assume complete modality availability for each patient or only deal with missing modalities. In reality, both modalities and labels can be absent. To address this, we propose MUSE, which builds a patient-modall leams to extract -ggoostty-agnostc and label-dccisive feature via mutual-consisten graph contrastive learning. Evaluations of MUSE on three medical datasets reveal its superiority over baseline methods. Furthermore, we supplement results with comprehensive qualitative analyses.
bipartite graph and

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479. We thank the patients and their families who contributed this research. This research was supported in part by the Intramural Research Program of the National Institute on Aging (NIA) and National Institute of Neurological Disorders and Stroke (NINDS), both part of the National Institutes of Health, within the Department of Health and Human Services project number ZIAAG000534, This work uses data obtained from the Alzheimer's 's Disease Neuroimaging Initiative (ADNI) database. ADNI funded by the National Institute Aging, the National Institute of Biomedical Imaging and Bioengineer- ing, and through generous contributions from the following: AbbVie, Alzheimer's Association; Alzheimer's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company: CereSpir, Inc.; Cogstate: Eisai Inc.; Elan Pharmaccuticals, Inc.; Eli Lilly and Company; Eurolmmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fu-
jirebio; GE Healthcare; IXICO Ltns:Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck:
Merck & Co., Inc.:Meso Scale Diagnosties, LLC.: NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging: Servier; Takeda Pharmaceuti- cal Company; and Transition Therapeutics. The Canadian Institutes of Health Research is provid- ing funds to support ADNI clinical sites Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org) The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNI data disseminated by the Laboratory for Neuro Imaging at the University of Southern California. The investigators within the ADNI did not participate in analysis or writing of this manuscript.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Emily Alsentzer, John Murphy, William Boag. Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72-78, Minncapolis, Minnesota, USA, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19. 1909. URL https:
Jiayi Chen and Aidong Zhang. HGMF: Heterogencous Graph-based Fusion for Multimodal Data with Incompleteness. In Proceedings of the 26th ACM SIGKDD International Conference on
Knowledge Discovery & Data Mining, pp. 1295-1305, Virtual Event CA USA, August 2020.
ACM. ISBN 978-1 4508--998-44 doi: 10.1 1114453339448633001153333 URL https: acm.
http://oo.1...111//3.33 340388
Yasha Ektefaie, George Dasoulas, Ayush Noori, Maha Farhat, and Marinka Zitnik. Multimodal learning with graphs. Nature Machine Intelligence, 5(4):340-350, April 2023. ISSN 2522- 5839. doi: 10.
URL https:/ www. nature. com/ hammarticles/
112256-023-00622
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. NIPS, 2017.
Clifford R Jack, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L Whitwell, Chad Ward, et al. The alzheimer's disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imaging, 27(4): 685-691, 2008.
Adrian Javaloy, Maryam Meghdadi, and Isabel Valera. Mitigating modality collapse in multimodal VAEs via impartial optimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9938 9964. PMLR, 17-23 Jul 2022. URL http:::/prroceeinngggee/oooo//ee mlr. press/v162/ javaloy22a html
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. CoRR, hts/1999.11351.. 2019. URL http::/ttapiiii/ org/ 19t9//9999..... 10351
J Jing, E d' A'grremmn,, S Zafar, ES Rosenthal, M Tabaeizadeh, S Ebrahim, J Dauwels, and MB Westover. Rapid annotation of seizures and interictal-ictal continuum eeg patterns. In Annu Int Conf IEEE Eng Med Biol Soc, pp. 3394-3397, Jul 2018. doi: 10.1109/EMBC.20088.5. 1015599 PMID: 30441116; PMCID: PMC6776236.
Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth Berkowitz, and Steven Horng. Mimic- cxr-jpg, large publicly available database of labeled chest radiographs, 2019.
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lchman, Leo A. Celi, and Roger G. Mark. MIMIC-IV. freely accessible electronic health record dataset. Scientific Data, doi:
10(1):1, January 2023. ISSN 4463.
URL https:
//doi.org// 10. 1038/ 022 0289-1199-11 x
Michael Kahn. Diabetes. http://oo...ooooo2.233////////
UCI
Machine
Learning Repository, 1994.
DOI:
B Kemp, AH Zwinderman, B Tuk, HAC Kamphuisen, and JJL Oberyé. Analysis of sleep- dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering, 179):1185-1194 2000.
Zaven S. Khachaturian. Diagnosis of Alzheimer's Disease. Archives of Neurology. 42(11):1097 1105, 11 1985. ISSN 000039942.. doi: 10. h0..000/acchhcur1198555 www.. 10083309. URL https: 7/doi. org hatp///coo00 198.. 0985004060100883022

---

Published as a conference paper at ICLR 2024
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- volution or region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 35383-5594, PMLR, 18-24 Jul 2021. URL https: htttps://proceeeiing mll.
transformer without con-
press/vv 39/ htt1k/2/kmmmmm m21 html
A. Kline. H. Wang. Y. Li, et al. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5:171, 2022. doi: 10. //doi. 10.g//10.0 10081008///00 0058//4114885 022---0---0000------2----
URL https:
Kwanhyung Lee, Soojeong Lee, Sangchul Hahn, Heejung Hyun, Edward Choi, Byungeun Ahn, and Joohyung Lee. Learning Missing Modal Electronic Health Records with Unified Multi- modal Data Embedding and Modality-Aware Attention, May 2023. URL http:://rxiv: org/aba//l aba/ 2305. 02504 arXiv:2305.02500 [cs].
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. SMIL: Multimodal Learning with Severely Missing Modality, March 2021. URL httt:://axttii org/ abs/ 2103. 05677 arXiv:2103.05677 [es].
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transform- ers robust to missing modality?, 2022.
RV Marinescu, NP Oxtoby, AL Young, EE Bron, AW Toga, MW Weiner, F Barkhof, NC Fox, P Golland, S Klein, and DC Alexander. Tadpole challenge: Accurate alzheimer's s disease predic- tion through crowdsourced forecasting future data. Predict Intell Med, 11 11843:1-10, Oct 2019. Epub 2019 Oct 10.
doi: 1-10077978-3-000--333 32281-66-211111111
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Multi- modal deep learning. In Proceedings of the 28th International Conference International Con- ference on Machine Learning, ICML' 11, pp. 689-696, Madison, WI, USA, 2011. Omnipress. ISBN 978 9781450306195.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http: httpppaarrrseeeeeea wwwppppppsss.rppsppeeeeeeeeeeeee 901 5 wywtrch-tr.--.. mperat ve sty e- pwwforhheeeeeoooeeeeeeeeeeeee leaemaaeeeeeeeeeeeeeennnnn learning Library.
pdf
Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific Data, 5(1):180178, September 2018. ISSN 2052-4463, doi:
10. 10.1038/sdata.20 1788. htRpssttt:///// /ww. http::/:////o.....0100.10000
sdata. 2018.178
Dhanesh Ramachandram and Graham W. Taylor. Deep multimodal learning: survey on recent advances and trends. IEEE Signal Processing Magazine, 34(6):96-108, 2017. doi: 10.1
2017.2738401.
Jonathan M. Schott and Jonathan W. Bartlett. Investigating missing data in alzheimer disease studies.
Neurology, 78(18) 137711371,111111 2012. ISSN 0028- 3878. doi: neurology.
URL https::p/////. URL.:ipps///..eeeooooo..... cttp://///..oooooooo////////////
Chao Shang, Aaron Palmer, Jiangwen Sun, Ko-Shin Chen, Jin Lu, and Jinbo Bi. Vigan: Missing view imputation with generative adversarial networks. In 2017 IEEE International Conference on Big Data (Big Data), pp. 766-775 2017. doi: 10. Btt.:BiiDaaa.227.82555555
M Sung, S Hahn, CH Han, JM Lee, J Lee, Yoo, Heo, YS Kim, and KS Chung. Event prediction model considering time and input error using electronic medical records in the intensive care unit: Retrospective study. JMIR Med Inform, 9(1 26622,, Nov 4 2021. doi: 10.2 10.2196/26426.
12

---

Published as a conference paper at ICLR 2024
Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4971-4980, 2017. doi: 10. 10.11099VPP22
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computationall Linguistics, pp. 6558 6569,,656,,660 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653. 11/619-1656. URL https: / ac antho lgg.. 9-1656
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 1186660469
Dallas P. Veitch, Michael W. Weiner, Paul S. Aisen, Laurel A. Beckett, Nigel Cairns, Robert C. Green, Danielle Harvey, Clifford R. Jack, William Jagust, John C. Morris. Ronald C. Petersen, Andrew J. Saykin, Leslie M. Shaw, Arthur W. Toga, and John Q. Trojanowski. Understand- ing disease progression and improving alzheimer's s disease clinical trials: Recent highlights from the alzheimer's disease neuroimaging initiative. Alzheimer's Dementia, 15(1):106 152, 2019. ISSN 1552-5260. doi: http://dww.gg//00000oooo/2000000 URL https:
www.ssccieceeirect.... wwwtwwwwwwww...........ttt
7/www.
com
Peng Xu, Xiatian Zhu, and David A. Clifton. Multimodal learning with transformers: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(10):12113-22111 2023. doi: N0.1199/PAAMMI2223333755555
Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. THE MODALITY FOCUSING HY- POTHESIS: TOWARDS UNDERSTANDING CROSSMODAL KNOWLEDGE DISTIL- LA- TION. 2023.
Jiaxuan You, Xiaobai Ma, Daisy Yi Ding, Mykel Kochenderfer, and Jure Leskovec. Handling Miss- ing Data with Graph Representation Learning, October 2020. URL /attp://axxiv.or
Tos/2010.16411 arXiv:2010.16418 [cs, stat].
Chaohe Zhang, Xu Chu, Liantao Ma. Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao M3Care: Learning with Missing Modalities in Multimodal Healthcare Data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2418-2422 Washington DC USA, August 2022. ACM ISBN 978-1-4500-93388- doi: 10.1145/3534678. 3539388. URL
ddl acm. 1ttp///...........//.....1111//. 111.1....113333 3539388
13

---

Published as a conference paper at ICLR 2024
Contents of Appendix
Pseudocode of MUSE
15
Additional Details on Baselines
15
Additional Details on Implementations
16
D Additional Details on EHR Data
16
D.1 Datasets
16
D.2 Tasks
17
D.3 Metrics
17
D.4 Backbone Encoders
17
Additional Details on AD Data
17
Datasets
17
Tasks
18
3 Metries
18
Backbone Encoders
18
Additional experiments
18
Additional evaluation on the missing not at random setting
18
F2
Additional analysis of the use of the supervised contrastive loss
18
3
Additional metric of the distance between patient representations
19
Limitations
19
Notations
20
14

---

Published as a conference paper at ICLR 2024
A PSEUDOCODE OF MUSE
Algorithm 1: Training and Inference for MUSE.
// Training
Require: Training dataset Drr, modalities missingness matrix M. label missingness matrix L
2: for each epoch do
3: for each batch do
4: Sample batch data from Der
5:
Transform batch data to bipartite graph Initialize graph embeddings by Eq. 2 Obtain augmented graph G' via edge dropout Feed both graphs through Siamese GNN by Eq. Compute consual-cnssstent contrastive loss by Eq. 5
6:
7:
8:
9:
10:
Obtain the predicted label via MI. and calculate the cross-entropy loss Update model parameters with the weighted sum of all losses
12: end for
13: end for
14: I/ Inference
Require: Testing dataset Die. modalities missingness matrix M
15: for each batch do
16:
Sample batch data from Dtc
17:
Transform batch data bipartite graph g
18: Initialize graph embeddings by Eq.
Feed graph & through GNN by Eq.
20: Obtain the predicted label via MLP
end for
B ADDITIONAL DETAILS ON BASELINES
We compare MUSE with the following baselines:
CM-AE (Ngiam et al., 2011) leverages cross-modality auto-encoder to recover the missed modalities. Specifically, we first sample a subset of patients with complete modalities. Then, train CM-AE to reconstruct the masked-out modality. Next, the trained CM-AE model to impute missing modalities for all patients. Finally, we perform the downstream tasks on the imputed dataset.
SMIL (Ma et al., 202T) further incorporates Bayesian meta-learning techniques to perturb the la- tent feature space that embeddings with missing modalities approximate the ones with com- plete modalities. Specifically, it approximates missing modality with a weighted sum of modality priors learned from the modality-complete dataset together with feature regularizer.
MT (a et al., 2022) utilizes Transformer layer fuse the embeddings from different modal- ities. Specifically, we use the late-interaction version where the Transformer fuses the learned modality embeddings from modality-specific encoders.
GRAPE You et al., 2020) utilizes similar bipartite graph as MUSE to fuse the information across modalities and patients. We directly feed the fused patient representations from GNN to downstream MI. for the predictive tasks.
heterogeneous hypernode graph model the patient- modality relationship and to accommodate different modality missing patterns. Modalities from the same patients are modeled via an intra-node encoder. And the interaction among patients is modeled with the hypergraph.
HGMF (Chen & Zhang 2020
M3Care (Zhang et al., 2022) learns patient similarity for each modality, and then builds per- modality similarity graph. The cross postients interaction is modeled via a GNN. And the modali- ties embeddings for each patient aggregated via Transformer head.
15

---

Published as a conference paper at ICLR 2024
C ADDITIONAL DETAILS ON IMPLEMENTATIONS
We train all models for 100 epochs on the training set, and select the best model by monitoring the performance the validation set. The final results reported the test set. The specific metric monitored varies depending on the task: we used the PR-AUC score for mortality and readmission tasks and the one-vs-one macro AUC-ROC score for the Alzheimer's disease prediction task. MUSE uses the GraphSage (Hamilton et al. 2017) with edge attributes version in You et al. (2020) as the GNN. Specifically, in the 1-th layer propagation layer, the node h, aggregates information from its neighbours nodes via the connecting edges, as in Eq.
(1) Coneat h (h("),, Mean (f (RELU(W())) h + + e. VVj e e(o)))
h2
(7)
where U, W, O are learnable parameters. And the edge embeddings are updated as well, as in Eq.
(1) PPP (n.-"",,"""" Oje off"" h.
(8)
where is a learnable parameter matrix. We set the number of GNN layers to 2. And we add a MLP classification head to each of the GNN layer. The cosine similarity lemperature is set 0.05. We would like to note that we did not extensively tune the hyperparameters for the experiments. Specifi- cally, we only tune the learning rate for both MUSE and the baseline methods while keeping the other hyperparameters aligned to ensure a fair comparison. We implement MUSE using PyTorch Paszke jet al. (219)) 1.1 and Python 3.8. The model is trained on CentOS Linux 7 machine with 128 AMD EPYC 7513 32-Core Processors, 512 GB memory, and eight NVIDIA RTX A6000 GPUs. The code MUSE will be released after acceptance.
D ADDITIONAL DETAILS ON EHR DATA
Table 4: ICU Dataset statistics.
Item
MIMIC-IV
eCCU
#Patients
172113
120892
#Admissions
389761
158248
Mortality Positive
0.09
0.04
Moctality Missing
0.58
0.50
Readmission Positive %
0.15
Readmission Missing &
0.41
0.50
Age
Gender Race
F: 0.52, M: 0.48
F: 0.46, M: 0.54
African American: 0, Asian: 0.04, Coucasion: 0.66,
African American: 0. 0.02, Caucasian: 0.77,
Hispanic: 0.06, Native American: 0.00, Other: 0.06
Hispanic: 0.04, Native American: 0.01, Other: 0.06
Diagnosis Missing &6
0.00
0.08
Procedure Missing %
0.50
0.18
Medication Missing %
0.17
0.17
Lah Missing &
0.03
Notes Missing &
0.31
Vital Signal Missing
0.11
D.1 DATASETS
MIMIC-IV (Johnson et al. 2023). We extract the data from the patients, admissions, prescrip- tions, diagnoses ied, procedures ied, labevents, and discharge tables. Prescriptions, diagnoses, and procedures are recorded as medical codes. For labevents, following [Sung et al. (202T). we gather ten groups of data related to Hematocrit, Platelet, WBC, Bilirubin, pH, Bicarbonate, Creatinine, Lactate, Potassium, and Sodium. A total of 111 lab items are included. For clinical notes, we use the text from the discharge summary section. We select our cohorts by filtering out visits of patients younger than 18 older than 89 years old, and visits that last longer than 10 days. We exclude visits where the patient ultimately passed away, as the predictions are made upon discharge.
eICU Pollard et al. 2018). We extract data from the patient, diagnosis, treatment, medication, lab, and ApscheAps Var tables. Diagnosis, treatment, and medication are recorded as medical codes. We select our cohorts by filtering out visits of patients younger than 18 older than 89 years old, visits that last longer than days, and visits lasting shorter than 12 hours, as the predictions are made 12 hours after admission.
16

---

Published as a conference paper at ICLR 2024
D.2 TASKS
We focus on two common clinical predictive tasks: readmission prediction and mortality prediction.
For the MIMIC-IV dataset, the predictions made at the time of discharge. Mortality prediction is defined as predicting whether a patient will pass away within 90 days after discharge. Readmission prediction aims determine whether a patient will be readmitted within the next 15 days following discharge. We extract the mortality label from the date of death information. For patients without date of death information, we further check their next admission date. If their next admission date is after 90 days, we set the mortality label to false. Otherwise, we set the mortality label as missing. For the readmission label, we compare the current discharge date versus next admission date. We set the readmission label to missing for each patient's last visit, since it is possible that the patient is transferred a different hospital.
In the case of the eICU dataset, the predictions made 12 hours after admission. Mortality predic- tion aims to predict whether a patient will pass away upon ICU discharge. Readmission prediction, on the other hand, aims to determine whether a patient will be readmitted to ICU within the next 15 days during the same hospital stay. Both mortality and readmission labels are extracted from the current hospital stay, and there are missing labels. Thus, simulate the missing label scenario by masking out 50% labels randomly.
D.3 METRICS
We use the metric functions from the sklearn library. The AUC-ROC and AUC-PRC scores are calculated follows:
from kalen import metrics
auc_roc = metrics.roc_auce rar_srr_o_auccorroooe (y_true, y_score)
precision, recall, thresholds metrics, metrics. wwwpseprrrriiiooeeeelooooooo 4 y_true, y_score)
auc_prc metrics. auc (recall, precision)
D.4 BACKBONE ENCODERS
For demographic (age, gender, ethnicity) and sequential medical coding data (diagnosis, procedure, and medication). we use the Transformer (Vaswani et al., (017)) architecture as the backbone en- coder. The number of layers is 2, the embedding dimension 128, and the number of attention heads is 2. The event embedding look-up table is initialized with ClinicalBERT Alsentzer et al. (209)) embeddings of the event name and then projected down to 128 dimension with linear layer. Patient demographics features (age. gender, and ethnicity) are separately embedded with another embedding look-up table. The medical and patient demographies are added together to form the overall sequence embedding. For times series data (lab values and vital signals), we use the Recur- rent Neural Network (RNN). set the number of layers to 2 and enable bidirectional propagation. For text data (clinical notes), length to 256. We add a project layer to map the modality embeddings to the same latent space of dimension 128.
the TinyBERT (Jiao et al. (019)) encoder. We set the maximum
E ADDITIONAL DETAILS ON AD DATA
E. DATASETS
ADNI (Jack et al., 2008). We use the TADPOLE.D1 D1_22 table from the TADPOLE data chal- lenge MMaiiessuu et al. 2019), which integrates data from the ADNIMERGE spreadsheet as well as additional MRI, PET (FDG, AV45 and AV1451), DTI and CSF biomarkers. Specifically, we use the MRI biomarker features from the CCSFFSL.02201.
and UCSFFSLSIALL.O0 01_16 tables, the PET ROI biomarker features from the BAIPETNMRC.09.1222
UCBERKELEYNV55.0... and UCBERKELEYAVIAS 1451 10 17 16 tables, and ROI summary measures (e.g. mean diffusivity MD, axial diffusivity AD) from the DTIROI 04 30 14 table.
17

---

Published as a conference paper at ICLR 2024
Table 5: AD Dataset statistics.
Item
ADNI
#Patients
#Admissions
Label Distribution
Normal Control: 0.03, Mild Cognitive Impairment:
0.46, Alzheimer's Discase: 0.22
Label Missing
0.08
Age Gender Race
74
F: 0.45. M: 0.55
Indian/Alaskan: 0.00, Asian: 0.02. Black: 0.04, Hawai-
ian/Other PI: 0.00, More than one: 0.01. Unknown:
0.00, White: 0.93
DTI Biornarkers Missing %
0.87
PET Biomarkers Missing %
0.65
Biomarkers Missing 0.27
E.2 TASKS
We focus on the Alzheimer's disease progression prediction task. The goal classify each patient visit into normal control, mild cognitive impairment, and Alzheimer's 's disease. The label information extracted from the TADPOLE DI.DD. D2 table. There are no missing labels. Thus, we simulate the missing label scenario by randomly masking out 50% of the labels.
E.3 METRICS
We adopt the official metrics: the balanced accuracy score and the one-vs-one macro AUC-ROC score. They calculated follows:
from sklearn import metrics
b_acc metrica. ba bel.aacceeeeecccrrccccoooro (ttp:/ssorrryyytyooo (y_true, y-pred)
rer_roo_macro___ = metrics roc_iicsrro_uccsscoooo (y_true, y_score, average "mmarro,, catti_clas==ovo"")
E.4 BACKBONE ENCODERS
Given that the TADPOLE challenge supplies extracted features for each modality, we implement two-layer perceptron with ReLU activation and batch normalization as the backbone encoder. The hidden dimension is set to 128.
F ADDITIONAL EXPERIMENTS
F.1 ADDITIONAL EVALUATION ON THE MISSING NOT AT RANDOM SETTING
In this paper, follow existing works Zhang al. 2022 You et al. 2020) and mainly focus the missing completely at random setting. The modalities we consider are typically collected routinely for ICU patients, and idcally, they should be available for each patient. Missingness our context is primarily attributed to administrative issues or time gaps.
Additionally, we simulate scenario where label missingness correlated with modality missing- ness and the true label for the eICU mortality prediction task. The results can be found in Tab, 6 Results show that MUSE can also adapt to this non-random missingness scenario as the correlation could be learned in data-driven manner.
F.2 ADDITIONAL ANALYSIS OF THE USE OF THE SUPERVISED CONTRASTIVE LOSS
We would like to note that the supervised contrastive loss and the classification loss are calculated two different tensors. Specifically, calculating the contrastive loss, while the classification loss is based on the original patient embed- ding. Our intuition behind this design is that the contrastive losses encourage the model learn both
add projection layer the patient embedding before
18

---

Published as a conference paper at ICLR 2024
Table 6: Additional results the eICU mortality prediction task under the missing random setting. We simulated this setting by assigning a higher label-missing rate to deceased patients with missing vital signals while keeping the overall label-missing rate still 50%.
Model
AUC-ROC
CM-AE
0.8140
SMIL
0,8121
MT
0.8410
GRAPE
0.8698
HGMF
0.8511
M3Care
0.8700
MUSE
0.8753
MUSE+
0.8911
Table 7: Additional ablation study on the supervised contrastive loss terms of the AUC-ROC score.
MIMIC-IV
eCCU
ADNI
ID Method
Mortality
Readmission Mortality Readmission AD Progression
A3 MUSE w/o Supervised Contrastive Loss
0.8934
0.7088
0.8916
0.7699
0.9086
AS A3 w/ higher Classification Loss
0.8955
0.7002
0.8902
0.7713
0.9033
MUSE
0.9004
0.7152
0.9017
0.7709 0.8003
0.9158
MUSE
0.9201
0.7351
0.9332
0.9309
monality-agnostic and label-decisive features. In contrast, the classification loss leverages model- specific and label-decisive features to enhance prediction performance. To empirically support this, provide additional ablation study the supervised contrastive loss in terms of the AUC-ROC score in Tab. 7 This ablation study demonstrates the contribution of the supervised contrastive loss the overall model performance.
F.3 ADDITIONAL METRIC OF THE DISTANCE BETWEEN PATIENT REPRESENTATIONS
In our experiment analyzing the learned representations (Sec. 4.5p. calculate the average cosine similarity across all patients. Additionally, we include the average Euclidean distance between representations in Tab. g
Table 8: Fuclidean distance the representations from the same patients with different modalities.
Model
Euclidean Distance
CM-AE
0.7352
SMIL
0.6337
MT
0,5839
GRAPE
0.3001
HGMF
0.4426
M3Care
0.3286
MUSE
0.2437
MUSE+
0.2232
G
LIMITATIONS
Our paper primarily addresses the challenge of handling missing modalities and labels. However, it important to note that for given task, there is no guarantee that multimodal models will consis- tently outperform unimodal methods. Investigating the conditions under which multimodal models
19

---

Published as a conference paper at ICLR 2024
outperform unimodal ones is still open research question and falls outside the scope of this work. Instead, our focus is on scenarios where multimodal models have the potential to outperform uni- modal models, but their performance is limited by the absence of certain modalities and labels. For other applications, users should assess the necessity of employing multimodal learning before considering the application of proposed method to address the issue of missing modalities and labels.
H NOTATIONS
Notation
Meaning
(()
multimodal observation data for patient p
(x1
XM
(p)
M multimodal observations for patient p specific patient
P
m
a specific modality
M
total number of modalities
y(m)
label for patient p a multimodal model with parameter θ general loss function
fe(-)
2(-)
Drr = [(x{
(1) )
a training set of N patients
M
M
mortality missingness matrix whether patient has the modality m
A jp, m]
L
label missingness matrix whether the label is available for the patient p a bipartite graph with node V and edge set set of patient nodes
Lpp]
G (V,
Vp [uI,... UN
VM
(UI..... UM
set of modality nodes the edge between patient and modality backbone feature encoder for modality m initialized edge embedding
Cupum
Encoder (-)
h
(0)
OneHot(mm)
initialized modality node embedding initialized patient node embedding graph neural network
hug
GNN(.)
the learned patient representations embeddings for node i layer / /
h (1) h (2-1)
(1-1)
embedding for node j at leyer 1
(-1)
embedding for edge ejt between nodes and at layer neighboring nodes of message extractor at layer I message aggregator layer an augmented version of graph & through edge dropout
(()
Message (1)
Aggregate (1)
g'
Z
the augmented patient representations unsupervised mutual-consistent contrastive loss cosine similarity function
Cinsup (Z,Z') Z')
(
temperature hyperparameter supervised mutual-consistent contrastive loss
Lsup (Z, Y, L)
1(.)
indicator function
[L] 1)
select patient p and g with labels
[[L 1)
select patient with label
Y
predicted labels
MLP(-)
multi- layer perceptron the classification loss
CCE(Y, Y)
CE(:))
the cross-entropy loss
20


Reviews:
[{"review_id": "O6mQ4Ux9Du", "reviewer": "Reviewer_se67", "summary": "The authors tackle the problem of multimodal self-supervised representation learning using clinical data, where some modalities and some labels may be missing. They introduce a new contrastive earning method, which represents the dataset as a bipartite graph between patients and observed modalities. They initialize edge embeddings with observed data, and run a graph neural network over this graph, using a loss that combines supervised and unsupervised contrastive loss with the downstream classification loss. They evaluate their method on three clinical datasets, finding that they outperform the baselines.", "questions": "Please address the weaknesses above. There is also a minor typo -- the second $\\mathcal{G}$ should say $\\mathcal{G}'$ in Section 3.2 paragraph \"Mutual-Consistent Contrastive Loss\" .", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is intuitive and easy to understand.\n- The paper is generally well-written.\n- The authors motivate the gain of jointly modelling missingness in modalities and labels both intuitively and empirically.", "weaknesses": "1. The datasets studied in the paper are fairly limited from a modality perspective in the classical sense (e.g. MIMIC-IV has time series and text, and eICU has only time series). Though the authors split the time-series into multiple modalities (e.g. labs, vitals, diagnoses), it is unclear what a modality has to encompass in order for the method to work well. In the extreme case, could each individual measurement have been its own modality?\n\n2. The method proposed in the paper tackles a healthcare problem, but could definitely be applied to datasets beyond healthcare. The authors should consider benchmarking their method on datasets from [1], potentially by masking out labels and modalities.\n\n3. It was not clear to me reading through the paper exactly how the method handles completely new patients at test-time. Are they added as new patient nodes and edges in the bipartite graph, and the GNN is run in inference with all of the pre-training patient nodes still in the graph?\n\n4. The authors should add [2] as a baseline which does not make use of any labels during pretraining, which can then be finetuned on the labelled set. The authors should also add in baselines corresponding to pre-training an encoder on each modality separately (e.g. as in [3]), either with contrastive learning or with supervised learning, to show the gain of modelling modalities jointly in the pre-training stage.\n\n5. In healthcare, missingness is often informative, e.g. a patient did not receive a chest X-ray because the physician did not suspect any pulmonary conditions, and such patients are thus less likely to have a pulmonary condition. The authors should consider probing this effect in the experiments. In particular, on the eICU dataset (where the authors currently add missingness to labels randomly), they could instead correlate label missingness with modality missingness and the true label.\n\n[1] MULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning. NeurIPS 2021.\n\n[2] Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data. AISTATS 2023.\n\n[3] MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images. MLHC 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PDqTi6hzT", "reviewer": "Reviewer_3igL", "summary": "This work creates a contrastive learning approach to learning patient representations across data modalities, and augments the learning procedures by creating bipartite patient-modality graphs with various dropout to represent missing data modalities. Through the contrastive objectives, it learns to keep similar representations aligned in the embedded space despite a variety of modality differences. Additionally the authors learn to represent similar and dissimilar representations in relation to downstream labels, in order to provide an ability to represent patients even without said labels. They then demonstrate their model effectiveness over a variety of tasks, compared against a number of baselines.", "questions": "- Is it possible to achieve the same results by removing the supervised contrastive loss and adding a higher weight to the classification loss? If it is necessary, could you provide a justification for why we need to classification losses?\n\n - For the experiment analyzing the learned representation (Section 4.5) are you using the average cosine similarity across all patients? Could you also provide average Euclidean distance between representations since GRAPE minimizes MSE loss for continuous values?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The approach to learn representations that may allow for some variance in the features collected per person is an important finding for clinical applications\n\n- There is novel application and design of the contrastive objectives, allowing for modality differences as well as using labels to find representations guided by the strongest risk factors\n\n- There are a comprehensive set of baselines and the Muse+ model outperforms on all tasks\n\n- The paper is extremely well written and clear to follow", "weaknesses": "Major\n\n - The use of the supervised contrastive loss and the classification loss seems redundant. This is essentially the cross-entropy loss of the representations followed by the cross-entropy loss of a non-linear transformation of the representation. I appreciate ablation study showing that the removal of the supervised contrastive loss results in worse performance. I see that the final loss is the unweighted sum of all the losses. Is it possible to achieve the same results by removing the supervised contrastive loss and adding a higher weight to the classification loss? If it is necessary, could you provide a justification for why we need to classification losses?\n\nMinor\n - While ablation studies were provided that showed an array of dropout, did authors find a rate of missingness that ultimately resulted in the model ignoring certain features? (e.g. really rare lab examinations)\n - Typo in Appendix D.1 first paragraph. “For clinical nodes” should be “For clinical notes” I think.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWnmsFnyHa", "reviewer": "Reviewer_eaku", "summary": "The paper presents MUSE, a mutual-consistent graph contrastive learning method, which is designed to account for multimodal scenarios where both the label and different modalities are missing. This method is used in the context of clinical decision support and is validated on three different medical datasets (MIMIC-IV, eICU, and ADNI), against existing imputation and direct prediction approaches. Furthermore, the authors perform ablation experiments to better understand the value of each component in the model, examine the similarity of representations across methods, and discuss the run-time efficiency of methods. Overall, MUSE presents performance increases in the presented tasks without exceeding existing methods’ compute times.", "questions": "+ In the Missing Modalities and Labels paragraph, it is unclear what the text means by “fail to fully use inter-patient relationships?” \n\n+ Could it be clarified what is meant by storing and reusing similarity calculations? Where is the similarity calculation reused, and why?\n\n+ It is stated in the Training and Inference Section that for MUSE, the authors only feed the original graph through a GNN, but this contradicts the model diagram, which shows both augmented and original graphs going through a GNN. Can this be explained or clarified in the text?\n\n+ The implementation details are unclear. “We train all models for 100 epochs on the training set, and select the best model by monitoring the performance on the validation set.” What is meant by “all models”? Does this include the baselines or only differently initialized MUSE models? Are the authors using the validation performance to determine all hyperparameters, and what specific metric is used? Is there early stopping for epochs, or are models trained for the full 100 epochs? \n\n\n+ The paper cites the Transformer paper (Vaswani et al.) as the backbone encoder for several tabular modalities. The original Transformer paper cited is meant for text and not necessarily tabular data, so were there further adjustments made for this encoder?\n\n\n+ In the Related Works Section, the authors mention MulT, ViLT, and UMSE, as other existing methods. Why were those methods not used as baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper presents a strong argument for needing methods that can account for both missing modalities and missing labels. MUSE, based on the cited literature, is an original idea that has clear implications for the medical community and could be potentially extended to multimodal learning in other scenarios. \n\n\n+ MUSE is validated on three datasets, which is important in demonstrating its generalizability. Furthermore, the additional experiments presented in the paper (missing data, ablation, similarity, and run-time analyses), add interesting insights into the different methods and are valuable in understanding what types of models work well in these multimodal medical settings. \n\n\n+ The model architecture is straightforward and well-described. Representing patient-modality relationships in a graph and using edge dropout as a way to simulate missing modalities is creative yet easy to conceptualize. Incorporating both supervised and unsupervised contrastive learning is a great way to correct the modality collapse problem, as contrastive learning has been shown to be very effective in multimodal scenarios. \n\n\n+ The authors describe the method in great detail and include the exact parameter values they used, which is a great community contribution as this method can be easily reproduced.", "weaknesses": "+ A weakness of the paper is that some of the architecture, experimental design, and parameter choices are not explained and not justified. For example, there are many ways to compute similarity (such as the normalized dot product used in the well-established InfoNCE loss), so a discussion on the choice of cosine as the similarity function is needed. \n\n+ Similarly, while it is helpful for reproducibility to list the hyperparameters used in the model, this alone is not enough as their choice is not justified. Searching through the main paper as well as the appendix, there is no mention of hyperparameter tuning. Hyperparameter tuning is important both for the baselines and for the proposed MUSE to establish the fairness of comparison, as the wrong choice of parameter could lead to a baseline performing worse than it could. This is especially true as the baseline architectures are vastly different (ranging from auto-encoders to Transformers)\n\n+ Other unexplained choices include using 15% as the edge dropout rate, the use of Siamese GNN over other GNN methods, doing the missing ratio experiment only on the mortality prediction task for MIMIC, and choosing 30% randomly masked modalities in the analysis of the learned representations. \n\n\n+ Even if the MIMIC mortality prediction task is the most interesting, it would be great to include the same experiment for all other datasets and tasks in the appendix. Overall, anytime there is a choice of number, task, or a specific function, there should at least be a sentence explaining that choice, at a minimum backed by logic and intuition and ideally backed by experiments, to justify the omission of alternative choices. \n\n+ The paper mentions the use of bootstrapping 1000 times, however, it is unclear if the reported performance is an average over the random weight initializations.\n\n\n+ While the dataset statistics are mentioned in the Appendix, it is important to point out the class label imbalance present in the datasets in the main paper. This contextualizes the performance of the models; for example, MUSE+ performs with ~92% AUC-ROC on MIMIC mortality prediction, but only 9% of the data has a positive mortality prediction. Meaning, a model that does not learn truly learn the task, can simply guess the majority label and get 91% accuracy on the task. This may not be the case here, but this information is important to better understand performance.\n\nMinor points:\n\n+ The paper will benefit from a more comprehensive Related Works Section, not just on methods pertaining to missing modalities, but on other multimodal methods both supervised and unsupervised, and other contrastive loss functions. This would help the reader understand how the proposed formulas differ from existing formulas and why the author’s choices are better.\n\n+ The authors do not discuss any limitations. The paper would benefit from a Limitations and a Discussion Section to address many of the aforementioned points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwrPQVZCxI", "reviewer": "Reviewer_D1yA", "summary": "In this paper, the authors propose a novel framework, namely MUSE, to handle the missing modalities and labels while learning the multimodal representation of patients. The authors extend the conventional missing modalities problem to the missing labels issue and tackle both challenges concurrently using GNN-based contrastive learning. The authors conduct extensive experiments and analyses to demonstrate the superiority of the proposed MUSE.", "questions": "Besides the concerns raised above, I have the following questions regarding this work:\n1. If we regard labels as a subset of features, the whole architecture can be perceived as a pretraining framework combining two contrastive learning loss terms, along with a classification loss serving downstream tasks. I wonder if the authors have conducted experiments to pre-train MUSE with two contrastive learning terms and then fine-tune it using the classification loss term. The two-stage pattern might help the model to take a further step towards better performance by separating pretraining and fine-tuning. \n2. For consistency, it is better to replace the “modality-general” in the abstract with “modality-agnostic.” \n3. For readability, it might be better to explicitly introduce how the trio of losses are combined using an individual mathematical formula rather than verbally describing it in Section 3.3.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The missing label problem widely exists in medical representation learning. However, previous works in this area do not handle it appropriately. This work provides excellent problem formulation to describe this research gap and proposes valuable solutions that can be inspirable to subsequent research.\n2. The paper is well-organized. The research problem is described comprehensively. The necessity of emphasizing modality-agnostic and label-decisive features is clearly illustrated, solidifying the motivation of adopting supervised and unsupervised learning accordingly. \n3. Modeling the modality missing and label missing problem is ingenious. Through contrastive learning, the proposed MUSE appropriately leverages alignments corresponding to these two problems, i.e., alignment between original and augmented graphs and between patients sharing different labels. \n4. Experiments are extensive. Sections 4.4 – 4.7 significantly help readers to understand better how MUSE works, as well as advantages that cannot be simply reflected in major experiments in Sections 4.1 and 4.2.", "weaknesses": "1. More baselines should have been added for a more comprehensive comparison, e.g., MMIN (Zhao et al., Missing modality imagination network for emotion recognition with uncertain missing modalities.) and Robust Multimodal Transformer (Ma et al., Are multimodal transformers robust to missing modality?).\n2. Other clinical tasks for ICU data can also be explored, such as ARF prediction and shock prediction (Tang et al.,  Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)