Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources (2024)

Abstract: Although pre-training has become a prevalent approach for addressing various biomedical tasks, the current efficacy of pre-trained models is hindered by their reliance on a limited scope of medical sources. This limitation results in data scarcity during pre-training and restricts the range of applicable downstream tasks. In response to these challenges, we develop Medical Cross-Source Pre-training (MEDCSP1), a new pre-training strategy designed to bridge the gap between multimodal medical sources. MEDCSP employs modalitylevel aggregation to unify patient data within individual sources. Additionally, leveraging temporal information and diagnosis history, MEDCSP effectively captures explicit and implicit correlations between patients across different sources. To evaluate the proposed strategy, we conduct comprehensive experiments, where the experiments are based on 6 modalities from 2 real-world medical data sources, and MEDCSP is evaluated on 4 tasks against 19 baselines, marking an initial yet essential step towards cross-source modeling in the medical domain.

heading: 1 Introduction
text: Pre-training, a widely adopted technique with the primary objective of enhancing the performance of downstream tasks, is a practice extensively employed in natural language processing (Kenton and Toutanova, 2019; Radford et al., 2018). In the medical domain, researchers have dedicated efforts in pretraining powerful models, including ClinicalBERT (Huang et al., 2019), ClinicalT5 (Lehman and Johnson, 2023), and MedHMP (Wang et al., 2023). While these pre-training techniques benefiting from unlabeled data have showcased superiority in diverse medical downstream tasks, they still suffer from the following challenges:
∗Corresponding author. 1Source codes are available at https://github.com/
XiaochenWang-PSU/MedCSP.
Data scarcity. Training a robust pre-trained model typically requires a substantial corpus, particularly in a multimodal approach. However, obtaining a sizable training dataset in the medical domain poses challenges owing to concerns surrounding data privacy. Hence, exploring innovative approaches for integrating more medical data into the pre-training process becomes imperative. Limited downstream tasks. Current pre-trained models in the medical domain are often trained using data from a single source, thus limiting the spectrum of applicable downstream tasks. For example, MedHMP (Wang et al., 2023) pretrained on electronic health records (EHRs) source is only suitable for predictive modeling tasks involving EHR data. In contrast, pre-trained models in the general domain are usually applicable to various tasks. For instance, Flamingo (Alayrac et al., 2022), a visual-language model, achieves state-of-the-art performance on 16 few-shot learning tasks. Therefore, considering the multimodal nature of medical data, an ideal pre-trained model should be equipped to address as many tasks as possible.
To tackle these issues simultaneously, a promising approach involves training a model using diverse medical data sources from various datasets. This strategy not only augments the volume of training data but also broadens the spectrum of tasks. Nevertheless, achieving this objective is inherently challenging due to the following reasons:
Firstly, the number of patients who have data across multiple data sources is significantly limited. For example, this number is 14,620 between the MIMIC-IV and MIMIC-CXR databases, representing only 22.36% and 45.19% of these two databases, respectively. The scarcity of patients with data spanning multiple sources further diminishes the limited connectivity between these sources, adding complexity to cross-source integration efforts. Thus, designing an effective model that proficiently leverages the overlapped patients
3644
as a bridge to facilitate the training of other patients simultaneously is essential.
Secondly, modeling patients with data from multiple sources is challenging due to the intrinsic complexity of medical data. For example, a patient’s chest X-ray images and reports may be housed in the MIMIC-CXR database, while diagnosis and treatment information are stored in the MIMICIV database. However, owing to the temporal nature of medical data, their recorded times of information across various sources may not align perfectly. Therefore, exploring a reasonable approach to model these relationships is urgently needed.
Finally, implicit yet informative relationships often exist among patients across different sources, requiring appropriate handling. For instance, patients with analogous conditions may exhibit similar symptoms, despite being in separate databases. Recognizing and leveraging these implicit connections is essential for facilitating cross-source training, as they hold significant potential for enhancing model performance through the aggregation of similar patient profiles.
To address the aforementioned challenges inherent in multimodal medical records from diverse sources, we introduce a pioneering pre-training framework in this paper, named Medical CrossSource Pre-training (MEDCSP), as shown in Figure 1. MEDCSP first encodes each modality from each source using modality-specific encoders in Section 3.2. Subsequently, it employs two distinct pre-training tasks. The first task explores modalitylevel relations among patients within individual sources (Section 3.3), while the second task focuses on discovering relationships among patients across different sources (Section 3.4). Specifically, MEDCSP models relations for overlapped patients across sources by considering their record times in Section 3.4.1 and establishes connections among
patients in similar cohorts using their diagnosis similarities in Section 3.4.2.
Through interactive modeling, MEDCSP acquires the capability to generate informative and representative medical embeddings for diverse downstream tasks. Our exhaustive experiments across six modalities within two sources demonstrate the effectiveness of our pre-training strategy, providing an initial yet crucial solution for unifying diverse modalities across multiple medical sources.

heading: 2 Related Work
text: Multimodal Pre-training on Medical Data. Pretraining on multimodal medical data, although it has seen significant development in recent years, remains fragmented across various sources. The predominant approach to multimodal pre-training involves aligning images with text (Hervella et al., 2021, 2022a,b; Khare et al., 2021). Additionally, with the emergence of Large Language Models (LLMs), some pioneering studies have endeavored to integrate images into the semantic space of LLMs (Li et al., 2023; Moor et al., 2023). However, due to the constraints imposed by their pretraining data sources, applying these pretrained models to tasks devoid of images proves challenging.
Thus, research on pre-training with multimodal medical data excluding images remains relatively limited. Some researchers have achieved success by aligning numerical clinical features with diagnosis codes (Li et al., 2022, 2020), while others have explored the correlation between clinical language and codes. Recent advancements include modeling complex interactions within EHR data, incorporating multiple modalities such as diagnosis codes, demographics, clinical notes, medication codes, and clinical monitoring data (Meng et al., 2021; Wang et al., 2023). Nonetheless, these endeavors face challenges akin to those encountered in imagerelated pre-training, compounded by the issue of data scarcity within EHR data, which significantly restricts their broader applicability. Multi-source Multimodal Pre-training. Conventional pre-training techniques typically leverage diverse data sources to enhance the generalizability of representations, a principle that extends to multimodal settings. Previous studies (Lu et al., 2019; Cho et al., 2021; Su et al., 2019; Lee et al., 2023a; Tan and Bansal, 2019) have demonstrated this by combining image-text pairs from multiple sources, thereby enhancing model performance across vari-
ous domains. However, these models face limitations when confronted with new modalities, as they are built upon uniform data sources.
Recognizing the shortcomings of homogeneous multimodal pre-training approaches, recent endeavors have shifted focus towards harnessing more diverse and heterogeneous sources. Recent studies (Liang et al., 2022; Reed et al., 2022) have embraced data from varied modalities for joint pretraining, resulting in improved modality-specific encoders. Despite these advancements, designed to cater to general fields, these approaches struggle to capture latent medical correlations within multimodal health data, thereby impeding the generation of domain-specific representations.

heading: 3 Methodology

heading: 3.1 Model Input
text: Let p ∈ P represent a patient in the patient set P . The patient’s data may be distributed across multiple medical sources, as illustrated in Figure 1. We use Dps to represent data stored in the s-th source for patient p. Each patient’s data from a source s may contain multiple records, i.e., Dps = {Dps,r}R p s r=1, where R p s represents the number of records in Dps . In addition, each record usually consists of multimodal modalities. Let Dps,r,m denote the m-th modality in the r-th record from the s-th source for patient p. With these inputs, the subsequent step involves modality-level encoding.

heading: 3.2 Modality Encoding
text: Due to the significant differences among modalities in the data sources, employing a uniform encoder to embed them poses challenges. Hence, we adopt modality-specific encoders to map the modalitylevel data to a shared latent space, formulated as follows:
eps,r,m = Encoderm(Dps,r,m), (1)
where the specifics of each encoder Encoderm(·) are detailed in Appendix A. By averaging embeddings of modalities, we then obtain the record-level representation as follows:
cps,r = 1
M
M∑
m=1
eps,r,m, (2)
where M is the number of modalities.

heading: 3.3 Intra-source Pre-training
text: To conduct pre-training across multiple sources, the primary challenge lies in modeling the relationships among both modality-level and source-level data. Despite the different formats of modalities in sources, they inherently exhibit alignment. For instance, a chest X-ray image typically corresponds to a radiological report detailing the findings from the image, and a patient’s in-hospital visit correlates with a set of diagnosis codes, procedure codes, clinical notes, and so forth. These alignments indicate that corresponding data in different modalities convey information about the same clinical event or patient admission. Consequently, it is imperative that these modality-level embeddings are mapped as closely as possible.
An ideal approach to capture the relationships among modalities is through pair-wise modalitylevel contrastive learning. However, the pair-wise learning paradigm encounters a drawback: the computational complexity escalates significantly with a large value of M . To address this challenge, we propose conducting record-modality-level contrastive learning. Intuitively, as illustrated in Eq. (2), the record representation cps,r serves as the centroid of all modality-level representations. Ideally, each modality eps,r,m should be proximate to its corresponding centroid cps,r but distant from others, as shown in Figure 2.
Based on this intuition, we design our alignmentbased loss. The loss is based on InfoNCE (Oord et al., 2018) and functions on a record-modality pair (eps,r,m, c p s,r) for intra-source pre-training as follows:
La = − log exp(sim(eps,r,m, cps,r)/τ)∑
c p′ s′,r′∈Na
exp(sim(eps,r,m, cp ′ s′,r′)/τ) , (3)
where sim(·, ·) is the cosine similarity function, cp
′ s′,r′ denotes a randomly selected record within the batch, Na denotes the negative set, and τ is a
temperature hyperparameter. Thus, the total alignment loss is defined based on Eq. (3) as follows:
LA = ∑
p∈P
Sp∑
s=1
Rps∑
r=1
Ms∑
m=1
La, (4)
where Sp is the number of medical sources containing patient p’s data, and Ms is the number of modalities within the source s.

heading: 3.4 Cross-source Pre-training
text: The training objectives outlined in Eq. (4) serve to direct the model in capturing explicit alignment between different modalities for the same patient within the same source comprehensively. However, an unresolved issue remains: what if no explicit alignment is defined? This issue becomes particularly prominent in cross-source settings, where distributed medical data often represent distinct admissions and studies. To address this issue, we propose two additional loss functions to capture relationships among patients across medical sources. The first loss term focuses on modeling relationships for patients present in different sources, while the second loss term aims to learn the relationships of patients in similar cohorts among different sources.

heading: 3.4.1 Same Patients Across Different Sources
text: Intuitively, the data of the same patient across different sources should exhibit similar patterns, particularly for records archived within the same time window. Let us assume that a patient’s data in two distinct sources are denoted as Dps,r and Dpŝ,r̂, and the timestamps of these two records satisfy |T ps,r − T pŝ,r̂| ≤ θ, where θ represents a predefined time window threshold. The similarity between Dps,r and Dpŝ,r̂ represented by sim(c p s,r, c p ŝ,r̂) should be larger than that between Dps,r and a record Dp′s′,r′ randomly selected from different sources, i.e, s ̸= s′, if p ̸= p′. As illustrated in Figure 3,
we design a record-level cross-source contrastive learning loss for the same patients as follows:
LP = ∑
p∈P
Sp∑
s=1
Rps∑
r=1
Lp,
Lp = − log exp(sim(cps,r, c p ŝ,r̂)/τ)∑
c p′ s′,r′∈Np
exp(sim(cps,r, cp ′ s′,r′)/τ) ,
s.t. |T ps,r − T pŝ,r̂| ≤ θ,
(5)
where Np is the randomly selected pairs.

heading: 3.4.2 Patients with Similar Cohorts Across Different Sources
text: In addition to within-patient interactions, as shown in Eq. (5), relationships between records that neither share the same patient nor belong to the same source still require appropriate analysis. When considering two data samples from distinct sources, denoted as Dps,r and Dp̂ŝ,r̂, the absence of explicit similarity poses a challenge for understanding their relationship.
To ensure that no potential relationships across the medical domain are overlooked, we leverage diagnostic history from different patients to further capture implicit cross-source interactions. Intuitively, if Dps,r and Dp̂ŝ,r̂ belong to patients with the same medical history — such as two patients both suffering from schizophrenia and bipolar disorder — the symptoms manifested through their records should exhibit similarity. Conversely, data without any overlap in diagnoses, i.e., Dps,r and Dp ′ s′,r′ , are
unlikely to have similar recorded contents. Let us denote the multi-hot vector representing all distinct diagnosis codes related to the patient p as hp ∈ R|H|, where |H| denotes the distinct number of diagnosis codes. hp serves as the diagnostic history of patient p. By calculating the cosine similarity between two patients, p and p̂, we extend the definition of diagnostic similarity as follows:
αp,p̂ = sim(hp,hp̂). (6)
In Eqs. (3) and (5), we employ a strategy of directly choosing negative pairs with hard negative labels. This is because the positive labels exclusively originate from identical records (i.e., Eq.(3)) or patients (i.e., Eq. (5)). Consequently, pairs randomly selected in this manner exhibit a high confidence of being negative. Nevertheless, discerning positive and negative labels for similar cohorts drawn from distinct patients across diverse sources poses a challenge. To address this, we are prompted to
adopt the similarity score calculated by Eq. (6) as a soft label for each pair, which contains more finegrained information than a hard label.
As illustrated in Figure 4, given any cross-source pair (cps,r, c p ŝ,r̂), we can derive its diagnostic relationship αp,p̂ using Eq. (6). Building on previous research (Wu et al., 2021), we define the loss function designed for aggregating records associated with similar cohorts as follows:
LD = ∑
p∈P
Sp∑
s=1
Rps∑
r=1
Ld,
Ld = −αp,p̂ log exp(sim(cps,r, c p̂ ŝ,r̂)/τ)∑
c p′ s′,r′∈R
exp(sim(cps,r, cp ′ s′,r′)/τ) ,
(7)
where R is the relation set across sources.

heading: 3.5 Training Objective of MEDCSP
text: The final pre-training objective of MEDCSP is the weighted summation of alignment-based, patientbased, and disease-based contrastive learning terms, expressed as:
L = LA + λPLP + λDLD, (8)
where λP and λD are two hyperparameters aiding in balancing the loss terms. This aggregated optimization objective balances intra- and cross-source modeling on health data, catering to sources with diverse cohorts and modalities. We will showcase the effectiveness of MEDCSP through numerous experiments in the subsequent sections.

heading: 4 Experiments
text: In this section, we first outline the configuration of our pretraining process in Section 4.1 and then demonstrate evaluation with downstream tasks on EHR source (Section 4.2) and medical image source (Section 4.3), respectively. Due to the space limitation, we put more experimental results in Appendix D and E.

heading: 4.1 Pretraining Setting

heading: 4.1.1 Datasets of Pretraining
text: For our pretraining, we engage with two distinct sources: the MIMIC-IV dataset (Johnson et al., 2023), which acts as a proxy for EHR data, and the MIMIC-CXR dataset (Johnson et al., 2019), which represents sources of medical imaging. These datasets span six modalities: text, images, temporal clinical data, demographics, diagnosis codes, and medication codes.

heading: 4.1.2 Data Processing
text: We adopt existing pipeline (Tang et al., 2020) for preprocessing EHR data. We follow existing work (Wang et al., 2023) to set the values of hyperparameters. To showcase the model’s capability to manage non-overlapping cohorts across sources, we retain patients who do not appear in the MIMICCXR dataset. Regarding the CXR data source, we omit records from pre-training if their corresponding patients are not featured in the processed MIMIC-IV dataset, prioritizing efficiency. These excluded records are then utilized for zero-shot textimage retrieval tasks. This approach ensures the complete avoidance of any potential data leakage issues. From the pool of patients excluded from pretraining, we randomly select 1% and subsequently gather 1,202 records to form the evaluation set for the text-image retrieval task in Table 2. Comprehensive details on the datasets used for pretraining and fine-tuning across downstream tasks are provided in Table 1.

heading: 4.1.3 Implementation Details of Pretraining
text: We subject the introduced model to pretraining over 10 epochs with a learning rate of 1e-5. The batch size is configured at 128, optimized for the NVIDIA A100 GPU. Setups of modality-specific encoders are outlined in Appendix A. Throughout the training phase, we adjust all parameters, setting the balancing hyperparameters λP and λD to 0.5 and 0.2, respectively. Temperature hyperparameter τ is set to 0.1. Furthermore, we establish a time gap threshold θ of 30 days for the training process.

heading: 4.2 Evaluation on EHR Source

heading: 4.2.1 EHR Tasks
text: In-ICU Criticality Prediction. This experiment focuses on forecasting in-ICU activities by utilizing temporal clinical data and demographic information as inputs. We use three predictive tasks in this
evaluation, including acute renal failure (ARF) prediction, shock prediction, and mortality prediction within a 48-hour window.
Readmission Prediction. The goal here is to forecast the likelihood of a patient’s readmission within 30 days. This prediction’s input includes temporal clinical data, clinical notes, demographic details, diagnosis codes, and medication codes.

heading: 4.2.2 Experimental Setups for EHR Tasks
text: The data used in the evaluation of these tasks are extracted from the MIMIC-III dataset (Johnson et al., 2016), which avoids the label leakage issue. We divide the dataset into training, validation, and testing subsets at an 80%/10%/10% split. The baselines include F-LSTM (Tang et al., 2020), F-CNN (Tang et al., 2020), RAIM (Xu et al., 2018), DCMN (Feng et al., 2019), and MedHMP (Wang et al., 2023) for the In-ICU Criticality Prediction task. For the Readmission Prediction task, we use eight multimodal approaches present in existing study (Yang
and Wu, 2021) and MedHMP as baselines. Note that only MedHMP and the proposed MEDCSP are pre-trained models. However, MedHMP uses both MIMIC-III and MIMIC-IV databases for the pretraining, while MEDCSP conducts the pre-training with MIMIC-IV and MIMIC-CXR databases. In other words, MedHMP uses more training EHR data than MEDCSP for the EHR tasks.
To evaluate the effectiveness of our pretrained encoder, we merge its output embeddings for each task and employ a Multi-layer Perceptron (MLP) module for task-specific classification. We determined the optimal learning rate and batch size through a grid search, with the batch size ranging from 32 to 256 and the learning rate from 2e-5 to 2e-2. We employ the area under the PrecisionRecall (PR) curve (AUPR) and the area under the receiver operating characteristic curve (AUROC) as the evaluation metrics. The higher, the better. We obtain the final results as the mean values of five runs.

heading: 4.2.3 Results of Evaluation on EHR Source
text: The experiment results on the in-ICU criticality prediction task are depicted in Figure 5. The pretrained models, MedHMP and MEDCSP, outperform other non-pre-trained baselines. MEDCSP, pre-trained with less EHR data but taking the lead in all three tasks, demonstrates its superiority by using cross-sourced pre-training. This observation consolidates the correct rationale behind our design of a cross-source pre-training strategy. We can observe similar patterns for the readmission prediction task, as shown in Figure 6.

heading: 4.3 Evaluation on Radiological Source
text: One advantage of the proposed MEDCSP is to increase the diversity of downstream tasks by leveraging multi-source pre-training. To validate this advantage, we also conduct experiments to assess our model’s proficiency in analyzing radiological images and the corresponding reports.

heading: 4.3.1 Radiological Tasks
text: Text-image Retrieval. This task assesses the model’s ability to associate radiological images with corresponding textual descriptions correctly. It measures the model’s comprehension of visual elements and textual information.
Zero-shot Image Classification. The model’s accuracy in categorizing medical images into established categories without fine-tuning is evaluated. This ability is vital for healthcare applications and medical diagnostics, offering insights into the model’s utility in real-world scenarios.

heading: 4.3.2 Radiological Datasets
text: We utilize a subset of the MIMIC-CXR dataset which is NOT included in the pretraining phase for the text-image retrieval task. Extra experiments
on Open-I (Demner-Fushman et al., 2016) can be found in Appendix D. The text queries came from X-ray reports, and the corresponding X-ray images act as the ground truth for image candidates.
For zero-shot image classification, we use the COVID-19 dataset (Chowdhury et al., 2020; Rahman et al., 2021), consisting of COVID and nonCOVID lung X-ray images, as the evaluation task. Additional experiments on CheXpert (Irvin et al., 2019) are covered in Appendix E.

heading: 4.3.3 Implementation Details
text: We maintain the original configuration settings for all CLIP-like baseline models, including MEDCSP. Specifically, for LLaVA-Med, we utilize models designed for pure text input to encode textual data. To process images, we employ a summarizing prompt in conjunction with the radiological image as input. The final aggregated outputs from these processes serve as the encoded embeddings for both text and image modalities. We then calculate the similarity between these modalities using the cosine distance metric, facilitating a comprehensive evaluation of the model’s ability to bridge the gap between textual descriptions and visual representations. For the text-image retrieval task, we measure and report precision and recall at K scores, aligning with the methodologies established in previous studies, such as those detailed in (Wang et al., 2022) and (Zhang et al., 2023). In the image classification task, we document the precision, recall, and F1 score to evaluate model performance comprehensively.

heading: 4.3.4 Result Analysis
text: The findings from our Text-Image Retrieval task experiments, detailed in Table 2, indicate that our model, MEDCSP, significantly outshines CLIPlike baseline models with similar architecture in all assessed precision metrics at every k value. Impressively, MEDCSP even exceeds the performance of CXRCLIP (Lee et al., 2023b), another model pretrained on the MIMIC-CXR dataset, evidencing the superior advantage of our multi-source pre-training approach. This advantage is particularly noteworthy because it suggests that our model’s effectiveness is not merely due to its alignment with the test data’s origin.
Similar to the results listed in Table 2, MEDCSP outperforms baselines on the zero-shot image classification task, as shown in Table 3. These observations highlight the robustness of MEDCSP’s pre-training processes in forging strong correla-
Table 2: Results (%) on Text-image Retrieval Task
Methods Precision @K Recall @K 1 5 10 20 50 100 1 5 10 20 50 100 CLIP 0.17 0.18 0.17 0.13 0.14 0.12 0.08 0.67 1.16 1.75 4.63 7.79 MedCLIP 0.08 0.10 0.08 0.09 0.08 0.08 0.04 0.23 0.44 1.03 2.07 4.21 BiomedCLIP 0.50 0.53 0.43 0.39 0.31 0.26 0.46 2.29 3.49 5.89 11.79 18.73 PubMedCLIP 0.25 0.13 0.16 0.15 0.15 0.12 0.11 0.39 0.96 1.71 4.30 7.42 CXRCLIP 0.08 0.10 0.11 0.09 0.09 0.08 0.03 0.24 0.58 0.96 2.77 4.61 LLaVAMed 0.17 0.13 0.12 0.12 0.11 0.10 0.11 0.44 0.82 1.66 3.90 7.00 MEDCSP 12.06 6.41 4.45 2.97 1.64 1.04 8.74 21.91 29.51 38.04 50.49 61.74
tions between image and text modalities. Consequently, MEDCSP emerges as a powerful asset for addressing complex medical issues, demonstrating its particular strength in the field of radiological image analysis.

heading: 4.4 Ablation Study
text: Our ablation study is conducted from two distinct angles: loss-wise and source-wise. This approach allows us to examine not just the impact of each individual loss term but also the benefits derived from a cross-source setting. For this purpose, we utilize the COVID-19 image classification task as a means to analyze the effectiveness of our pretraining strategy. Through this methodical examination, we aim to uncover the specific contributions of different loss components and the value added by leveraging diverse data sources to enhance our model’s performance on a critical healthcare challenge. We report the F1 score of different settings in Figure 7. Source-wise Comparison. MEDCSPsingle, which represents the version of our model pretrained solely on the MIMIC-CXR dataset, exhibits a noticeable decrease in performance (↓15.36%) compared to its multi-source pretrained counterpart, MEDCSP. This comparison starkly highlights the indispensable role of cross-source pretraining, demonstrating the substantial benefits that accrue from incorporating a variety of data sources to im-
0.62
0.60
0.58
0.56
0.54
0.52
Figure 7: Results of ablation study. The X-axis denotes different settings, and the Y-axis represents the F1 score.
prove pretraining outcomes. Loss-wise Comparison. Delving deeper into the architecture of our model, we introduce notations MEDCSPA, MEDCSPA+P, and, MEDCSPA+D to represent only keeping LA, LA+LP , and LA+LD in the loss function (Eq. (8)), respectively. There are several observations: (1) The omission of any of these components results in a significant drop in performance metrics, emphasizing the essential contribution of each term to the model’s comprehensive efficacy. (2) Only utilization of MEDCSPA+P causes the most significant drop (↓11.56%) of the F1 score based on the experiment. The ablation study demonstrates the importance of disease-oriented metric learning terms. This analysis further elucidates the synergistic impact of these components in enhancing the model’s ability to navigate complex medical data landscapes.

heading: 4.5 Case Study
text: To better illustrate how MEDCSP benefits from patient-centric and disease-oriented learning objectives, we visualize the case study performance in this section. Patient-wise Modeling. The cross-source pretraining strategy aims to consolidate records from different sources linked by common patient identifiers. To evaluate the impact of alignment- and patient-centric training objectives, we focus on two
patients with the highest number of radiological records in the MIMIC-CXR dataset. We then visualize the embeddings of both modalities, CXR images, and their corresponding reports to facilitate a detailed analysis, as presented in Figure 8. We can observe that data from the same patients are organized according to modality rather than patient identity when using solely the alignment-based loss function Eq. (4). This observation suggests the insufficient modeling of the unique latent medical patterns specific to each patient. In contrast, our model, designed to capture patient-level consistency, effectively clusters data by patient rather than by modality. This comparison vividly demonstrates that our model acquires an excellent understanding of patient latent medical patterns through the targeted design of our loss function. Diagnosis-wise Modeling. We further explore our model’s ability to forge connections between patients diagnosed with similar diseases. The analysis, illustrated in Figure 9, focuses on the record representations from three distinct patients. Patients 1 and 2 exhibit diagnostic similarities, whereas Patient 3 does not share any common diseases. Our findings reveal that when our model is pre-trained in the comprehensive setting, it effectively clusters records of patients with similar diagnoses. In contrast, when the model is pretrained solely with the alignment-based loss, it faces challenges in forming consistent connections within disease-specific cohorts. This outcome underscores MEDCSP’s proficiency in capturing the relationships between patients with similar diagnostic profiles, thereby generating meaningful representations for diverse downstream tasks.

heading: 5 Conclusion
text: This paper introduces a novel pre-training framework, MEDCSP, specifically designed for the complexities of diverse and highly heterogeneous medical sources. MEDCSP aggregates patient data within individual sources by aligning different
modalities and subsequently captures patient relationships across multiple medical sources by leveraging temporal information and diagnosis history. Our experiments across a range of medical tasks and sources demonstrate that MEDCSP achieves superior performance. The observations are further supported by ablation studies and case analyses, underscoring the potential of MEDCSP in advancing medical cross-source modeling.

heading: 6 Ethic Consideration
text: The data utilized in our study have been appropriately de-identified according to Health Insurance Portability and Accountability Act (HIPAA) standards, which mandate the removal of all sensitive information as outlined in the HIPAA guidelines. As such, privacy concerns regarding the data we employ are mitigated. Additionally, the pretrained checkpoints of MEDCSP will be released following a thorough assessment of privacy, ethnicity, and security considerations.

heading: 7 Limitations
text: This study is constrained by computational resources, leading to the inclusion of only two medical databases during the pre-training phase. Recognizing the importance of diverse data for comprehensive learning, we aim to incorporate a wider array of medical sources in future research endeavors. Furthermore, we are considering an upgrade of our text encoding system by integrating advanced large language models (LLMs), as detailed in Appendix C. This strategic enhancement is expected to augment the learning capabilities of our framework, paving the way for more sophisticated analyses and applications in the medical domain.

heading: Acknowledgements
text: This work is partially supported by the National Science Foundation under Grant No. 2238275 and the National Institutes of Health under Grant No. R01AG077016.

heading: A Details of Encoders
text: As detailed in Section 3.2, we utilize modalityspecific encoders to handle data from different modalities. Specific details about these encoders and the modalities they correspond to are provided in Table 4. For initialization, we employ BiomedCLIP checkpoints for both the image and language encoders. Throughout the pretraining phase, the language encoders designated for clinical notes and radiological reports are set up to share parameters.

heading: B Baselines
text: B.1 Baselines for EHR Tasks The following multimodal approaches designed to handle clinical tasks serve as our baselines in EHRrelated evaluation: F-LSTM (Tang et al., 2020) is a Long-short Term Memory (LSTM) architecture that processes inputs consisting of concatenated demographic and clinical temporal features. F-CNN (Tang et al., 2020) is a conventional Convolutional Neural Network (CNN) operating on the concatenation of clinical time series and demographic information for prediction on down-
stream tasks. Raim (Xu et al., 2018) is an advanced architecture engineered to process clinical information with a multi-channel attention mechanism. DCMN (Feng et al., 2019) is a combination of two distinct memory networks, with one focusing on temporal information and the other on static demographic data, allowing for comprehensive analysis. MedHMP (Wang et al., 2023) leverages a hierarchical pretraining strategy for boosting the model’s performance in medical downstream tasks. Representations of modalities are aggregated through an attention mechanism for pretraining and fine-tuning. BertLstm et al. (Yang and Wu, 2021) contains different combinations of modality-specific encoders, including BERT, StarTransformer, LSTM, and MLP. Multimodal representations are aggregated through summation for prediction tasks.
B.2 Baselines for Radiological Tasks
We adopt the following baselines for our evaluation of the radiological source: CLIP (Radford et al., 2021) is the backbone architecture developed by OpenAI. By performing contrastive learning between aligned image-text pairs, the model marks a significant step towards the unification of vision and language domains. MedCLIP (Wang et al., 2022) is pretrained on multiple datasets in a multitasking pattern.It relies on labeled images for extracting medical knowledge, thus performing contrastive learning without leveraging alignment between image and text. BiomedCLIP (Zhang et al., 2023) leverages PMC-15M dataset for deepening CLIP’s adaptation in the biomedical domain, pretraining with InfoNCE loss (Radford et al., 2021). PubMedCLIP (Eslami et al., 2023) performs pairwise pretraining based on ROCO dataset (Pelka et al., 2018), following the conventional CLIP design. CXRCLIP (You et al., 2023) is pretrained on MIMIC-CXR dataset. The authors utilize contrastive learning loss between image and text, as well as multi-view of images, to achieve competitive performance. LLaVAMed(Li et al., 2023) is a multimodal Large Language Model (LLM) built upon pretrained LLaVA(Liu et al., 2023). It leverages the PMC-15M dataset for additional pretraining in a generative pattern.
We adopt image processors and tokenizers corresponding to each baseline for a fair comparison, as introduced in the original papers.
C Implementing MEDCSP with Large Language Model (LLM)
Inspired by recent findings demonstrating the efficacy of applying LLM in the medical domain (Li et al., 2023), we sought to integrate our pretraining strategies with proficient LLM. Specifically, for modalities other than text, we employ the modality-specific encoders detailed in Table 4 to generate uniform embeddings, which are then concatenated with modality-specific prompts as input for LLaMA (Touvron et al., 2023). Textual contents are directly encoded alongside the prompt. Our pre-training approach, encompassing various data modalities, records, and patient information, aligns with the methodologies outlined in Sections 3.3, 3.4, and 3.5. We adopt the LLaVA-med model (Li et al., 2023) as the structural foundation for our exploration.
During this exploration, we solely fine-tune the projection layer between the frozen visual encoder and the fixed LLM, comprising only 3.15 million parameters. We observed a notable performance enhancement in the text-image retrieval task. This improvement is particularly evident when comparing our results to those obtained with LLaVA-med, achieving a significant increase in Recall@100 on the MIMIC-CXR dataset (16.19% versus 7.00%). This exploratory investigation underscores the effectiveness of MEDCSP’s pre-training strategy and hints at its potential for integration with various backbone architectures.

heading: D Extra Text-image Retrieval Results
text: Although we meticulously executed data splitting that absolutely eliminates data leakage concerns in our text-image retrieval task experiments, there might still be skepticism regarding whether MEDCSP truly surpasses baseline models on the MIMIC-CXR dataset, given its pre-training on the
same source. To address this and demonstrate that the robust performance of MEDCSP is attributed to our strategically crafted pre-training approach, we conducted additional experiments on the Open-I dataset (Demner-Fushman et al., 2016). The results are presented in Table 5. Echoing the findings detailed in Table 2, MEDCSP consistently exceeds the performance of all comparison models across various metrics, further validating the effectiveness and soundness of our well-designed pre-training strategies.

heading: E Extra Experiments for Zero-shot Image Classification
text: To further demonstrate the effectiveness of MEDCSP in the zero-shot image classification task, we conducted experiments on the CheXpert dataset (Irvin et al., 2019). In these experiments, MEDCSP and baseline models are tasked with predicting the presence of an enlarged cardiomediastinum in images without any fine-tuning. The results are presented in Table 6. Consistent with our findings from the COVID-19 dataset experiments (Table 3), MEDCSP surpasses both CLIP-like baselines and the Large Language Model (LLaVAMed), underscoring its superior performance resulting from well-devised pretraining strategies.

heading: Future Work
text: Recognizing the importance of diverse data for comprehensive learning, we aim to incorporate a wider array of medical sources in future research endeavors. Furthermore, we are considering an upgrade of our text encoding system by integrating advanced large language models (LLMs), as detailed in Appendix C. This strategic enhancement is expected to augment the learning capabilities of our framework, paving the way for more sophisticated analyses and applications in the medical domain