CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Authors: Wei Dai
Peilin Chen
Malinda Lu
Daniel A Li
Haowen Wei
Hejie Cui
Paul Pu Liang
Venue: ICML 2025 poster (2025)
Keywords: Dataset
Evaluation
Benchmark
Multimodal Learning
Healthcare
Primary area: applications->health_medicine
TL;DR: We created CLIMB, a massive multimodal medical dataset with 4.51 million patient samples, and showed that training AI models on diverse clinical data types simultaneously improves performance by up to 29% compared to single-task approaches.

Abstract: Recent advances in clinical AI have enabled remarkable progress across many clinical domains. However, existing benchmarks and models are primarily limited to a small set of modalities and tasks, which hinders the development of large-scale multimodal methods that can make holistic assessments of patient health and well-being. To bridge this gap, we introduce Clinical Large-scale Integrative Multimodal Benchmark (CLIMB), a comprehensive clinical benchmark unifying diverse clinical data across imaging, language, temporal, and graph modalities. CLIMB comprises 4.51 million patient samples totaling 19.01 terabytes distributed across 2D imaging, 3D video, time series, graphs, and multimodal data. Through extensive empirical evaluation, we demonstrate that multitask pretraining significantly improves performance on understudied domains, achieving up to 29% improvement in ultrasound and 23% in ECG analysis over single-task learning. Pretraining on CLIMB also effectively improves models' generalization capability to new tasks, and strong unimodal encoder performance translates well to multimodal performance when paired with task-appropriate fusion strategies. Our findings provide a foundation for new architecture designs and pretraining strategies to adavance clinical AI research. Code is released at https://github.com/DDVD233/climb.

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation
Models
Wei Dai Peilin Chen Malinda Lu Daniel Li!
Haowen 2ei
Hejie Cui Paul Pu Liang'
Abstract
These domains include analyzing clinical images (Johnson et al., 2019; Irvin et al., 2019a), processing clinical notes (Liu et al., 2025; Johnson et al., 2016), and predicting patient outcomes (Yan et al., 2024). Despite these achievements, most current approaches remain limited to a few modalities primarily in the image and text domain (Thakoor et al., 2019; Jing et al., 2023b; Sharma al., 2022), failing to capture the interactions between many medical indicators that clinicians routinely combine to make holistic assessments on patient health and well-being (Liang et al., 2024b; Rajendran et al., 2023; Shaik et al., 2024).
Recent advances in clinical AI have enabled re- markable progress across many clinical domains. However, existing benchmarks and models are primarily limited a small set of modalities and tasks, which hinders the development of large-scale multimodal methods that can make holistic assessments of patient health and well- being. To bridge this gap, we introduce Clinical Large-scale Integrative Multimodal Benchmark (CLIMB), a comprehensive clinical benchmark unifying diverse clinical data across imaging, lan- guage, temporal, and graph modalities. CLIMB comprises 4.51 million patient samples totaling 19.01 terabytes distributed across 2D imaging, 3D video, time series, graphs, and multimodal data. Through extensive empirical evaluation. we demonstrate that multitask pretraining signif- icantly improves performance on understudied domains, achieving up 29% improvement in ultrasound and 23% in ECG analysis over single- task learning, Pretraining CLIMB also effec- tively improves models' generalization capability to new tasks, and strong unimodal encoder perfor- mance translates well to multimodal performance when paired with task-appropriate fusion strate- gies. Our findings provide a foundation for new architecture designs and pretraining strategies to advance clinical AI research. Code is released at this link.
To develop the next generation of holistic multimodal clin- ical foundation models, we introduce Clinical Large-scale Integrative Multi-modal Benchmark (CLIMB), compre- hensive multimodal clinical benchmark that unifies data across imaging, language, temporal, and genomic modal- ities. Our dataset comprises 4.51 million patient samples, totaling 19.01 terabytes of data, with a diverse spectrum of modalities: 707K 2D imaging data (including X-rays, dermoscopy images, fundus images, and pathology slides), 1.83M 3D or video samples (ultrasounds, CT scans, endo- scopic images and MRI images), 871K 1D data (electronic health records, EEG, ECG, gait and genomic data), 69.3K graph data (brain networks, molecules) and 1.03M multi- modal data combining multiple of the above modalities. We accomplish this through a novel data collection and pre- processing pipeline that standardizes diverse data formats from 33 different medical institutions while preserving the natural patterns of missing data. The dataset encompasses 96 different clinical conditions across 13 clinical domains, making it one of the largest and most diverse public clinical benchmarks to date.
1. Introduction
Advances in AI for clinical data have significantly helped doctors process and analyze complex clinical information for diagnosis. treatment planning, and decision support (Es- teva et al., 2019; Liu et al., 2025; Rajpurkar et al., 2022).
Through extensive empirical evaluation CLIMB, we es- tablish comprehensive benchmarks and best practices for clinical multimodal learning. As illustrated in Fig. 3, our analysis yields three key insights:
"aassachusetts Institute of Technology Athinoula A. Marti- nos Center for Biomedical Imaging 'Harvard Medical School "Stanford University. Correspondence to: Wei Dai <dv- dai @ edt.
1. Multitask pretraining:
We evaluate single encoders trained jointly on all tasks within each modality in CLIMB. Our experiments show that multitask pretrain- ing significantly improves performance across clinical tasks, achieving up to 32.54% AUC improvement in COVID ultrasound and other understudied areas. Fur-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Training
Testing
CLIMB
Encoder
Fusion
Multimodal
Dowustream
Graph
Time
Seriee
Layer
Midboritar
KIN; KEB
Geoettle
Disease Diagnosis
Graph
Transfer
OOD Dataset
X-nay
Demoo
Mastmo
Palio
Poccalned
1000
Vision
Patient Risk Prediction
CT Scan
XBRI
Eadoropii
Ulterootd
Figure 1. Overview of the CLIMB framework for training and testing multimodal datasets. The training phase incorporates diverse data modalities: graphs (molecular, BrainNet), 1D signals (ECG/EEG, genomics, EHR), 2D images (X-rays, dermoscopy, fundus, mammograms, pathology), and 3D scans (CT, MRI, endoscopy, ultrasound). Through multitask training on hetcrogeneous clinical data, our framework enhances model performance across individual tasks, particularly for understudied modalities defined Fig. 2(b). This approach improves both generalization to novel tasks and multimodal understanding when combined with appropriate fusion strategies, ultimately advancing performance on critical clinical applications like disease diagnosis and patient risk prediction.
thermore, pretraining on CLIMB drastically improves model performance in novel and understudied tasks for general-domain encoders, specialized clinical encoders, and clinical large vision language models (LVLMs).
2. Related Work
We cover related work in unimodal and multimodal clinical benchmarks and models.
2. Few-shot transfer: We test how models pretrained CLIMB generalize to new clinical tasks with limited labeled data. Models pretrained on CLIMB demonstrate sigmificant improvements in few-shot learning scenarios, achieving up to 29% improvement in ultrasound and 23% ECG tasks under few-shot settings compared to pretraining on existing datasets.
2.1. Unified Multimodal Clinical Benchmarks
The convergence of computational advances and large clini- cal datasets (Johnson et al., 2016; 2019; Irvin et al., 2019a) has enabled Al systems to match human performance across various medical tasks, from retinopathy detection drug discovery (Tsiknakis et al., 2021: Sone al., 2021; Rajko- mar et al., 2019). While large-scale multimodal foundation models have shown promise in learning unified clinical representations (Liang et al., 2024a; Yang et al., 2024; Phil- iastides et al., 2021), current benchmarks typically focus on limited modalities like X-rays, pathology, or their combi- nations (Moses, 2021; Schneider et al., 2022; Nasir et al., 2023). Large benchmarks include BenchMI (Wantlin et al., 2023) and CARES (Xia et al., 2024), covering 7 clinical modalities (1D, 2D, and 3D) and 16 different 2D and 3D im- age modalities, respectively. As shown in Table 1, however, our dataset uniquely incorporates time series and graph data alongside traditional clinical imaging while maintaining the widest coverage across each of the modalities.
3. Multimodal fusion:
Finally. investigate different strategies for combining multimodal clinical data, in- cluding imaging, text, and time series on MIMIC-IV, multimodal clinical benchmark. Our results show that modaliyyy pretraining on CLIMB enhances multi- modal learning performance, leading successful trans- fer MIMIC-IV, and that complex fusion strategies perform better challenging tasks.
In light of these findings, we release our vision, EEG, and ECG unimodal and multimodal models trained on CLIMB, which achieve state-of-the performance on multiple clin- ical tasks. We also provide detailed recommendations for model architecture selection and pretraining strategies across clinical modalities, establishing practical frame- work for future clinical Al development. All code for data collection, training, evaluation, and pretrained weights is available at this link.
2.2. Multimodal Clinical Models
Recent clinical Al models broadly fall into categories: LLM--baee multimodal systems and specialized vision coders. LLM-based approaches like Llava-Med (Li et al., 2023) and Med- Med-Flamingo (Moor et al., 2023) combine frozen vision encoders with language models for clinical
2

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
(a)
( Focus of CLIMB Datasel
from torch ut Sata ieport lataset
Novel Tasks
Undrestudied Mobiliis
Unkmepronttet Megaee
fros transforaers isport Tratner
fros 41im datanets ieport
frox (11m) aodels teport Contestiiiaasstt
ISRA,
COVID-19
datasetsi
dowelocd-rrie,
Ultrosound
South Ammeica
base /sath/to/sstaeess
rodel conteery2class if
trainer Trainert Ladee meedsl....
Other Viral
Premmonis
Grapha
Vietzam
trainor. traie() evalunte))
train, dataset-train baaderr loader,
eval, detasetural, Teader1
(d)
TUR
(e)
Data Samples
Sounon: COVID-BLATE
Tabel CIVID. Bevecity 2
Conmment:: teoken
000
pleur lnce rntall connchdated)
Label
Zone
Figure Overview CLIMB benchmark and code. (a) Visualization of CLIMB datasef composition. The inner ring displays the primary data modalities (2D, ID, Graph, Multimodal). The middle ring represents major clinical modalities within cach modality. The outer ring shows the names of specific datasets within each category, with the outer bar plot representing the number of samples in that dataset. A detailed description of each modality and datasets are included in App. A. (b) Focus of dataset collection. During the collection of CLIMB, we aim collect diverse range of datasets, with a special focus on novel tasks, datasets from understudied modalities, and datasets from underrepresented regions. (c) Distribution of data collection sites in CLIMB. Red regions indicate areas where clinical datasets are commonly collected, whereas blue regions indicate places where clinical dataset collections are rare. (d) Example code usage CLIMB framework. This code example loads custom mixed subset of CLIMB spanning across three modalities, then trains 2env2extv2 classifier on the dataset mixture with unified vocabulary. (e) Sample data from CLIMB. CLIMB preserves detailed labels, metadata and comments explaining the diagnosis.
tasks, but notably do not optimize the visual components. In contrast, vision-focused works like Swin-Unet (Cao et al., 2022) develop specialized encoders for clinical imaging, though typically for single modalities. M4oE (Jiang & Shen, 2024) represents a rare exception, using Swin Transform- ers (Liu et al., 2021) to create mixture of experts model across both CT and MRI modalities, but they failed ex- pand it further into more modalities. While general vision architectures like ConvNeXt and EVA-2 (Liu et al., 2022: Fang et al., 2024) have demonstrated strong performance on natural images, efforts to develop clinical-specific en- coders have largely focused on adapting older architectures, as seen in CLiP-based PMC-CLIP (Lin et al., 2023) and Vision Transformer-basee MedViT (Manzari et al., 2023), leaving the potential of modern architectures for clinical tasks largely unexplored.
3.1. Dataset Selection Criteria
CLIMB unifies diverse public clinical datasets into uni- fied benchmark designed specifically for developing and evaluating multimodal medical AI systems. To maximize the diversity of the data, we established three key criteria to guide our dataset selection process. As illustrated in Figure 2(b), we prioritize datasets that address one or more of the following objectives:
1. Novel tasks: Recent emerging clinical challenges, such as COVID-19 diagnosis from chest imaging.
2. Understudied modalities: Data types traditionally un- derrepresented in clinical AI, including electroencephalo- grams (EEG), endoscopic videos, and graphs.
3. Underrepresented regions: Clinical data from geo-
graphic areas with limited representation in existing benchmarks, particularly South America and developing regions of South Asia.
3. Dataset
A detailed description of selection methodology and inclusion criteria is provided in App. A.2.
In this section, we provide an overview of the CLIMB dataset, sourced from 44 public datasets spanning 15 modalities in 13 clinical domains. A detailed description of each modality and dataset is included in App. A, and a visual overview of the dataset is provided in Fig. 1. We first introduce the selection criteria of the CLIMB dataset, followed by the dataset information and statisties. In the end, we provide simple code snippet demonstrating the of CLIMB for model training and inference.
3.2. Dataset Construction
Based on the selection criteria above, CLIMB was even- tually sourced from 44 public datasets. Fig. 2(c) shows three examples in CLIMB, with labels COVID-19, Parkin- son's 's disease and Seizure, from ultrasound, brain network, and EEG, respectively (Wie, 2021; Cui et al., 2022; Jing et al., 2023a). Notably, additional metadata and explanation labels for the datasets also preserved, shown in the
3

---

d CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table Comparison of clinical benchmarks. Abbreviations: BN Brain Networks. Mol Molecules, ECG = Electrocardiogram, EEG
Ellettrenceephalogrmm Genom Genomics, Mammo = Mammoggapphy Derm = Dermoscopy, Fund = Fundus, Path = Pathology, CT Computed Tomography, MRI Magnetic Resonance Imaging, US Ultrasound, Endo Endoscopy. CLIMB (ours) the the most diverse, comprehensive, largest clinical public multimodal dataset up to date, which enables holistic studies on multiple modalities and provides data foundation for large-scale clinical pretraining across vision, language, time series and graphs. For BenchMD, 5.IM out of
5.2M samples are EEGs, with only 0.1M samples from other modalitics.
Dataset
#Samples
Graph
1D
2D
3D
BN Mol
ECG
EEG
Genom
Gait Text
X-ray Mammo Derm Fund Puth
CT
MRI
US Endo
BenchMD (Wantlin et al., 2023)
5.2M*
x
x
v
x
x
PMC-VQA (Zhang al., 2023)
149K
x
GMAI-MMBench (Chen et al., 2024a)
26K
x
X
X
X
X
X
v
X
CARES (Xia et al., 2024)
18K
x
X
X
x
x
CLIMB (ours)
4.51M
<
V
v
architectures, this information is not exposed to the model in the experiments, although we hope future works could utilize it to develop more robust and fair methods.
Table Comparison of dataset statistics. CLIMB-QA (ours)
demonstrates the largest scale in terms of number of words, QA pairs, and overall dataset size.
Dataset
#Words
#QA Pairs
Size of Dataset
3.3. Dataset Statistics
PMC-VQA
10.2M
227K
CLIMB contains 4.51 million samples totaling 19.01 ter- abytes, with the following composition: 871K (19.31%) ID time series and text data (including electronic health records, EEG, ECG, gait and genomic data), 707K (15.68%) 2D im- ages (including X-rays, dermoscopy images, fundus images, and pathology slides), 1.83M (40.56%) 3D or video data (including ultrasounds, CT scans, endoscopic images and MRI images), 69.3K (1.54%) graph data (including brain networks, molecules), and 1 1.03M (22.90%) multi-modality data combining multiple of the above modalities.
GMAI-MMBench
980K
26K
49 GB
CARES
1.74M
41K
21.61 GB
CLIMB-QA (ours)
129.IM
4.51M
19.01 TB
COVID-19 ultrasound example.
To enable holistic training and benchmarking, we need to unify the input data loading and prediction tasks. To unify the label space, we consider two options. The first is to pose all tasks as multi-label classification given clinical data samples from different modalities. We combine the vocab- ularies in each dataset while merging semantically equiv- alent labels. Specifically, given a heterogeneous dataset collection D = (Di,... } with mixed annotation types (multi-label/multi-dd)
As shown in Table 1, CLIMB has the widest range of modal- ities while incorporating time series and graph data, which distinguishes it from existing multimodal benchmarks in the field that typically only include images and text. Table 2 pro- vides a quantitative comparison with other clinical datasets, demonstrating that CLIMB significantly exceeds existing benchmarks in scale, containing 129.1M words and 4.51 QA pairs across 19.01 TB of data, substantially larger than other multimodal clinical QA datasets.
we define a unified label vocab- ulary V~ = U V where Vi represents the label set of dataset Dr. To ensure consistency, we standardize terminol- ogy variations and combine similar concepts such as Lung Opacity from Chexpert and Infiltration from Vindr CXR to maintain clean, unified vocabulary while preserving the original clinical meaning. A detailed description of how we standardize the taxonomy is included in App. C. The list of classes are included in App. Table 9.
Figure 2(a) shows the distribution across primary modalities and the size of individual datasets. We carefully balance the dataset such that each modality contains 3-5 datasets, pro- viding multiple data sources per modality while maintaining diversity within each category. The geographic distribu- tion of data sources is shown in Figure 2(c). The dataset includes data from 37 medical institutions across 18 coun- tries, including contributions from Vietnam, Iraq, India, and Brazil, expanding the representation beyond traditionally well-represented regions.
The second option is to pose everything as QA. We also built closed choice question-answering version of the dataset, namely CLIMB- -A, for comparable LVLM evaluation. In CLIMB-QA, each dataset is preprocessed with QA pairs containing close-ended multiple-choice questions. A de- tailed description of CLIMB-QA along with examples are included in App. B.
In addition, we preserve the metadata, demographic infor- mation, segmentation masks, and associated clinical reports from the original dataset and link them to every sample where applicable. To ensure comparability across model
3.4. Dataset Interface and Code Usage
We present unified interface to download and process our dataset into unified format for mixed large-scale pretrain- ing, given that the user has provided agreement consents
4

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
on individual dataset websites. Figure 2(d) shows the code for standard workflow of loading multiple medical imag- ing datasets and training classification model with our CLIMB framework. The entire training and evaluation seript can be completed in under ten lines of code while maintaining the flexibility for any custom models or train- ing loops. We also provide a standardized training pipeline that is easily reproducible and parallelizable across multiple machines and instances.
RQ2: Few-shot transfer. We investigate how well mod- els pretrained on CLIMB can generalize novel clinical tasks with limited labeled data. To evaluate few-shot gen- eralization, we test on out-of iittof-distribution (OOD) datasets 'D Dodd & Dtrain- The OOD datasets are selected to reflect either a new task or a different data source within the same modality, simulating real-world scenarios where models must adapt to novel diagnostic tasks with limited labeled examples (1, 8, and 32 samples). We curated diverse set of 10 datasets spanning 9 modalities, as detailed in App. D.7.I, A quantitative analysis of the OOD datasets is included in App. F. For each modality, we use the best-performing model from our RQI experiments: ConvNextv2 for vision e tasks, ECG JEPA for ECG analysis, and BIOT for EEG pro- cessing. We compare two scenarios: (1) models initialized with publicly released pretrained weights, and (2) models pretrained on CLIMB. Both are then fine-tuned using few- shot samples from the target OOD dataset.
4. Experiments
We run extensive experiments to investigate the core tech- nical challenges for developing clinical foundation models with CLIMB. Specifically, we ask the following questions:
1. RQ1: Can multitask pretrained clinical models work across multiple tasks consistently, especially for under- studied tasks?
2. RQ2: How well do multitask pretrained clinical models transfer to new tasks within the same clinical modality, especially tasks with limited data?
RQ3: Unimodal pretraining to multimodal fusion. We investigate how to effectively combine information from different clinical modalities (imaging. text, and time series) to improve patient outcome prediction. This addresses the practical clinical scenario where multiple types of medical data are available for diagnosis and prognosis. Our exper- iments pretrain models on CLIMB and transfer them to MIMIC-IV (Johnson et al., 2023), a large-scale multimodal clinical dataset. We evaluate three fusion strategies with increasing levels of mooa interaction: Late fusion, MLP fusion, and cross-attention fusion. Detailed architec- tural specifications are provided in App. D.2.3. We fix the encoder architectures across all experiments: 2envNextv2 for visual inputs, ClinicalB et al., 2025) for text, and ECG-JEPA for time series data. In addition, to evaluate how large scale pretraining helps with multimodal tasks across different encoders, we compare models initialized with our CLIMB pretrained weights against those using publicly available pretrained weights. We evaluate on two common clinical prediction tasks: length of stay (LOS) prediction and 48-hour in-hospital mortality prediction (48 THM). These tasks are clinically significant and require inte- grating information across modalities.
3. RQ3: Can multitask pretrained unimodal models be fused effectively to tackle multimodal clinical tasks?
4.1. Experimental setup
To answer the above research questions, we design our experiments as follows:
RQ1: Multitask pretraining, We investigate whether mul- titask learning can enable robust universal encoders for clin- ical tasks. For each input modality (vision 2D/3D, graph, EEG, ECG), we train single encoder jointly on all re- lated tasks in CLIMB, Each encoder is combined with a classification head that predicts task-specific labels from an aggregated vocabulary V. which encompasses diagnos- tic terms across all tasks within that modality. For each modality, we evaluate both specialized medical models and general-domain architectures. In vision, we compare medical-specific encoders (MedViT (Manzari et al., 2023), PMC-CLIP (Lin et al., 2023), RAD-DINO (Prrez-Gareía et al., 2025)) against general vision models (ConvNeXTv2 (Liu et al., 2022), SBB2 (Radford et al., 2021). Swin Trans- former (Liu et al., 2021), EVA-2 (Fang et al., 2024), Intern ViT (Chen et al., 2024b)). For time-series data, we evaluate ECG-specific model, ECG JEPA (Kim, 2024), against gen- eral time-series architectures, UniTS (Gao et al., 2024). In the EEG domain, we test specialized architectures includ- ing SPARCNet (Jing et al., 2023b), CNNTransformer (Peh et al., 2022), FFCL (Li et al., 2022), ContraWR (Yang et al., 2023), STTransformer (Song et al., 2021), and BIOT (Yang et al., 2024). We also evaluate SoTA clinical VLM, LLaVa- Med (I.i et al., 2023), on CLIMB-QA, a muestion-answering version of CLIMB designed for comparability across large VLMs and traditional encoders. Details on construction are included in App. B.
Evaluation metrics. For consistency, we evaluate all classi- fication tasks with balanced AUC, sensitivity, and specificity. Regression tasks (c.g., length of stay prediction) are evalu- ated with mean absolute error (MAE). Task specifications, metrics considerations and experimental setups are provided in App. D.
4.2. On RQI: Multitask Pretraining Performance Importance of multitask pretraining. Our experiments demonstrate that multitask pretraining yields substantial and consistent improvements across diverse clinical scenarios, as shown in Fig. 4. Notably, the most significant gains
5

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Eval
Eval
Vision Encoder
Multitank
Transfor
Stries Encader
Few Samplees
Undiestudie
RQL: Can mallitask pre-truined clinical mnodels work across musitiple tasks cousisteutly, especially undernttddidd tasks!
RQ2: How do muititask pre trained elinical models tronseer to tasles within the cluiced modaliyy, repecially tasks with limited data?"
RQ3: Can maltitaek pesttraiued unimeddal modela be fused alfoctävvly to tacklo snultmndds clintcad toels?
(a)
(b)
(c)
Figure 3. Experimental setup for evaluating (a) multitask, (b) transfer, and (c) fusion learning strategies, addressing RQ1, 2, 3 respectively. We show inputs to the model the left and display evaluation on the right. (a) trains single model on multiple clinical tasks across different medical modalities and evaluate on each individual dataset to see if model can generate across diverse tasks, including understudied ones. We compare the model's performance across different achitectures, as well as between the performance single task learning versus multitask learning. (b) explores how well multitask pretrained clinical models transfer to new tasks, despite having few samples the same modality. In this experiment, each model trained on a few samples of target out-of-distribution (OOD) datasct, as well as other tasks across different clinical modalities. (c) experiments whether multitask pretrained unimodal models can be fused effectively to tackle multimodal clinical tasks. The source encoders are first separately trained with diverse data in their on modality, before being fused and evaluated on new multimodal dataset (MIMIC-IV).
Table 3, Performance comparison of trained multitask models across various medical imaging modalities. Sen: Sensitivity, Spe: Specificity, AUC: Area under the receiver operating characteristic curve. CLIP-L2B: CLIP ViT-Laion 2B. The best performance for each metric is bolded. Detailed breakdowns of each dataset are attached in App. E.I. The experiment shows that general domain SoTA encoders perform better than specialized medical encoders, with ConvNeXTv2 offering the best performance-compul tradeoff.
Model
Chest X-Ray
Mammography
Dermoscopy
CT Scan
Fundus
Ultrasound
Overall
AUC Sen Spe
AUC Sen Spe AUC Sen Spe AUC
Sen
Spe
AUC
Sen Spe
AUC
Sen Spe
AUC
Sen
Spe
Clinical Encoders
MedVIT
.670
.253
833
.627
.417
.583
.522
.361
.639
.604
.382
.616
.320
.317
.688
.452
.500
.583
.579
.364
.690
PMC-CLIP
.725
.251
888
614
.322
.710
.674
.355
.706
.619
.407
.593
.508
.220
.785
.521
.384
.609
.635
.341
.724
RAD-DINO
.818
.406
.928
.666
.314
.701
.717
348
.715
.653
.408 .594
.606
.221
.786
.639
.431
.619
.681
.368
.729
General Domain Encoders
SBB2
.791
.401
922
.388
.262
.633
.784
.362
.724
.691
.403
.590
.732
.293
821
.711
.45
.689
.730
4420
.754
Swin Transformer
.795
.389
-226
.13
.200
.599
.815
.43
.747
.685
.429
.615
.770
.327
838
.705
.545
.712
.765
.436
.775
EVA-2
.863
.382
-929
.516
.320
.699
.716
.353
.744
.531
.496
.496
.780
.295
822
.462
.340
.659
.685
.372
.737
VIT
8155
.413
.30
.32
340
.713
.868
.543
.770
.706
.469
.652
.839
431
851
.735
.549
.718
.772
.492
.789
ConyNeXTv2
.817
.436
.939
558
.30
.706
.011
.568
.777
.611
466
.641
.833
.563
888
.774
.641
.770
.787
.537
.806
absolute AUC improvement of 23 percentage points (from 0.474 to 0.704) when comparing single-task to multitask pretraining approaches, as shown in App. Table 30. These results suggest that multitask learning is particularly effec- tive for scenarios where data or research attention has been historically limited.
Table 4. Performance comparison of graph neural networks across brain networks and protein structures. The best perfor- mance of each model is bolded. Graph transformer offers the best performance in terms of AUC and sensitivity specifiiiiy balance.
BrainNet
Molecular
Overall
Model
AUCC
Sen Spe
AUC
Sen
Spe
AUC Sen Spe
GCN
.804
.966
800
.763
.532
.760
.783 .644 .780
Comparison of encoder models. Our analysis reveals coun- terintuitive patterns in encoder effectiveness across differ- ent domains. In the visual domain, image-based models consistently outperform video models across both image and video tasks, which attribute to the substantially larger pretraining datasets available for image models. In addition, general-purposs architectures like ConvNextv2 significantly outperform clinical-specific encoders such as MedViT, achieving a 35.99 35.9% performance improvement. We hypothesize this superiority stems from the more diverse pretraining distribution encountered by general-domatt en-
GAT
.705
.916
.044
.83
.551
801
764
.733 602
Graph Transformer
.852
810
826
789
.381
.20
.820
.555 .833
are observed in the three understudied categories: novel tasks, understudied modalities, and datasets from under- represented regions, as defined in Sec. 3.1. Datasets that intersect multiple categories show the highest performance improvements, as exemplified by COVID-US with an AUC gain of 0.3254. In temporal modalities, particularly ECG analysis, the Ga dataset demonstrates this trend with an
6

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table EEG Model Performance on Different EEG Datasets. general. pretrained models (starting with BI0T-pretraing achieve better performances than models trained from scratch. However, there's s no clear relationship between the number of pretrained datasets the performance, indicating that quality and relevance of pretraining datasets may play a bigger role than the quantity of data.
IIC
TUEV
TUAB
Overall
Model Name
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
SPARCNet
.846
.525
.955
.589
.801
.495
.877
.286
.868
.796
.79
.797
.838
.605
.859
.557
CNNTraasformer
.809
.455
.890
.399
.874
.473
.913
.386
.879
.797
.797
.799
.854
.569
.867
.528
FFCL
.841
.588
.805
.448
.801
.443
.906
.347
.874
.787
.787
.789
.839
.562
.833
.528
ContraWR
.832
.446
.892
.410
.847
.439
.898
.370
.872
.782
.782
.781
.850
.556
.858
.520
STTransformer
.785
412
.884
.407
.701
.371
.874
.262
.864
.785
.785
.787
.783
.522
.848
.485
BIOT
.854
.510
.905
.499
.856
.466
.908
.371
.879
.798
.798
.799
.863
.591
.870
.556
BIOT-pretrain-PREES
.844
.496
.902
.486
.898
.580
.918
.373
.878
.797
.797
.799
.873
.624
.872
.552
848
.523
-906
.507
.880
.586
.914
.415
.882
.806
.806
.808
.870
.639
.875
.577
B10T-pretraing six-datasets
.862
.546
.911
.531
.78
.549
.917
.397
.869
.794
.794
.755
.870
.630
.874
.574
Table 6. Performance comparison of zero-shot and fine-tuned LLa Va-Med. LLaVa-MMd is the current SoTA LVLM model for clinical QA tasks. Both CLIMB-Con/Nexxv2 and LLaVa- Med are trained and evaluated on CLIMB in a comparable, close- ended manner. While directly fine-tuning the LLM CLIMB- QA improves performance over zero-shot cases, they still lag behind SoTA multitask encoders. namely LLLMB-ConNNextv2, SoTA
0.3
Novel Task
Undenstidiee Modallty
0.25
Underrspeesentee Regions
Multiple Categories
AUC Difference
0.2
Other Detssets
0.1
multitask encoder trained on CLIMB.
0.05
Zero-Shot
Fine- Fine-Tuned
Dataset
0
Acc
Sens
Spe
Acc
Sens
Spe
0.05
Chest X-ray
.088
.192
.808
.309
.207
.795
COVID-US
COVID mn gLSiE
Broo Turner
BUSI Brain
PADA LRAEEPES. 20
ciiis 1221
coviD CT INSPEC
Chapannn Ce
PTB.A
Pandes APTOS Gromatbed
MiNIC- you
LADD Hi
Vight Mtanmo
RSPET
ISI 3000 Piraau RiCH
MRI
.363
.473
.650
.480
.375
.625
Ultrasound
.448
.427
.640
.579
.389
.611
Mammography
.049
.203
.800
.741
.300
.700
Dermoscopy
.466
.296
.700
.673
.245
.756
Dataset
Fundus
.434
.202
.794
.578
.217
.783
Figure 4. Difference AUC achieved by the multitask model compared to single-task training. Novel Task represent emerg- ing clinical challenges like COVID-19; Underrepresented Regions indicates datasets from underrepresented regions in developing countries such as Brazil and China; Understudied Modalities in- cludes less common imaging types such ultrasound and CT scans. Datasets belonging to multiple categories are highlighted in pink. In general, multitask learning helps the model reach a better performance, with the greatest improvement observed in understudied tasks, regions, and/or modalities.
CT
.448
.424
.575
.788
.435
.585
Endoscopic
.000
1.00
.000
.296
.143
.857
Average
.287
.402
.621
.555
.289
.714
CLIMB-ConyNextv2
.877
.806
.787
modality and the availability of domain-specific pretraining data. In the graph domain, graph transformers work the best with the highest score across all metrics, as shown in Tab. 4.
coders. In the EEG domain, we see a similar trend, with pretrained models achieving better overall performance than all models trained from scratch, but models trained on more datasets (BIOT-pretruinn-iix-eee daxttassssssssssssss does not guaran- tee better performance than models trained on less datasets. This indicates that in the EEG domain, the quality and rele- vance of the pretraining datasets is of importance. On the other hand, in the ECG domain, specialized architectures demonstrate clear advantages over general models, with ECG JEPA outperforming the general-purpose time series model UniTS by 36.8%. This dichotomy suggests that the optimal choice of architecture depends heavily on both the
Comparing Vision Encoders with Large VLMs. We evaluated LLaVA VA-Meed on CLIMB-QA to assess current clinical VLMs' capabilities in multimodal understanding. While fine-tuning on CLIMB- LLMB--Q improves performance over zero-shot inference by 28.7 percentage points, these results significantly lag behind FLIMB-ConNNeXA
our SoTA vision encoder trained on CLIMB, by 32.2 per- centage points in accuracy. The gap is particularly evident in modalities requiring fine-grained visual understanding, such as chest X-rays (30.9% accuracy) and endoscopic images (29.6% accuracy), where the model struggles to maintain balanced sensitivity and specificity. These results suggest
7

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
BIOT (EEG)
ECG-JEPA (BCG)
UniTS (ECG)
0.9
0.9
P1 on Standard
CT Scan
0.8
0.8
PT on CT JMR
Fundus
on CLIMB
AUC
0.7
PT Suaddrdd Dutssets
0.55
Dermoscopy
0.6
0.45
8
32
Full
8
Full
Full
Pathology
Shots
Shots
Shots
Figure 5. Few-shot learning performance comparison across different time series domains. We evaluate the few-shot perfor- mance (I-shot, 8-shot, and full dataset) three representative models: BIOT for EEGs, ECG-JEPA and UniTS for ECGs. PT on Standard shows the performance when pretrained on their datasets from the original paper, while PT on CLIMB shows the perfor- mance when pretrained on our CLIMB dataset. Models pretrained on CLIMB demonstrate consistent improvements over the original ECG domain-specific models.
CXR
0,4
0.6
0.8
Mammo
EEG
Ultrasound
ECG
Figure 7. Few-shot performance of models across different pre- training (PT) datasets. CXR: Chest X-ray. We use ConvNextv2 for vision, ECG JEPA for ECG data, and BIOT for EEGs. PT on Standard Datasets is the performance of training one shot exam- ple on top of their own pretrained weights, while PT on CLIMB is the performance of training one shot example on top of our CLIMB dataset. For EEGs, shot results are used instead due to the inherent complexity of the task.
Chest X- X-Ray
Dermoscopy
CT Scan
0.85
0.65
0.7
IntermV
AUC
0.9
0.8
ConyNext
0.85
0.75
0.55
0.5
0.8
0.7
0.45
Shots
Shot
8 Shots
Full
Shot Shots
Fundus
Pathology
Overall
0.8
0.8
AUC
0.75
AUC
0.7
AUC
0.75
ically trained on this domain, which again illustrates the benefits of our large-scale pertaining data.
0.7
0.6
0.5
0.65
Shor 8 Shots
Full
1 Shoot
8 Shots
Full
1 Shot
8
Shots
Pull
Architecture comparisons. As shown in Figure 6, in the vision domain, we again found both ConvNextv2 and ViT- based models perform similarly under both full and few shot settings, as shown in Figure 6. ConvNext exhibit better performance in fundus and dermoscopy images, while ViT performs better for CT scans. We also found specialized ECG models like ECG-JEPA transfer better than universal time series models like UnITS, as demonstrated in Figure 5,
Figure 6. Few-shot performance on out-of-distribution datasets. InternVL. represents the biT-based model, while ConvNext repre- sents the convolution based model. In general, ViT-based models and ConyNext- based models exhibit similar overall performances across all shots, with VIT excelling at CT Scan tasks while Con- vNext performs better at diagnosing fundus images.
that CLIMB is a rich data source for clinical AI that has a substantially different distribution than the pretraining dataset of current LLMs, and that training on CLIMB-QA can substantially improve existing models. However, the fine-tuned results also make it evident that current vision- language models require fundamental architectural innova- tions and novel training paradigms for the model to match the performance of fine-tuned dedicated vision encoders.
Unimodal Pretraining to Multimodal Fusion Pretraining results. Experimental results in Table 7 demon- strate that encoders pretrained on CLIMB consistently out- perform those pretrained on other datasets across all eval- uation settings. This performance advantage is maintained across diverse tasks, including length of stay prediction and in- in-hospital mortality prediction, both in full-data and few- shot scenarios. Notably, our pretrained encoders achieve 5.78% lower MAE (2.61 vs 2.77) in LOS prediction and 15.9% higher AUC in 8-shot mortality prediction, suggest- ing that the diverse modalities in CLIMB enable more robust feature representations.
4.4. RQ3:
4.3. On RQ2: Few-shot Transfer Performance
Strength of few-shot transfer. As illustrated Figure 7, our large-scale pretraining dataset enables efficient learning of novel tasks with limited samples, demonstrating consis- tent performance improvements across all modalities. The impact is particularly significant in traditionally challenging modalities such as CT scans and ultrasound imaging, where models achieved substantial gains in AUC of 28.7% and 29.1%. 16, respectively. In time series domains such as ECG and EEG, our model outperformed SoTA weights specif-
Fusion comparisons. Our analysis reveals that the effec- tiveness of different fusion strategies varies with task com- plexity. For complex regression tasks like length of stay prediction, cross-attention mechanisms demonstrate supe- rior performance, achieving the lowest MAE of 2.61. In contrast, for binary classification tasks such as 48-hour in-
8

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
provements in few-shot scenarios across modalities. Finally, unimodal pretraining on CLIMB consistently enhances per- formance on downstream multimodal tasks. Based on these findings, we recommend leveraging general-domate archi- tectures for visual tasks and emphasize the importance of multitask pretraining, especially for understudied domains. For multimodal applications, we suggest matching fusion complexity to task requirements and utilizing large-scale unimodal pretraining before multimodal integration.
Table Performance comparison of different multimodal fu- sion approaches on length of stay (LOS) prediction and 48- hour in-hospital mortality (48 IHM) prediction tasks. CrossAtt: Cross Attention. Unimodal encoders trained on CLIMB transfers to better multimodal performance, given proper fusion strategies are used. Complex tasks like LOS require complex fusions like cross- tross-attention, while simple tasks like 48 IHM work well under simple MLP fusion. FT Fine Tuning.
LOS
48 IHM (Full FT)
48 IHM (8-Shots)
Enc.
Fusion
MAE
AUC
Sens
Spec
AUC
Sens
Spec
Looking ahead, our findings point to several emerging re- search directions: developing novel architectures that better balance general and domain-specinf features, finding new ways to combine unexplored modality combinations, and creating fusion mechanisms that adjust to task complexity. While dataset-specific models currently achieve higher per- formance through specialized optimizations, we encourage future research to develop general multitask encoders that can effectively adapt across diverse modalities and tasks. By releasing our dataset, code, and models, we hope to acceler- ate progress in these directions and advance the development of holistic clinical AI systems.
Late
4.78
0.689
0.495 0.760 0.524 0.001 0.994
SoTA
MLP
2.98
0.957
0.806 0.979
0,556
0.536
0.538
CrossAtt
2.77
0.786
0.628 0.814
0,580
0.286
0.766
Late
4.71
0.859
0.017 0,983
0.628
0.022 0.993
Ours
MLP
2.84
0.961
0.824 0.975
0.672
0.295
0.858
CrossAut
2.61
0.796
0.822 0.590
0,570
0.294
0.753
hospital-mortality (48 IHM) prediction, MLP-based concate- nation proves more effective, achieving the highest AUC while maintaining balanced sensitivity (0.824) and speci- ficity (0.975). While late fusion appears to achieve higher specificity in some cases, its near-7ero sensitivity indicates that it effectively defaults to predicting the majority class without any meaningful information. This pattern persists in the challenging 8-shot setting, where MLP fusion maintains the best performance while preserving reasonable balance between sensitivity and specificity.
Impact Statement
This paper presents empirical benchmarking, analysis, and development of multimodal clinical datasets and models. Multimodal AI can help clinicians analyze large-scale longi- tudinal data, make predictions, and investigate interventions. Furthermore, increasingly many indicators are no longer taken in the doctor's office, but daily, such as physiolog- ical sensors that track sleep, mood, stress, diet, exercise, and social interactions. Our findings can have a broad im- pact on developing holistic AI models of human health and wellness.
4.5. Comparison with Sptasit- SoTAs
To contextualize the performance of our pretrained unimodal and multimodal models, we compare them to prior reported results from state-of- -ttt models specifically optimized for individual datasets in App. Table 36. While dataset- specific architectures sometimes outperform pretrained mod- els through specialized optimizations, they often struggle to adapt to new tasks, even within the same modality it was trained on (see Sec. 4.2). The highly specialized nature of their architectures, like the multi-stage category-wise fine- tuning utilized in CFT (Chong et al., 2023), makes them difficult to adapt or retrain for different clinical tasks. There- fore, there is much value in training generalizable unimodal and multimodal models that can effectively adapt across diverse clinical modalities, tasks, and scenarios.
At the same time, data privacy and model fairness are crit- ical qualities. There may be privacy risks associated with collecting and making predictions from multimodal clinical data. We have taken appropriate steps to only access data that participants have consented to public release, and to the best of our knowledge, all data was anonymized and stripped of all personal (e.g., personally identifiable infor- mation) and protected attributes (e.g., race, gender). There is also the risk that clinical AI models capture spurious fea- tures from race, gender, and other demographic variables, especially more clinical data is provided. To deploy these algorithms at scale in the real world, it is also important to keep data and features secure without public sharing.
5. Conclusion
We present CLIMB, a comprehensive multimodal clinical benchmark unifying 4.51M samples across 44 datasets, 15 modalities, and 13 clinical domains. Our extensive em- pirical evaluation revealed several key insights for clini- cal AI development. First, multitask pretraining signifi- cantly improves performance on understudied modalities and novel tasks. Second, general-domas domaan architectures out- perform clinical-specffic ones in multitask settings. Third, models pretrained on CLIMB demonstrate substantial im-
Overall, CLIMB offers opportunities to study the promises of multimodal AI while mitigating potential risks at scale across clinical modalities, tasks, and domains. We will con- tinue expanding CLIMB to rigorously test for these social impacts and improve the safety and reliability of multimodal clinical models.
9

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Acknowledgments
tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185-24196 2024b.
We gratefully acknowledge NVIDIA for their generous sup- port through GPU computing credits. We also extend our sincere thanks to Dr. Farzan Vahedifard (Neuroscientist, Athinoula A. Martinos Center for Biomedical Imaging, Har- vard Medical School) for his insightful feedback on the clinical terminology used in our classification tasks, which significantly strengthened the rigor and clinical relevance of this work.
Chereshnev, R. and Kertész-Farkas, A. Hugadb: Human gait database for activity recognition from wearable inertial sensor networks. In Analysis of Images, Social Networks and Texts: 6th International Conference, AIST 2017, Moscow, Russia, July 27-29, 2017, Revised Selected Pupers 6. pp. 131-141. Springer. 2018.
Chong, C. F. Yang, X., Wang, T., Ke, W., and Wang, Y. Category- wise fine-tuning for image multi-label classification with partial labels. In International Conference Neural Information Processing, pp. 332 345. Springer, 2023.
References
Abråmoff, M. D., Folk. J. C.,, Han. D. P., Walker, J. D., Williams, D. Russell, S. R., Massin, P., Cochener, B., Gain, P., Tang, L., Lamard, M.. Moga. D. C., Quellec, G., and Niemeijer, M. Au- tomated analysis of retinal images for detection of referable dia- betic retinopathy. JAMA Ophthalmology, 131(3)351-351) 2013, doi: 10. 10.1.001jaaamm 20001000//ammpphhhhooooo0....... ori.org/10.0 1001 / jtwp://ttaaaaaaaaaaaaaaaaaaaaaa
Cohen, J. P., Morrison, P., and Dao, L. Covid-19 image data collection, arXiv 2003. 1/597, 2020. htt.ttps://giitat com/ http:/eeeee0////// wtw.riiiieeeeeteeeeeeeeeeee
Colak, E., Kitamura, F. C.. Hobbs, S. B., Wu, C. C., Lungren, M. P., Prevedello, L. M., Celpathy-Cramed J., Ball, R. L., Shih, G., Stein, A., et al. The rsma pulmonary embolism ct dataset. Radiology: Arrificial Intelligence, 3(2)xe200254, 2021.
URL https:// 221.2.0111.1 2013.
Al-Dhabyani, W., Gomaa, M., Khaled, H., and Fahmy, A. Dataset of breast ultrasound images. Data in brief. 28:104863, 2020.
Cui, C., Li, L.. Cai, H., Fan, Z., Zhang, L.., Dan, T., Li, J., and Wang. J. The chinese mammography database (cmmd): An online mammography database with biopsy confirmed types for machine diagnosis of breast. The Cancer Imaging Archive, 2021. doi: 10. F URL attps:://ol. org/ 0. 7937/ /aca www.e/eaaaaaeeeeeeeeeeeeeeeeeeea 4b16. 6.
Alday, E. A. P., Gu, A., Shah, A. J., Robichaux, C., Wong, A.-K. I., Liu, C., Liu, F,, Rad, A. B., Elola, A., Seyedi, S., et al. Classifi- cation of 12-lead ecgs: the physiiont/compuuing in cardiology challenge 2020. Physiological measurement, 122:2132, 2020.
Asia Pacifie Tele-Ophthalmolgyy Society. APTOS 2019
Cui, H., Dai, W., Zhu, Y., Kan, X., Gu, A. A. C.,, Lukemire, J., Zhan, L., He, L., Guo, Y., and Yang. C. Braingb: benchmark for brain network analysis with graph neural networks. IEEE transactions on medical imaging. 42(2):493-55) 506, 2022.
blindness detection. https://www/ https/:/wwwwwwwggl.eppp/ com/c/ aptos2019 1et20019-b1le b-li---bbnddsssseeeeeeeeeeeeeeee baliidness-ddeeetttin//eeee [Dataset].
2019.
Bhuvaji, S., Kadam, A., Bhumkar, P., Dedge, S., and Kanchan, S.
Cui, H., Mao, I.., Liang, X., Zhang. J., Ren, H., Li, Q., Li, X., and Yang, C. Biomedical visual instruction tuning with clini- cian preference alignment. Conference on Neural Information Processing Systems, 2024.
Brain tumor classification (mri), 2020. URL https: / www.
kaggle. . ctm/cc/////..c... wttpcc//////ttttccccttcctttwtttt 11.
Borgwardt, K. M., Ong, C. S., Schönauer, S., Vishwanathan, S., Smola, A. J., and Kriegel, H.-P. Protein function prediction via
Decencière, E., LaGraize, C.,, Pélégrin, P., Benassi, F. Régér, C., and Vautrin, T. Feedback on publicly distributed database: the messidor datahase. Image Analysis Stereology, 33(3): 231-234, 2014, ISSN 1854-5165. doi: 10.5566/as..1 155. URL. http:/ /ttp://dx.dee doi wot../g///////////////// 10. 55w6655665666 1155.
graph kernels. Bioinformatics, 21(suppl_ (e)pspppp_):i7-----1 1):147-i56, 2005.
Borkowski, A. A., Bui, M. M.. Thomas, L. B., Wilson, C. P., DeLand, L. A., and Mastorides, S. M. Lung and colon can- cer histopathological image datasct (le25000), arXiv preprint arXiv: 2019.
Diederik, P. K. Adam: A method for stochastic optimization. (No Title), 2014.
Cao, H. Wang. Y., Chen, J., Jiang. D., Zhang, X.. Tian, Q. and Wang. M. Swin-unet: Unet-like pure transformer for medical im- age segmentation. In Karlinsky, L., Michaeli, T., and Nishino, K. (eds.). Computer Vision ECCV 2022 Workshops- Tel Aviv, Is- rael, October 23-27, 2022, Proceedings, Part III, volume 13803 of Lecture Notes in Computer Science, pp. 205-218. Spriner, 2022. doi: 10. 10111009778-3-01-5556665 //doi.org// 10. 1007 000.000007777033 25061---5-66888 8_9.
Fhadi, A., Haider, W., Karargyris, I., Rizvi, S. A., Dandu, V., Chaudhary, M., Giardino, S., Mathur, A., Lake, D., Venugopal, V.. and MyhmaoMa T. COVIDx-US: An Open-Access Benchmark Dataset of Ultrasound Imaging Data for Al-Driven COVID-19 Analytics. Frontiers in Bioscience Landmark Edition, 27(7):198, 2022, doi: 00.31083/j.b1270777
9. URL https:
Chen, P., Ye, J., Wang. G., Li, Y., Deng, Z., Li, W. Li, T., Duan, H., Huang. Z., Su, Y. Wang, B.. Zhang, S., Fu, B.. Cai, J. Zhuang, B., Seihel, E. He, J., and Qiao, Y. :mmaicmmeench: A comprchensive multimodal evaluation benchmark towards general medical ai. arXiv 24ep 2024a. URL https: : / doi. .adi..re crg/10. 28w50/aa2iv....... 2408. 03361.
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., and Dean, J. guide to deep learning in healthcare. Nature medicine, 25(1): 24-29, 2019.
Fang, Y., Sun, Q. Wang. X., Huang. T., Wang, X., and Cao, Y. EVA-02: A visual representation for neon genesis. Image Vis. Comput., 149:105171, 2024. doi: 10.1016/.IMAVIS.222 2024.
Chen, Z., Wu, J., Wang, W. Su, W. Chen, G., Xing. S., Zhong, M., Zhang, Q. Zhu, X., Lu, I.., al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic
105171. h https: http://oo...ogg//// 105171.
. imavis.
10

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Gao, S., Koker, T., Queen, O., Hartvigsen, T., Tsiligkaridis, T., and Zitnik, M. Units: A unified multi-task time series model, 2024, URL https::///// // /awxivivvv...////a//.../aaaaavva wwt/oogg.//.......o... 4 403. 3...00130
of expert-level classification of seizures and rhythmic and pe- riodic patterns during eeg interpretation. Neurology, 100(17); e c1750-c1 762, 2023a.
Goldberger, A. L., Amaral, L. A., Glass, L.,, Hausdorff, J. M., Ivanov, C., Mark, R. G., Mietus, J. E., Moady, G. B., Peng. C.- K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals, circulation, 101(23):=21--2222 2000.
Jing, J., Ge, W., Hong. S., Fernandes, M. B., Lin, Z., Yang, C., An, S., Struck, A. F., Herlopian, A., Karakis, I., al. Development of expert-level classification seizures and rhythmic and pe- riodic patterns during eeg interpretation. Neurology, 100(17): c1750-c c1762 2023b.
Heller. N., Isensee, F., Trofimova, D., Tejpaul, R., Zhao, Z., Chen, H., Wang, L., Golts, A., Khapun, D., Shats, D., et al. The kits21 challenge: Automatic segmentation of kidneys, renal tumors. and renal cysts in corticomedullary-pp -haas ct. arXiv preprint arXiv:2307.01966 2023.
Johnson, A. E.,, Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L.. and Mark, R. G. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Johnson, A. E.,, Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y.. Mark, R. G.,, and Homg. S. Mimic- exr, a de-identified publicly available database of chest radio- graphs with free-text reports. Scientific data, 6(1):117, 2019.
Hssayeni, M., Croock, M., Salman, A., Al- Al-khafaji, H., Yahya, Z.,. and Ghoraani, B. Computed tomography images for intracranial hemorrhage detection and segmentation. Intracranial hemor- rhage segmentation using deep convolutional model. Data, 5 (1):14, 2020.
Johnson, A. E.. Bulgarelli, L.. Shen, L.. Gayles, A., Shammout, A. Horng, S., Pollard, J. Hao, S., Moody, B., Gow, B., et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Huang. S.-C., Huo, Z., Steinberg. E.,, Chiang, C.-C., Langlotz, C., Lungren, M., Yeung, S., Shah, N., and Fries, J. Inspect: a multimodal dataset for patient outcome prediction of pulmonary embolisms. Advances in Neural Information Processing Sys- tems, 36:17742 17772. 2023.
Kim, S. Learning general representation of 12-lead electrocar- diogram with a joint- embedding predictive architecture. arXiv preprint arXiv:2410.00 08559, 2024.
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-llcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019a.
Li, C., Wong, C.,, Zhang, S., Usuyama, N., Liu, H., Yang. J., Nau- mann, T., Poon, H., and Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Oh, A., Naumann, T., Gloherson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-lleus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R. I.., Shpanskaya, K. S., Seekins, J., Mong, D. A., Halabi, S. S., Sandherg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P.. and Ng. A. Y. Chexpert: A large chest ra- diograph dataset with uncertainty labels and expert compari- son. In The Thirty-Thire AAAI Conference on Artificial Intel- ligence, AAAI 2019, The Thirty- Sairty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2019, Honolulu, Hawaii, USA, Jamuary 27 February I, 2019, pp. 590-597. AAAI Press, 2019b. doi: URL https: :/tttp::/ee http///o..oo/ org/
Li, H., Ding, M., Zhang, R., and Xiu, C. Motor imagery eeg clas- sification algorithm based on enn-lstm feature fusion network. Biomedical signal processing and control, 72: 12:310342, 2022.
Liang, P. P.. Goindani. A.. Chafekar, T., Mathur, L., Yu, H., Salakhutdinov, R., and Morency, L.-P. Hemm: Holistic eval- uation of multimodal foundation models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
10. A1600//AALLV333003330000 10. 10011699//// /aa.i.aaaa...../ v31101. 3301590.
Liang, P. P., Zadch, A., and Moreney, L.-P. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10): 2024b.
Jang. J.. Kyung. D., Kim, S. H., Lee. H., Bae, K., and Choi, E. Significantly improving zero-shot x-ray pathology classifica- tion via tne-ttning pre-trained image-text encoders. Scientific Reports, 14(1) 1):23199, 2024.
Lin. W.. Zhao, Z.. Zhang, X., Wu, C., Zhang, Y., Wang, Y., and Xie, W. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In MICCAI, 2023.
Jiang Y. and Shen, Y. M oc: A foundation model for medi- cal multimodal image segmentation with mixture of experts. In Linguraru, M. G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., and Schnabel, J. A. (eds.), Medical Im- age Computing and Computer Assisted Intervention- - MICCAI 2024 - 27th International Conference, Marrakesh, Morocco, October 6-10, 2024, Proceedings, Part XII, volume 15012 of Lecture Notes in Computer Science, pp. 621-631, Springer, 2024. doi: 10. 10.1007/9988---031--229---- __58 URL https: //doi. org/ / 0. 10010007777/ 1117//98--3-000 031 72390- 2_.5.
Liu, F. Liu, C., Zhao, L., Zhang, X., Wu, X., Xu, X., Liu, Y., Ma, C., Wei, S., He, Z., et al. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):1668-373,, 2018
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song. Z., et al. generalist medical language model for disease diagnosis assistance. Nature Medicine, pp. 1-11, 2025.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision trans- former using shifted windows. In 2021 IEEE/CVP Interna-
Jing, J., Ge, W., Hong. S., Fernandes, M. B., Lin, Z., Yang. C., An, S., Struck, A. F., Herlopian, A., Karakis, I., et al. Development

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
tional Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9992-10002. IEEE, 2021. doi: 10.1 10.100/CCV48992.200 202.,000802 .00986. URL https: //doi. org/ 100. 2w/.0//CCC99222222 2021. 00986. 2trgg/....
2020. doi: 10. 10110. 106221. URL attps://ddn org/ org/10.1 1016/ dib. 2020. 2020.1
106221.
Papatheodorou, I., Fonseca, N. A., Keays, M., Tang. Y. A., Barrera, E., Bazant, W., Burke, M., Füllgrabe, A., Fuentes, A. M.-P. George, N., et al. Expression atlas: gene and protein expression across multiple studies and organisms. Nucleic acids research, 46(DI):D246-D35 2018.
Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., and Xie,
S. A convnct for the 2020s. In IEEE/CVF Conference on
Computer Vision and Pattemn Recognition, CVPR 2022, New Orleans, LA. USA, June 18-24, 2022, pp. 11966-11976 IEEE, 2022. doi: 10.1 10.C 2022.01167. URL https: //doi. org/ / 10. 1109/111//101 (0T.09//VVR222628 2022. 01167.
Pedrosa. J., Guilherme, C., Márcio. P., André, J., Eduardo, I., and António, A. Lndb dataset (version 4). In 17th International Conference on Image Analysis and Recognition (ICIAR 2020). Zenodo, 2023. doi: 10.5281/zemodo. 8348419. URL https: / //doi. org/ 10. 3281 zenodo. 8348419.
Lopez, S., Suarez, G., Jungreis, D., Obeid, and Picone, J. Au- tomated identification of abnormal adult eegs. In 2015 IEEE signal processing in medicine and biology symposium (SPMB), pp. 1-5. IEEE, 2015.
Peh, W. Y. Yao, Y., and Dauwels, J. Transformer convolutional neural networks for automated artifact detection in scalp eeg. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3599- 3602. IEEE, 2022.
Lopez, S., Gross, A., Vang. S.,, Golmohammadi, M., Obeid, I., and Picone, J. An amalysis of two common reference points for eegs. In 2016 IEEE signal processing in medicine and biology symposium (SPMB). pp. 1-5, IEEE, 2016.
Pérez-Garche F., Sharma, H., Bond-Taylor, S., Bouzid, K., Sal- vatelli, V., llsc, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence, 2025.
Manzari, O. N., Ahmadabadi, H., Kashiani, H., Shokouhi, S. B., and Ayatollahi, A. Medvit: robust vision transformer for generalized medical image classification. Computers Biology and Medicine. 157:106791, 2023.
pp. 1-12, 2025.
Moor, M.., Huang, Q. Wu, S., Yasunaga, M.. Dalmia, Y.. Leskovec, J.. Zakka, C., Reis. E. P., and Rajpurkar, P. Med-flaming a multimodal medical few-shot learner. Hegselmann, S., Parziale, A., Shanmugam, D., Tang, S., Asiedu, M. N., Chang, S., Hartvigsen, T., and Singh, H. (eds.), Machine Learning for Health, ML4H@ NeurlPS 2023, 10 December 2023, New Or- leans, Louisiana, USA, volume 225 of Proceedings of Machine Learning Research, pp. 353-367. PMLR, 2023. URL https: //proceedings mlr press / v225 mmo0//mmoo2222 2a.. htm.l.
Pham, H. H., Nguyen, T. H., and Nguyen, H. Q. mudd--mammo: large-scale benchmark datasct for computer-aided detection and diagnosis in full-ficld digital mammography. PhysioNer, 2022. URL https: //w.//ooooo///o/ org/ / /00. 1w026 7517.
Philiastides. M. G., Tu. T., and Sajda, P. Inferring macroscale brain dynamics via fusion of simultaneous eeg-fmrii Annual Review of Neuroxcience, 44(1):315-334, 2021.
R., S.LLL. F.. G., A.. H., and D., R. Curated breast imaging subset of digital database for screening mammography (cbis-ddsm) [data set), 2016.
Moses, D. A. Deep learning applicd to automatic discase detection using chest x-rays. Journal of Medical Imaging and Radiation Oncology, 65(5):498-517, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language su- pervision. In International conference on machine learning, pp. 8748-88 1788-8763 PMLR, 2021.
Nasir, N., Kansal, A., Barneih, F., Al-Shaltone, O., Bonny, T., Al-Shabi, M., and Al Shammaa, A. Multi-modal image clas- sification of covid-19 cases using computed tomography and x-rays scans. Intelligent Systems with Applications, 17:200160, 2023.
Rajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., and Wang, F. Patchwork learning: A paradigm towards integrative analysis across diverse biomedical data sources. arXiv preprint a7Xiv:2335..06217, 2023.
Nasser, A. A. and Akhloufi, M. A. Deep learning methods for chest disease detection using radiography images. SN Computer Science, 4(4):388, 2023.
Rajkomar, A., Dean, J., and Kohane, I. Machine learning in medicine. New England Journal of Medicine, 380(14):1347- 1358, 2019.
Nguyen, H. Q. Pham, H. H., Le, T. L., Dao, M., and Lam, K. Vindr-exr: An open dataset of chest x-rays with radiologist annotations. PhysioNet, 2021. doi: 10.13326/3akn 6287. URL https: //dcs: . erg/10. / 32kn-b287.
Rajpurkar, P., Chen, E.,, Banerjee, O., and Topol, E, J. Ai in health and medicine. Nature medicine, 28(1):31-33) 2022.
Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C. P.P. Heidenreich, P. A., Harrington, R. A., Liang. D. H., Ash- ley, E. A., et al. Video- based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802)2552 256, 2020.
Rotemberg, V., Kurtansky, N., Betz-Stablein, B., and al. A
patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific Data, 8(1):4, 2021. doi: 10. doi wrr.orgg////// /10 1038 /t/////119999777 0002111000022222-
URL https://
Pacheco, A. G., Lima, G. R., Salomão, S., Krohling, B., Biral, I. P., de Angelo, G. G., Alves Jr, C., Esgario, J. G., Simora, A. C., Castro, P. B., Rodrigues, F. B., Frasson, P. H., Krohling, R. A., Knidel, H., Santos, M. C., do Espírito Santo, R. B., Macedo, T. I.., Canuto, T. R., and de Barros, L. F. Pad-ufes- 20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in Brief, 32:106221,
Schneider, L., Sediouar-Pedarti S., Kuntz, S., Krieghoff-Henning. E., Hekler, A., Kather, J. N., Gaiser, T., Frochling. S., and Brinker, T. J. Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. Guropeanjouraall of cancer, 160:80-91, 2022.
12

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Shaik, T., Tao, X., Li, L., Xie, H., and Velásquez, A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom. Information Fusion, 102: 102040, 2024.
available electrocardiography dataset. Scientific data, 7(1):1- 15, 2020.
Wantlin, K., Wu, C.,, Huang, S.-C., Banerjee, O., Dadabhoy, F. Mehta, V. V., Han, R. W. Cao, F., Narayan, R. R.,, Colak, E., et al. Benchmd: A benchmark for unified learning on medical images and sensors. arXiv preprint axXiv:2304.0844
Sharma, S., Gupta, S., Cupta, D., Juneja. S., Gupta. P., Dhiman, G., and Kautish, S. [retracted] deep learning model for the automatic classification of white blood cells. Computational Intelligence and Neuroscience, 2022(1) ):7384131, 2022.
2023.
Wie, N. Covid-blues: A large-scale lung ultrasound datasct for covid-19 diagnosis. https:/ /ggittittb.u//.. com/ com/Ninati /innaiiiieee e/ COVID-BLJJ 2021. Maastricht University Medical Center.
Sone, K., Toyohara, Y., Taguchi, A., Miyamoto, Y., Tanikawa, M., Uchino-Mort M., Iriyama, T., Tsuruga, T., and Osuga, Y. Ap- plication of artificial intelligence gynecologic malignancies: A review. Journal of Obstetrics and Gynaecology Research, 47 (8):2577-2585, 2021.
Xia, P., Chen, Z., Tian, J., Gong, Y., Hou, R., Xu, Y., Wu, Z., Fan, Z., Zhou, Y., Zhu, K., ct al. Cares: comprehensive benchmark of trustworthiness in medical vision language models. Advances in Neural Information Processing Systems, 37:1 17:144033-140365, 2024.
Song, Y., Jia, X., Yang. L.., and Xie, L. Transformer-based spatial- temporal feature learning for eeg decoding. arXiv preprint arXiv:2106. 11170, 2021.
Yan, X., Wang. W., Xiao, M.. Li, Y., and Gao, M. Survival prediction across diverse cancer types using neural networks. In Proceedings of the 2024 7th International Conference on Machine Vision and Applications, pp. 134-138, 2024.
Song, Y., Zheng, Q. Liu, B., and Gao, X. Eeg conformer: Convo- lutional transformer for eeg decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineer- ing. 31: 31:710-719, 2022.
Yang, C., Xiao, C., Westover, M. B., and Sun, J. Self-supervised electroeneephalogran representation learning for automatic sleep staging: Model development and evaluation study. JMIR AI, 2:e46769, Jul 2023. ISSN 2817-1705, doi: 10,2196/46769. URL https: / /ai jmi r. wttpo///////// /e//////4466/ 6769.
Stark, C., Breitkreutz, B.-J., Reguly, T., Boucher, L.,, Breitkreutz, A., and Tyers. M. Biogrid: a general repository for interac- tion datasets. Nucleic acids research, 34(suppL.)::5535 DD335--DD399 2006.
Yang. C.,, Westover, M., and Sun, J. Biot: Biosignal transformer for cross-data learning in the wild. Advances in Neural Information Processing Systems, 36, 2024.
Takahashi, H., Tampo, H., Arai, Y., Inoue, Y., and Kawashima, H. Applying artificial intelligence to disease staging: Deep learning for improved staging of diahetic retinopathy. PloS one, 12(6)=01797900 2017.
Zhang, X., Wu, C., Zhao, Z., Lin, W., 7hang, Y., Wang, Y., and Xic, W. Pme-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint: 2305. 10415, 2023. URL https:/ //tp:///ooii./ooo//// http://ooo.oo///////// 10. 48550/ atr//riiiiiv///...//eaaeeeaaee.e 2305 10415.
Tayebi Arasteh, S.. Isfort, P., Saehn, M., Mueller-Franzer G. Khader, F., Kather, J. N., Kuhl, C., Nebelung. S., and Truhn, D. Collaborative training of medical artificial intelligence models with non-uniform labels. Scientific Reports, (:646, 2023.
Zheng, J., Chu, H., Struppa. D., Zhang, J., Yacoub, S. M.. El- Askary, H., Chang, A., Ehwerhemuepha, L., Abudayych, I., Barrett, A., ct al. Optimal multi- arrhythmia classification approach. Scientific reports, 10(1):2898, 2020.
Thakoor, K. A., Li, X., Tsamis, E., Sajda, P., and Hood, D. C. Enhancing the accuracy of glaucoma detection from act proba- bility maps using convolutional neural networks. In 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp. 2036-2040. IEEE, 2019.
Tschandl, P., Rosendahl, C., and Kittler, H. The ham ham10000 10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018
Tsiknakis, N., Theodoropoulos, D., Manikis, G., Ktistakis, E., Boutsora, O.,, Berto, A., Scarpa, F,, Scarpa, A., Fotiadis, D. I, and Marias, K. Deep lcarning for diabetic retinopathy detection and classification based on fundus images: A review. Computers in biology and medicine, 135:104599, 2021.
Twinanda, A. P., Shehata, S., Mutter, D., Marescaux, J., De Mathe- lin, M., and Padoy, N. Endonet: deep architecture for recogni- tion tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86 (661))86-99 97, 2016.
Vyas, N., Morwani. D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. Soap: Improving and stabilizing shampoo using adam. The Thirteenth International Conference on Learning Representations, 2024.
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D., Lunze, I., Samek, W., and Schaeffter, T. Ptb-xl, a large publicly
13

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
A. Individual Dataset Details
Table Breakdown of CLIMB dataset by Modalities
Dataset
# Samples
Clinical Domain
Modality
Task
Fine-grained
PTB-XL
Cardiology
ECG
Diagnosties. Attribute Recognition
Soapaan-Shaxxin
40K
Cardiology
ECG
Diagnostics
Georgia
20K
Cardiology
BCG
Diagnosties
CPSC
6K
Sleep Cardiology
ECG
A Abnormality Detection
IIC
Neurllggccal Disorders
Diagnostics
TUAB
409.5k
Neurological Disorders
EEG
A Abnormulity Detection
TUEV
111.9k
Neurological Disorders
EEG
Diagnostics
MIMIC-CXR
356K
Radiology
Chest X-ray
Diagnostics, Clussification
No
CheXpert
212K
Radiology
Chest X-ray
Diagnosties, Classification
No
VinDr-CXR
18K
Radiology
Chest X-ray
Diagnostics, Abnormality Detection
Both
COVID-19
2.9K
Radiology
X-ray
Diaguostics
No
CoronaHack
5.9K
Radiology
Chest X-ray
Diagnosties
No
VinD=-Mammo
20K
Radiology. Oncology
Manmography
Diagnostics, Abnommality Detection
Both
CBIS-DDSM
2.8K
Radiology, Oncology
Mammography
Diagnostics, Abnormality Detection. Classification
Both
CMMD
1.8K
Radiology, Oncology
Mammography
Diagmostics,
Segmentation, Abnormality Detection
Both
ISIC-2020
33K
Demattooogy. Oncology
Demoscopy
Diagnostics. Classification
No
HAM10000
10K
Deratooooy. Oncology
Demossopy
Diagnosties, Segmentation
Both
PAD-UFES-20
2.3K
Deratologyy. Oncology
Demoscopy
Diagnosties, Classification
No
Messidor-2
1.7K
Ophihalmology
Fundus
Diagnostics
No
APTOS 2019
3.6K
Ophthalmology
Fundus
Diagnostics
Jichi
9.9K
Ophthalmology
Fundus
Diagnosties, Prognostics, Severity Grading
No
LND6
5.6K
Radhology, Oncology
CT
Diagnostics, Abnornality Detection
Yes
INSPECT
23K
Radiology
CT
Diagnosties, Prognostics
No
KFTS23
Radinlogy, Oncology
Segmentation
Hemorrhage
2.5K
Radiology
CT
Diagnosties, Segmentation
Both
RSPECT
1.79M
Radiology
CT
Diagnosties, Classification
Yes
EchoNet-Dynamic
10K
Radiology
Uluasound
Sogmentation
Yes
BUSI
780
Radiology, Oncology
Ultrasound
Diagnosties, Segmentation
Both
COVID-BLUES
362
Radiology
Ultrasound
Diagnostics
No
COVID-US
242
Radiology
Ultrasound
Diagnostics. Severity Grading
No
Bruin Tumor
3,2K
Radiology, Oncology
MRI
Diagnostics
No
Brain MRI
253
Radiology. Oncology
MRI
Diagnostics
No
ABCD
9.5K
Radiology
BruinNet
Classification
Yes
ABIDE
1009
Radiology
BruinNet
Classification
Yes
PPMI
Radiology
BrainNet
Classification
PROTBINS
1113
Moleculur Biology
Molecule
Classification
No
PPI
57K
Molecular Biology
Molecule
Classification
No
L.C25000
18:7K
Pathology
Tissues
Classification
No
BCSS
5.26K
Pathology
Tissues
Classification
No
Cholec80
14.4K
Surgery
Video
Workflow Analysis, Segmentation
Yes
HuGaDB
364
Physical Medicine, Rehabilitation
IMU
Motion Analysis, Classification
Yes
Expression Atlas
4.5K
Molecular Biology, Genetics
Gene Expression
Expression Analysis, Classification
Yes
Goo
126K
Molecular Biology. Geneties
Gene Expression
Expression Analysis, Classificution Diagnostics
Yes
Vital
210K
Multiple CardiologyRattologg
Multimodal
Both
MIMIC-IV
800K
EHR, ECG, X-ruy
Diagmostics, Prognostics, Severity Gruding
Both
Table 8 shows a breakdown of the data sources in CLIMB. In this section, we provide details for each dataset. We describe the source and structure of the datasets, split used to evaluate the models, QA prompts, access restrictions, and licenses. A list of classes we constructed is included in Table 9. We also include a list of medical institutions and locations where the data was collected, as well as the demographic info available from each dataset Table 10.
The definitions of each column are as follows:
Clinical Domain and Modality. Clinical Domain involves the clinical specialty and the exact diseases described by the data. Our dataset aims cover as many clinical domains as possible. The final dataset spans 13 domains, including radiology, cardiology, pathology, dermatology, oncology, ophthalmology, molecular biology, sleep cardiology and neurological disorders. Within radiology, our dataset spans diseases such as breast cancer, kidney cancer, and pncumonia. Modality involves the type of data at play. Our data ranges from time series (ID), vision (2D& & 3D), text records and graphs, making one of the most diverse datasets up to date.
Task describes the applicable tasks for multi-modal model. CLIMB spans diagnostics (disease labels), attribute recognition (recognition of inherent structures in the data), abnormality detection (localized abnomality labels), segmentation, prognostics (labels related improvement of condition over time), severity grading, and plain classification (non-disease related labels).
Granularity specifies the level of localization the task involved. For example, tasks such segmentation and abnormality
14

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
detection requiring localized reasoning and reference specific object in an image are classified as fine-grained. In contrast, tasks that make prediction based the entire image (e.g. diagnostics) are considered coarse-grained.
Here, we first introduce the list of modalities in CLIMB, followed by a detailed description of each individual dataset.
A.1. List Modalities
ECG (Electrocardiograam) Elletrooardiogrrm is a cardiac diagnostic tool that records the electrical activity of the heart over time. The datasets in this modality include (PTB-XL, Chapman-Shaaxing, Georgia, CPSC), primarily focusing on cardiac diagnostics and abnormality detection. The collective classes across these datasets encompass Normal, Conduction Delay (CD), Hypertrophy (HYP), Myocardial Infarction (MI) Sinus TachyeanhaaBBaayenttaaCCCuototor (STTC), Atrial Fibrillation/ Atrial Flutter (A. Fib/Aflutter), and Other conditions.
EEG (Electroencephaloggran) EEG is a neurological monitoring method that records brain's 's electrical activity through electrodes placed on the scalp. The datasets in this category include {III. TUAB, TUEV), ), used for diagnosing neurological disorders and detecting abnormalities. The classes across these datasets include Seizure (SZ), Lateralized Periodic Discharges (LPD), Generalized Periodic Discharges (GPD), Lateralized Rhythmic Delta Activity (LRDA), Generalized Rhythmic Delta Activity (GRDA), Spike and Slow Wave (SPSW), Generalized Periodic Epileptiform Discharge (GPED), Periodic Lateralized Epileptiform Discharge (PLED), Eye Movement (EYEM), Artifact (ARTF), Background (BCKG), and simple Normal/Abnormal classifications.
Chest X-ray Chest X-ray a radiological imaging technique used to examine the chest cavity, including the heart, lungs, and surrounding structures. The datasets in this modality include (MIMIC-CXR, CheXpert, VinDr-CXR, COVID-19, CoronaHack]. The comprehensive set of classes across these datasets includes Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion, Pneumonia (including Bacterial and Viral), Pneumothorax, Pleural Other, Support Devices, Lung tumor, Tuberculosis, COPD, COVID-19, and No Finding.
Mammography Mammography is specialized medical imaging that uses low-dose X-rays to examine breast tissue for early detection of breast cancer. The datasets in this category include { (VinDr-Mammo, CBIS-DDSM, CMMD). The classes are primarily based the BI-RADS scoring system (ranging from 0-5) and binary classification of Benign versus Malignant lesions.
Dermoscopy Dermoscopy is a non-invasive skin imaging technique used for examining skin lesions and early detection of skin cancer. The datasets this modality include (12IC-2020, HAM00000, PAD-UFES-20). The classes across these datasets include Melanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Keratosis/Intraepithellall Carcinoma (AKIEC), and Other categories, with some datasets using simple Malignant/Benign classification.
Fundus Fundus photography is a specialized technique for capturing detailed images of the interior surface of the eye, particularly useful in diagnosing retinal conditions. The datasets include (Messidor-2, APTOS 2019, Jichij. The classes focus on different stages of Diabetic Retinopathy (DR), including None/No DR, Mild DR, Moderate DR. Severe DR, PDR (Proliferative DR), as well as SDR (simple diabetic retinopathy) and PPDR (pre-proliferative diabetic retinopathy).
CT (Computed Tomography) CT is advanced imaging technique that produces detailed sectional images of the body. The datasets in this modality include (LNDb, INSPECT, KiTS23, Hemorrhage, RSPECT). The classes across these datasets cover various conditions including nodule classification (23mm, <3mm, non-nodule), Pulmonary Embolism (PE) categories (No PE, Acute PE, Chronic PE, Subsegmental PE), Hemorrhage detection, and tumor classification (Benign, Malignant).
Ultrasound imaging uses high-frequency sound waves to produce real-time images of the inside of the body, The datasets include (EchoNet-Dynaml) BUSI, HOVID-BLUES COVID-US) The classes across these datasets include Normal, Malignant, Benign, COVID-19, and Pneumonia, with some datasets focused on segmentation tasks rather than classification.
Ultrasound
MRI (Magnetic Resonance Imaging) MRI uses magnetic fields and radio waves to create detailed images of organs and tissues. The datasets in this category include (Brain Tumor, Brain MRI). The classes focus on tumor detection and
15

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
classification, including No Tumor, Pituitary Tumor, Glioma Tumor, Meningioma Tumor, and simple presence/absence of tumors.
BrainNet Brain Network represents brain connectivity networks derived from neuroimaging data. The datasets include {ABCD, ABIDE, PPMI). The classes focus on binary classifications including Normal/Abnormal, ASD/Typical controls, and Control/PD patients.
Molecule Molecular data represents structural and functional information about biological molecules. The datasets include [PROTEINS, PPIJ, with classification focusing on enzyme/non-enzyme categorization for proteins and molecule property prediction.
Tissues
Tissue imaging from pathology involves microscopic examination of biological tissue samples. The datasets include (LC25000, BCSS). The classes include various types of adenocarcinomas (Colon, Lung), Benign tissue (Colon, Lung), Lung squamous cell carcinomas, and tissue components (Tumor, Stroma, Lymphocytic infiltrate, Necrosis/debris),
Video Medical video data captures dynamic medical procedures. The dataset in this category is {Cholec80), which focuses on surgery phase annotations and tool labels rather than traditional classification tasks.
IMU (Inertial Measurement Unit) IMU data captures motion and orientation information. The dataset (HuGaDB) includes classes for basic physical activities: Sitting, Standing, Sitting down, and Standing up.
Gene Expression Gene expression data measures the activity levels of genes. The datasets (Expression Atlas, Geo} are primarily used for classification tasks but rather for expression analysis.
Multimodal Multimodal datasets combine multiple types of medical data. The datasets include {Vital, MIMIC-IV), with MIMIC-IV specifically focusing on 48-hour Hospital-Mortalii prediction (Yes/No) while combining EHR, ECG, and X-ray data.
A.2. List of Datasets
1. PTB-XI (Wagner et al., 2020) a dataset of 12-lead ECGs from 18,869 patients of 10 second length. The raw waveform data was annotated by two cardiologists, who assigned and validated diagnostic classification, form, and rhythm statements to each record. We provide grouped label of 7 classes: Normal, CD, HYP, MI, STTC, A. Fib/ Aflutter and Other, following conventions from (Wantlin al., 2023). For out-of-domain transfer learning, we utilize the subclass diagnostic labels from the PTB-XL dataset, which provides more challenging 24-label classification task. Split: For multitask training, use the BenchMD split, which includes label remapping to 7 diagnostic categories. This split consists of 17,476 records in the training set and 4,361 records in the test set, totaling 21,837 records.
Access restrictions: The dataset is available to download from
Licenses: ECG records under this dataset are available in Creative Commons Attribution 4.0 International Public License http:/raaeeeccoooooooogo//ieeeee Dllealcoo
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
2. Chapman Shaoxing (Zheng et al., 2020) consists of 12-lead ECGs from 10,646 patients, created under the auspices of Chapman University and Shaoxing People's ' Hospital. We provide a grouped label of 7 classes: Normal, CD, HYP, MI, STTC, A. Fib/ Aflutter and Other, following conventions from (Wantlin et al., 2023).
Split: For multitask training, we use the BenchMD split, which includes label remapping to 7 diagnostic categories. The split consists of 38,207 records in the training set and 2,051 records in the test set, totaling 40,258 records. Access restrictions: The dataset is available to download from https//www.kaggle.omddaasssss ruteyyamereezommicer//maaaaaaano
Licenses: ECG records under this dataset are available in Creative Commons Attribution 4.0 International Public License http:///atiiccoooooooo/////eeeee
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
3. Georgia (Alday et al., 2020) is a database from the 2020 Physionet Computing in Cardiology Challenge, curated by Emory University. It consists of 12-lead ECGs from 15.742 patients of 10 second lengths and 500 Hz frequency, representing a unique demographic of the Southeastern United States. We provide grouped label of 7 classes: Normal,
16

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
CD, HYP. MI, STTC, A. Fib/ Aflutter and Other, following conventions from (Wantlin al., 2023).
Split: For multitask training, we use the BenchMD split, which includes label remapping to 7 diagnostic categories. The split consists of 18,622 records in the training set and 2,067 records in the test set, totaling 20,689 records.
Access restrictions: The dataset is available to download from http:///wwwwwwll..ooo///
bjoernjostein/georgaa- I dlleaddceg-chlllene-ddieee
Licenses: ECG records under this dataset are available in
Creative Commons Public Domain License https:/creativeoommnns.oc hwt.://mmmoo.oogpoooooooooo////e
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
4. CPSC (Liu et al., 201 8) is database from the 2021 China Physiological Signal Challenge. It consists of 12-lead Holter and 3-lead wearable ECG monitoring device recordings of variable lengths, each sampled at 200 Hz. We provide a grouped label of 7 classes: Normal, CD, HYP, MI, STTC, A. Fib/ Aflutter and Other, following conventions from (Wantlin et al., 2023).
Split: For multitask training, we use the BenchMD split, which includes label remapping to 7 diagnostic categories.
The split consists of 4,815 records in the training and 1,377 records in the test set, totaling 6, 192 records.
Access restrictions: The dataset is available to download from
Licenses: ECG records under this dataset are available in Creative Commons Attribution 4.0 International Public License hutps://crativecoommonoooo www.://mmooooorg/iiccsess//////e Detieeesseeeb//44..elaalleeee Ethical considerations: No personally identifiable information, hospital identification, or offensive content is present in the dataset.
5. IC et al., 2023b) consists of EEG samples collected from 2,711 patients at Massachusetts General Hospital, annotated by 124 raters. The publicly available version includes 134,450 EEG segments from 1,950 patients, each segment lasting 10 seconds. The test set is non-public. Our evaluation focuses on the 6 diagnostic categories: seizure
(SZ), lateralized periodic discharges (LPD), generalized periodic discharges (GPD), lateralized rhythmic delta activity (LRDA), generalized rhythmic delta activity (GRDA), and "Other" if none of those patterns was present.
Split: Since the test dataset not publicly available, we divide patient groups into training/validationnesst sets by 60%:20%:20%.
Access restrictions: The dataset is available to download from
Licenses: This dataset is available in the BDSP Restricted Health Data License 1.0.0.
sparcnet/view-licens// 1.1// 1.1/.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
6. TUAB(Lopez et al., 2015) is dataset from the Temple University EEG Corpus. The dataset consists of 276 EEG recording sessions from 253 subjects. Each session is segmented into 10-second samples using event markers. We evaluate the models based on the binary classification labels "Normal" and "Abnormal." Split: The training and test separation is provided by the dataset.
Access restrictions: The dataset is available to download from
http:///wwppss.cooooooo/c///edea http:/ppwess.ooooopppptctttttttt Users must apply using the form described on wtwp://ppspppponpppesss.ss httpp//s.c.cooppttsssccc///////a No personally identifiable information or offensive content is present in the dataset.
Licenses:
Ethical considerations:
7. TU et al., 2016) is a dataset from the Temple University EEG Corpus. The dataset consists of 518 EEG recording sessions from 368 subjects. Each session is segmented into 5-second samples using event markers. We evaluate the models based on the 6 event categories: spike and slow wave (SPSW), generalized periodic epileptiform discharge (GPED), periodic lateralized epileptiform discharge (PLED), eye movement (EYEM), artifact (ARTF), and background (BCKG).
Split: The training and test separation is provided by the dataset.
Access restrictions: The dataset is available to download from https::/isip. wwwp://wwwwesssoooooo/c/////c/ca eeg//
Licenses:
Users must apply using the form described on https://sipppiconepress. piconepress comm//pggecssiccccchmmtttt...... No personally identifiable information or offensive content is present in the dataset.
Ethical considerations:
8. MIMIC-CXI (Johnson et al., 2019) is dataset of chest X-rays in JPG format. Our evaluation utilizes the 14 disease labels: " "Atelectasis", "Cardiomegaly". "Consolidation", "Edema", "Enlarged Cardiomediastinum" "Fracture", "Luug Lesion", "Lung Opacity", "Pleural Effusion", "Pneumonia", "Pneumothorax", "Pleural Other", "Support Devices", and "No Finding"
Split: We use a training set of 348,516 records, test set of 7,709 records, and total size of 356,225 records. Access restrictions: The dataset is available to download from http:///wynooctttcccmmeeemmmmmnn with credentialed account and CITI Data or Specimens Only Research training
17

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Licenses: Radiology images under this dataset are available in PhysioNet Credentialed Health Data License 1.5.0 http://iiintttooooommmmnennnnine wwwp:liieeee/1O....
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
9. CheXpert (Irvin et al., 2019b) is a chest radiology dataset collected from Stanford Hospital, covering 65,240 patients and 224,316 radiographs. The original dataset labels each record with uncertainty level for 14 diagnostic observations including Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion, Pneumonia, Pneumothorax, Pleural Other, Support Device and No Finding. Our evaluation focuses on predicting the labels that are "positive".
Split: We use a training set of 212,243 records, a test set 225 records, and total size of 212,498 records. Access restrictions: The dataset is available to download from
2eb9-4565-affe- 1cf4/7ebe2 with registration
Licenses: Radiology images under this dataset are available under the Stanford University Dataset Research Use Agreement http:://saanfrrddiiiazzzeweeeeee www.wwwbbeetss........ww........ www.///wwwwweeeee.eeeaaaaaateeea
1 1cf417ebe2
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
10. VinDr-CXR (Nguyen et al., 2021) consists of adult chest X-rays collected from Hospital 108 and Hanoi Medical University Hospital in Vietnam. The dataset contains local labels for bounding boxes, however we evaluate our models based on the global labels: "Lung tumor", "Pneumonia", "Tuberculosis", "COPD", "Other diseases", and "No finding", all annotated by 17 radiologists with at least years of experience.
Split: We use a training set of 15,000 records, test set of 3,000 records, and total size of 18,000 records. 1.0.0/ with a
Access restrictions: The dataset is available to download from http:://bysioone.ooo//ctttm///// credentialed account and CITI Data or Specimens Only Research training
Licenses: Radiology images under this dataset are available in PhysioNet Credentialed Health Data License 1.5.0 http://pwsooonttttoooc/cooon/m//
wwwewwewleeeeeeleeee .0.0/
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
11. COVID-19 is chest X-ray dataset for COVID-19 related diseases. We evaluate the models based on the diagnostic labels:
"Normal", "Bacterial Pneumonia", "COVID-19", and "Viral Pneumonia".
Split: We use a training set of 2,002 records, a test set of 988 records, and a total size of 2,990 records.
Access restrictions: The dataset is available to download from
w0w.aarrshaaa11//oiiii/oo 1504/ovvi19-
detection-xray-datasee
Licenses: This dataset is available in the Creative Commons Attribution 4.0 International
License
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
12. CoronaHack (Cohen al., 2020) is chest X-ray dataset compiled at the University of Montreal. Our evaluation utilizes the diagnosis labels: "Normal", "Bacterial Pneumonia", and "Viral Pneumonia". Split:
We use training set of 5,284 records, a test set of 624 records, and total size of 5,908 records.
Access restrictions: The dataset is available to download from chest-xraddataser.
Licenses: This dataset is available in the Creative Commons Attribution 4.0 International License https//creativecommonns http:///ttticoooooooooo///////// wrgg/licesss/by/4..00
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
13. VinDr-Mammo (Pham et al.. 2022) consists of mammography collected from Hospital 108 and Hanoi Medical University Hospital in Vietnam. The dataset contains local labels for bounding boxes, however we evaluate our models based on the global labels for BI-RAD 1-5.
Split: We use a training set of 16,000 records, test set of 4,000 records, and a total size of 20,000 records. Access restrictions: The dataset is available to download from http::/wwww..hiooooooooo///////a with a data agreement
1.0.0/
Licenses: Images under this dataset are available in PhysioNet Restricted Health Data License 1.5.0 www.://pyysoont.oo//oeeeeeeeeeee wwwadd/mmmmooowiiemessssee 1.0.0/
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
14. CBIS-DDSM (R. al., 2016) curated subset of the Digital Database for Screening Mammography (DDSM). Our evaluation focuses on the BI-RAD labels (0-5).
18

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Split: We a training set of 2230 records, test set of 595 records, and a total of 2,825 records.
Access restrictions: The dataset available to download from http://wwwwwwemm..iiiii////e////
ddsm/
Licenses: Images under this dataset are available in Creative Commons Attribution 3.0 Unported License htwp://tiiiiinoooooooooniiiie//e Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
15. CMMD (Cui et al., 2021) is breast mammography dataset for 1,775 patients from China. Our evaluation utilizes the diagnostic labels, "Benign" and "Malignant", which are confirmed through biopsy. However, due to issues found in the provided labels, while this dataset is a part of CLIMB, it is not part of the experiment while pending expert verifications.
Split: We use a training set of 1,404 records, a test set of 468 records, and a total size of 1,872 records.
Access restrictions: The dataset is available to download from https:/www.ccnnerrmaaigge wtt.eeemmnnnggicciiicceccloeteao Licenses: This dataset is available in the Creative Commons Attribution 4.0 International License https://creativecoommossss
http://wwwwooooooooooo////eeee/a org///icenses/b//..0//
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
16. ISIC-2020 (Rotemberg et al., 2021) consists of dermoscopy of skin lesions from over 2000 patients, generated by the International Skin Imaging Collaboration (ISIC). We evaluate the models on the binary classification ("Malignant" or "Benign") for each image, where all malignant diagnoses have been confirmed through histopathology, and benign diagnoses have been confirmed using either expert agreement, longitudinal follow-up, histopathology.
Split: We use a training set of 26,501 records, a test set of 6,625 records, and total size of 33,126 records.
Access restrictions: The dataset is available to download from https:/cchalleene22000iiiieeeeee
Licenses: Images under this dataset are available in Creative Commons Attribution- 4.0 International License https:/crrettvvecommmos. org/licenses/by-nc44...
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
17. HAM10000 (Tschandl et al., 2018) is dataset from the ISIC 2018 classification challenge, comprising dermoscopy
images of pigmented lesions from from the ISIC archive. Our evaluation focuses on the 5 diagnostic categories: Melanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Keratosis/Intraepitheelall Carcinoma (AKIEC),
Other (OTHER)
Split: We use a training set of 8,012 records, a test set of 2,003 records, and a total size of 10,015 records. Access restrictions: The dataset is available to download from http://www..aaggg.ccc/m/ttttttt/ mmist-ham10000
Licenses:
Images under this dataset are available in Creative Commons International License https:/crreaiveeoommnns hwtps//iiieessssy--neeeeeeeee
4.0
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
18. PAD-UFES-20 (Pacheco et al., 2020) consists of dermoscopy images of 1641 skin lesions from 1373 patients. We evaluate the models on the 5 skin diagnostics, three of which are skin disease and three of which are skin cancers: Melanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic KeraatsiisInttrappiheellll Carcinoma (AKIEC), Other (OTHER). All of the skin cancers are biopsy-proven, and more than half of the skin disease are biopsy-proven as well.
Split: We use a training set of 1,839 records, a test set of 459 records, and a total size of 2,298 records. Access restrictions: The dataset is available download from http:://www.kgggl...comatttttsts
1002/// 202/skin-
cancer
Licenses: Images under this dataset are available in Creative Commons Attribution 4.0 International License https://creaiivccomooo.../iiiiit
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
19. Messidor-2 (Abràmoff al., 2013) (Decencière et al., 2014) is dataset of Diabetic Retinopathy (DR) examinations, where each record consists of two macula-centered eye fundus images. The dataset is kindly provided by the Messidor We utilize the 5 point ICDR grades: "None",
program partners (see https://www.adcis.nr http://wwww..diiiieeee///ttiiiaa
"Mild DR", "Moderate DR", "Severe DR", and "PDR". Split: We use a training set of 1,394 records, a test set of 350 records, and a total size of 1,744 records. Access restrictions: The dataset is available to download from brain/meesidoor--ddggddes..
19

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
License Licenses: Images under this dataset are available in Creative Commons
Public Domain
https:/creativecommnn...gg hwwpm//mmmn...oooooooooooooa/ooo
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
20. APTOS 2019 (Asia Pacific Tele-Ophthalmology Society, 2019) is a dataset from the 4th Asia Pacific Tele- Ophthalmology Society Symposium, collected from rural India. The dataset consists of fundus images under variety of imaging conditions. Our evaluation focuses the 5 diabetic retinopathy ratings: "No DR", "Mild", "Moderate", "Severe", and "Proliferative DR".
Split: We use a training set of 2,929 records, a test set of 733 records, and total size of 3,662 records.
Access restrictions: The dataset is available to download from
binndnes--eetectioon/aaa
Licenses:
Images under this dataset are available under the Kaggle Competition Rules cwwp///mpppiiioooo/ooo//0----ii- wew.iii---biiinnsddeeeeeeee wttss//deecccciouses////-oeeeeee
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
21. Jichi (Takahashi et al., 2017) is a posterior pole fundus photography dataset collected at the Jichi Medical University in Japan, covering a total of 2740 patients. We evaluate the models based on the David Grading for each image: SDR (simple diabetic retinopathy), PPDR (pre-proliferative diabetic retinopathy), and PDR (proliferative diabetic retinopathy).
Split: We training set of 7,950 records, test set of 1,989 records, and total size of 9.939 records. Access restrictions: The dataset is available to download from https://pmc.ncbbbb www.:nnnciiniiiii........ wwwp//rricees/MC54880088//////// Licenses: Images under this dataset are available under the Creative Commons Attribution 4.0 International License.
https:/creativeoommos..rggg wttp:/aawweooooooooooooo////ee/e
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
22. LNDb (Pedrosa et al., 2023) is lung cancer CT scan dataset collected the Centro Hispitalar Universitario de Sao Joao in Portugal between 2016 and 2018. Our evaluation focuses on the pulmonary nodule labels created by radiologists, including "nodule 33mm", "nodule <3mm", and "non-nodule".
We use a training set of 4,130 records, a test set of 1,431 records, and a total size of 5,561 records. Access restrictions: The dataset is available to download from https://zeoddoorrrrreord//////
Split:
Licenses: Images under this dataset are available under the Creative Commons Attribution 4.0 International License.
https//crreatvvccommmns http///ciwwccccoooooooooo//iic//
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
23. INSPECT (Huang et al., 2023) is multi-modal dataset containing CT images, radiology report impression sections, and structured electronic health records (EHR) from 19,438 patients. We focus on the pulmonary embolism (PE) labels which include "No PE". "Acute Subsegmental-only PE", "Acute PE", "Subsegmennallonlly PE", and "Chronic PE". Split:
We a training set of 17,434 records, test of 5,806 records, and total size of 23,240 records. Access restrictions: The dataset is available to download from c4b6-4006 1044400669566---85 9856-6f48ba40ad66 with registration.
Licenses: This dataset is available under the Stanford University Dataset Research Use Agreement
https//ssanffrdaimi..oo wttp:rdaaiiii..ueewbbssse...
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
24. KiTS23 (Heller et al., 2023) is dataset from the 2023 Kidney Tumor Segmentation Challenge. The dataset consists of CT vidcos showing kidney tumors. Although the data contains metrics and labels for segmentation tasks, we evaluate the models based on the "Benign" and "Malignant" key of each patient.
Split:
We use a training set of 361 records, a test set of 117 records, and a total size of 478 records. Access restrictions: The dataset is available to download from
Licenses: This dataset is available in the Creative Commons License https://creativecoomonn htw.///mmoooooog/i/ieeeee/////// http://ieennesyyn--caaaa44
4.0 International
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
25. Hemorrhage (Hssayeni et al., 2020) consists of Intracranial hemorrhage CT images for 82 patients at Al Hilla Teaching Hospital, Iraq, each with brain and bone window images and approximately 30 image slices in total. We evaluate the models the diagnosis labels "No Hemorrhage" and "Has Hemorrhage". Split: We use training set of 1,986 records, a test set of 515 patient records, and total size of 2,501 records.
20

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Access restrictions: The dataset is available to download from http:s//www....ggkooooomacccsoos tomography-ct-imag)
Licenses: This dataset is available in the Creative Commons Attribution 4.0 International Public License
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
26. RSPECT (Colak al., 2021) consists of CT scans for patients from five different countries suspected of Pulmonary Embolism (PE), created by the Radiological Society of North America (RSNA) and the Society of Thoracic Radiology (STR). We evaluate the models on the diagnosis labels "No PE", "Chronic PE", and "Acute PE", which are annotated by expert thoracic radiologists.
Split: We use a training set of 1,342,945 records, a test set of 447,649 records, and a total size of 1,790,594 records.
Access restrictions: The dataset is available to download from http://www..aaggge.ccc//ooooooos embolism-dettecioonnan.
Licenses: This dataset is available in under the Kaggle competition rules https://www.kggttt
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
27. EchoNet-Dynamic (Ouyang et al., 2020) consists of 10,030 apical-4-chamee echocardiography videos from patients who underwent imaging between 2016 and 2018 part of routine clinical care Stanford University Hospital. Each video comes with two pairs of human tracings used to estimate ventricular volume-the the first pair representing the left ventricle, and subsequent coordinate pairs representing short axis linear distances starting from the apex of the heart to the mitral apparatus.
Split: We use a training set of 8, 196 records, a test of 2,051 records, and a total size of 10,247 records. Access restrictions: The dataset is available to download from Oa with registration.
9217-4268-9daa-d355 daae-adaaadd999955
Licenses: This dataset is available under the Stanford University Dataset Research Use Agreement 177-4468--daa-d33555
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
28. BUSI (Al-Dhabyani et al., 2020) a breast cancer ultrasound image dataset from 600 femaile patients between and
75 years old in 2018. We utilize the labels "Normal", "Mallgnant", and "Benign". Split:
We use a training set of 583 records,
of 197 records, and total size of 780 records.
Access restrictions: The dataset is available to download from https:/www..kaggee.com hww.ee/aettttttttttaaaaaaaaaaaaa
titrasound-images-tttaee
Licenses: This dataset is available in the Creative Commons
Public
Domain
License
https:/creativecommnnss
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
29. COVID-BLUES (Wie, 2021) consists of bluepoint-speciffc lung ultrasound videos for 63 patients at the Maastricht University Medical Center in the Netherlands, each with 6 recordings. Our evaluation focuses on two labels: the diagnostic label ("Has COVID", "No COVID"), and the patient age label.
Split: We use a training set of 266 records, a test set of 96 records, and a total size of 362 records.
Access restrictions: The dataset is available to download from https///iittbbboommiiiaaaiccooDD BLUES?tab=readme-- -v-iiee
Licenses: This dataset is available in the Creative Commons Attihuro--Manooomeecaaaaaaeeaila 4.0 Interna- tional License https://creativecoommoss http:///ccoooooooooooorc/ceee/// nnnd////0
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
30. COVID-US (Ebadi et al., 2022) consists of 150 COVID-related lung ultrasound videos. We evaluate the models based on the diagnostic labels: "Covid", "Pneumonia", and "Normal". Split: We use training set of 74 records, a test set of 25 records, and a total size of 99 records.
Access restrictions: The dataset is available to download from https///gihubc.ooo///cccc//C/C/ Licenses: This dataset is available in the GNU Affero General Public License 3.0 http://wwwww.oggriicssssee/.ell 3.0.en.html
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
31. Brain Tumor (Bhuvaji et al., 2020) consists of brain MRI images. Each image is labeled either "No Tumor",
21

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
"Pimiitar Tumor", "Glioma Tumor", "Meningioma Tumor".
Split: We training set of 2,870 records, test set of 394 records, and total size of 3,264 records.
Access restrictions: The dataset is available to download from tataasifiiatiiommiiseeeeeee
Licenses: This dataset is available in the MIT License https:/www.mitt.edu amini/LICENNE..n
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
32. Brain MRI is brain MRI image dataset where each image is labeled with the presence of tumors, either "Yes" "No".
Split: We use a training set of 202 records, test set of 51 records, and total size of 253 records.
Access restrictions: The dataset is available to download from http::/www...gggg..ooooaneeteppp mri-
mmages-for-brain-tmorr
Licenses: This dataset does not have a license.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
33. ABCD (Cui et al., 2022) is a study supported by the NIH on adolescent brain cognitive development on nearly 12,000 youths of ages 9-10, who were studied for 10 years. The dataset contains MRI images, behavioral and cogmitive assessments, mental health, and other environmental factors data, with labels such as mental health diagnosis. In our dataset, do mental health diagnosis that classifies the sample into normal and abnormal.
Split: The dataset has total of 9563 records. We randomly split the dataset into train set of 7650 and test dataset of 1613 records.
Access restrictions: The dataset is available to download from Link. Licenses:
The license is available at Link.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
34. ABIDE (Cui al., 2022) is autism brain MRI diagnosis dataset with 1112 12 samples, including 539 from individuals with ASD and 573 from typical controls.
Split: We use random split to build a training set of 807 records, a test set of 202 records, and a total size of 1009 records.
Access restrictions:
with Licenses: The dataset is available in the Creative Commons AttrbationnNonCCCmmeraalllllll Alike License https://creativecommon.cck
The dataset available to download from account registration.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
35. PPMI (Cui et al., 2022) is a multi-center, longitudinal study dedicated to understanding the progression of Parkinson's disease. This data is derived from cohort of 1694 subjects, broken down into 309 controls and 1385 PD patients. Split:
After curation, size of 718 records.
random split to build training set of 572 records, test set of 143 records, and total
Access restrictions:
The dataset is available to download from Link. The license is available in the Link.
Licenses:
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
36. PROTEINS (Borgwardt et al., 2005) consists of 1,113 graphs where the nodes represent amino acids, and two nodes are connected by an edge if they are less than 6 Angstroms apart. We evaluate our models the binary classification labels of whether protein is an enzyme or not.
Split: We use random split to build a training set of 890 records, a test set of 223 records, and a total size of 1,113 records.
Access restrictions:
The dataset is available to download from http:///www...eeeeecccc////////s The dataset does not have a license.
Licenses:
Ethical considerations:
No personally identifiable information or offensive content is present in the dataset.
37. PPI (Stark et al., 2006) is a protein dataset from BioGRID covering physical and genetic interaction of proteins.
Split:
We use a training set of 45555 records, a test set of 11 11389 records, and total size of 56944 records.
Access restrictions: The dataset is available to download from Licenses: The dataset is available in the MIT License
httpapptttttooo..c//aaappppppppa https://biogrid-
22

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
38. LC25000 (Borkowski et al., 2019) is lung and colon histopathological image dataset containg labels for colon adenocareinomas, benign colon, lung adenocarcinomas, lung squamous cell carcinomas, and benign lung. We use a training set of 15,000 records, a test set of 3,750 records, and a total size of 18,750 records.
Split:
Access restrictions: The dataset is available to download from Licenses: The dataset does not have license.
colon image set.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
39. BCSS contains 151 breast cancer slides from 25 participants. We curate 5264 non-overlapping samples, with labels tumor, stroma, lymphocytic infiltrate, and necrosis/debris
Split:
We use a training set of 4211 records, test set of 1053 records, and total size of 5264 records.
Access restrictions: The dataset is available to download from Link.
Licenses: This dataset is licensed under a CCO 1.0 Universal (CCO 1.0) license.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
40. Cholec80 (Twinanda et al., 2016) consists of 80 videos of cholecystectomy surgeries performed by 13 surgeons. We evaluate the models based on the surgery phase annotations (at 25 fps) and the surgery tool labels (at I fps).
Split: We use a training set of 5,760 records, a test set of 1,440, and total size of 7 7,200 records.
Access restrictions: The dataset is available to download from http://camma..-sttaeee www.ssttt....attaasseseeetsteete through request form.
Licenses: This dataset is available in the Creative Commons haabbuutionnoonncomrriiiaaa-----
4.0 International
License https:/ccreaivveoommmms http://iwooooooooooooinnnnnnnnaa wtwpo/bb-nnnaa//.A////llel.eeeee
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
41. HuGaDB (Chereshnev & Kertész-Farkas, 2018) is a dataset for human gait analysis collected from 18 healthy young adults using six wearable inertial sensors and two EMG sensors. It contains labels "sitting", "standing", "sitting down", and "standing up". However, due to the limitation of the number of samples, this gait dataset is not included in the experiments.
of 73 records, and total size of 364 records. Access restrictions: The dataset is available to download from http:///thuub.cooo/ommameeeeeeee Licenses: The dataset does not have a license.
Split: We use a training set of 291 records,
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
42. Expression Atlas (Papatheodorou et al., 2018) consists of RNA gene expression data across species and biological conditions.
Split: We use training set of 3605 records, test set of 901 records, and total size of 4,506 records. Access restrictions: The dataset is available to download from http://www.bbbb.....//////ddwwoo Licenses: This dataset is available in the Creative Commons Taabbuooooooroommmcciii-oooiiiio tional License
4.0 Interna-
httprraawwwccooooooooonnnneeeeee nncnn/4.0/.
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
43. Geo is a functional genomics dataset supporting MIAME-compliant data submissions. Split: We use a training set of 101 162 records, test set of 25290 records, and a total size of 126,452 records. Access restrictions: The dataset is available to download from http://www.ncbb.ll...n www..ii..ooogee.... Licenses: The license is available at Link.
Ethical considerations:
No personally identifiable information or offensive content is present in the dataset.
44. Vital (Cui et al., 2024) is a medical image-language dataset based on PMC-15, where instructional data is generated using the gpt-4-vision-previee
API.
Split: We use a training set of 42000 records, a test set 168000 records, and a total size of 210,000 records. Access restrictions: The dataset is available to download from
VITAL-instructions:.
Licenses: The dataset is available in Apache License 2.0 https///wwwaaapcpccooooccccccscs wtt.//cccccce////CCEE--2000
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
45. MIMIC-IV (Johnson et al., 2023) (Johnson et al., 2016) (Goldberger et al., 2000) is multimodal medical dataset on patients admitted the emergency department or intesnive care unit the Beth Israel Deaconess Medical Center in Boston, MA. We train and evaluate our models using the EHR, vital sign, and chest-xray modalities.
23

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Split: CLIMB provide training of 640,000 records, test of 160,000 records, and total size of 800,000 records. In the fusion experiment, however, we focus the multimodal subset of MIMIC-IV, which contains 166,215 training samples, 41,554 test samples, and 207769 total samples.
Access restrictions: The dataset is available to download from http:///piissonntooooccooteeemmm with credentialed account and CITI training.
Licenses: The dataset http:///wyoooeeooooooo/ttttnnnnn
available in PhysioNet Credentialed
Health
Data
License
1.5.0
Ethical considerations: No personally identifiable information or offensive content is present in the dataset.
Dataset Selection Methodology
Our dataset collection and curation process follows systematic two-stage approach to ensure both comprehensive coverage and accessibility while maintaining data quality and diversity.
Stage I: Initial Dataset Identification
first conducted a comprehensive literature review of publicly available clinical datasets across different modalities. Our inclusion criteria at this stage focused on accessibility:
Datasets with direct download access through public repositories
Datasets requiring application but with clear, less restrictive licensing terms
Datasets with well-documented data collection protocols and annotation procedures
We explicitly excluded datasets that:
Require institutional review board (IRB) approval
Have restrictive licensing terms limiting research use
Stage 2: Selection and Prioritization
In the second stage, we applied a two-tier selection process:
Tier 1: We first included widely-cited benchmark datasets that serve as standard evaluation metries in their respective domains. These datasets were identified based citation count and frequency of in published literature. Examples include CheXpert and MIMIC-IV.
Tier 2: We then systematically identified and included datasets that addressed our three key criteria:
Novel tasks:
Datasets covering emerging clinical challenges (e.g., COVID-19 diagnosis)
Understudied modalities: Datasets from underrepresented data types (e.g., EEG, endoscopic videos)
Underrepresented regions: Datasets from developing regions with limited representation
Selection Criteria
Here, elaborate methodology for identifying understudied modalities and underrepresenteed regions during the stage 2 selection.
Understudied Modalities.
We evaluated modalities from two complementary perspectives:
1. Research Attention: We quantified research activity by aggregating Google Scholar search results using standardized queries (e.g.. '[Modaity]] classification', '[Modalit]] machine learning'). Our analysis revealed significant disparities
in research attention across modalities:
High attention (> 1M articles): Pathology (1.38M), X-ray (1.08M), CT Scan (2.83M), Endoscopic (1.74M)
Medium attention (300K-1M): MRI (931K), Dermoscopy (530K), Ultrasound (636K), Fundus (848K), ERG (450K)
Low attention (<500K): Mammography (56. ECG (265K)
2. Data Availability: We analyzed the total number of publicly available samples per modality:
High availability (>500K): X-ray (595,264), CT Scan (1,810,256). EEG (655,786)
24

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Medium availability (50K-500K): ECG (84,172), Genomic (130,958)
Low availability (5K-50K): Mammography (24,697), Dermoscopy (45,439), Fundus (18,067), Endoscopic
(14,400). Pathology (24.014), MRI (14,807)
Very low availability (<5k): Ultrasound (1,633)
Based on this analysis, we identified several critically understudied modalities. Mammography and ECG emerged as understudied based on research attention, while ultrasound was identified as understudied due to extremely limited public data availability (1,633 samples). These findings guided our focused efforts to collect additional datasets in these modalities.
Underrepresented Regions. We also employed two-step approach identify geographic gaps dataset coverage:
1. Geographic Distribution: We created global heatmap of dataset origins, revealing significant underrepresenttaionn in:
Africa
South America
Parts of South and Southeast Asia
2. Economic Development: We mapped datasets to their countries of origin, specifically identifying datasets from developing nations. This analysis highlighted the importance of including datasets from:
Asia: India, Vietnam
Middle East: Iraq
South America: Brazil
This analysis informed our targeted efforts to include datasets from these underrepresented regions, aiming to improve the geographic and demographic diversity of our benchmark. The complete list of included datasets and their geographic distribution is provided in Table X.
Data Preprocessing and Standardization To preserve data fidelity while ensuring usability, implemented minimal preprocessing steps:
Time Series Data (ECG/EEG):
Standardized sampling rates across datasets
Normalized amplitude ranges
Preserved original waveform characteristics
Imaging Data:
Maintained original image resolution and quality
Created standardized JSON metadata files linking:
Clinical labels
Demographic information
Multi-view relationships
Additional annotations (where available)
No preprocessing was done on graph data, as they are already processed and ready use.
All additional metadata and multi-view images are preserved and made available, though our benchmark experiments utilize only the primary labels to ensure fair comparison across models. The complete preprocessing scripts and documentation are available in our code repository.
B. CLIMB-QA Construction
To enable standardized evaluation of large vision-language models (VLMs), we construct CLIMB-QA, a question-answering version of our dataset. For each sample x; E D& with label set yı C V&, we generate a question-answer pair (4, a,) where:
ai is constructed as a natural language question incorporating:
25

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Task description (e.g.. "grade the diabetic retinopathy")
Input modality context (e.g., "in the retinal image")
Available choices from the label vocabulary Vi
03 the ground truth answer derived from y/, formatted as either:
Single-label: ag Vi for mutually exclusive classes
Multi-label: a; Vi for compatible conditions
Formally, define mapping function & (zt, Va. V&.) - (9:; a:) that generates question-anse answer pairs while preserving the original classification task structure. For example, in the APTOS dataset for diabetic retinopathy grading:
qi = " "Above is retinal image patient. Grade the diabetic retinopathy on the Davis Scale, choosing from: No DR, Mild DR, Moderate DR, Severe DR, Proliferative DR."
a, "Moderate DR"
For multi-label classification tasks, we evaluate predictions using order-agnostic matching: given a predicted answer set à. and ground truth aę, consider the prediction correct if à; ag regardless of the order in which the labels are listed. This ensures fair evaluation when multiple conditions are present and can be enumerated in any order.
C. Taxonomy Standardization
The standardization of label taxonomies across diverse medical imaging modalities represents significant methodological challenge. Our standardization efforts required careful balancing of two competing objectives:
1. Merging semantically similar terms to facilitate effective learning and enable cross-modality knowledge transfer
2. Minimizing information loss and avoiding the introduction of inaccuracies when modifying existing labels
Our standardization efforts concentrated particularly on ECG and chest X-ray modalities, as these domains offer fine- linegrained labels with highly variable terminologies across different datasets. For ECG standardization, we adopted the methodology established in BenchMD (Wantlin al., 2023).
For chest X-ray standardization, we developed novel mapping framework with input from radiologists, building upon established practices from recent literature (Nasser & Akhloufi, 2023; Tayebi Arastch et al., 2023; Jang et al., 2024). We consolidated all general chest X-ray labels into the CheXpert 14 categories, as shown in Table 11:
We acknowledge that this standardization process necessarily affected label gramularity, particularly for the lung opacity and lung lesion classes where multiple distinct pathologies were consolidated. To accommodate diverse research needs, CLIMB includes both standardized and raw labels, enabling researchers to prioritize either cross-dataset consistency or fine-grained granularity based on their specific requirements. All experiments reported in our main manuscript utilized the standardized labels.
C.1. Empirical Evaluation of Standardization Impact
To quantitatively assess the effects of label standardization, we conducted comparative evaluation using identical vision encoders trained raw versus standardized labels. The results, presented in Table 12, demonstrate the impact across multiple imaging modalities:
The standardized labels yielded superior overall performance with 3.6% improvement in AUC, 10.3% improvement in sensitivity, and 4.0% improvement in specificity. Interestingly, the benefits of standardization were more pronounced in modalities other than chest X-ray, despite chest X-ray being the modality where most relabeling occurred. This suggests that standardized labels enable the model to more effectively establish conceptual connections across different imaging modalities, compensating for the inevitable loss of some fine-graines distinctions. We believe these findings underscore the importance of continued efforts toward terminology standardization and systematic relabeling of public clinical imaging datasets to facilitate more effective multi-modal learning approaches.
26

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
D. Detailed Experimental Setup
D.1. Problem Definition
RQ1/RQ2: We use the definition of the dataset as well as the vocabularies as defined in Sec. 3.2. For each input sample D E X. which can be an image, video, or multi-channel time series, we extract n sequential elements, The input processing varies by modality:
1. Images: Vimg BBXXXX + InxoxaxC
2. Videos: Pvid BRXHXWXC + BaxaxoxC
3. Time Series: Oes IRTXX BOxC
where denotes the model-specific input size, and H, W, C, represent spatial dimensions, channels, and temporal length, respectively. The objective is to learn a function : + {0, 13 that maps each input to a binary vector over V.
RQ3: Given multimodal inputs M Muse, Miang Maa, where each modality input x;" E x corresponds to visual, language, and time series data respectively, the model aims to learn prediction funetions:
1. In-hospital Mortality in 48 hours (IHM 48): Binary prediction film mortality
IImEM Xmm + {0, 1 predicting 48-hour
2. Length of Stay: Regression function flos IImE xm + R+ estimating the expected duration of hospitalization.
D.2. Experimental Procedures
To answer the above research questions, we design our experiments as follows:
D.2.1. RQI: UNIVERSAL ENCODERS PERFORMANCE UNDER MULTITASK LEARNING.
We investigate how graph and time series encoders the complete 2> to assess general diagnostic capabilities. For each sample x; E DR. we evaluate performance using dataset-specific vocabulary mask 1 v. E {0. 1)1M) where only predictions corresponding to labels in V are considered in the evaluation metrics. Performance is measured using Area Under the ROC Curve (AUC) and binary classification metrics (specificity and sensitivity) with a decision threshold of 0.5 over the masked label space O lv.- In addition, we compare the results pre-trained on our entire datasets against those just pre- pre-trained on each target dataset, exploring the effect of large-scale pre-training on the model's performance.
build universal encoder for each input type across all clinical tasks. Specifically, train vision,
D.2.2. RQ2: TRANSFER LEARNING UNDER RESOURCE CONSTRAINTS.
Second, we evaluate few-shot generalization on out-of-distribution (OOD) datasets Dood & Dtrains where Dood contains novel label sets Vood such that Vood n V = / within the same modality. This setup simulates the practical scenario where models must adapt to novel diagnostic tasks with limited labeled examples while leveraging pre-trained representations from related but distinct tasks. A detailed description of the dataset composition, as well as the experimental procedure, are included in App. D.7.1.
D.2.3. RQ3: SINGLE MODALITY TO MULTIMODALITY TRANSFER VIA ROBUST FUSION STRATEGY. Finally, we investigate optimal fusion mechanisms for integrating heterogeneous modalities as defined in App. D.1. We evaluate three fusion architectures ge IIIEM Rdm Rdd
1. late fusion Piate that averages predictions from modality-speciffe classifiers: glate = 1
[MI ΣmEm MLP (hmm.
2. simple concatenation followed by two-layer MLP Imip (hmm) = MLP MLP(Concat((Awia Manng (t.l)),, and
3. cross-modal attention Jattn that uses text features form queries and concatenated vision/time--sries features for keys and values, computing Jatta (hm) = FFN(softmax/ QK"
Vd V) where Q htang Wq. K Concat([h Wr.
V CConcat(hvis Wy
For controlled comparison, we fix the backbone encoders across all fusion strategies: ConvNextv2 (Liu et al., 2022) for visual inputs, ClinicalBERT (Liu et al., 2025) for text, and ECG-JEPA (Kim, 2024) for time series data.
D.3. Evaluation Metrics
For binary classification tasks f (0, 1). we employ three complementary metrics to assess model performance:
27

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
1. Area Under the ROC Curve (AUC): Given predicted probabilities D, [0, and true labels yss E (0,1). AUC measures the model's ability to discriminate between classes across all possible decision thresholds:
AUC J.'
(PR()) (I)dI
where TPR(() and FPR(1) are the true positive and false positive rates at threshold t.
2. Sensitivity (also known as recall): Measures the model's ability to correctly identify positive cases:
TP
Sensitivity
TP + FN
where TP and FN denote true positives and false negatives respectively.
3. Specificity: Quantifies the model's 's ability to correctly identify negative cases:
TN TN FP
Specificity
where TN and FP denote true negatives and false positives respectively.
The choice of these three metrics is particularly motivated by clinical considerations. In medical diagnosis, there is often an inherent trade-off between sensitivity and specificity, where improving one typically comes at the cost of the other. Sensitivity is crucial in cases where missing positive diagnosis (false negative) could have severe consequences for patient outcomes, such as failing to detect a life-threettening condition. Conversely, specificity is vital when false positives could lead to unnecessary interventions, psychological distress, or resource waste.
AUC provides threshold-indesn -ndepeedeent measure of discriminative ability, making it particularly valuable for comparing models across different operating points and clinical contexts. This is especially relevant in our multi-task setting where different clinical applications may require different sensitivity-specifiity trade-offs.
For regression tasks such as length of stay prediction (flos), we use Mean Absolute Error (MAB):
MAE
ys
Dil
where y; and 9; represent the true and predicted values respectively. All metrics are computed using the dataset-specific vocabulary masks lv, as defined in Sec. D.I.
Below, we describe in detail about the setup and procedures of vision, time series and graph experiments.
D.4. Vision Model Experiments
D.4.1. VISION MODEL DETAILS
MedViT (Manzari et al., 2023) is Vision Transformer variant specifically designed for medical imaging tasks. It incorporates a hierarchical structure with varying token sizes across different stages and employs medical-specific attention mechanisms optimized for capturing fine-grained anatomical details.
PMC-CLIP (Lin al., 2023) adapts the CLIP architecture for medical imaging by pretraining on PubMed Central articles and their associated figures. It maintains the original dual-encoder structure but incorporates medical domain knowledge through specialized text-image contrastive learning.
RAD-DINO Pérez-García al., 2025) extends the DINO self-supervised learning framework to radiology images. employs specialized augmentation strategies and anatomical consistency constraints during the self-supervised pretraining process to better capture medical imaging characteristics.
SBB2 (Radford et al., 2021) is an enhanced vision backbone that builds upon the clip architecture. It introduces improved spatial mixing operations and hierarchical feature representations while maintaining computational efficiency.
28

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Swin Transformer (Liu et al., 2021) is hierarchical vision transformer that computes self-attention within shifted windows. It introduces hierarchical architecture with varying window sizes across different stages, enabling efficient modeling of both local and global dependencies.
EVA-2 (Fang et al., 2024) a large-scale vision foundation model that extends the original EVA architecture. It utilizes masked image modeling and contrastive learning objectives, incorporating improvements in model scaling and training strategies.
InternViT (Chen et al., 2024b) is a vision transformer model that introduces internalized attention mechanisms. It optimizes the traditional transformer architecture for improved efficiency while maintaining performance across diverse visual tasks.
V2 (Liu et al., 2022) is pure convolutional architecture that modernizes traditional CNN design principles. It incorporates fully convolutional design, global response normalization, and gradient checkpointing, achieving strong performance across various vision tasks.
D.4.2. VISION HYPERPARAMETER AND EXPERIMENTAL PROCEDURES All experiments are
GPU server with 8xH200 141GB GPUs. We used the SOAP optimizer (Vyas et al., 2024) as it offers the best performance. Depending the model sizes, rate from 1 x 10-5 to 1 x 10-3 for all experiments. The weight decay was set to 1 x 10-3 We use parameter search to find the largest GPU that could fit on a single server. All experiments were conducted using the PyTorch framework. We saved the model with the lowest CrossEntropy loss over 5 epochs for evaluation on the test split. We report the AUROC, Sensitivity, Specificity, F1 Score and accuracy of our experiments in App. E.
parameter search to identify the optimal learning
D.5. EEG Model Experiments
We investigate the performance of five baseline models and four variants of foundational time series models for EEG classifications. Our EEG experiment is built upon the repository sheared by (Yang et al., 2024), and the pre-trained weights were downloaded from https: /gthhtitthhh/////// http://ccccccyc////......
1. EEG DATASETS AND PREPROCESSING We evaluated the models on IIIC (Jing et al., 2023b), TUAB (Lopez et al., 2015), and TUEV (Lopez et al., 2016). All EEG channels, STi]. in each individual sample were resampled 200 Hz and normalized using the 95th percentile of the absolute amplitude:
S
percentile[[[Sji. 1]I. [S|i 2]
..... S [[1
For details dataset access and data splitting, please refer App. A.
D.5.2. EEG HYPERPARAMETERS AND EXPERIMENTAL PROCEDURES
We used the Adam optimizer (Diederik, 2014) with a learning rate of 1 x 10-3 3 for all experiments. The weight decay was set to 1 x 10-5, For most experiments, the batch size was set 512, while for few-shot experiments, was set to 4. All experiments were conducted using the PyTorch framework. We saved the model with the lowest Cross Entropy loss over 20 epochs for evaluation on the test split. We report the AUROC, Sensitivity, Specificity, and F1 Score of our experiments in App. E.
D.5.3. EEG MODEL DETAILS
SPaRCNet (Jing et al., 2023b) a ID-CNN designed for EEG classification. It employs a hierarchical feature extraction process, beginning with an initial convolutional layer followed by multiple densely connected blocks and transition layers. The model integrates ELU activation functions, batch normalization, and dropout for improved generalization.
(Peh et al., 2022) is a hybrid deep learning model that combines convolutional neural networks (CNNs) with Transformer encoder for EEG classification. The model first applies short-time Fourier transform (STFT) to extract spectral representations, which are then processed through deep residual CNN with four stacked ResBlocks for hierarchical feature extraction. The CNN embeddings are segmented and passed through Transformer encoder with positional encoding, allowing the model to capture long-ran rrange temporal dependencies.
CNNTransformer
ContraWR (Yang et al., 2023) is a EEG classification model that integrates STFT with a 2D-CNN for sleep staging. The model first converts raw EEG signals into spectrograms using STFT, which are then processed through a deep residual CNN
29

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
with four stacked ResBlocks, each employing batch normalization, dropout, and max pooling for feature extraction.
FFCL (Li et al., 2022) is a hybrid CNN-LSTM model designed for EEG classification, integrating both spectral and temporal feature representations. The model first apply STFT to extract frequency-domall features, which are then processed through a deep residual CNN with four stacked ResBlocks. using a downsampling operation before being fed into a bidirectional LSTM for sequential feature extraction. The final representation is obtained by concatenating CNN and LSTM embeddings, which are passed through fully connected classification layer.
parallel, the raw EEG signals undergo temporal compression
STrransformer (Song et al., 2021) is a spatiotemporal Transformer model for EEG classification that integrates channel- wise attention with Transformer-base sequence modeling- The model first applies a ChameelAtention mechanism to capture inter-channel dependencies, followed by a PatchSTEmbedding module that encodes local temporal structures. These embeddings then processed through a deep Transformer encoder, which models long-range dependencies and contextual information. The v2.0 of STTransformer is known as EEG-Conformer (Song et al., 2022).
BIOT (Yang et al., 2024) is biosignal transformer model designed for cross-data learning in the wild, enabling robust representation learning across diverse biosignal modalities such as EEG and ECG. The model utilizes frequency- tokenization approach, where biosignals are first transformed into spectrogram representations via STFT. Channel-specific positional embedding and temporal positional embedding are added to the tokens to enhance both temporal and spatial representations. These spectral embeddings are then processed using Linear Attention Transformer.
D.6. ECG Model Experiments
We compare the performance of ECG-JEPA, a time series specific model, and UniTS, a generalized time series model for ECG classifciations. Our ECG experiment and pretrained encoder weights were adopted from the following repositories: https: and https: : //ggt/gttttt htgpttutbbb/////////////// com/mims harvard/Uni7S.
//github. com/ www.///eofrooooooooooccccooooooo www.nfooommmgggeeeaeeoo
D. ECG DATASETS AND PREPROCESSING
We evaluated the models on PTB-XL (Wagner et al., 2020), CPSC (Liu et al., 2018) 8), Chapman-Shaoxing (Zheng et al., 2020), and Ga (Alday et al., 2020). All ECG signals were resampled to 500 Hz and standardized to a length of 2500 timesteps. Samples where the first 15 timesteps contained only zeros across all channels were removed. For ECG-JEPA, the number of ECG channels was reduced from 12 to 8, as the remaining 4 channels can be derived using linear combinations of the selected leads.
For details on dataset access and data splitting, please refer to App. A.
D.6.2. HYPERPARAMETERS AND EXPERIMENTAL PROCEDURES
For the ECG-JEPA model, we use the Adam optimizer (Diederik, 2014) with a learning rate of x 10-3 and weight decay of 1 10-2, 22 For the UniTS model, we use the Adam optimizer with learning rate of 1 x 10-4 and weight decay of 5 x 10-6, For all experiments, the batch size set to 32. All experiments were conducted using the PyTorch framework. We report the AUCROC, Sensitivity, and Specificity Score of our experiments in App. E.
D.6.3. ECG MODEL. DETAILS
ECG-JEPA (Kim, 2024) is a self-supervised ECG representation learning model that predicts in the latent space rather than Attention (CroPA), masked attention mechanism that prioritizes critical ECG features across multiple leads, enhancing performance downstream tasks.
reconstructing raw signals. It introduces
UnITS (Gao et al., 2024) is a unified multitask time series model that integrates predictive and generative tasks using task tokenization within a single framework. It employs a modified transformer block to learn transferable time series representations across diverse domains, handling variations in sampling rates and temporal patterns.
D.7. Out-of-distribution Experiment Details
In this section, we describe the details on how ran the OOD transfer experiment, which addresses RQ2. We first define the list of datasets we selected for the transfer experiments, and then explain the detailed procedure of the experiment.
D.7.1. DEFINITION OF OUT-OF-DISTRIBIIT
DATASETS
The list of datasets we selected for -istriibuitonooi (OOD) transfer experiments is described in Table 13. general, we select the datasets such that they reflect different task within modality it was trained on. The OOD may be a novel task
30

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
(like COVID-19), new task (cancer pulmonary embolism), or different granularity of the same task (6-way BI-RADS classification instead of 5-way).
D.7.2. EXPERIMENTAL PROCEDURES
Vision Encoders.
For vision encoders, we run a mixed training on the full dataset, with the OOD dataset filtered out.
E. Full Experimental Results
E.1. Full Dataset Multitask Training Results In this section, we report the detailed model's performance for each dataset in Table 3.
F. Validation of Out-of-Distribution Dataset Selection
We provide additional empirical validation of the distinctiveness of out-of-distribttion (OOD) datasets. Our OOD dataset selection was primarily guided by fundamental task differences rather than superficial variations. As detailed in Table 13, we assembled 10 OOD datasets spanning diverse clinical domains and imaging modalities. Notably, 7 out of these 10 datasets (COVID-19, 19, CoronaHack, ISIC-2020, BCSS, BUSI, LNDb, and -TB-eXl-Finegrained) share zero label overlap with any other datasets within their respective clinical domains, while the remaining datasets exhibit distinct task granularities that fundamentally alter the nature of the prediction problem.
To quantitatively validate the heterogeneity of our designated OOD datasets, we conducted comprehensive membership inference experiment. Using ConvNeXT-v2-bass as the backbone architecture, we trained a classifier to predict dataset membership essentialll testing whether the model could distinguish which dataset a given sample originated from. High performance on this task would indicate that the datasets possess distinct characteristics that make them easily distinguishable, thus supporting our designation of these datasets as distrii-ditiiiiinonn
The results, presented in Table 32, demonstrate that our selected OOD datasets exhibit substantial distinctiveness:
The consistently high balanced accuracy and AUC scores across most datasets confirm that these datasets possess unique characteristics that make them readily distinguishable from one another. This empirical evidence strongly supports methodological choice to designate these datasets out-of-distribution, they represent genuinely distinct data distributions rather than minor variations of the same underlying distribution.
G. Exploration on Self-Supervised Learning
Self-supervised pretraining represents fundamental component of modern representation learning. We conducted compre- hensive experiments comparing unsupervised pretraining with supervised multitask learning across different modalities to understand their relative contributions to model performance.
G.1. Time Series Modalities: ECG and EEG
Our experiments revealed that time series modalities benefited substantially from unsupervised pretraining. In the ECG domain, we implemented masked autoencoder (MAE) pretraining on the CLIMB dataset and compared it with pretraining exclusively on the target dataset. As shown in Table 33, pretraining on the diverse CLIMB dataset yielded consistent improvements:
Similarly, for EEG datasets, we compared three training strategies: no pretraining, multitask learning (MTL) only, and combined pretraining with multitask learning (PT+MTI.). The results in Table 34 show that pretraining on diverse data improved performance for two out of three datasets:
G.2. Vision Modalities
While our main manuscript (Figure 4) demonstrated that multitask learning effectively improved vision encoder perfor- mance, we found surprisingly different results for self-supervised pretraining in vision modalities. Table 35 presents our comprehensive comparison of different pretraining strategies:
We hypothesize that this phenomenon occurs because these vision models have already undergone extensive pretraining on massive unlabeled natural image corpora. Consequently, an additional phase masked image modeling or contrastive learning on clinical data may not substantially shift enrich their learned representations. This finding suggests that for vision models with strong natural image pretraining, supervised multitask learning on diverse labeled clinical data provides
31

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
more effective domain adaptation than additional self-supervised pretraining phases. We encourage the community explore alternative approaches for better leveraging the diverse labeled data available in CLIMB further improve vision model performance in clinical domains.
32

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 9. Dataset Classes for Multilabel Classification with CLIMB.
Dataset
Classes
Classes
PTB-XL
Normal, Conduction Delay (CD), Hypertrophy (HYP), Myocardial Infarction (MJ), Sinus Tachy-
candia/BrudycarcdaaaCCnccti (STTC), Atrial Fihrillation/Ani Flutter (A, Fib/Aflutter), Other
Chapman-Shaoxinng
Same as PTB-XI.
Georgia
Same PTB-XI.
CPSC
Same PTB-XL
IIIC
Seizure (SZ), Lateralized Periodic Discharges (LPD), Generalized Periodic Discharges (GPD), Lateralized Rhythmic Delta Activity (LRDA), Generalized Rhythmic Delta Activity (GRDA),
Other
TUAB
Normal, Abnormal
TUEV
Spike and Slow Wave (SPSW), Generalized Periodic Epileptiform Discharge (GPED), Periodic Lateralized Epileptiform Discharge (PLED), Eye Movement (EYEM), Artifact (ARTF), Back- ground (BCKG)
CheXpert
14
Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinmm Fracture, Lung Lesion, Lung Opacity, Pleural Effusion, Pneumonia, Pneumothorax, Pleural Other, Support Devices, No Finding
MIMIC-CXR
14 Same as CheXpert
VinDr-CXR
Lung tumor, Pneumonia, Tuberculosis, COPD, Other diseases, No finding Normal, Bacterial Pneumonia, COVID-19, Viral Pneumonia Normal, Bacterial Pncumonia, Viral Pncumonia
COVID-19
4
CoronaHack
Mammr
BI-RAD 1-5
CBIS-DDSM
BI-RAD 0-5
CMMD
Benign, Malignant
ISIC-2020
Malignant, Benign
HAM10000
Mclanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Keraaosis/Intrrepithelial
Carcinoma (AKIEC), Other (OTHER)
PAD-UFES-20
Melanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Keratosis/Intraepiheeiaal Carcinoma (AKTEC), Other (OTHER)
Messidor-2
None, Mild DR, Moderate DR, Severe DR, PDR
APTOS 2019
No DR, Mild, Moderate, Severe, Proliferative DR SDR (simple diabetic retinopathy). PPDR (pre-proliferatiie diabetic retinopathy). PDR (prolifer- ative diabetic retinopathy)
Jichi
LNDb
nodule 2 3mm. nodule ;3mm, non-nodule
INSPECT
No PE, Acute Subsegmental-ong PE, Acute PE, Subsegmental-only PE, Chronic PE Benign, Malignant
KiTS23
Hemorrhage
No Hemorrhage, Has Hemorrhage
RSPECT
No PE, Chronic PE, Acute PE
EchoNet-Dynamic
Not classification Normal, Malignant, Benign Has COVID, No COVID Covid, Pneumonia, Normal
BUSI
COVID-BLUU
COVID-US
Brain Tumor
No Tumor, Pituitary Tumor, Glioma Tumor, Meningioma Tumor Yes, No (presence of tumors)
Brain MRT
ABCD
Normal, Abnormal
ABIDE
ASD, Typical controls
PPMI
Control, PD patients
PROTEINS
Enzyme, Not enzyme
PPI
LC25000
Colon adenocarcinomas, Benign colon, Lung adenocarcinomas, Lung squamous cell carcinomas, Benign lung
BCSS
Tumor, Stroma, Lymphocytic infiltrate, Necrosis/debris
Cholec80
Surgery phase annotations and surgery tool labels Sitting, Standing, Sitting down, Standing up Not classification
HuGaDB
Expression Atlas
Geo
Not classification
Vital
Not classification
MIMIC-IV
Hour In Hostital-Mortality Mortality (48 IHM) (Yes/No)
33

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table Dataset Demographics and Location Information.
Dataset
Locations
Demographic Information
PTB-XL
Multiple
sex: 52% male, 48% female; age range: 0-95 (median:
62. IQR: 22); height; weight
Chapman-Shaoxing
Shaoxing. Zhejiang, China
sex: male: 22.599 (56%): female: 17,659 (44%))
age groups: 51-60 (19.8%), 61-70 (24%), 71-80 (17.3%)
Georgia
Emory University, Atlanta, Georgia, USA
age; SeX unknown unknown
CPSC
China
MIMIC-CXR
Beth Israel Deaconess Medical Center in Boston, MA
CheXpert
Stanford, California, US The Hospital 108 and the Hanoi Medical University Hospital in Vietnam
Sex; age
VinDr-CXR
Training set: median age: 43.77; sex: 52.21% male, 47.79% female; Test set: median age: 31.80; sex: 55.90% male, 44.10% female
VinDr-Mammo
The Institutional Review Board of Hanoi Medical Uni- versity Hospital (HMUH) and Hospital 108 (H108) Stanford, California, US
age; imaging device's model
CBIS-DDSM
Unknown
ISIC-2020
Hospital Clinic de Barcelona, Medical University of Vienna, Memorial Sloan Kettering Cancer Center, Melanoma Institute Australia, University of Queens- land, and the University of Athens Medical School unknown
sex: female: 15981 (48%), male: 17080 (52%); age range: 0-90 (median: 48.87)
HAM10000
unknown
PAD-UFES-20
Federal University Espírito Santo (UFES), Espírito Santo, Brazil
country parents; age: gender; access to piped water; access to sewage system; region
Messidor-2
Brest University Hospital
unknown
APTOS 2019
Aravind Eye Hospital, India
unknown unknown
Jinchi
Jichi Medical University
LNDb
Centro Hospitalar e Universitário de São João (CHUSI) in Porto, Portugal
unknown
ABIDE
California Institute of Technology, Carnegic Mcllon
age; handedness; full-scalc IQ
University, Kennedy Krieger Institute, and more
ABCD
unknown
gender identity; environmental factors sex: 54.5% male, 45.5% female; race; age
PPMI
Multiple
HuGaDB
unknown
sex: 4 females, 14 males; age: average: 23.67; height:
average: 179.06 em; weight: average: 73.44 kg
INSPECT
Stanford Medicine (2000-2021)
gender: female: 10,733, male: 8,666, unknown: age:
18-39: 2,912. 39-69: 9,974, 69-89: 5,859, :89: 657;
race: white: 10,704, asian: 2.976, black: 1,103, native:
415, unknown: 2,404; ethnicity: hispanic: 3,018, not
hispanic: 15,628, unknown: 756
EchoNet-Dynamic BUSI
Stanford University
unknown
unknown
unknown
COVID-19
University of Montreal, Canada Eindhoven University of Technology, Netherlands Minnesota, US
unknown
Brain Tumor
unknown
KiTS23
unknown
Hemorrhage
AI Hilla Teaching Hospital, Iraq
age: mean: 27.8; gender: male: 46, female: 36
CMMD
China
age
CoronaHack
Mila, University of Montreal
unknown
COVID-BLUES
Maastricht University Medical Center (UMC+) in the Netherlands
weight; sex; height; bmi; age
COVID-US
unknown
gender; age; alcoholic; drug use
IT
Massachusetts General Hospital, Harvard Medical
1950 patients; labeled by 124 raters; 20 of the are physician experts
School, Boston, USA
TUAB
The Temple University Hospital, Philadelphia, Penn- sylvania. USA
Evaluation Dataset: Total: 276 files, 253 subjects; Ab- normal Female: 63 files. 51 subjects; Abnormal Male: 63 files. 54 subjects; Normal Female: 85 files. 84 sub-
jects; Normal Male: files. subjects; Train Dataset: Total: 2,717 files, 2,130 subjects
TUEV
The Temple University Hospital, Philadelphia, Penn- sylvania, USA Beth Israel Deaconess Medical Center, Massachusetts Institute of Technology
290 patients in train; 78 patients in eval
MIMIC-IV
age; gender; insurance type (modicaid, medicare, other)
34

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table
Chest X-ray label standardization mapping
Raw Label
Standardized Label
Aortic enlargement, Enlarged PA
Enlarged Cardiomediastinum
Cardiomegaly
Cardiomegaly
Atelectasis
Atelectasis
Consolidation
Consolidation
Edema
Edema
Infiltration, Lung Opacity, ILD, Pulmonary fibrosis
Lung Opacity
Nodule/Mass, Other lesion, Lung cavity, Lung cyst, Lung tumor
Lung Lesion
Pleural effusson
Pleural Effusion
Pleural thickening
Pleural Other
Pneumothorax
Pncumothorax
Rib fracture, Clavicle fracture
Fracture
No finding
No Finding
Support Devices
Support Devices
Pneumonia
Pneumonia
Table 12, Performance comparison between and standardized label training
CXR
Marmo
Demm
CT
Fundus Sen
US
Overall
Model
Spc
Spc
Spe
Cony NomnNNetV2-RRoooe Label
820
.388
9935
.543
.203
6933
8833
.492
.757
.690
.442
6674
.794
.351
.811
.699
.519
.713
.71
.344
.766
ComoNextV22 StanaV2-Saaddrdd..aab
.436
.939
.330
.706
.901
.568
.671
.641
.873
.563
.888
.774
.641
.770
.787
.537
.806
35

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 13. Out-of-Distribution Datasets
Dataset
Classes
Outtof-Distribution Characteristics
COVID-19
Normal, Bacterial Pneumonia, COVID-19, Viral Pneumonia
Novel disease class (COVID-19)
CoronaHack
Normal, Bacterial Pneumonia, Viral Pneumonia, COVID-19
Novel disease class (COVID- 19))
CBIS-DDSM
BI-RAD 0-5
More fine-grained classification (6 BI-RADS categories) com-
pared to VinDr-Mamma (5 categories) and CMMD (binary
classification)
ISIC-2020
Malignant, Benign
Different task granularity (binary classification)
Jichi
SDR, PPDR, PDR
Different classification scheme for diabetic retinopathy pro- gression compared to Messidor-2 and APTOS 2019's five- stage classification
BCSS
Tumor, Stroma, Lymphocytic infiltrate, Necro- sis/debris
Different task type (tissue component classification) compared LC25000's focus cancer type classification
BUSI
Normal, Malignant, Benign
Different task focus (breast lesion classification) compared other ultrasound datasets (COVID-BLL BLUES, COVID- US) which focus on lung pathology
LNDb
nodule > 3mm, nodule 3mm, non-nodule
Different task focus (nodule size classification) compared to other datasets like INSPECT and RSPECT which focus on pulmonary embolism
XXL-Fnnnggaieed
Normal ECG (NORM). Ischemic inferior leads (ISCI), Non-specific ST changes (NST.). Ischemic in anterior Icads (ISCA), Non-specific ischemie (ISC.), ST-T changes (STTC), Right ventricular hypertrophy (RVH), Right atrial over- load/enlargement (RAO/RAE). Septal hyper- trophy (SEHYP), Left atrial overload/enlarge- ment (LAO//LAE), Anterior myocardial infare- tion (AMI), Inferior myocardial infarction (IMI), Lateral myocardial infarction (LMI), Posterior myocardial infarction (PMI), Left anterior/left posterior fascicular block (LAFB/LPFRD) In- complete right bundle branch block (IRBBB), AV block (CAVB), Non-speeific intraventricu- lar conduction disturbance (IVCD), Complete right bundle branch block (CRBBB), Com- plete left bundle branch block (CLBBB). Wolff- Parkinson-Whin syndrome (WPW), Incomplete left bundle branch block (ILBBB)
More fine-grained classification (24 categories) compared to PTB-XL superclass (7 categories in BenchMD).
TUEV
Spike and slow (SPSW). Generalized pe- riodic epileptiform discharge (GPED), Periodic lateralized epileptiform discharge (PLED), Eye movement (EYEM), artifact (ARTF). and Back- ground (BCKG)
Data consists of pathological patterns, human artifacts, and normal background activity.
36

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 14. Performance metrics of Med Vit across different medical imaging datasets.
VedVit
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.7609
0.3333
0.6667
0.3248
0,9667
CT INSPECT
0.5495
0,2000
0.8000
0.1764
0.9156
MIMIC-CK
0.6743
0.1447
0.9432
0.1444
0.8737
Fundus JICHI
0.5464
0.2500
0.7625
0.1951
0.8077
Fundus APTOS
0,4141
0.2000
0,8000
0.0750
0,6923
CT LNDB
0.7009
0.4960
0.4960
0.4395
0.7841
ISIC 2020
0.2676
0.5000
0.5000
0.4931
0.9726
CBIS-DDSM
0.8333
0.5000
0.5000
0.3000
0.4286
BUSI
0.1111
0.5000
0.5000
0,3333
0,5000
LC25000
0.9448
0.4444
0.8631
0.3218
0.7836
HAM10000
0.6446
0.2500
0.7500
0.2143
0.8750
VinDr CXR
0.6546
0.1607
0.9005
0.1278
0.8587
CoronaHack
0.6907
0.4583
0.7526
0.4242
0.7333
BCSS
0.5833
0.3333
0.6667
0.2727
0.7949
VinDr Mammo
0.4205
0.3333
0.6667
0.2716
0.7917
COVID-BLUES
0.3556
0,5000
0,5000
0.3448
0.5263
Brain Tumor
0.8927
0.4375
0.8229
0.3167
0.7500
KiTS23
0.4063
0.5000
0.5000
0.4706
0,8889
CheXpert
0.6333
0.1686
0.9021
0.1693
0.8455
PAD-UFES-20
0.6528
0.3333
0.6667
0.0667
0,4074
COVID-19 CXR
0.6972
0.3333
0.6667
0,2222
0.6667
COVID-US
0.8889
0,5000
0,7500
0,3556
0.6000
Messidor-2
0.0000
0.5000
0.5000
0.4286
0.7500
Overall
0.5793
0.3642
0.6903
0.2821
0.7484
37

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 15. Performance metries of PMC CLIP across different medical imaging datasets.
PMC CLIP
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.7928
0.3333
0.6667
0.3241
0,9639
CT INSPECT
0.5484
0,2000
0,8000
0.1779
0,9205
MIMIC-CRR
0.6613
0.1261
0,9411
0.1234
0.8438
Fundus JINCHI
0.5492
0.2560
0.7513
0.2110
0.8308
Fundus APTOS
0.5863
0.2033
0.8023
0.0928
0.7124
CT LNDB
0.7012
0.5000
0.5000
0,4515
0.8232
ISIC 2020
0.7351
0,5000
0,5000
0.4955
0,9823
CBIS-DDSM
0.6475
0.2069
0.8064
0.1798
0.8024
BUSI
0.5371
0.3233
0.6596
0.2847
0.6616
LC25000
0.9842
0.7328
0.9332
0.7058
0.8931
HAM10000
0.7439
0.2191
0.8050
0.1921
0.8670
VinDr CXR
0.6233
0.0935
0.9235
0.0720
0.9110
CoronaHack
0.8480
0.5062
0.7729
0)4458
0.7212
BCSS
0.6432
0.2555
0.7534
0.1855
0.7289
VinDr Mammo
0.5807
0.2062
0.8007
0.1685
0,8651
COVID-BLUES
0.4175
0.4941
0.4941
0.2304
0,2708
Brain Tumor
0.6160
0.3210
0.7771
0.2656
0.6802
KiTS23
0.4712
0.5000
0.5000
0.4658
0.8718
CheXpert
0.6680
0.1170
0.9191
0.1287
0.7665
PAD-UFES-20
0.5427
0.2552
0.8121
0.1436
0.7089
COVID-19 CXR
0.8240
0.4106
0.8579
0.3550
0,8138
CMMD
0.6140
0.5226
0.5226
0.0499
0.0513
COVID-US
0.6091
0.3333
0,6730
0.1961
0.6000
Messidor-2
0.3898
0.2000
0,8000
0.1473
0.8331
CT Hemorrhage
0.5805
0.5000
0,5000
0.4550
0.8350
Brain Tumor 2
0.5848
0.5441
0.5441
0.5433
0.5882
Overall
0.6346
0.3408
0.7237
0.2727
0.7518
38

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 16. Performance metries of RAD-DINO across different medical imaging datasets.
RAD-DINO
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.8357
0.3333
0.6667
0.3241
0,9639
CT INSPECT
0.5752
0,2000
0,8000
0.1779
0,9205
MIMIC-CRR
0.7281
0.1873
0.9404
0.2034
0.8583
Fundus JICHI
0.6093
0.2500
0.7500
0.1988
0.8301
Fundus APTOS
0.6589
0.2000
0.8000
0.1320
0.7970
CT LNDB
0.6950
0.5000
0.5000
0,4515
0.8232
ISIC 2020
0.7753
0.4999
0.4999
0.4955
0,9822
CBIS-DDSM
0.5935
0.1988
0.8015
0.1438
0.8225
BUSI
0.6570
0.3333
0.6667
0.2389
0,7056
LC25000
0.9809
0.6853
0.9213
0.6288
0.8741
HAM10000
0.8128
0,2392
0.8221
0.2337
0.8752
VinDr CXR
0.7605
0.1179
0.9376
0.1003
0.9152
CoronaHack
0.9173
0.6078
0.7995
0.5739
0.7436
BCSS
0.6502
0.2622
0.7535
0.1526
0.6768
VinDr Mammo
0.5756
0.2000
0.8000
0.1606
0,8682
COVID-BLUSE
0.5775
0,5000
0,5000
0.2066
0.2604
Brain Tumor
0.6725
0.3334
0.7760
0.2437
0.6459
KiTS23
0.4565
0,5000
0.5000
0.4658
0.8718
CheXpert
0.7516
0.2124
0.9264
0.2347
0.8068
PAD-UFES-20
0.6125
0.2666
0,8217
0.1722
0.7028
COVID-19 CXR
0,8946
0.4199
0.8357
0.3952
0.7753
CMMD
0.5065
0,5000
0.5000
0.0064
0.0064
COVID-US
0.6750
0.5803
0.7397
0.5056
0.6800
Messidor-2
0.5311
0.2000
0,8000
0.1473
0.8331
CT Hemorrhage
0.7054
0.5000
0,5000
0.4550
0.8350
Brain Tumor 2
0.5519
0.5000
0.5000
0.2500
0.3333
Overall
0.6831
0.3588
0.7253
0.2807
0.7464
39

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 17. Performance metries of SBB2 across different medical imaging datasets.
SBB2
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.8556
0.3343
0.6676
0.3260
0,9640
CT INSPECT
0.5624
0,2000
0,8000
0.1779
0,9204
MIMIC-CRR
0.7227
0.1714
0,9429
0.1875
0.8561
Fundus JINCHI
0.7215
0.3261
0.7697
0.3071
0.8389
Fundus APTOS
0.8212
0.3514
0.8927
0.3292
0.8647
CT LNDB
0.6932
0.5000
0.5000
0,4515
0.8232
ISIC 2020
0,8026
0,5000
0,5000
0.4955
0,9823
CBIS-DDSM
0.6366
0.1994
0.8007
0.1437
0.8232
BUSI
0.6649
0.3561
0.6773
0.3040
0.7056
LC25000
0.9977
0.9622
0.9906
0.9622
0.9849
HAM10000
0.8641
0.2797
0.8438
0.2902
0.8824
VinDr CXR
0.6377
0.0845
0.9188
0.0717
0.9261
CoronaHack
0.9127
0.7521
0.8830
0.7532
0.8462
BCSS
0.7249
0.4504
0.8171
0.3938
0.7550
VinDr Mammo
0.5981
0.2003
0.8013
0.1668
0,8653
COVID-BLUES
0.6890
0.5977
0,5977
0.6063
0.7500
Brain Tumor
0.7723
0.5411
0.8434
0.4945
0.7640
KiTS23
0.6137
0,5000
0.5000
0.4658
0.8718
CheXpert
0.7325
0.2204
0.9238
0.2520
0.8027
PAD-UFES-20
0.6844
0.3048
0,8280
0.2064
0.7264
COVID-19 CXR
0.9487
0.7784
0.9393
0.7951
0,9216
CMMD
0.3803
0.3871
0.3871
0,4348
0.7692
COVID-US
0.7789
0.5318
0.7921
0,5258
0.7333
Messidor-2
0.6529
0,2000
0,8000
0.1473
0.8331
CT Hemorrhage
0.7307
0.4815
0.4815
0.4583
0.7883
Brain Tumor 2
0.7734
0.7059
0.7059
0.7230
0.7843
Overall
0.7297
0.4199
0.7540
0.4027
0.8378
40

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 18. Performance metries of Swin Transformer across different medical imaging datasets.
Swin Transformer
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.8962
0.4318
0,7605
0.4613
0,9683
CT INSPECT
0.6468
0.2049
0.8035
0.1878
0.9219
MIMIC-CXI
0.7620
0.1748
0,9459
0.2113
0.8571
Fundus JICHI
0.7821
0.3663
0.7967
0.3392
0.8396
Fundus APTOS
0.8636
0.3632
0.9089
0.3123
0.8849
CT LNDB
0,6660
0.5113
0,5113
0.4778
0.8239
ISIC 2020
0.7654
0.5313
0,5313
0.5438
0,9780
CBIS-DDSM
0.7078
0.2019
0.7973
0.1688
0.8071
BUSI
0.6264
0.3627
0.6742
0.1955
0.5228
LC25000
0.9987
0.9573
0.9893
0.9572
0.9829
HAM10000
0.8764
0.3743
0.8557
0.3842
0.8860
VinDr CXR
0.6206
0.0920
0.9210
0.0811
0.9248
CoronaHack
0.9231
0.7725
0.8909
0.7580
0.8419
BCSS
0.7141
0.3573
0.7713
0.2427
0,6434
VinDr Mammo
0.6330
0,2000
0,8000
0.1606
0.8682
COVID-BLUES
0.6175
0.5989
0.5989
0.6069
0.7708
Brain Tumor
0.8555
0.6215
0.8729
0.6311
0.8160
KiTS23
0.4621
0.5000
0.5000
0.4658
0.8718
CheXpert
0.7395
0.2009
0.9438
0.2395
0,8140
PAD-UFES-20
0.8042
0.3996
0.8542
0.3654
0,8100
COVID-19 CXR
0,9319
0.7029
0.9268
0.7396
0,8978
CT Hemorrhage
0.7562
0.5000
0.5000
0.4550
0.8350
COVID-US
0.8721
0.6727
0,8619
0.6672
0.8133
Messidor-2
0.6645
0.2524
0.8072
0.2183
0.8366
Brain Tumor 2
0.9360
0.5588
0.5588
0.5149
0.7059
Overall
0.7649
0.4364
0.7753
0.4154
0.8369

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 19. Performance metries of EVA-2 across different medical imaging datascts.
EVA-2
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
Fundus JINCHI
0.6808
0.2500
0.7500
0.5250
0.8301
COVID-19 CXR
0.9007
0.4372
0.8751
0.6081
0.8335
ISIC 2020
0.7291
0.5000
0.5000
0.9736
0.9823
HAM10000
0,8003
0.2625
0.8204
0.5575
0,8684
MIMIC-CRR
0.7636
0.2018
0.9291
0.3616
0.8546
Brain Tumor
0.6221
0.3531
0.7756
0.2297
0.6523
CT LNDB
0.6405
0.5000
0.5000
0.7434
0.8232
CBIS-DDSM
0.5590
0.2916
0.8046
0.4000
0.8151
VinDr Mammo
0.5240
0.2013
0.8002
0.5388
0.8683
CT Hemorrhage
0.6970
0.5000
0.5000
0.7599
0.8350
KiTS23
0.6337
0.5196
0.5196
0.0958
0.1624
VinDr CXR
0.6986
0.1136
0.9326
0.4718
0,9223
Messidor-2
0.5972
0,2000
0,8000
0.4293
0.8331
CoronaHack
0.8826
0.4654
0.7472
0.4578
0.6902
Fundus APTOS
0.8674
0.2773
0.8562
0.4195
0.8117
PAD-UFES-21
0.5795
0.2386
0.8073
0.2656
0.7673
CheXpert
0.8056
0.2596
0,8940
0.3897
0.8034
CMMD
0,4932
0.5000
0.5000
0.0001
0,0064
BUSI
0.5925
0.3333
0.6667
0.4001
0.7056
Brain Tumor 2
0.6739
0,5000
0,5000
0,5333
0.6667
Overall
0.6871
0.3452
0.7239
0.4580
0.7366
42

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 20. Performance metrics of 2onvNextv2 across different medical imaging datasets.
ConvNextv2
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.9400
0.5598
0.8337
0.6080
0,9748
CT INSPECT
0.5867
0.2098
0.8084
0.1983
0,9217
MIMIC-CRR
0.7997
0.2400
0,9480
0.2740
0.8721
Fundus JICHI
0.8568
0.5949
0.8492
0.5997
0.8798
Fundus APTOS
0.9380
0.5835
0.9455
0.6105
0.9214
CT LNDB
0.6670
0.5311
0.5311
0.5322
0.7722
ISIC 2020
0,8529
0.5042
0,5042
0.5039
0,9823
CBIS-DDSM
0.7070
0.2435
0.8170
0.2214
0.8299
BUSI
0.7653
0.4862
0.7431
0.4623
0.6853
LC25000
0.9999
0.9926
0.9982
0.9926
0.9971
HAM10000
0.9423
0.6485
0.9298
0.6511
0.9251
VinDr CXR
0.5609
0.0852
0.9175
0.0726
0.9270
CoronaHack
0.9573
0.8369
0.9260
0.8381
0.9017
BCSS
0.8098
0.5076
0.8180
0.4981
0.7773
VinDr Mammo
0.6732
0.2536
0.8069
0.2414
0,8693
COVID-BLURE
0.7318
0.6814
0.6814
0.6742
0.7396
Brain Tumor
0.9293
0.7257
0.9069
0.7136
0.8655
KiTS23
0.4108
0,5000
0.5000
0.4658
0.8718
CheXpert
0.8037
0.2722
0.9521
0.3160
0.8236
PAD-UFES-20
0.9068
0.5511
0.8978
0.5873
0,8693
COVID-19 CXR
0.9639
0.7474
0.9495
0.7545
0,9337
CMMD
0.4935
0.4935
0,4951
0,9808
COVID-US
0.8254
0.7561
0,8841
0,7313
0,8400
Messidor-2
0.8245
0.5116
0.8684
0,5116
0.8606
CT Hemorrhage
0.7485
0.5306
0.5306
0,5188
0.8388
Brain Tumor 2
0.9585
0.9265
0.9265
0.9328
0.9412
Overall
0.7867
0.5374
0.8064
0.5387
0.8770
43

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 21, Performance metrics of InternVi ViT across different medical imaging datasets.
Intern VIT
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.9005
0.4407
0.7597
0.4791
0.9679
CT INSPECT
0.5954
0.2013
0.8008
0.1808
0.9206
MIMIC-CR
0.7780
0.2196
0,9426
0.2397
0.8667
Fundus JINCHI
0.8258
0.5438
0.8224
0,4625
0.7293
Fundus APTOS
0.8786
0.4505
0.9260
0,4115
0.8941
CT LNDB
0.6810
0.5448
0,5448
0,5470
0.8050
ISIC 2020
0.8427
0.5040
0,5040
0.5037
0.9820
CBIS-DDSM
0.6701
0.2714
0.8217
0.2488
0.8205
BUSI
0.7310
0.5016
0.7391
0.4475
0.6311
LC25000
0.9990
0.9738
0,9934
0.9737
0.9895
HAM10000
0.9012
0.6092
0.9210
0.5381
0.8820
VinDr CXR
0.6309
0.0899
0.9184
0.0808
0,9272
CoronaHack
0.9303
0.7680
0.8910
0.7515
0.8419
BCSS
0.7856
0.4696
0,8235
0.3876
0.7584
VinDr Mammo
0.6450
0.2473
0,8176
0.2423
0.7962
COVID-BLU -LUES
0.6772
0.6592
0.6592
0.6364
0.6875
Brain Tumor
0.8236
0.6134
0.8734
0.5869
0.8135
KiTS23
0.5722
0.5000
0.5000
0)46588
0.8718
CheXpert
0.7830
0.2333
0,9502
0.2757
0,8168
PAD-UFES-22 20
0,8600
0.5162
0.8844
0.5476
0.8501
COVID-19 CXR
0.9526
0.7525
0.9484
0.7360
0.9281
CMMD
0.2817
0.5000
0.5000
0.4984
0.9936
COVID-US
0.7978
0.4864
0.7556
0.4680
0.7067
Messidor-2
0.8119
0.2977
0,8050
0.1526
0.6720
CT Hemorrhage
0.7790
0.6562
0.6562
0.6398
0.7806
Brain Tumor 2
0.9377
0.7500
0.7500
0.7733
0,8235
Overall
0.7720
0.4923
0.7888
0.4721
0.8368
Table 22, Performance evaluation of GCN across different modalities and datasets.
GCN
Performance Metrics
Modality
Dataset
AUC Sensitivity Specificity
PPMI
0.973
0.922
0.897
ABIDE
0.626
0.596
0,586
Brain Networks
ABCD
0.814
0.570
0.916
Average
0.804
0.696
0.800
PPI
0.807
0.496
0.716
Molecular
PROTEINS
0.718
0.568
0.803
Average
0.763
0.532
0.760
Overall
Average
0.783
0.614
0.780
44

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 23. Performance evaluation of GAT across different modalities and datasets.
GAT
Performance Metries
Modality
Dataset
AUC Sensitivity
Specificity
PPMI
0.927
0.931
0.828
ABIDE
0.688
0.818
0.385
Brain Networks
ABCD
0.500
1.000
0.000
Average
0.705
0.916
0.404
PPI 0.926
0.572
0.798
Molecular
PROTEINS
0.719
0,529
0.803
Average
0.823
0,551
0.801
Overall
Average
0.764
0.733
0.602
Table 24. Performance evaluation of Graph Transformers across different modalities and datasets. Graph Transformers
Performance Metries
Modality
Dataset
AUC Sensitivity Specificity
PPMI
0.950
0.862
0.957
ABIDE
0.743
0.707
0.683
Brain Networks
ABCD
0.864
0.860
0.837
Average
0.852
0.810
0.826
PPI
0.997
0.606
0.873
Molecular
PROTEINS
0.580
0.156
0.967
Average
0.789
0.381
0.920
Overall
Average
0.820
0.595
0.873
45

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 25. Performance metries across different medical imaging datasets. ConvNextv2 (Single Task Training)
Dataset
AUC
Sensitivity
Specificity
F1 Score
Accuracy
CT RSPECT
0.9680
0.7388
0,9096
0.7472
0.9675
CT INSPECT
0.5524
0.2170
0.8103
0.2070
0.7930
MIMIC-CH
0.8036
0.8100
0.6526
0.3735
0.6805
Fundus JINCHI
0.8994
0.6689
0.8794
0.6465
0.7763
Fundus APTOS
0.9162
0.4880
0.9309
0.4969
0.7613
CT LNDB
0.6750
0.4983
0.4983
0.4507
0.8204
ISIC 2020
0.8936
0.4957
0,9559
0.6120
0,9478
CBIS-DDSM
0.6375
0,2173
0.8070
0.1837
0.5563
BUSI
0.6370
0.4359
0.7013
0.4064
0.5381
LC25000
1.0000
0.9971
0.9993
0.9971
0.9971
HAM10000
0.9531
0.7154
0.9416
0.6850
0.8018
VinDr CXR
0.3353
0.5809
0.1281
0.7099
CoronaHack
0,9461
0.8404
0.9225
0.8400
0,8462
BCSS
0.8431
0.6276
0,8571
0.6053
0.6219
VinDr Mammo
0.6864
0.3138
0.8179
0.2802
0.6747
COVID-BLUES
0.5282
0.5000
0.5000
0.2066
0.2604
Brain Tumor
0.8291
0.4801
0.8188
0.4555
0.4721
KiTS23
0.3235
0.5000
0.5000
0.4658
0,8718
CheXpert
0.6351
0.7669
0.4516
0.7603
PAD-UFES-20
0.8112
0.3295
0.8531
0.3430
0.5425
COVID-19 CXR
0.9168
0.5689
0.8947
0.6013
0.7419
CMMD
0.5204
0.4860
0.4860
0.4913
0.9658
COVID-US
0.5000
0.3333
0.6667
0.0920
0.1600
Messidor-2
0.8301
0.2590
0.8073
0.2117
0.5886
CT Hemorrhage
0.7666
0.8884
0.2471
0.5724
0.7825
Brain Tumor 2
0.7734
0.0000
1.0000
0.4000
0.6667
46

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 26. Performance metries of Llava-Med across different medical imaging datasets. Note the overall row is averaged across dataset, which is different from the model average in Table 6.
Llava-Med
Zero-Shot
Fine-Tuned
Dataset
Accuracy
Sensitivity
Specificity
Score
Accuracy
Sensitivity
Specificity
Score
MIMIC-CXR
0.001
0.214
0.786
0.063
0.001
0.214
0.794
0.090
CheXpert
0.000
0,077
0.924
0.036
0.000
0.154
0.847
0.093
VinDr-CXR
0.045
0.083
0.916
0.016
0.702
0.083
0.917
0.069
COVID-19
0,009
0,250
0.750
0,005
0,455
0,250
0.750
0,156
CoronaHack
0.388
0.333
0.667
0.186
0.388
0.333
0.667
0.186
Brain Tumor
0.393
0.445
0.799
0.300
0.292
0,250
0.750
0.113
Brain Tumor
0.333
0,500
0.500
0,250
0.667
0,500
0.500
0.400
BUSI
0.579
0.391
0.697
0.357
0.558
0.333
0.667
0.239
COVID-BLUES
0.365
0.557
0.557
0.353
0.740
0.500
0.500
0,425
COVID-US
0.400
0.333
0.667
0.190
0,440
0.333
0.667
0.204
CBIS
0.008
0.202
0.799
0.005
0.560
0.200
0.800
0.144
VinDr-Mammo
0.047
0.200
0.800
0.018
0.670
0.200
0.800
0.161
CMMD
0.091
0,206
0.800
0.045
0.994
0.500
0.500
0.498
ISIC 2020
0.982
0,500
0.500
0,496
0.982
0.333
0.667
0.330
HAM10000
0.110
0.201
0.800
0.047
0.670
0.201
0.800
0.162
PAD-UFES-20
0.307
0.186
0.800
0.107
0.368
0.200
0.800
0.108
Messidor-2
0.151
0.157
0.832
0.048
0.583
0.200
0,800
0.147
APTOS
0,492
0,200
0.800
0.132
0,492
0.200
0.800
0.132
Jichi
0,660
0.250
0.750
0.199
0.660
0.250
0.750
0.199
LNDb
0.823
0,500
0.500
0,452
0.645
0,541
0.541
0,520
Kits23
0.128
0,500
0.500
0.114
0.872
0,500
0,500
0,466
Brain CT
0.835
0.500
0.500
0.455
0.835
0.500
0.500
0,455
INSPECT
0.004
0.194
0.800
0.006
0.801
0.200
0.800
0.178
Cholec 80
0.000
1.000
0.000
0.243
0.296
0.143
0.857
0.069
Overall
0.298
0.333
0.685
0.172
0.570
0.297
0.707
0.231
Table 27. Model Performance of B1OT Variants with TUEV Finetuning
Num Shots Pretrain Encoder
Finetune
AUC
Sens
Spec
.589
.232
.811
8
.688
.298
.859
BIOT
TUEV
32
.740
.359
.833
full
.856
.466
.908
.609
.239
.835
.754
.372
.871
BIOT-pretrain PREST
TUEV
32
.781
.363
.899
full
.898
.580
.918
.630
.239
.836
.723
.305
.861
BIOT:peerrann-SSHIS-PPPEPTT
TUEV
32
.777
.382
.879
full
.880
.586
.914
.553
.179
.836
.759
.332
.873
BIOT-pretrain-IIIC-TUAB
TUEV
32
.807
.410
.894
full
.869
.510
.905
47

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 28. Model Performance Different ECG Datasets
PTB-XL
ChapmanShao
CPSC
Ga
Overall
Model Name
AUC
Sens
Spe
AUC
Sens
Spc
AUC
Sens
Spe
AUC
Sens
Spe
AUC
Sens
Spe
Transformer ECG-JEPA
.785
.239
.885
.797
.312
.199
.579
.213
.883
.671
.251
.891
.708
.253
.895
906
.591
.918
858
.392
.935
.979
.797
.980
.767
.294
.899
.87
.518
.877
UniTS
.669
.150
861
.656
.146
.859
.641
.143
.857
.598
.158
.863
.641
.149
.860
Table 29, Model Performance ECG-JEPA UnITS Variants with PTB-XL Finetuning. CLIMB dataset with PTB-XL removed.
model pretrained on
Model
Pretrain Encoder
Num Shots
Finetune
AUC
Sens
Spe
.633
.048
.956
Ours
PTB-XL
.760
.113
.966
full
.895
.210
.980
ECG-JEPA
.512
.043
.956
PTB-XL
PTB-XL
.472
.043
.956
full
.868
.195
.979
.527
.430
.537
Ours
PTB-XL
.549
.641
.370
full
.673
.025
.993
UniTS
.512
.674
.371
PTB-XL
8
PTB-XL
.470
.322
.658
full
.688
.053
.984
Table 30. Model Performance of ECG-JEPA Variants with Different Pretraining, Ours is the model pretrained CLIMB dataset.
Pretrain
Finetuned
AUC
Sens
Spe
PTB-XI.
PTB-XI.
.776
.237
.883
CPSC
CPSC
.682
.144
.857
ChapmanShao
ChapmanShao
.771
.248
.899
Ga
Ga
.474
.143
.857
PTB-XL
.805
.329
.902
CPSC
.843
.343
.918
Ours
ChapmanShao
.831
.355
.932
Ga
.704
.273
.894
Table 31. Model Performance of UniTS Variants with Different Pretraining, Ours is the model pretrained CLIMB dataset and
Original is the pretrained encoder provided by the UniTS model.
Pretrain
Train
Eval
AUC
Sens
Spe
PTB-XL
PTB-XL
.669
.150
.861
CPSC
CPSC
.641
.143
.857
Original
ChapmanShao
ChapmanShao
.656
.146
.859
Ga
Ga
.598
.158
.863
PTB-XL
.772
.242
.866
CPSC
.871
.336
.999
Original
Ours
ChapmanShao
.812
.333
.927
Ga
.742
.271
.894
48

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 32. Dataset membership prediction results validating OOD dataset distinctiveness
OOD Dataset
Balanced Accuracy
AUC
Fl Score
BCSS
99.8
1.000
0.928
CBIS-DDSM
99.9
1.000
0,801
CoronaHack
99.7
0.999
0.371
COVID-19
53.2
0.953
0.112
BUSI
99.9
1.000
0.997
Jichi
99.8
0.999
0.758
ISIC-2020
99.7
0.999
0.841
Table 33, ECG unsupervised pretraining results comparing dataset-specific CLIMB pretraining, The results demonstrate that
pretraining on a diverse collection effectively improved downstream performance on the target dataset.
Model
Pretrain Dataset
Evaluation Dataset
AUC
Sensitivity
Specificity
ECG-JEPA
PTB-XL.
PTB-XI.
.868
.195
.979
ECG-JEPA
CLIMB
PTB-XI.
.895
.210
.980
Table 34, EEG pretraining comparison across different training strategies. The combination of pretraining and multitask learning achieved best overall results, with pretraining appearing to play more significant role than multitask learning in EEG model performance.
IIC
TUEV
TUAB
Overall
Model Name
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
AUC
Sens
Spe
F1
No Pretrain
.854
.510
.905
.499
.856
.466
.908
.371
.879
.798
.798
.799
.863
.591
.870
.556
MTL Only
.48
.484
.001
.475
.033
.386
.932
.387
.844
.764
.764
.761
.865
.545
.866
.541
PT+MTL
.862
.546
.911
.531
.878
.549
.917
.397
.869
.794
.794
.795
.870
.630
.874
.574
Table 35, Vision model performance comparison across different pretraining strategies. MAE: Masked Autoencoder following ConvNeXTv2 (Liu et al., 2022), CL: Contrastive Learning using CLIP-style approach from Intern (Chen et al., 2024b), MTL: MultiTask Learning. Contrary expectations. neither MAE contrastive learning pretraining improved model performance. The multitask learning approach alone achieved the best results across all metrics.
Model
CXR
CT
Fundus
Overall
AUC
Sen Spe
AUC
Sen
Spe
AUC
Sen
Spe
AUCC
Sen
Spe
AUC
Sen
Spe
AUC
Sen
Spe
AUC
Sen Spe
ConvNeXTv2
.835
MAE + CI. MTL. on CLIMR IntemViT
.753
.338
906
.500
.287
.89
.767
353
715
678
409
.55
683
.98
.25
683
.532
689
.697
.394
.743
Only MTL CLIMB
ConNNeXTv2
.817
.939
.588
.330
.706
.011
.568
777
671
.83
563
.888
.774
.641
.770
.787
.537 8016
49

---

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
Table 36. Performance Comparison with Dataset Specific Encoders. Ours is the result of multitask universal encoders trained on CLIMB, whereas Dataset SoTA is the state-of-the-art encoder specially tuned optimized for that particular task. Note we aim to preserve as much information as possible from each datasets and avoids binning classes, so the task for some datasets, like VinDr-CXR and CBIS-DDSMN may be different from the ones used in other works. The results are taken directly from the papers.
Dataset
Ours (AUC)
Dataset SoTA (AUC)
Source/Method Name
Reference
PTB-XL
0.895
0.896
ECG JEPA X3ECG w/ HC DDI
Link
Chapman-Shane Shanxxxng
0.858
0.979
Georgia
0.767
CPSC
0.978
0.974
ECG JEPA
Link
MIMIC-CXR
0.800
0.834
ChexClusion
Link
CheXpert
0.783
0.933
CFT
Link
VinDr-CXR
0.631 (14 classes)
0.961 (6 classes)
Paper
Link
VinDo-Mammo
0.673 (5 classes)
0.840 classes)
MaMT4
Link
CBIS-DDSM
0.707 (5 classes)
0.900 (2 classes)
MEWOA
Link
ISIC 2020
0.853
0.943
Kaggle Leaderboard
Link
HAM10000
0.942
0.943
Paper
Link
PAD-UFES-20
0.907
0.920
EfficientNetB3
Link
Messidor-2
0,825 (5 classes) 0.938
0.971 (2 classes) 0.920
Paper
Link
APTOS 2019
Survey
Link
Jichi
0.857
LNDb
0,681
0.831 1 (Fleischner kw)
Leaderboard
Link
INSPECT
0.595
0.771 (Accuracy)
Paper
Link
BUSI
0.765
COVID-19
0.964
Brain Tumor
0.929
Brain MRI
0.959
Kits23
0,572
0.835 (DICE)
Leaderboard
Link
Hemorrhage
0.779
CoronaHack
0.957
COVID-BLUES
0.732
COVID-US
0.825
0.94 0.580 (Balanced Acc.)
Review
Link
IIC
0.862
TUAB
0.882
0.882 0.528 (Balanced Acc.)
TUEV
0.898
PROTEINS
0.719
0.849 (Acc) 0.997 (Acc)
HGP-SL SL
Link
PPI
0.997
g2-MLP
Link
RSPECT
0.94
LC25000
1.000
1.000 0.710 (mloU)
SE Networks
Link
BCSS
0.810
MLP-MF
Link
50


Reviews:
[{"review_id": "WCCJOXKNQb", "reviewer": "Reviewer_pTYB", "summary": "The paper introduces CLIMB (Clinical Large-scale Integrative Multimodal Benchmark), a clinical benchmark that puts together a large number of existing datasets across different modalities with a strong focus on vision, including 1D, 2D, and 3D signals, as well as graph data. The authors conduct a thorough comparison of models trained on CLIMB improve performance across a number of different tasks compared to the best existing models in the literature for these tasks (often either pretrained within a single modality). CLIMB is very large scale (including over 4.5M patients and 19TB of data). To obtain CLIMB, one must go through the steps necessary for each individual dataset.\n\n## Update after rebuttal\nI have read the rebuttal and will keep my recommendation as is.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pTYB for the positive reviews and constructive feedback. We have addressed all typographical errors in our manuscript.\n\nRegarding the definition of OOD datasets (Claim 1), we selected OOD datasets primarily based on task differences and provided a comprehensive list with justifications in the App. Table 9. Of the 10 OOD datasets, 7 (COVID-19, CoronaHack, ISIC-2020, BCSS, BUSI, LNDb, PTB-XL-Finegrained) have zero label overlap with other datasets in the same clinical domain, while the remainder feature distinct task granularity. To quantify this heterogeneity, we conducted a membership experiment using ConvNeXT-v2-base as the backbone to predict dataset membership, following hyperparameter settings in Appendix C.4.2. Results confirm our designated OOD datasets exhibit substantial distinctiveness:\n\n|OOD Dataset|Balanced Accuracy|AUC|F1 Score|\n|-----------|----------------|---|--------|\n|BCSS|99.8|1.000|0.928|\n|CBIS-DDSM|99.9|1.000|0.801|\n|CoronaHack|99.7|0.999|0.371|\n|COVID-19|53.2|0.953|0.112|\n|BUSI|99.9|1.000|0.997|\n|Jichi|99.8|0.999|0.758|\n|ISIC-2020|99.7|0.999|0.841|\n\nClaim 2: In Table 5, \"SoTA encoders\" refers to the best pre-trained encoders tested across clinical-specific and general domain models. Specifically, as detailed in Sec. 4.1, we employed ConvNextV2, ECG JEPA, and ClinicalBERT. Our experimental protocol follows [FuseMoE](https://arxiv.org/abs/2402.03226), except we implemented regression without binning for LOS to increase granularity, and utilized AUC, sensitivity and specificity for 48 IHM to maintain consistency throughout the paper. We will incorporate a comprehensive results table for experiment 3 in the appendix.\n\nClaim 3: We will remove lines 470-473 from the impact statement. As typical of impact statements, this referred to potential future work. The dataset samples contain demographic information where available, which could facilitate future evaluation of robustness, fairness, and privacy in clinical AI.\n\nMethods 1: Data accessibility was a key consideration in dataset selection, as outlined on page 23 (Dataset Selection Methodology). We specifically selected datasets that do not require lengthy approval processes. Researchers can access 37 of 44 datasets in CLIMB after completing these steps, requiring less than 4 hours total:\n\n- Register for PhysioNet (15 mins) and Kaggle (10 mins) accounts\n\n- Complete CITI certification training (3 hours)\n\n- Acknowledge dataset agreements (10 seconds each)\n\nOur framework then manages all downloads and processing automatically. \n\nMethods 2: CLIMB integrates all metadata, labels and text reports with each sample. We did an analysis of the number of words, the number of QA pairs and the total size of the dataset. Comparing with other multimodal clinical QA datasets: \n\n| Dataset        | Number of Words | Num QA Pairs | Size of Dataset |\n|----------------|------------------|----------------|------------------|\n| CLIMB-QA      | 129.1M           | 4.51M          | 19.01 TB         |\n| PMC-VQA        | 10.2M            | 227K           | -                |\n| GMAI-MMBench   | 980K             | 26K            | 49 GB            |\n| CARES          | 1.74M            | 41K            | 21.61 GB         |\n\nOur dataset exceeds others by at least an order of magnitude in all three metrics. \n\nMethods 3: We have added accuracy and F-1 score to Table 5. We include a subset here due to space limit and will add the full table in Appendix of final manuscript:\n\n| | | LOS | 48 IHM (Full) | | | | | 48 IHM (8-Shots) | | | | |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| **Enc** | **Fusion** | **MAE** | **AUC** | **Sens** | **Spec** | **Accuracy** | **F1 Score** | **AUC** | **Sens** | **Spec** | **Accuracy** | **F1 Score** |\n| SoTA | CrossAtt | 2.77 | 0.786 | 0.628 | 0.814 | 0.792 | 0.416646562 | 0.58 | 0.286 | 0.766 | 0.763 | 0.014860231 |\n| Ours | MLP | 2.84 | 0.961 | 0.824 | 0.975 | 0.968 | 0.704789834 | 0.672 | 0.295 | 0.858 | 0.767 | 0.290421866 |\n| Ours | CrossAtt | 2.61 | 0.796 | 0.822 | 0.59 | 0.825 | 0.90490467 | 0.57 | 0.294 | 0.753 | 0.728 | 0.105340098 |\n\nQuestion 2: In this work, our focus is on vision, time series and graphs, where large pretraining efforts are particularly scarce. We focused less on textual modality since multiple textual LLMs trained on diverse medical data already exist, including [ClinicalBERT](https://huggingface.co/medicalai/ClinicalBERT) and [BioMistral](https://huggingface.co/BioMistral/BioMistral-7B). \n\nQuestion 1,3: The EHR data encompasses all textual data available in MIMIC-IV within 48 hours of admission: vital signs, lab measurements, treatments, medications, and demographics. We excluded data without timestamps (e.g., diagnoses) and omitted discharge summaries and radiology reports. We performed no normalization or binning on LOS regression. The text is JSON-formatted and fed directly into the text model for embedding.", "claims_and_evidence": "- Lines 237-239 (column 2): \"To evaluate few-shot generalization, we test on out-of-distribution (OOD) datasets $D_\\text{ood} \\not\\subset D_\\text{train}$.\" => You are claiming that if a dataset is not part of the training datasets used, it is out-of-distribution, but that may not always be the case. Authors themselves explain in lines 185-188 (column 2) that they \"balance the dataset such that each modality contains 3-5 datasets, providing multiple data sources per modality while maintaining diversity within each category.\" It is likely that another dataset will still have a lot in common with the training datasets. Please do not use OOD to describe these datasets unless you formally quantify their \"OOD-ness\", e.g., by training a membership model.\n- In Table 5, you show results for the \"SoTA\" encoder compared to \"Ours\". Please include the citation of the paper(s) that introduce the SoTA encoder(s) instead of only \"SoTA\". Where can we find these papers with SoTA results for LOS and IHM?\n- In your impact statement, lines 470-473, you state: \"Our holistic evaluation metrics will also encourage the research community to quantify the tradeoffs between performance, complexity, robustness, fairness, and privacy in clinical AI.\" => I only see evaluation metrics related to standard model performance, and even in that case only simple metrics like AUC, accuracy, precision, and recall are used for classification, and MAE for regression. Please remove that from your impact statement.", "methods_and_evaluation_criteria": "- Not sure where to mention this, but in my opinion an issue with CLIMB is not explicitly discussing how the user can obtain the data upfront. Anyone working with applications in healthcare know this is a major bottleneck. There are details in the github repository shared and in the appendices about the licences, but it would be nice to have a paragraph early on in the main paper where authors clearly state that users need to obtain approval to access specific datasets, that upon doing that they can use their codebase to download all datasets automatically, that it can take \"X\" weeks/months to obtain approval, etc.\n- One part of the data I find the authors did not explore well (even in the appendices) is textual and EHR data. In fact, CLIMB is mostly an image-first benchmark, perhaps also a time-series-second benchmark. However, there is very little textual or EHR data in CLIMB, but this is not at all something clear from the paper. (For example, in Table 1 \"Comparison of clinical benchmarks\" it seems that CLIMB \"fully covers\" the text modality, which I find does not fully tell the truth.) Free-text clinical notes and EHR data are, to the best of my knowledge, only available from MIMIC-IV (one of the numerous datasets used in CLIMB). Even in Appendix A.2, where the \"Understudied modalities\" are discussed, only images are mentioned. These downsides, such as limited EHR/text data, should be more clearly highlighted to the reader in the main paper. => I am not sure what would be the alternative to showcase this more clearly, but perhaps you could have a plot like a bar plot or a radar chart, where in each dimension you have proxies to the amount of text in the dataset (e.g., \"number of words\", \"number of documents\", \"size in GB\", etc). That way you could compare CLIMB with existing benchmarks (like PMC-VQA, GMAI-MMBench, CARES) in more detail when it comes to the amount of text available in the benchmarks. For this, you should also include, for example, \"comments\" in the COVID-BLUE dataset, or any other free-text data available.\n- For evaluation of in-hospital mortality in Table 5, you should probably include the F-1 score and the area under the precision-recall curve. (Perhaps in the appendix, if that does not fit the main paper.) I find that the answer to RQ3 is a bit lacking compared to the other 2 RQs, RQ1 and RQ3.", "theoretical_claims": "No theoretical claims are made in this paper that I could verify.", "experimental_designs_or_analyses": "The experimental design used in the paper is comprehensive. The set of baselines compared against is extensive, and it is really nice to see this large-scale, wide comparison being made. The main research questions looked into in this paper have to do with whether pretraining a model on CLIMB improves downstream performance across tasks, and to what extent do models transfer to unseen datasets.", "relation_to_broader_scientific_literature": "Good comparisons to existing benchmarks (Table 1) to the best of my knowledge.", "essential_references_not_discussed": "No crucial references missing that I noticed to the best of my knowledge.", "other_strengths_and_weaknesses": "Strengths worth mentioning:\n- Very large scale clinical benchmark\n- Extensive experiments showing improvements on downstream tasks\n\nWeaknesses specific to each dimension of the review are discussed in the dimension's section.", "other_comments_or_suggestions": "- In line 325, you refer to Table D.1 but it links to Table 25 in Appendix D.1.\n- If you are going to focus on precision and recall (like in Tables 2, 3, etc), you should probably also include an F-score. The F-score helps quickly see the trade-off between both quantities (which is of course do-able in with only P and R, but considerably slower).\n- In all your Tables, only embolden the overall best-performing results (using bold-face). In Table 2, for instance, you embolden InternViT Mammography Sen (.340), but MedViT obtained .417 Sen. You also say in Table 2's caption that \"The best performance of each model in AUC is bolded.\" However, it is not just the AUC that is emboldened. In Table 5, SoTa MLP for 48 IHM (8-Shots) has recall of .536, whereas the entry that is emboldened is Ours MLP with a recall of 0.295. Please fix these typos/mistakes/inconsistencies throughout the paper.\n- Figures and Tables in your paper do not appear in the order they are mentioned in the text. Please fix that.\n- You often compare few-shot performance with \"full dataset\" performance, like for instance in Figure 6 or Table 5. I may have missed that, but in \"full dataset\" do you mean the model is fine-tuned on the full training data available for the task? If that is the case, please do not use \"1 shot\", \"8 shots\", and \"full\", but for the latter use \"full FT\" instead (and explain in the figure/table caption that FT means fine-tuning).", "questions_for_authors": "For MIMIC-IV experiments under RQ3: what exactly are the clinical notes you used (radiology reports, discharge summaries)? For modelling length of stay as a regression task, did you do any normalisation of the stay duration?\n\nIs it fair to say that this is an image-first, time-series second benchmark? For instance, in Appendix C4, for your vision experiments you have 8 baseline models, for your EEG/ECG experiments 8 more models, that cover a number of different architectures and strategies. For text and EHR structured data, you have ClinicalBERT.\n\nYou mention you use \"EHR data\" from MIMIC-IV, but I could not find a clear explanation of what exact variables do you mean by that. What are the variables used in your experiments that you refer to as \"EHR data\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSRjqVrR67", "reviewer": "Reviewer_SVNB", "summary": "This paper introduces the Clinical Large-scale Integrative Multimodal Benchmark (CLIMB), which integrates diverse clinical data across imaging, language, time-series, and graph modalities. CLIMB consists of 4.51 million patient samples (19.01 terabytes), covering 2D imaging, 3D video, and multimodal data. Empirical evaluations demonstrate that multitask pretraining significantly enhances performance, with improvements of 29% in ultrasound analysis and 23% in ECG analysis over single-task learning. Additionally, pretraining on CLIMB improves generalization to new tasks, while strong unimodal encoders effectively contribute to multimodal models when combined with appropriate fusion strategies.\n\n\n**update after rebuttal**\n\nIn my initial review, I had some concerns about the novelty of the proposed unified framework, and the results appeared fairly predictable.\n\nIn their response, the authors provided additional clarification on the taxonomy standardization, which I found helpful. Moreover, the improved performance on underrepresented regions and modalities (ultrasound, CT, EEG) is particularly meaningful in the healthcare domain.\n\nTherefore, I am increasing my initial score to 3, leaning toward acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer SVNB's feedback. Besides our scale and focus on multimodal, our work distinguishes itself from related research in several ways:\n\n- Our focus extends beyond confirming general pretraining benefits by specifically targeting underrepresented regions and modalities (ultrasound, CT, EEG). These more rare, OOD tasks have demonstrated superior performance gain as compared to tasks that occur often in the datasets. This challenges the traditional idea that pretraining mainly helps with popular tasks while struggling to improve perf. on OOD tasks (https://arxiv.org/abs/2211.08411, https://arxiv.org/abs/2212.10511). \n\n- In addition, we show that traditional vision encoders still perform better than medical VLLMs by a large margin, whereas related works mentioned in the review mainly focus on improving VLLMs. \n\n- We release a unified framework that streamlines downloading, processing, and model training across vision, time series, and graph domains, enabling researchers to rapidly replicate and iterate on methods against diverse clinical tasks. Our work is of a much larger scale than related works, as shown in our response to Reviewer pTYB. \n\nComment on Figure 4: We define underrepresented regions and understudied modalities in Lines 1219-1290 using a two-step approach that identified geographic and economic gaps in dataset coverage. While 8/30 (26.7%) datasets with known collection sites come from these regions, this representation still lags behind developed countries. We will clarify that our classification of understudied modalities reflects public dataset availability rather than clinical importance.\n\nOn dataset construction: We put extensive efforts into standardizing the taxonomy. Standardization is difficult as it requires balancing two competing objectives:\n\n- Merging similar terms to facilitate learning and cross-modality transfer\n- Minimizing information loss and avoiding inaccuracies when modifying labels\n\nOur standardization efforts concentrated on ECG and chest X-ray, which offer fine-grained labels with varying terminologies. For ECG, we followed approaches from [arXiv:2304.08486](https://arxiv.org/abs/2304.08486) and formulated a mapping [here](https://anonymous.4open.science/r/climb_submission-5D0E/Label_Processing.md) in our anonymous repo. \n\nFor Chest X-rays, we developed a new mapping with radiologist input. Following practices from [1](https://pmc.ncbi.nlm.nih.gov/articles/PMC10173935), [2](https://www.nature.com/articles/s41598-023-33303-y) and [3](https://pmc.ncbi.nlm.nih.gov/articles/PMC11455863/), we consolidated all Chest X-ray labels into the CheXpert 14 categories:\n\n| Raw Label | Standardized Label |\n|-----------|-------------------|\n| Aortic enlargement, Enlarged PA | Enlarged Cardiomediastinum |\n| Cardiomegaly | Cardiomegaly |\n| Atelectasis | Atelectasis |\n| Consolidation | Consolidation |\n| Edema | Edema |\n| Infiltration, Lung Opacity, ILD, Pulmonary fibrosis | Lung Opacity |\n| Nodule/Mass, Other lesion, Lung cavity, Lung cyst, Lung tumor | Lung Lesion |\n| Pleural effusion | Pleural Effusion |\n| Pleural thickening | Pleural Other |\n| Pneumothorax | Pneumothorax |\n| Rib fracture, Clavicle fracture | Fracture |\n| No finding | No Finding |\n| Support Devices | Support Devices |\n| Pneumonia | Pneumonia |\n\nLabels from other modalities with the same name are then merged with these classes. We acknowledge standardization may affect label granularity, particularly for lung opacity and lung lesion classes. CLIMB provides both standard and raw labels, allowing researchers to prioritize either granularity or standardization. All experiments in our work used standard labels.\n\nTo evaluate standardization effects, we compared vision encoders trained on raw versus standardized labels:\n\n| Model | CXR | | | Mammo | | | Derm | | | CT | | | Fundus | | | US | | | Overall | | |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe |\n| ConvNextV2-RawLabel | **.820** | .358 | .935 | .543 | .293 | .693 | .853 | .492 | .757 | **.690** | .442 | .624 | .794 | .351 | .841 | .689 | .519 | .713 | .751 | .434 | .766 |\n| ConvNextV2-StandardLabel | .817 | **.436** | **.939** | **.558** | **.330** | **.706** | **.901** | **.568** | **.777** | .671 | **.466** | **.641** | **.873** | **.563** | **.888** | **.774** | **.641** | **.770** | **.787** | **.537** | **.806** |\n\nStandardized labels yielded better performance: 3.6% improvement in overall AUC, 10.3% in sensitivity, and 4.0% in specificity. Notably, standardization benefited other modalities more than Chest X-ray itself, where most relabeling occurred. We believe standardized labels help the model connect concepts across modalities more effectively. We hope this work motivates further efforts in terminology standardization and fine-grained relabeling of public clinical datasets.", "claims_and_evidence": "The primary contribution of this work is the standardization of multiple publicly available datasets to demonstrate that pretraining on a large-scale medical dataset enhances downstream task performance. However, the results appear somewhat predictable, as prior research has already established that large-scale pretraining generally improves model performance.", "methods_and_evaluation_criteria": "This paper does not propose a novel method, so evaluating a specific methodology is not applicable. However, in terms of evaluation criteria, this work makes contributions by introducing a unified framework for holistic training and benchmarking of clinical models through standardization of data loading and prediction tasks.\nThe key contributions include:\n\n**Standardized Task Formulation**\n- All tasks are framed as multi-label classification across different clinical modalities.\n- Terminology variations are standardized, and similar concepts (e.g., Lung Opacity from CheXpert and Infiltration from VinDR-CXR) are merged to create a consistent and clinically meaningful vocabulary.\n\n**Question-Answering (QA) Reformulation**\n\nThe dataset is also structured as a closed-choice question-answering (QA) task, named CLIMB-QA, to support comparative evaluation of large vision-language models (LVLMs).\n\n**Unified Data Processing Interface**\n- A standardized pipeline is provided for downloading and processing datasets into a unified format, ensuring compatibility for large-scale mixed-modality pretraining (subject to dataset-specific consent agreements).\n\nBeyond the significance and novelty of these contributions, the rationale behind formulating multiple heterogeneous tasks in a unified manner is well-grounded and conceptually sound.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are designed to address the following three research questions:\nQ1: Can multitask-pretrained clinical models perform consistently across multiple tasks, particularly for understudied tasks?\nQ2: How well do multitask-pretrained clinical models transfer to new tasks within the same clinical modality, especially when data is limited?\nQ3: Can multitask-pretrained unimodal models be effectively fused to tackle multimodal clinical tasks?\n\nIf the novelty and significance of these research questions are set aside, the experiments themselves are adequately designed to address them. However, regarding the first and third questions, the conclusion is fairly predictable—multitask learning tends to be particularly beneficial in scenarios where data or research focus has historically been limited.\n\nFor the second question, the paper concludes that the large-scale pretraining dataset enables efficient learning of novel tasks with limited samples, yielding consistent performance improvements across all modalities. While this finding aligns with the results presented, it largely re-claims well-established principles in machine learning.", "relation_to_broader_scientific_literature": "It’s related to ML4H.", "essential_references_not_discussed": "Regarding the primary research questions addressed with CLIMB, I believe that similar findings have already been demonstrated in prior work. Specifically, previous studies have shown that training a foundation model on large-scale medical datasets improves downstream task performance. The following papers provide strong evidence in support of this claim:\n- Med Gemini (arXiv:2405.03162)\n- BiomedGPT (arXiv:2305.17100)\n- RadFM (arXiv:2308.02463)\nI assume that the authors are already well aware of these works. Given this, a clearer differentiation of CLIMB’s unique contributions—beyond reaffirming known benefits of large-scale pretraining—would further strengthen the impact of this study.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "**Comment on Figure 4**\n\nThe authors define underrepresented regions and understudied modalities. For underrepresented regions, Brazil and China are mentioned. Could you clarify why these countries are considered underrepresented in the medical domain? Do the individual datasets listed in Table 6 lack sufficient samples from Asian and South American cohorts? Providing statistical evidence would make this argument more convincing.\n\nRegarding understudied modalities, ultrasound and CT scans are included. However, this characterization may be somewhat misleading, as both modalities play a crucial role in daily clinical practice. I believe the authors intended to highlight the relative scarcity of publicly available datasets for these modalities compared to others. It would be helpful to clarify this point more explicitly.\n\n**Comment on dataset construction**\n\nI am particularly interested in how similar medical concepts are merged or how terminology is standardized, as this is a crucial preprocessing step when handling large-scale medical data for model training. \n\nFor chest X-ray datasets, did you merge lung opacity and infiltration into a single category, such as lung opacity? If so, could you provide the final list of combined classes used in the dataset? \n\nAdditionally, VinDR defines diseases (e.g., pneumonia, cancer, tuberculosis) based on findings such as nodules, fibrosis, consolidations, and etc. How did you establish the mapping between diseases and their corresponding findings? Understanding the rationale behind these classifications would provide better insight into the dataset's structure and its impact on model performance.\n\n**Suggestion**\n\nCombining and refining publicly available datasets is valuable and provides significant benefits to the research community. However, in my opinion, a more substantial revision of the dataset should be undertaken. For example, the relabeled version of ImageNet (Beyer et al., 2020) has been widely appreciated for improving data quality. Similarly, in image-text datasets, re-captioning existing datasets such as LAION or Conceptual Captions (CC) has proven helpful for researchers and practitioners.\n\nFollowing this direction, this work would greatly benefit from further efforts to refine the existing datasets. One possible approach is to standardize terminology more rigorously in collaboration with domain experts, such as thoracic radiologists. Although this work has made some efforts toward standardization, I could not find detailed information on the extent of these efforts. Specifically, it remains unclear how many radiologists were involved and what process was followed for standardization. Additionally, rephrasing existing Chest X-ray reports using standardized medical terminology would be highly valuable for deep learning applications in radiology. These enhancements would improve the dataset’s usability and reliability, ultimately benefiting clinical AI research.", "questions_for_authors": "Please refer to the comment section above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYAsysjCII", "reviewer": "Reviewer_ZboR", "summary": "This paper introduces a large-scale clinical multimodal benchmark. The authors conduct multitask pretraining, few-shot transfer, and multimodal fusion. Based on the constructed data, they provide extensive experiment results to answer the proposed research questions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback regarding Figures 3.a and 3.b. Both figures illustrate two different experiments we conducted on CLIMB: multitask learning and transfer learning. The figures display example data from CLIMB (such as x-rays, CT scans, etc.) as inputs to the model and display evaluation on the right. For Figure 3.a, we train a single model on multiple clinical tasks across different medical modalities and evaluate it on each individual dataset. The goal is to evaluate whether a multitask pre-train model can generalize across diverse tasks, including understudied ones. This setup helps assess if shared representations from CLIMB improve generalization within each dataset. In Figure 3.b, a multitask pre-trained model is applied to a new task with limited data to see whether the model adapts its learned representations to the new task, despite having few samples available. The goal is to determine if exposure to a broader range of tasks in CLIMB helps compensate for data scarcity in specific datasets.\n\nTo improve clarity, we will implement the following revisions: 1) provide a more detailed figure description 2) move labels for \"understudied\" and “few samples\" to the side to avoid implying they apply to the entire column. We will also cite the recent works mentioned in the review. \n\nRegarding the computational resources, as described in Appendix C.4.2, all experiments are performed on a server with 8xH200 GPUs for the best performance. At least 20TB of storage is needed if you would like to train a model on the entire dataset. With that said, our model is small enough to fit on one GPU with 24GB of VRAM, and the entire training on one GPU would take less than a week. \n\nAll data used in this work is public and can be accessed easily using our framework, as outlined in our response to Reviewer pTYB. In summary, a user only needs to complete one-time registrations of two accounts and complete one CITI training, which takes less than 4 hours. Our framework will then prompt the user to agree to agreements, and downloading each dataset will take less than 10 seconds of human labor. Our framework will also handle the data cleaning, processing and standardization of formats.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no proofs or theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This work provides contribution to the medical and healthcare domain.", "essential_references_not_discussed": "They cover most of the recent works. But lack some of them, such as Wang, Xiaochen, et al. \"Unity in diversity: Collaborative pre-training across multimodal medical sources.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.", "other_strengths_and_weaknesses": "Strengths:\n\n1, This is a comprehensive multimodal clinical benchmark covering different modalities. \n\n2, This paper introduces the data construction, experiment, evaluation, and discussion.\n\n3, The writing is easy to follow.\n\nWeakness:\n\n1, Figure 3.a and 3.b seem not that clear to understand.\n\n2, I am wondering what computation resources do the users need to implement this approach.", "other_comments_or_suggestions": "No", "questions_for_authors": "Please see the weakness.\n\nAlso, is all the data used in this work public? Are the data collection and cleaning scripts provided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZGHVLGfOKe", "reviewer": "Reviewer_ovwX", "summary": "This paper introduces the Clinical Large-scale Integrative Multi-modal Benchmark (CLIMB), a benchmark unifying diverse clinical data across imaging, language, temporal, and graph modalities. The dataset comprises 4.51 million patients distributed across multiple modalities. The authors conduct extensive empirical evaluations and demonstrate three key findings: (1) Multitask pretraining significantly improves performance on understudied domains. (2) Models pretrained on CLIMB demonstrate improved few-shot transfer learning capabilities, and (3) Unimodal encoder performance translates well to multimodal tasks when paired with appropriate fusion strategies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1-2, Q1-2, C1-2: In our previous experiments, both pretraining and multitask learning are performed in Exp. 1. We added experiments below comparing unsupervised pretraining vs supervised multitask learning. \n\nFirst, we found time series models benefited substantially from unsupervised pretraining. As illustrated in App. Tab. 26, we conducted unsupervised masked autoencoder (MAE) pretraining and compared it with pretraining on target dataset only:\n\n| Model | PT Dataset | Eval. Dataset | AUC | Sens. | Spec. |\n| --- | --- | --- | --- | --- | --- |\n| ECG-JEPA | PTB-XL | PTB-XL | .868 | .195 | .979 |\n| ECG-JEPA | CLIMB | PTB-XL | .895 | .210 | .980 |\n\nWe found pretraining on diverse data effectively improved the downstream outcome on the target dataset. Similarly, for EEG datasets, as shown in App. Table 24, pretraining on a diverse range of data improved the performance for 2/3 datasets and overall:\n\n| Model Name | IIIC |  |  |  | TUEV |  |  |  | TUAB |  |  |  | Overall |  |  |  |\n|------------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n|  | AUC | Sens | Spe | F1 | AUC | Sens | Spe | F1 | AUC | Sens | Spe | F1 | AUC | Sens | Spe | F1 |\n| Single Task | .854 | .510 | .905 | .499 | .856 | .466 | .908 | .371 | **.879** | **.798** | **.798** | **.799** | .863 | .591 | .870 | .556 |\n| MTL Only | .848 | .484 | .901 | .475 | .903 | .386 | .932 | .387 | .844 | .764 | .764 | .761 | .865 | .545 | .866 | .541 |\n| Pretrain+MTL | **.862** | **.546** | **.911** | **.531** | **.878** | **.549** | **.917** | **.397** | .869 | .794 | .794 | .795 | **.870** | **.630** | **.874** | **.574** |\n\nWhile combining pretrain and multitask learning achieves the best results, pretraining seems to play a larger role than multitask learning in EEG models.\n\nOn the other hand, we showed in Fig. 4 that multitask learning effectively improved the vision encoder’s performance. Surprisingly, vision models did not benefit from further pretraining, and only multitask learning helps:\n\n| PT Method | Model | CXR | | | Mammo | | | Derm | | | CT | | | Fundus | | | US | | | Overall | | |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| | | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe | AUC | Sen | Spe |\n| MAE + MTL on CLIMB | ConvNeXTv2 | 0.801 | 0.379 | 0.923 | 0.489 | 0.276 | 0.671 | 0.795 | 0.414 | 0.738 | **0.699** | 0.430 | 0.614 | 0.757 | 0.325 | 0.835 | 0.705 | 0.484 | 0.687 | 0.733 | 0.433 | 0.766 |\n| MAE + CL + MTL on CLIMB | InternViT | 0.753 | 0.338 | 0.906 | 0.500 | 0.287 | 0.689 | 0.767 | 0.353 | 0.715 | 0.678 | 0.409 | 0.595 | 0.683 | 0.298 | 0.825 | 0.683 | 0.532 | 0.689 | 0.697 | 0.394 | 0.743 |\n| **Only MTL on CLIMB** | ConvNeXTv2 | **0.817** | **0.436** | **0.939** | **0.558** | **0.330** | **0.706** | **0.901** | **0.568** | **0.777** | 0.671 | **0.466** | **0.641** | **0.873** | **0.563** | **0.888** | **0.774** | **0.641** | **0.770** | **0.787** | **0.537** | **0.806** |\n\nHere, MAE = Masked Autoencoder, CL = Contrastive Learning and MTL = Multitask Learning. For MAE, we followed the same approach as in [ConvNeXTv2](https://arxiv.org/pdf/2301.00808). For contrastive learning, we followed the CLIP-style approach as outlined in [InternVL](https://arxiv.org/abs/2312.14238). In the above vision experiment, neither MAE nor CL pretraining improved the model’s performance for downstream tasks. One hypothesis is that these models are already heavily pre-trained on massive unlabeled natural image corpora, so an additional masked image or contrastive-style phase on clinical data doesn’t substantially shift or enrich their learned representation. We welcome the community’s contribution to better leveraging the diverse labeled data available in CLIMB. \n\nQ3, C3: We found that datasets with novel tasks, Understudied modalities or from Underrepresented regions, as defined in Lines 1219-1290, benefit the most from multitask learning. We believe this is due to the scarcity of data in these categories, which makes data from other modalities or tasks, albeit not closely related, beneficial. On the other hand, if a dataset has a large number of samples or fairly saturated performance on a specialized diagnostic task, then multitask learning may not add much beyond what single-task training already learns. \n\nQ4: During the construction of CLIMB, we aim to include as many datasets from underrepresented regions as possible. This includes datasets from geographically underrepresented regions, as well as data from developing countries where the data has been historically scarce, as defined in Lines 1274-1290. We included a list of dataset collection locations in the App. Table 8. In summary, 8 out of 30 (26.7%) of the datasets with known source locations come from underrepresented regions, a percentage significantly higher than all datasets available from public sources.", "claims_and_evidence": "Generally, the claims made in this submission are well-supported. \n1. Multitask pretraining improving performance on understudied domains: the authors provide evidence in Figure 4, showing substantial AUC improvements for understudied modalities. \n2. The second claim regarding few-shot transfer is supported by experiments in Figure 7, demonstrating improvements across various modalities including ultrasound, CT, and ECG domains when using CLIMB pretraining versus standard pertaining.\n3. The third claim about fusion strategies is supported by results in Table 5, showing that different fusion methods (late fusion, MLP, cross-attention) exhibit varying effectiveness depending on the task complexity.\nHowever,", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem.", "theoretical_claims": "The paper does not contain formal proofs for theoretical claims.", "experimental_designs_or_analyses": "I reviewed the experimental designs and analyses in detail and found them to be generally sound. The authors use appropriate statistical measures and the multitask training setup is well-designed.", "relation_to_broader_scientific_literature": "The paper positions itself well within the broader scientific literature on clinical AI and multimodal learning. The authors compare CLIMB to existing medical benchmarks like BenchMD, PMC-VQA, GMAI-MMBench, and CARES.", "essential_references_not_discussed": "A few references that should be included like recent work on multimodal foundation models in healthcare like BiomedCLIP published in NEJM AI (https://ai.nejm.org/doi/full/10.1056/AIoa2400640), and works on self-supervised learning for medical imaging.", "other_strengths_and_weaknesses": "**Strengths**\n1. The scale and diversity of the CLIMB benchmark is impressive.\n2. The focus on understudied modalities and underrepresented regions is valuable for addressing biases in clinical AI.\n\n\n**Weaknesses**\n1. The paper focuses on supervised multitask pretraining rather than exploring self-supervised approaches, which might be more data-efficient and better leverage the unlabeled portions of clinical data, also no bias problem.\n2. The improvement in understudied domains might be primarily attributable to increased data exposure rather than true cross-task knowledge transfer.", "other_comments_or_suggestions": "1. The authors should discuss why multitask pre-training instead of self-supervised methods (like masked image modeling or contrastive learning), which might better leverage the large-scale nature of CLIMB.\n2. A more thorough analysis separating the effects of increased data quantity from the benefits of multitask learning would strengthen the paper. For example, comparing with models trained on equivalent amounts of data but without task sharing.\n3. The paper would benefit from a more detailed analysis of when multitask learning helps versus when it potentially causes negative transfer, as Figure 4 shows varying impacts across datasets.", "questions_for_authors": "1. Have you explored self-supervised pretraining approaches as an alternative to supervised multitask learning?\n2. To what extent are the improvements in understudied domains attributable to multitask learning specifically versus simply having access to more training data? Have you conducted ablation studies with equivalent data quantities in single-task settings?\n3. Figure 4 shows that some datasets experience minimal gains or even slight performance decreases with multitask pretraining. What factors determine whether dataset benefits from multitask learning, and could you elaborate on potential cases of negative transfer?\n4. How did you address potential biases in the datasets, particularly for underrepresented regions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript introduces CLIMB (Clinical Large-scale Integrative Multimodal Benchmark), a comprehensive clinical benchmark integrating diverse data across imaging, language, temporal and graph modalities, comprising 4.51 million patient samples (19.01 TB). The work demonstrates that multitask pretraining on CLIMB significantly improves performance on understudied domains and modalities, with notable gains in ultrasound (29%) and ECG analysis (23%). The reviewers highlighted several key strengths, including the extensive scale and diversity of the benchmark, standardised taxonomy development with radiologist input, and improved performance on underrepresented regions and modalities. Through their rebuttal, the authors provided additional experimental evidence comparing supervised multitask learning with self-supervised approaches, demonstrating that vision models benefited primarily from multitask learning while time-series models showed substantial gains from unsupervised pretraining. Some limitations noted by reviewers included the predictability of certain findings regarding pretraining benefits and the need for more thorough analysis of negative transfer cases. The rebuttal effectively addressed concerns about data accessibility and standardisation approaches, leading two reviewers to increase their scores. Following the rebuttal, reviewers reached general consensus supporting acceptance, with scores ranging from weak to full accept, particularly appreciating the meaningful improvements demonstrated for understudied modalities and underrepresented regions.
Decision: Accept (poster)