MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

## Abstract

Abstract This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities, with multigranular annotations for more than 65 diseases. These enriched annotations encompass both global textual information, such as disease/lesion type, modality, region-specific descriptions, and inter-regional relationships, as well as detailed local annotations for regions of interest (ROIs), including bounding boxes, segmentation masks. Unlike existing approach which is limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and texual annotations (in the form of image-ROI-description triplets) without the need for any paired text descriptions. Specifically, data from over 90 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions.
We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular texual descriptions.
Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation.
Pretraining on MedTrinity-25M, our model achieves state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal large language models and other representative SoTA approaches.
This dataset can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain.
The dataset is publicly available at https://yunfeixie233.github.io/MedTrinity-25M/ .

## 1 Introduction

Large-scale multimodal foundation models [, , , , ] have demonstrated remarkable success across various domains due to their ability to understand complex visual patterns in conjunction with natural language. This success has sparked significant interest in applying such models to medical vision-language tasks.
Much progress has been made to improve the medical capacity of general domain multimodal foundation models by constructing medical datasets with image-text pairs and fine-tuning general domain models on these datasets [, , , , ].

However, current medical datasets have several limitations.
Firstly, these datasets lack multigranular annotations that reveal the correlation between local and global information within medical images.
Medical images often contain detailed cues, such as regional abnormal textures or structures, which may indicate specific types of lesions. Therefore, multimodal models need the ability to infer global information, such as disease or lesion type, from local details. The absence of such data limits the models’ capacity to comprehensively understand medical images.
Moreover, current dataset construction methods heavily rely on medical images paired with reports or captions, which restricts their scalability.

In this paper, we address the above challenges by proposing an automated data construction pipeline using multimodal large language models (MLLMss) without relying on paired text descriptions.
To address the lack of comprehensive medical knowledge in general-purpose MLLMs, we leverage domain-specific expert grounding models and retrieval-augmented generation (RAG) to extract relevant medical knowledge. We then prompt MLLMs to generate multigranular visual and textual annotations enriched with this knowledge based on identified regions of interest (ROIs).
We utilize this pipeline to transform the collected data, including large-scale unpaired images, into image-ROI-description triplets.
These triplets provide multigranular annotations that encompass both global textual information, such as disease/lesion type, modality, and inter-regional relationships, as well as detailed local annotations for ROIs, including bounding boxes, segmentation masks, and region-specific textual descriptions.
Using the proposed pipeline, we create a large-scale multimodal multigranular medical dataset containing over 25 million triplets, named MedTrinity-25M. To our best knowledge, this is the largest multimodal dataset in medicine to date.

Initially, we assemble a large amount of medical data from over 90 online resources such as
[TCIA](https://www.cancerimagingarchive.net/), [Kaggle](https://www.kaggle.com/), [Zenodo](https://zenodo.org/), [Synapse](https://www.synapse.org/), etc.
In addition to images with a small amount of high-quality paired manual reports, this assembled data also includes two types of coarse medical data: 1) Image data with segmentation masks, lesion bounding boxes, or only disease types but lacking detailed textual descriptions, and 2) Images paired with coarse captions that describe only global modality or disease information, but lack detailed descriptions of local regions.
To generate multigranular annotations from the massive coarse medical data, we first identify ROIs that contain disease or lesion patterns by applying expert grounding models. We then build a comprehensive knowledge base from online corpora (e.g., PubMed) and retrieve image-related medical knowledge. Finally, we prompt MLLMs to integrate medical knowledge with guidance of identified ROIs to generate multigranular textual descriptions.

## 2 Related Work

Medical Multimodal Foundation Models.
Due to the effectiveness of multimodal foundation models in understanding visual features, adapting these models to perform medical vision-language tasks has garnered increasing attention in recent years [, , , ].
Several papers attempt to adapt general domain multimodal foundation models with varying architecture to medical domain through end-to-end training on medical datasets.
For example, Med-Flamingo [] enhances the medical capacity of OpenFlamingo-9B [] by fine-tuning it with 0.8M interleaved and 1.6M paired medical image-text data.
While Med-PalM [] adapts PaLM-E [] to medical domain using approximately 1M medical data points, demonstrating competitive or surpassing performance compared to state-of-the-art models.
Additionally, LLaVA-Med [] employs end-to-end visual instruction tuning [] with two stages, achieving remarkable results in medical Visual Question Answering (VQA) tasks.
Similarly, Med-Gemini [] employs a long-form question answering dataset to enhance the multimodal and long-context capabilities of baseline Gemini [].
Although these models have achieved remarkable performance, they are still limited by the scale of training data.
Prior research [] has shown that scaling up the training data improves the performance of large multimodal foundation models.
In this paper, we aim to build a large-scale medical dataset to facilitate the development of more powerful medical multimodal foundation models.

Multimodal Datasets for medicine.
The significance of construting comprehensive medical multimodal datasets has garnered considerable attention [, , , ].
Several works attempt to collect images and paired clinical reports prepared by pathology specialist [, , ], which provide comprehensive descriptions of images, including disease types and corresponding reasoning.
For example, MIMIC-CXR[] comprises 227,835 images for 65,379 patients, containing pathological findings and impressions in reports paired with each images.
However, manually constructing such reports is both time-consuming and expensive, thereby limiting the scale of these datasets.
PMC-OA [] aims to expand the dataset scale by extracting a large number of image-caption pairs from medical papers, increasing the number of data samples to 1.65 million. However, the extracted captions are less detailed compared to manual clinical reports, resulting in a lack of multigranular annotations. RadGenome-Chest CT [] includes more detailed annotations, such as segmentation masks and medical reports generated by MLLMs. Nonetheless, its construction method still relies on paired image-text data, which limits its scalability.
Unlike these existing methods, we devise the first automated data construction pipeline to generate multigranular annotations for unpaired images, achieving a comprehensive multigranular dataset with 25 million data samples.

## 3 MedTrinity-25M Dataset

Figure: (a) Qualitative Comparison with sample in radiology report of chest x-rays dataset MIMIC-CXR [].
Refer to caption: x1.png

### 3.1 Data Triplet

Our dataset comprises triplets of $\{\texttt{image},\texttt{ROI},\texttt{description}\}$. Each ROI is associated with an abnormality and is represented by a bounding box or a segmentation mask, specifying the relevant region within the image. For each image, we provide a multigranular textual description, which includes the disease/lesion type, modality, region-specific description, and inter-regional relationships as illustrated in Figure 2.

##### Images.

We use the original medical image in the source dataset, we extensively collected medical datasets from the following sources: (1) online resources such as [TCIA](https://www.cancerimagingarchive.net/), [Kaggle](https://www.kaggle.com/), [Zenodo](https://zenodo.org/), [Synapse](https://www.synapse.org/), [Hugging Face](https://huggingface.co/),[Grand Challenge](https://grand-challenge.org/) , [GitHub](https://github.com/), etc. (2) relevant medical dataset research, such as CheXpert [] and DeepLesion []. These datasets were first categorized into two types: (1) datasets containing local annotations, such as MIMIC-CXR [] with corresponding radiology reports, and PMC-OA [] with corresponding captions, where the reports or captions provide analysis of specific local conditions in the images; another example is the 3D image segmentation dataset BraTS2024 [], which marks the tumor regions in CT scans with masks. (2) datasets containing global annotations: such as image classification datasets ISIC2019 [] and ISIC2020 [], whose classification labels reflect the overall pathological condition of tissue sections; another example is the CheXpert [] dataset, which provides detailed classification of disease types for each chest X-ray. We collect 25,001,668 samples spanning 10 modalities and
over 65 diseases. For 3D volumetric
images stored in DICOM or NIfTI formats, we converted
each 2D slice to PNG format.
Additional caption and annotations like masks and bounding boxes from these
datasets were utilized to construct ROIs and corresponding textual descriptions as below.

##### ROIs.

For each image, ROIs are highlighted using segmentation masks or bounding boxes. These ROIs mostly contain pathological findings such as lesions, inflammation, neoplasms, infections, or other potential abnormalities.
In the few cases without abnormalities, the ROIs generally indicate the primary object or organ in the image, as shown in examples in the supplementary material.

##### Textual Descriptions.

The textual descriptions for each image are provided with detailed information across various aspects. Unlike the unstructured free-text descriptions found in previous medical report datasets[, , ] or simple short sentences in visual QA dataset[, ] and caption dataset[, ], our textual descriptions are multigranular and structured. General attributes related to the image are described first, including the image modality, the specific organ depicted, and the type of disease presented. Subsequently, ROI-related information is provided, including their locations and the abnormal characteristics within them that indicate underlying pathology, such as distinctive color and texture. Additionally, comparisons between the ROIs and surrounding regions are presented to highlight differences in features and the extent of disease progression.

We also demonstrate the multigranular textual descriptions in our dataset with those in other common forms. As illustrated in  Figure 1, our textual description is multigranular with more attributes than radiology report of chest x-rays dataset MIMIC-CXR [], visual QA dataset SLAKE[] and radiology objects caption dataset ROCO[].

Figure: Figure 2: Data construction pipeline. 1) Data processing: extracting essential information from collected data, including metadata integration to generate coarse caption, ROI locating, and medical knowledge collection. 2) Multigranular textual description generation: using this information to prompt MLLMs to generate fine-grained captions.
Refer to caption: x4.png

Figure: Figure 3: A qualitative comparison example of generated textual description with and without coarse caption. Without a coarse caption, MLLMs fails to detect diseases. On the contrary, providing a caption mentioning “COVID-19” allows MLLMs to identify and categorize the disease, facilitating further analysis.
Refer to caption: x5.png

Figure: Figure 4: A qualitative comparison example of generated textual description with and without locating ROIs. Without ROIs, the caption offers only a brief global analysis; with ROIs, MLLMs conducts detailed local analysis and assesses the impact of lesion ROIs on adjacent normal regions.
Refer to caption: x6.png

Figure: Figure 5: A qualitative comparison example of generated textual description with and without external medical knowledge. MLLMs can standardize medical terminology in its expressions and refine its diagnosis based on disease progressions detailed in medical literature.
Refer to caption: x7.png

### 3.2 Data Construction Pipeline

Given a medical image, we aim to generate corresponding multigranular visual and texual annotations by leveraging MLLMs.
Specifically, as shown in Figure 2, our pipeline can be decomposed into two stages - Data Processing and Generation of Multigranular Text Description.
In the Data Processing stage (Section 3.2.1), we address the lack of domain-specific knowledge in general-purpose MLLMs by leveraging expert grounding models and retrieval-augmented generation (RAG). This stage includes three key steps:
1) Metadata Integration to produce coarse captions encapsulating fundamental image information such as modality and disease types;
2) ROI Locating to identify regions of abnormalities;
and 3) Medical Knowledge Retrieval to extract relevant fine-grained medical details.
Based on the processed data, we then prompt MLLMs to generate multigranular text descriptions, resulting in the creation of fine-grained captions, as detailed in Section 3.2.2.

#### 3.2.1 Data Processing

##### Coarse Caption Generation via Metadata Integration.

We aim to generate coarse captions that provide fundamental information for a given image, including modality, organ labels, disease types, and optionally, camera views and equipment information.
Instead of extracting features directly from the images, we generate these captions by integrating dataset metadata. We first extract metadata from the datasets and then apply a fixed rule to integrate this information into coarse captions.
For example, for an image from the QaTa-COV19 dataset(^1^11[https://www.kaggle.com/aysendegerli/qatacov19-dataset.](https://www.kaggle.com/aysendegerli/qatacov19-dataset.)), we derive metadata from the dataset’s accompanying paper or documentation, indicating that it consists of COVID-19 chest X-ray images.
Next, we construct coarse captions like “A chest X-ray image with COVID-19 in the lungs” highlighting the modality, organ types, and disease labels.
If the image contains additional textual information like radiological findings, this is also integrated to enhance the richness of the caption.
The effectiveness of adding coarse captions when generating fine-grained captions is illustrated in Figure 3.
In contrast to the scenario without a coarse caption where MLLMs fails to recognize the disease, providing MLLMs with a coarse caption that includes the disease type “COVID-19” enables it to identify and categorize the disease, thereby laying the foundation for further analysis.

##### ROI Locating.

We employ various strategies to locate Regions of Interest (ROIs) in images. For datasets that already include localization annotations, such as segmentation masks or bounding boxes, we derive the ROIs from these existing annotations. Specifically, bounding boxes are directly used as the ROIs, while segmentation masks are converted to ROIs by creating the smallest bounding box that covers the mask. When such localization annotations are not available, we apply different pretrained expert models listed in the Appendix to generate ROIs. For text-prompt driven grounding model[], we use disease and organ information in coarse captions as text prompts to guide the model in segmenting specific parts.
Examples of generated ROIs from various modalities with different models are demonstrated in Figure 6. It is important to note that for modalities such as X-ray and MRI scans viewed from the z-axis, our ROI localization employs a coordinate system relative to the human body, resulting in a left-right reversal in the image representation.

Without ROIs, the original description is limited to a brief global analysis of the image. However, with ROIs, MLLMs  can perform a more detailed local analysis of the ROIs and assess the impact of lesion ROIs on the surrounding normal regions, as demonstrated in Figure 4.

Figure: (a) Example of locating ROI via SAT[].
Refer to caption: x8.png

##### Medical Knowledge Retrieval.

General-purpose MLLMs often produce content that lacks specialized medical terminology and professional expression.
To address this issue,
we build a medical knowledge database following the approach in MedRAG []. We collect three main corpora: PubMed(^2^22[https://pubmed.ncbi.nlm.nih.gov/](https://pubmed.ncbi.nlm.nih.gov/)) for biomedical knowledge, StatPearls(^3^33[https://www.statpearls.com/](https://www.statpearls.com/)) for clinical decision support, and medical textbooks [] for domain-specific knowledge.
We segment these corpora into short snippets and encode them into high-dimensional vectors using the text encoder from Med-CPT []. These vectors are then indexed into a specialized vector knowledge base using Faiss[], optimized for efficient retrieval.

For a given image, we retrieve relevant medical knowledge by using its coarse caption, which is generated through metadata integration. Specifically, we encode the coarse captions, including disease and organ classifications, into vectors using the Med-CPT text encoder. We then perform a vector similarity search in the medical vector database, retrieving the top eight medical knowledge snippets that semantically match the query. These snippets provide the external medical knowledge paired with the image.
A qualitative example demonstrating the effectiveness of incorporating external medical knowledge is shown in Figure 7. With access to COVID-19-related medical knowledge, MLLMs can standardize medical terminology and refine diagnoses based on the disease progressions outlined in medical literature.

Figure: Figure 7: An example of the Top-8 retrieval results. By leveraging COVID-19-related medical knowledge, MLLMs can standardize medical terminology and enhance diagnoses according to the disease progressions described in medical literature.
Refer to caption: x11.png

#### 3.2.2 Generation of Multigranular Text Description

After data processing, a comprehensive prompt is utilized to guide the MLLMs in generating multi-granular descriptions.
The prompt template consists of a three-level hierarchical framework with questions to instruct MLLMs:
(1) a global description that captures all details of the image
(2) a local-focused analysis of specific ROIs that potentially are unusual;
and (3) a local-global examination of the interaction between local and global attributes to understand the impact of local abnormalities on the entire organ.
Detailed prompt template is presented in supplementary materials.

To ensure that the MLLMs are guided by relevant medical information not inherently present in their training data, we incorporate the processed data (coarse captions, ROIs, and retrieved medical knowledge) into the prompts.
Specifically, for global information, coarse captions are directly integrated into the prompt. For local information, ROIs on images are converted into textual descriptions based on their coordinates and area ratio within the images. Examples of these textual descriptions are shown in Figure 6, using terms such as “left-center” and “area ratio: 1.2%”.

To refine terminology and diagnosis within ROIs, relevant medical knowledge about specific diseases is incorporated into the prompt. Instead of merely inserting this knowledge, we instruct MLLMs to identify and align the relevant knowledge to ROIs that require analysis.

##### Choice of MLLMs

We first prompt GPT-4V with the provided medical coarse captions, ROIs, and medical knowledge to generate a subset of 200,000 samples, maintaining a similar modality and organ distribution to our full 25 million dataset.
The goal of curating this subset is to calibrate a medical knowledge-guided MLLM to adhere to the formatting instructions specified for our text. Subsequently, we employ our model, LLaVA-Med Captioner, which is based on LLAVA-Med [], the state-of-the-art medical MLLM. To further improve this model, we leverage the latest LLaMA3[] to enhance its linguistic capabilities, and incorporate multi-scale feature extraction [] to improve its vision capabilities.
LLaVA-Med Captioner undergoes continuous training on medical multimodal data and is fine-tuned using our multigranular annotations, resulting in a specialized medical model.

After fine-tuning, we then use this specialized model to generate the multigranular text descriptions on our entire dataset, resulting in 25 million image-ROI-description triplets. The fine-tuning process leverages the advanced language organization capabilities of GPT-4V, providing an effective template for fine-grained captions, which our model uses to learn the formatting of fine-grained captions. As a result, our model generates more detailed descriptions compared to GPT-4V, as illustrated in Figure 8. We also show a detailed quantitative comparison in appendix B in the supplementary material.

Figure: Figure 8: Qualitative Comparison with sample generated by GPT-4V. Compared to GPT-4V, our model generate more detailed caption.
Refer to caption: x12.png

Figure: (a) Modality distribution in MedTrinity-25M.
Refer to caption: x13.png

### 3.3 Dataset Analysis

##### Diversity

Our dataset encompasses a wide range of 10 imaging modalties, with more than 65 diseases across various anatomical structures in human. The distribution of Anatomical and biological structures in MedTrinity-25M is shown in Figure 9(b). Meanwhile, the number of samples in the dataset for each modality are shown in Figure 9(a), spanning from common ones with over 1 million samples each (CT, MRI, X-ray) to rare modalities(ultrasound, dermoscopy), demonstrating a much more balanced distribution compared to other large-scale dataset like SA-Med2D-20M[], which only contain thousands of ultrasound and dermoscopy samples.

##### Scale

Figure 9(c) shows the amount of our dataset, which is significantly larger than previous datasets. To the best of our knowledge, this is the largest open-source, multi-modal multigranular medical dataset to date.

##### Diseases

The datasets involved in constructing MedTrinity-25M primarily focus on disease diagnosis and medical discovery.
In MedTrinity-25M, diseases are given in the free-form text. The same disease may be referred to using different terms, allowing for elaborate identification and analysis. Figure 9(d) illustrates the frequently used words related to diseases in our dataset.

##### Richness

Figure: Table 1: Comparison of dataset types based on provided attributes of annotations.
Refer to caption: x17.png

We provide both quantitative analysis and qualitative examples to show the richness of our generated multigranular compare to other medical dataset. Qualitative examples are shown in Figure 1, our textual description is multigranular with more attributes than radiology report of chest x-rays dataset MIMIC-CXR [], visual QA dataset SLAKE[] and radiology objects caption dataset ROCO[]. To demonstrate the multi-granularity of our data, we compared the average word count of text descriptions in our dataset, MedTrinity-25M, with those in other medical datasets, as illustrated in Figure 10. The word count in our dataset is significantly higher, indicating greater richness.

##### Alignment with human

To evaluate the validity and quality of the generated multigranular annotations, we compared them with their original human annotations to assess the degree of alignment (for samples with human annotations).

Since the generated fine-grained captions  contains structured descriptions that may significantly differ from free-text radiology reports and question-answering pairs, we leveraged GPT-4V’s vision and language understanding capabilities.
Rather than focusing on the exact alignment of sentence structure or organization, GPT-4V assessed the alignment based on the accuracy of medical facts and diagnoses.
Specifically, the structure of the generated fine-grained captions  consists of five key attributes that characterize a medical image: modality, structure detection, ROI analysis, lesion texture, and local-global relation. To evaluate the generated data, we had GPT-4V perform a detailed comparison with human annotations based on these five attributes. Each attribute was scored on a scale from 0 to 2 points, with a maximum possible total score of 10 points.

**Table 2: Comparison of alignment scores between our generated fine-grained captionsand human annotations.**
|  |
| --- |
|  |

Figure: Figure 11: An example of a perfect score result evaluated by GPT-4V. GPT-4V assesses five criteria, each fully aligned with human annotations, resulting in perfect scores.
Refer to caption: x18.png

We conducted an alignment study on SLAKE [] and MIMIC-CXR [], randomly selecting 50 samples to compare with fine-grained captions for evaluating alignment scores against human annotations. As shown in Table 2(b), the alignment scores were 8.2 and 8.9 for SLAKE and MIMIC-CXR, respectively. The criteria of modality, structure detection, and ROI analysis nearly achieved perfect scores, demonstrating the validity and accuracy of the generated data compared to human annotations. An example of perfect alignment score results evaluated by GPT-4V is shown in Figure 11. In these examples, GPT-4V fully aligned with human annotations across all five criteria, resulting in perfect alignment scores.

The prompt used to query GPT-4V for evaluating the alignment score is shown in Figure 14 of supplementary.

## 4 LLaVA-Med++: Experimental Training with MedTrinity-25M

To further demonstrate the validity of our dataset, we compare the performance of LLaVA-Med++ with and without training on our dataset.
We select Visual Question Answering (VQA) as the evaluation task, which requires models to learn detailed visual and language representations. We assessed the performance of our model on three biomedical VQA datasets: VQA-RAD [], SLAKE [] and PathVQA [].

We initially pretrained LLaVA-Med++  using the methodology of LLaVA-Med [] as our baseline. Subsequently, for each VQA dataset evaluation, we further pretrained our model on the corresponding MedTrinity-25M  subset to achieve multigranular alignment. The model was then fine-tuned on VQA datasets for three epochs, with performance results presented in Table 3. A comparative experiment was conducted without pretraining on MedTrinity-25M , maintaining all other settings. Results clearly demonstrate that LLaVA-Med++ achieves state-of-the-art performance in two of the three VQA benchmarks and ranks third in the remaining one.
Pretraining on MedTrinity-25M exhibits performance improvements of approximately 10.75% on VQA-RAD, 6.1% on SLAKE, and 13.25% on PathVQA compared to the model trained without pretraining on it. This enhancement underscores the efficacy of pretraining on MedTrinity-25M for downstream multimodal medical tasks, particularly in visual question answering.

**Table 3: Comparison with Existing Supervised Methods.The notation w/ and w/o indicate models with and without pretraining on MedTrinity-25M, respectively. Employing multigranular alignment pretraining on MedTrinity-25M, LLaVA-Med++achieves state-of-the-art performance in two of the three VQA benchmarks and ranks third in the remaining one. Our model surpasses both multimodal large language models and other representative SoTA approaches.**
|  |  | VQA-RAD | SLAKE | PathVQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method |  | Ref | Open | Closed | Ref | Open | Closed | Ref | Open | Closed |
| Supervised finet-tuning results with our own experiment runs |  |  |  |  |  |  |  |  |  |  |
| GPT-4V [] |  | 39.5 | 78.9 |  | 33.6 | 43.6 |  | - | - |  |
| LLaVA |  | 50.0 | 65.1 |  | 78.2 | 63.2 |  | 7.7 | 63.2 |  |
| LLaVA-Med |  | 55.5 | 66.5 |  | 70.6 | 54.5 |  | 35.9 | 89.2 |  |
| LLaVA-Med++(Ours, w/o) |  | 64.6 | 77.0 |  | 79.3 | 84.0 |  | 55.0 | 94.0 |  |
| LLaVA-Med++(Ours, w/) |  | 77.1 ( + 12.5) | 86.0 ( + 9.0) |  | 86.2 ( + 6.9) | 89.3 ( + 5.3) |  | 66.5 ( + 11.5) | 99.0 ( + 5.0) |  |
| Representative & SoTA methods with numbers reported in the literature |  |  |  |  |  |  |  |  |  |  |
| VL Encoder–Decoder [] | 71.5 |  | 82.5 |  |  |  | 71.5 |  | 85.6 |  |
| Q2ATransformer [] | 79.2 |  | 81.2 |  |  |  | 54.9 |  | 88.9 |  |
| Prefix T. Medical LM [] |  |  |  | 84.3 |  | 82.0 | 40.0 |  | 87.0 |  |
| PubMedCLIP [] | 60.1 |  | 80.0 | 78.4 |  | 82.5 |  |  |  |  |
| BiomedCLIP [] | 67.6 |  | 79.8 | 82.1 |  | 89.7 |  |  |  |  |
| M2I2 [] | 66.5 |  | 83.5 | 74.7 |  | 91.1 | 36.3 |  | 88.0 |  |

## 5 Conclusion

This paper introduces MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million image-ROI-description triplets sourced from more than 90 online resources, spanning 10 modalities and covering over 65 diseases. Unlike existing dataset construction methods that rely on image-text pairs, we have developed the first automated pipeline to scale up multimodal data by generating multigranular visual and textual annotations from unpaired image inputs, leveraging expert grounding models, retrieval-augmented generation techniques, and advanced MLLMs. MedTrinity-25M’s enriched annotations have the potential to support a wide range of multimodal tasks, such as captioning, report generation, classification, and segmentation, as well as facilitate the large-scale pre-training of multimodal medical AI models.

## Acknowledgement

We thank the Microsoft Accelerate Foundation Models Research Program, the OpenAI Researcher Access Program, TPU Research Cloud (TRC) program, Google Cloud Research Credits program, AWS Cloud Credit for Research program, and Lambda Cloud for supporting our computing needs.

## Appendix

## Supplementary material

We present the following items in the supplementary material section:

- 1.
Data source about MedTrinity-25M. (Section A)
- 2.
Quantitative comparison between GPT-4V and LLaVA-Med Captioner (Section B).
- 3.
Example of ROI for normal regions (Section C).
- 4.
The list of expert ROI models (Section D).
- 5.
Prompt for evaluating MedTrinity-25M alignment with human annotations (Section E).
- 6.
Prompt for generating MedTrinity-25M. (Section F).
- 7.
A Datasheet [] for MedTrinity-25M (Section G).

## Appendix A Data Source

**Table 4: Data sources for MedTrinity-25M from various medical image datasets, detailing their modalities, biological structures, quantities, and annotations.**
| Dataset Name | Modality | Biological<br>Structures | Quantity | Text | Disease<br>Type | BBox | Mask |
| --- | --- | --- | --- | --- | --- | --- | --- |
| BHX[] | MRI | brain | 973908 | ✗ | ✗ | ✗ | ✓ |
| BRATS24-MICCAI[] | MRI | brain | 2535132 | ✗ | ✗ | ✓ | ✗ |
| BRATS-ISBI[] | MRI | brain | 987340 | ✗ | ✗ | ✓ | ✗ |
| breast histopathology[, ] | Histopathology | breast | 547403 | ✗ | ✓ | ✗ | ✗ |
| BreastCancer[] | Histopathology | breast | 1824 | ✗ | ✗ | ✓ | ✗ |
| CheXpert[] | X-Ray | lung | 183242 | ✗ | ✓ | ✗ | ✗ |
| CISC[] | Histopathology | Adrenal,<br>Bile duct,<br>Bladder,<br>Breast,<br>Colon,<br>Cervix,<br>Esophagus<br>Kidney,<br>Liver,etc | 16285 | ✗ | ✓ | ✓ | ✗ |
| CPD[] | Histopathology | skin | 204 | ✗ | ✗ | ✓ | ✗ |
| CT-RATE[] | CT | lung,<br>liver,<br>mediastinum,<br>kidney,<br>heart, etc. | 3869640 | ✓ | ✗ | ✗ | ✗ |
| DeepLesion[] | CT | bone,<br>abdomen,<br>mediastinum,<br>liver,<br>lung,<br>kidney,<br>soft tissue,<br>pelvis | 2889672 | ✗ | ✗ | ✗ | ✓ |
| FLARE23[] | CT | Liver,<br>kidney,<br>spleen,<br>pancreas,<br>Aorta,<br>adrenal gland,<br>Gallbladder,<br>esophagus,<br>stomach,<br>duodenum,etc. | 13770 | ✗ | ✓ | ✓ | ✗ |
| ihc4bc[] | Microscopy | cell | 102535 | ✗ | ✓ | ✗ | ✗ |
| KIPA22[, , , ] | CT | kidney,<br>cervix | 26878 | ✗ | ✗ | ✓ | ✗ |
| LLaVA-Med[] | CT,<br>MR,<br>Endoscopy,<br>X-Ray,<br>Ultrasound,<br>Histopathology,<br>Dermoscopy,<br>Microscopy,<br>Fundus,<br>PET | cell, rib,<br>tissue,<br>face,<br>brain,<br>vascular,<br>liver,<br>bone,<br>lymph, etc. | 22550 | ✓ | ✗ | ✗ | ✗ |
| LLD-MMRI[] | MRI | liver | 21523 | ✗ | ✗ | ✓ | ✗ |
| MAMA-MIA[] | MRI | breast | 316113 | ✗ | ✗ | ✓ | ✗ |
| MIMIC-CXR-JPG[] | X-Ray | lung | 240506 | ✓ | ✓ | ✗ | ✓ |
| NCT-CRC-HE-100K[] | Histopathology | colon | 100361 | ✗ | ✓ | ✗ | ✗ |
| NIH-CXR[, , ] | X-Ray | lung | 986 | ✗ | ✗ | ✗ | ✓ |
| PadChest[] | CT | lung | 96284 | ✓ | ✗ | ✗ | ✗ |
| PatchGastricADC22[] | MRI | brain | 98399 | ✗ | ✓ | ✗ | ✗ |
| Path-VQA training[] | Pathology | gastrointestinal,<br>colon,<br>appendix,<br>pinworm,etc. | 13375 | ✓ | ✓ | ✗ | ✗ |
| PMC-OA[] | CT,<br>MR,<br>Endoscopy,<br>X-Ray,<br>Ultrasound,<br>Histopathology,<br>Dermoscopy,<br>Microscopy,<br>Fundus,<br>PET | cell,<br>tissue,<br>vascular,<br>brain,<br>bone,<br>liver,<br>lymph,<br>eye,<br>epithelium, etc. | 856999 | ✓ | ✗ | ✗ | ✗ |
| PMC-VQA[] | CT,<br>MR,<br>Endoscopy,<br>X-Ray,<br>Ultrasound,<br>Histopathology,<br>Dermoscopy,<br>Microscopy,<br>Fundus,<br>PET | cell,<br>brain,<br>tissue,<br>artery,<br>bone,<br>face,<br>rib,<br>vascular,<br>liver,<br>eye, etc. | 144999 | ✓ | ✗ | ✗ | ✗ |
| PTCGA[] | Histopathology | brain,<br>breast,<br>uterine corpus,<br>kidney,<br>lung,<br>thyroid | 3293965 | ✗ | ✓ | ✓ | ✗ |
| Quilt-1M[] | Histopathology | skin,<br>lung,<br>soft tissue,<br>blood,<br>kidney,<br>bone, etc. | 643819 | ✓ | ✗ | ✗ | ✗ |
| SAMMed-20M[] | X-Ray,<br>PET,<br>CT,<br>MR,<br>Endoscopy,<br>dermoscopy | brain,<br>kidney,<br>liver,<br>lung,<br>pancreas,<br>pulmonary,<br>hepatic,<br>skin, etc. | 5491274 | ✗ | ✓ | ✓ | ✗ |
| SLAKE training[] | CT,<br>MRI,<br>X-Ray | brain,<br>liver,<br>kidney,<br>pelvic,<br>lung | 646 | ✓ | ✓ | ✓ | ✗ |
| TCGA[] | Histopathology | tissue | 1142221 | ✗ | ✗ | ✓ | ✗ |
| ULS23 | CT | lung,<br>lymph nodes,<br>bladder,<br>brain,<br>colon,<br>kidney,<br>lung,<br>pancreas. | 105669 | ✗ | ✗ | ✓ | ✗ |
| VALSET[] | Histopathology | oesophagus,<br>stomach | 277565 | ✗ | ✓ | ✗ | ✗ |
| VQA-RAD training[] | X-Ray,<br>MRI | brain,<br>lung,<br>abdomen,etc. | 1758 | ✓ | ✓ | ✗ | ✗ |
| Total |  |  | 25016845 |  |  |  |  |

## Appendix B Quantitative Comparison of LLaVA-Med++ with GPT-4V

As detailed in Section 3.2.2 of the main paper, we developed an enhanced version of LLaVA-Med [], called LLaVA-Med++. This enhancement leverages the latest LLaMA3 [] to boost linguistic capabilities and incorporates multi-scale feature extraction [] to improve vision capabilities.

To justify the selection of our specialized medical model, LLaVA-Med++, over GPT-4V for generating textual descriptions, we conducted a quantitative comparison of the outputs generated by both models. We assessed the level of detail by comparing the average word count of text descriptions generated for the same sample. As shown in Figure 12, LLaVA-Med++, after task-specific fine-tuning, outperformed GPT-4V by 3.6% in word count, indicating that the descriptions generated by LLaVA-Med++ are more detailed.
Based on these findings, we selected LLaVA-Med++  to generate fine-grained captions  for our entire MedTrinity-25M.

Figure: Figure 12: Qualitative comparison of the relative average word count of samples generated by LLaVA-Med++ and GPT-4V.
Refer to caption: extracted/5775656/figures_file/length_gpt.png

## Appendix C Examples of ROIs for Normal Regions

As detailed in Section 3.1 of the main paper, the regions of interest (ROIs) identified using expert grounding models predominantly contain pathological findings such as lesions, inflammation, neoplasms, infections, or other potential abnormalities. In the few instances where no abnormalities are present, the ROIs typically highlight the primary object or organ in the image. Examples of ROIs without abnormalities are shown in Figure 13(b).

Figure: Figure 13: Examples of ROIs for normal regions.
Refer to caption: extracted/5775656/figures_file/normal_roi.png

## Appendix D List of Expert models to locate ROIs

**Table 5: List of expert models used to generate ROIs for different datasets.**
| ID | Dataset Name | Model |
| --- | --- | --- |
| 1 | breast histopathology | HoverNet [] |
| 2 | BreastCancer |  |
| 3 | CISC |  |
| 4 | CPD |  |
| 5 | NCT-CRC-HE-100K |  |
| 6 | PTCGA |  |
| 7 | TCGA |  |
| 8 | VALSET |  |
| 9 | ihc4bc |  |
| 10 | Quilt-1M |  |
| 11 | CT-RATE | SAT [] |
| 12 | PMC-OA | DINO [] |
| 13 | PMC-VQA |  |
| 14 | LLaVA-Med |  |
| 15 | Path-VQA training |  |
| 16 | PadChest | CheXmask [] [] |
| 17 | MIMIC-CXR-JPG |  |
| 18 | CheXpert |  |

As detailed in Section 3.2.1 of the main paper, for datasets lacking localization information such as segmentation masks and bounding boxes, we employ various pretrained expert models to identify the ROIs. The specific expert models used for each dataset are listed in Table 5.

## Appendix E Evaluation Prompt of Alignment to Human Annotations

The prompt used to query GPT-4V for evaluating the alignment score is shown in Figure 14.

Figure: Figure 14: Prompt used to evaluate the alignment of generated fine-grained captions.
Refer to caption: x19.png

## Appendix F Prompt Template for Generation of Multigranular Text Description

[Uncaptioned image]: x20.png

To generate multigranular textual descriptions, we design a multi-task prompting approach, breaking down this task into several smaller descriptive tasks. The model’s responses to these different tasks collectively form the final fine-grained text description.

appendix F illustrates our prompt template consisting of a three-level hierarchical framework with questions to instruct MLLMs:

Step 1 - Global Understanding: Instruct MLLMs to provide a comprehensive description of the image, detailing all modalities, identified anatomical structures, and their approximate locations. This step ensures that MLLMs gains an overarching understanding and basic information about the image.

Step 2 - Local Analysis: Instruct MLLMs to conduct a detailed analysis of the regions of interest (ROI), including their locations, abnormalities, and textures. This step guides MLLMs to focus on specific lesions for a thorough assessment.

Step 3 - Local-Global Relationship: Instruct MLLMs to examine the relationship between local and global regions and predict how the surrounding areas will be affected by the lesions in the ROI. This step aims to understand the interaction between local and global attributes, assessing the impact of local abnormalities on the entire organ for accurate disease diagnosis.

## Appendix G Datasheet for MedTrinity-25M

In this section, we present a DataSheet [] for MedTrinity-25M, synthesizing many of the other analyses we performed in this paper.

- 1.
Motivation For Datasheet Creation
•
Why was the dataset created?
The dataset was created to provide a large-scale, multimodal, multigranular medical dataset to support a wide range of multimodal tasks such as captioning, report generation, classification, and segmentation. It aims to facilitate large-scale pre-training of multimodal medical AI models by providing enriched annotations from unpaired image inputs.
•
Has the dataset been used already?
Yes. Multigranular annotations enable a wide range of tasks like Medical Visual Question Answering, which we discuss in Section 4.
•
What (other) tasks could the dataset be used for?
The MedTrinity-25M dataset could be used for multiple medical imaging tasks such as classification, segmentation, detection, and medical report generation. Its extensive and detailed annotations make it suitable for training and evaluating machine learning models across these tasks.
•
Who funded dataset creation?
This work is partially supported by the OpenAI Researcher Access Program, AWS Cloud Credit for Research Program, TPU Research Cloud (TRC) program and Google Cloud Research Credits program.
- 2.
Data composition
•
What are the instances?
Each instance in the dataset is a triplet consisting of an image, a Region of Interest (ROI), and a multigranular textual description. The ROI is associated with abnormalities and represented by bounding boxes or segmentation masks.
•
How many instances are there?
The dataset comprises over 25 million image-ROI-description triplets sourced from more than 90 online resources, spanning 10 modalities and covering over 65 diseases.
•
What data does each instance consist of?
Each instance consists of a medical image, a corresponding ROI (highlighting abnormalities within the image), and a detailed, multigranular textual description that includes disease/lesion type, modality, region-specific description, and inter-regional relationships.
•
Is there a label or target associated with each instance?
Yes, the textual description serves as a detailed label or target, providing information about the disease or lesion type, as well as other relevant medical details.
•
Is any information missing from individual instances?
No.
•
Are relationships between individual instances made explicit?
Not applicable – we do not study relationships between disparate medical samples.
•
Does the dataset contain all possible instances or is it a sample?
Our generation pipeline includes all instances collected from available medical data sources. However, the current list of medical dataset sources is not exhaustive, indicating a high probability of collecting additional instances in the future.
•
Are there recommended data splits (e.g., training, development/validation, testing)?
There are no recommended data splits, as this data was curated mainly for pretraining rather than evaluation.
•
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
Yes. Despite multiple efforts to minimize errors using coarse captions and external medical knowledge, the textual descriptions generated by MLLMsmay still contain inaccuracies.
•
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
The dataset is largely self-contained. However, it was constructed using data from over 90 online resources such as TCIA, Kaggle, Zenodo, and Synapse. The images and related data were collected from these sources, but the dataset itself does not rely on external resources like websites or tweets for its primary functionality once compiled.
- 3.
Collection Process
•
What mechanisms or procedures were used to collect the data?
The data collection involved an automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations from unpaired images. Data was collected from over 90 different sources, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions.
•
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data?
The data associated with each instance was indirectly inferred and derived from the collected images using domain-specific expert models and multimodal large language models (MLLMs). The images were annotated with bounding boxes, segmentation masks, and textual descriptions, transforming them into image-ROI-description triplets.
•
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
The dataset is not a sample from a larger set but an extensive collection aggregated from multiple datasets and online sources. The strategy was to include as many diverse images and annotations as possible from a wide range of medical datasets.
•
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
Data collection was primarily done by the co-authors of this paper.
•
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
The data was collected from April 2024 to June 2024.
- 4.
Data Preprocessing
•
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
Extensive preprocessing and annotation were performed, including segmentation, bounding box creation, and generating multigranular textual descriptions. The preprocessing also involved integrating metadata and knowledge retrieval from sources like PubMed to create comprehensive descriptions.
•
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the ‘raw’ data.
The raw data was saved, but at this time we do not plan to release it directly due to copyright and privacy concerns.
•
Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.
The software for preprocessing and labeling, including the automated pipeline and MLLMs, is available at  [https://github.com/yunfeixie233/DataProcessingSystem](https://github.com/yunfeixie233/DataProcessingSystem).
•
Does this dataset collection/processing procedure achieve the motivation for creating the dataset stated in the first section of this datasheet? If not, what are the limitations?
Yes. The preprocessing and collection procedures align with the motivation of creating a comprehensive, large-scale multimodal dataset to support the development of advanced medical AI models. The dataset’s multigranular annotations enable a wide range of tasks like Medical Visual Question Answering, which we discuss in Section 4.
- 5.
Dataset Distribution
•
How will the dataset be distributed?
The dataset is publicly available and can be accessed via the provided link: MedTrinity-25M [https://yunfeixie233.github.io/MedTrinity-25M/](https://yunfeixie233.github.io/MedTrinity-25M/).
•
When will the dataset be released/first distributed? What license (if any) is it distributed under?
We will release it as soon as possible, using a permissible license for research-based use.
•
Are there any copyrights on the data?
We believe our use is ‘fair use,’ however, due to an abundance of caution, we will not be releasing any of the videos themselves.
•
Are there any fees or access restrictions?
No.
- 6.
Dataset Maintenance
•
Who is supporting/hosting/maintaining the dataset?
The first authors of this paper.
•
Will the dataset be updated? If so, how often and by whom?
We do not plan to update it at this time.
•
Is there a repository to link to any/all papers/systems that use this dataset?
Not right now, but we encourage anyone who uses the dataset to cite our paper so it can be easily found.
•
If others want to extend/augment/build on this dataset, is there a mechanism for them to do so?
Not at this time.
- 7.
Legal and Ethical Considerations
•
Were any ethical review processes conducted (e.g., by an institutional review board)?
No official processes were done, as our research is not on human subjects, however, because the dataset is in the medical domain we had significant internal discussions and deliberations when choosing the scraping strategy.
•
Does the dataset contain data that might be considered confidential?
The dataset does not contain data that might be considered confidential, as it uses publicly available sources and anonymized medical data.
•
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why?
The dataset does not contain data that might be offensive, insulting, threatening, or anxiety-inducing. It consists of medical images and associated annotations for clinical and research use.
•
Does the dataset relate to people?
The dataset relates to people as it involves medical images and data. However, it is anonymized and does not include identifiable information.
•
Does the dataset identify any subpopulations (e.g., by age, gender)?
Not explicitly (e.g. through labels)
•
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?
The dataset does not identify specific subpopulations directly in the provided description. Additionally, it is not possible to identify individuals from the dataset as it is anonymized and compiled from various sources.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 2
questions: 1.	The data construction pipeline consists of two main steps: first, generating 200,000 multigranular textual descriptions via GPT-4V, and then fine-tuning LLaVA with this dataset. In my opinion, the performance of the trained model LLaVA-MedCap is significantly influenced by the quality of the data generated by GPT-4V. However, in Table 3, GPT-4V's performance falls far short of LLaVA-Tri. Why did you choose to use GPT-4V to generate the supervised fine-tuning data for the fine-tuning process? In my recent review of the publicly released MedTrinity-25M data, I found that many of the generated data instances were of average quality, with even simple modalities being incorrect.
2.	In Section 4.1, it is stated that “The model is fine-tuned for three epochs on each VQA dataset and evaluated accordingly.” However, this evaluation setup is not fair when comparing with the methods presented in the table. Some results in Table 3 appear to be directly extracted from llava-med’s table, but these methods were not fine-tuned on the training set of the VQA benchmark. In contrast, the accuracy of the chosen llava-med method was achieved after fine-tuning for 15 epochs on the corresponding VQA benchmark training set, making this comparison inappropriate.
Currently, the evaluation paradigm for multimodal large models (MLLM) typically involves assessing the model directly on multiple benchmarks after two stages of pretraining and supervised fine-tuning. Therefore, I suggested distinguishing the results presented: there should be a comparison of the results not fine-tuned on the corresponding training set against the models that were not fine-tuned. Additionally, when comparing with methods fine-tuned on the corresponding training set, the number of fine-tuning epochs (including for the comparison methods) should be clearly indicated. If possible, please also conduct an ablation study regarding the number of fine-tuning epochs.
3.	In Table 3, comparing CLIP-like models with LLaVA, a multimodal large model, seems inappropriate. It would be better to categorize them into two groups: one for CLIP models and another for MLLM.
4.	The evaluation details in Table 3&4 are not clearly explained. What do "open" and "close" specifically refer to?  Please add a brief explanation of the "open" and "close" terms in the table caption or in the text describing the results.
soundness: 2
strengths: 1.	The paper introduces MedTrinity-25M, the largest multimodal medical dataset to date, featuring multigranular annotations and containing over 25 million triplets (image-ROI-description). The development of an automated data annotation pipeline significantly scales up medical image-text data.
2.	With the support of MedTrinity-25M, the pretrained CLIP and LLaVA models demonstrate better performance compared to previous methods.
3.	The release of this dataset contributes to the medical AI community, providing researchers and practitioners with a valuable resource for advancing multimodal tasks and improving healthcare applications.
summary: This paper introduces MedTrinity-25M, a comprehensive and large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with detailed annotations for more than 65 diseases. The dataset provides the most enriched annotations for various multimodal tasks, such as captioning, classification, and segmentation, and supports the pre-training of advanced multimodal medical AI models, achieving state-of-the-art performance.
weaknesses: 1.	The quality of the generated image-text data may not be sufficiently high. It is advisable to review the questions associated with this data. In the image-captioning constructed from MedTrinity-25M, we found that many basic imaging modalities were incorrectly identified. For instance, in the CT-RATE data used as the source, over 60,000 images were misidentified as MRI, more than 90,000 as X-ray, and even a small number were identified as endoscopy. Below, we provide some captions to illustrate this phenomenon present in MedTrinity-25M.
“””
{"image_path":"seg_train_126_a_2-right lung.nii-34.jpg","id":"b573995e-2d11-11ef-bbea-f02f74942466","caption":"The image is a magnetic resonance imaging (MRI) scan of the thoracic region, showing the heart, part of the lung, and the spine. The heart is centrally located with the lung on either side and the spine running vertically in the background. The region of interest, located in the left-center at the bottom of the image, appears to have a different texture and intensity compared to the surrounding tissue, suggesting a possible abnormality. This region's relative position is towards the lower left side of the image, adjacent to the lower part of the lung. The content within this region may indicate a disease process, which could be related to or affecting the adjacent lung tissue. Given the proximity to the lung, it is possible that the abnormality could be influencing or being influenced by the pulmonary structures.","source":"ct_rate"}
{"image_path":"seg_train_126_a_2-right lung.nii-11.jpg","id":"b57345c6-2d11-11ef-a899-f02f74942466","caption":"The image is a magnetic resonance imaging (MRI) scan of the thoracic region, showing the heart, part of the lung, and the spine. The heart is centrally located with the lung on either side and the spine visible posteriorly. The region of interest, located in the lower-middle left-center of the image, shows an area with altered signal intensity, which is indicative of a pathological condition. This area is situated within the lung tissue and is characterized by a texture and signal intensity that differ from the surrounding healthy lung tissue, suggesting the presence of a disease process such as consolidation, infection, or a mass.\n\nThe region of interest may be related to the surrounding lung tissue in that it could represent a localized disease process affecting the lung, potentially leading to or resulting from changes in the adjacent lung parenchyma. Given the nature of MRI imaging and the appearance of the region, it is possible that this area could be associated with a process such as inflammation, demyelination, or a neoplastic growth, which may have implications for the function of the adjacent lung tissue.","source":"ct_rate"}
{"image_path":"seg_train_14684_a_1-covid-19 infection.nii-76.jpg","id":"a77ee972-2d78-11ef-bdc4-f02f74942576","caption":"The image is a radiographic scan, likely a chest X-ray, showing the thoracic region with the trachea and main bronchi appearing open, indicating no occlusive pathology. The lungs exhibit minimal emphysematous changes, and there are pleuroparenchymal sequelae changes at the apex of both lungs. Additionally, there are linear atelectasis in the middle lobe of the right lung and the lingular segment of the left lung upper lobe. The heart contour and size are normal, and there are no pleural or pericardial effusions. The mediastinal structures are not optimally evaluated due to the absence of contrast material, and the mediastinal main vascular structures are of normal width. No enlarged lymph nodes or pathological wall thickness increase are observed in the esophagus. The thoracic vertebral corpus shows normal heights, alignment, and densities, with osteophytes at the vertebral corpus corners, and the neural foramina are open.\n\nThe region of interest, located centrally and in the upper-middle area of the image, occupying approximately 0.3% of the area, corresponds to the lung apices where pleuroparenchymal sequelae changes are noted. These changes are characterized by alterations in lung parenchyma texture, which may appear as irregularities or areas of increased density compared to the surrounding lung tissue.\n\nThe pleuroparenchymal sequelae changes in the lung apices may be related to the emphysematous changes in the lungs, as both conditions can result from chronic inflammatory processes, suggesting a possible pathophysiological connection between the two findings.","source":"ct_rate"}
{"image_path":"seg_train_9381_a_2-covid-19 infection.nii-145.jpg","id":"9880af92-2c1e-11ef-b9dd-f02f74942466","caption":"The image is a chest X-ray showing a cross-sectional view of the thoracic cavity, including the lungs, heart, and part of the spine. A region of interest is located at the periphery of the thoracic cavity, likely within the lung tissue, which appears as a darker area compared to the surrounding lung parenchyma. The region of interest, which is abnormal, shows an area of increased opacity that suggests the presence of a pathological condition, possibly a mass or lesion within the lung tissue. This abnormal area is indicative of a disease process, which could be related to the surrounding lung tissue either as a primary pathology or as a secondary effect of a systemic condition affecting the lung. The abnormality's proximity to other structures within the thoracic cavity, such as the pleura or lung tissue could imply a relationship where the disease process in the region of interest may have originated from or is affecting adjacent areas, potentially impacting nearby structures due to its location within the thoracic cavity.","source":"ct_rate"}
{"image_path":"seg_train_7973_a_1-right lung.nii-84.jpg","id":"88af61aa-2d11-11ef-959e-f02f74942466","caption":"The image is a sagittal section of an endoscopic view of the thoracic region, showing the spine, part of the lung, and the surrounding thoracic structures. A region of interest is located at the lower-middle part of the image, horizontally left-center, occupying approximately 0.6% of the area. The region of interest, located in the lower-middle left-center of the image, displays an abnormality in the lung tissue, which appears to be a small, localized area with a different texture and possibly altered density compared to the surrounding lung parenchyma, suggesting a pathological change. This abnormality could be related to the adjacent lung tissue, potentially indicating a localized disease process or lesion that may be affecting or resulting from the surrounding lung tissue, given the proximity and the nature of the disease knowledge provided.","source":"ct_rate"}
“””
Moreover, there are also issues with modality misidentification in the image-captioning derived from the quilt-1m dataset. For example, pathology images were misidentified as X-ray and MRI, as follow, 
“””
{"image_path":"G-tdJ0oZxJ4_image_e74e3372-a40e-40e3-ac06-e3d53eeab845.jpg","id":"f5ce616b-89ca-41f6-b820-b480bb3327af","caption":"The image is a magnetic resonance imaging (MRI) scan of the lung, showing a cross-sectional view of the thoracic cavity. A region of interest is located at the top of the image, which appears to be in the upper part of the lung field, likely within the upper lobe, given the position of the lung's anatomy. The region of interest, which is abnormal, exhibits an unusual appearance compared to the surrounding lung tissue, possibly indicating a neoplastic process. This abnormality is characterized by a difference in texture and density, which may suggest a pathological change, such as a mass or lesion within the lung tissue. The affected area's relationship to the rest of the lung tissue could imply a localized pathological process that may be the primary site of the disease, potentially impacting or being impacted by adjacent lung structures due to its proximity to other regions, although the exact relationship depends on the nature of the pathology and its progression.","source":"quilt_1m"}
{"image_path":"895313503048712192_0.jpg","id":"0b53a8c8-5661-449a-b39a-f92887fceb86","caption":"The image is a magnetic resonance imaging (MRI) scan of the brain, showing a cross-sectional view that includes brain tissue with various shades of gray indicating different structures and densities. A region of interest is located at left-center part of the image, which appears to be in the cerebral hemisphere, likely within the white matter of the brain. The region of interest exhibits an abnormality that differs in texture and possibly size from the surrounding brain tissue, suggesting the presence of a pathological change. This abnormal area could be related to the surrounding brain structures, potentially affecting or being affected by them due to its proximity and the nature of brain tissue, which may indicate a pathological process such as a tumor, edema, or other brain abnormalities. The abnormality's characteristics, such as altered signal intensity, could be indicative of the disease process affecting the brain tissue's function or structure.","source":"quilt_1m"} 
{"image_path":"962043872649011205_0.jpg","id":"f94e8ba4-7ddf-450e-9616-d4681e9dcf02","caption":"The image is a chest X-ray image of a 15-year-old boy's hand, showing the left side of the image is a chest X-ray showing the hand's anatomy, including the bones of the hand, with the focus on the epiphysis and possibly the bones of the hand's proximal and distal parts such as the clavicles, ribs, clavicles, and parts of the spine, which are essential for various activities like running, throwing, and punching, along with the presence of the epiphyseal plate and the development of the trapezius, which are crucial for a 15-year-old boy's hand, suggesting a focus on the skeletal structure and function of the hand, including the development of the hand's bones such as the clavicles and the development of the wrist bones, which are crucial for various activities like walking, running, and running, as well as the development of the wrist bones of the hand, which are essential for a growing hand and are indicative of a developing hand and are crucial for a growing hand and are typically found in activities such as playing with play and sports, suggesting a comprehensive evaluation of the hand's anatomy and function.","source":"quilt_1m"}
{"image_path":"1301506707483439105_1.jpg","id":"7b0161c9-1451-4f27-9278-aff87726bacb","caption":"The image is a lateral chest X-ray image of a 15-year-old boy's hand, showcasing the epiphysis of the wrist bones, which are the bones of the hand, including the radius of the wrist bones, clavicles and the bones of the hand's growth plate are visible, such as the clavicles, ribs, and vertebrae are the primary focus of the image is a close-up view of a lateral chest X-ray image, showcasing the epiphysis of the hand's anatomy, which are essential for various stages of development and growth patterns. The image is a detailed X-ray of the hand's anatomy, displaying the epiphyseal growth plate and development of the wrist bones of a 15-year-old boy's hand, with the bones of the epiphyseal plate visible in the image are the primary organ of interest, which includes the epiphyseal plate and epiphyseal plate, as well as the epiphyseal plate's development is crucial for diagnosing the stage of developmental stage of the wrist bones, typically found in the wrist bones of a 15-year-old boy's hand, which is crucial for understanding the zone of developmental development, and the image is likely to be a lateral chest X-ray image showing the epiphyseal plate's development is crucial for diagnosing the stage of developmental stage of the wrist, with the epiphyseal plate being the primary focus of the image.","source":"quilt_1m"}.
2.	The experiments lack comprehensiveness. The comparison of multimodal large models is limited to VQA-RAD, SLAKE, and PathVQA. A broader range of specialized benchmarks, such as the health subset of MMMU, could provide a more robust comparison of the multimodal large models’ performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Given the reliance on automated processes and external knowledge sources, how is labeling consistency ensured in the data generation process? Additionally, has it been validated by human experts?
2. Has a comparison with expert-labeled datasets been considered to further quantify the quality of automated labeling?
3. How does the dataset address potential biases in source data? For example, is there a mechanism to prevent overrepresentation of certain demographics?
4. What is the expandability of the labeling process for new, unrepresented modalities or diseases? Does it offer strong scalability?
5. Although LLaVA-Tri achieved good performance on RAD, PathVQA, and SLAKE, how do the authors ensure that MedTrinity-25M does not contain data from these datasets? If these datasets were included, the model may have seen the questions and answers during training, leading to artificially high accuracy.
soundness: 3
strengths: 1. The dataset uses an automated process with medical knowledge retrieval and domain-specific models, greatly reducing manual labeling needs.
2. With over 25 million image-ROI-description triplets, the dataset supports classification, segmentation, report generation, and spans multiple medical domains.
3. The LLaVA-Tri model, pre-trained on MedTrinity-25M, performs exceptionally well across multiple benchmarks, showcasing the dataset’s potential to enhance medical AI applications.
summary: The paper presents MedTrinity-25M, a large-scale multimodal medical dataset with 25 million images across 10 modalities and detailed annotations for over 65 diseases. Enhanced by automated multigranular annotations, it supports tasks like image captioning and report generation. The LLaVA-Tri model, pre-trained on MedTrinity-25M, achieved state-of-the-art results on multiple medical VQA benchmarks.
weaknesses: 1. Although the labeling process is automated, its reliance on domain-specific models may limit scalability when handling new modalities or emerging diseases.
2. The accuracy of automated labeling may fall short of expert-labeled datasets, potentially impacting performance in critical medical applications. How can a high standard of automated labeling be ensured?
3. With data from over 30 sources, potential biases in image quality, demographics, or disease distribution call for a deeper integration analysis.
4. Performance depends on external medical knowledge bases, risking inconsistency if not updated, and the LLaVA-Tri comparison may be biased due to overlap with MedTrinity-25M's data sources.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	In the initial step of data processing (coarse caption generation via metadata integration), what approach is taken if the metadata lacks organ or disease labels?
2.	Section 3.2.2 states that all textual descriptions in MedTrinity-25M are generated using LLaVA-Medcap, a model fine-tuned on data generated by GPT-4V and the proposed pipeline. Why was GPT-4V specifically chosen? As shown in Table 3, several other medical VLMs, such as the VL Encoder-Decoder and LLaVA-Med, appear to perform better than GPT-4V.
3.	For images without bounding boxes or masks from the original data source, how does the pipeline generate the ROI using expert models? Is ROI generation based on organ or disease labels, and how accurate are the generated ROIs? Additionally, if an image contains multiple ROI regions, how is this managed?
soundness: 3
strengths: 1.	The proposed MedTrinity-25M dataset is the largest multimodal medical dataset to date, substantially increasing the scale of accessible training data for medical vision-language tasks.
2.	MedTrinity-25M includes structured, multigranular annotations for each image, offering a level of detail superior to other medical vision-language datasets.
summary: This paper presents MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million images across 10 imaging modalities with fine-grained annotations for more than 65 diseases. MedTrinity-25M is developed through an automated pipeline introduced by the authors. Leveraging this dataset, the authors pre-train a vision-language model, named LLaVA-Tri, which achieves superior performance on three MedVQA benchmarks compared to other vision-language models.
weaknesses: 1.	The multigranular text descriptions in MedTrinity-25M are generated by the proposed automated pipeline, raising concerns about the accuracy of the generated text. As indicated in Table 2, an expert-based evaluation of 200 random samples resulted in an accuracy score of 85%.
2.	Several key details regarding the automated pipeline remain insufficiently discussed. Specific questions are listed under “Questions”.
3.	The pre-trained model LLaVA-Tri is evaluated solely on the MedVQA task alongside other vision-language models, overlooking its potential application to other important medical vision-language tasks, such as visual report generation.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - Handling of Visual Data Without ROIs: For medical images that lack explicit ROI annotations, how does your pipeline construct the corresponding multimodal data pairs? Is there a mechanism to generate or infer ROIs in such cases, or is the pipeline limited to images where ROIs are predefined?
- Assessing the Impact of Multigranular Information on Med-MLLMs: Considering that ROIs and multigranular annotations are central to your dataset, have you conducted experiments to evaluate how these features specifically affect the performance of medical MLLMs? Can you provide insights or results that demonstrate the advantages of including multigranular information in model training?
- Comparison with Existing Multigranular Benchmarks: Could you elaborate on how MedTrinity-25M compares with existing benchmarks that involve multigranular information? What distinguishes your dataset from these, and how does it contribute uniquely to the advancement of general medical AI?
- Evaluation on Multigranular-Specific Tasks: Given that the dataset is evaluated only on VQA-RAD, SLAKE, and PathVQA, which may not fully utilize multigranular annotations, do you have plans to test your dataset on tasks specifically designed for multigranular information? How can you demonstrate the effectiveness of your dataset's detailed annotations in improving model performance on such tasks?
- Addressing Unique Medical Image Presentations in RAG: Medical images often present unique and variable features even when categorized under the same disease label. How does your RAG approach handle the specificity and variability of individual medical image descriptions? For instance, with conditions like Atelectasis that can manifest differently in imaging, how does the system ensure that the generated annotations accurately reflect these variations?
soundness: 3
strengths: - Dataset Comprehensiveness: The dataset is exceptionally extensive, encompassing a wide array of imaging modalities, diseases, and anatomical structures. This diversity significantly enhances its comprehensiveness and utility for various medical AI applications.
- Construction Pipeline through Advanced Models: The utilization of domain-specific expert models and multimodal large language models (MLLMs) for annotation substantially enriches the dataset. This approach adds multigranular details that improve the quality and depth of the annotations, making the dataset more valuable for training sophisticated models.
summary: This paper introduces MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million images across 10 imaging modalities and covering more than 65 diseases. The dataset features multigranular annotations that include both global information and local information. The authors develop an automated pipeline that generates image-ROI-description triplets without the need for paired textual descriptions. This pipeline utilizes domain-specific expert models to identify ROIs and prompts multimodal large language models (MLLMs) to produce detailed annotations through retrieval-augmented generation (RAG).
weaknesses: - Limited Applicability for Visual Data without ROIs: The proposed pipeline relies heavily on regions of interest (ROIs) for constructing multimodal data pairs. For visual data that lack explicit ROI annotations, it is unclear how the pipeline can be effectively applied. This limitation may restrict the dataset's usability across the full spectrum of medical images, particularly those where ROI determination is challenging or subjective.
- Impact of Multigranular Information Not Fully Explored: While Tables 3 and 4 validate the effectiveness of the benchmark in general terms, the paper does not specifically assess how the inclusion of multigranular information, such as ROIs, influences the performance of medical MLLMs. Given that ROI is a pivotal component of the dataset, a focused evaluation comparing models trained with and without multigranular information would strengthen the claims about its benefits.
- Insufficient Discussion of Related Benchmarks: The paper lacks a thorough discussion of existing med-MLLM benchmarks that involve multigranular information. Notably, works like "GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI" and "A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models" are not cited or compared against. Including a comparative analysis would provide context and clarify how MedTrinity-25M advances the field relative to existing resources.
- Lack of Evaluation on Multigranular-Specific Tasks: The dataset is primarily used for training and is evaluated only on VQA-RAD, SLAKE, and PathVQA. These evaluations may not fully capture the effectiveness of the dataset's multigranular annotations since they do not specifically target tasks designed to leverage such detailed information. This omission raises questions about the practical benefits of the multigranular data provided.
- Challenges with Unique Medical Image Descriptions in RAG: The Retrieval-Augmented Generation (RAG) technique used for annotation relies on a general medical knowledge base. However, medical images often have unique features and presentations, even within the same disease category. For example, Atelectasis can manifest differently in chest X-rays depending on the affected lobe. The paper does not address how the RAG system accounts for these unique variations when the initial visual datasets typically offer only generic classification labels. This could impact the accuracy and specificity of the generated annotations.

Meta-review: The paper introduces MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million images across 10 modalities, with multigranular annotations covering more than 65 diseases. The annotations include both global information, such as modality and organ detection, and local details like ROI analysis, lesion texture, and region-wise correlations. The dataset is further enriched by leveraging advanced models to enhance annotation quality and depth, and the authors commit to releasing the data and code, enabling broader advancements in medical imaging foundation models.

Reviewers unanimously appreciated the comprehensiveness of the dataset, its scale, and the automated annotation pipeline, which significantly supports medical pretrained models like CLIP and LLaVA. The combination of diverse modalities, diseases, and anatomical structures makes the dataset a valuable resource for the community.

While several concerns were raised initially, these were thoroughly addressed during the extensive discussion phase. The reviewers reached a unanimous consensus that the concerns had been resolved, and the contributions of this work justify its acceptance. I, therefore, recommend acceptance.

Decision: Accept (Poster)