MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
Authors: Yunfei Xie
Ce Zhou
Lang Gao
Juncheng Wu
Xianhang Li
Hong-Yu Zhou
Sheng Liu
Lei Xing
James Zou
Cihang Xie
Yuyin Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Medical Foundation Model
Multimodal Dataset
Vision-Language Pretraining.
Primary area: datasets and benchmarks

Abstract: This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available. The dataset is publicly available at https://yunfeixie233.github.io/MedTrinity-25M/.

Published as a conference paper ICLR 2025
MEDTRINITY-25M: A LARGE-SCALE MULTIMODAL DATASET WITH MULTIGRANULAR ANNOTATIONS FOR MEDICINE
Yunfei Xie Ce "
Lang Gao
Juncheng Wu
Xianhang Li
Hong-Yu Zhou Sheng Liu Lei Xing
James Zou*
Cihang Xie² Yuyin Zhou
*equal technical contribution
HHuzzhong University of Science and Technology Harvard University
UC Santa Cruz
"Stanford University
ABSTRACT
This paper introduces MedTrinity-25M, a comprehensive. large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets. which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI ROI--eocriptiion triplets without the need for any paired text deseriptions. Specifically, data from over 30 different sources has been collected, preprocessed, and grounded using domain-speciff expert models to iden- tity ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmennted genera- tion with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared with existing datasets, MedTrinity-25A.. provides the most enriched annota- tions, supporting comprehensive range of multimodal tasks such as captioning and re- port generation, well as vision-centric tasks like classification and segmentation. We propose LLaaVA-Tr by pretraining LLaVA on Med'Trinity-25M. achieving state-of-the- performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, Med can also be utilized to support large-scale pre-trate training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. The dataset is publicly available at httphhtt:/// betttp:://yunfiixiiee/ wtt.:e......ttttt. github. oo/MedT ww/.MeeTrnnittt.et www.ddriiiittyyte
1 INTRODUCTION
Large-scale multimodal foundation models (Liu et al.) 2024 Achiam et al.) 2023 Tu et al. 2024b Team jt al.) (2023b Zhou et al. 2024) have demonstrated remarkable success across various domains due to their ability to understand complex visual patterns in conjunction with natural language. This success has sparked significant interest in applying such models to medical vision-language tasks. Much progress has been made in improving the medical capacity of general domain multimodal foundation models by constructing medical datasets with image-text pairs and fine-tuning general domain models on these datasets (Bustos
et al. 2020 Irvin al. 2019 Johnson al. 2019a Li al. 2024a [kezogwo et al. 2024). However,

---

Published as a conference paper ICLR 2025
current medical datasets have several limitations. Firstly, these datasets lack multigranular annotations that reveal the correlation between region-wise wise information within medical images. Medical images often contain detailed cues, such as regional abnormal textures or structures, which may indicate specific types of lesions. Therefore, multimodal models need the ability to infer global information, such as disease or lesion type, from local details. The absence of such data limits the models' capacity to comprehensively understand medical images. Moreover, current dataset construction methods heavily rely on medical images paired with reports or captions from human experts (Ieezogwo et al. 2024 2020a), which restricts their scalability.
Liu et al. 2021 Lau et al., 2018b He et al.,
In this paper, we address the above challenges by proposing an automated data construction pipeline us- ing multimodal large language models (MLLMs) without relying on paired text descriptions. To address the scarcity of medical knowledge within general-purpose MLLMs, we incorporate retrieval-augmented generation (RAG) to source relevant medical knowledge from medical database for MLLMs's reference. To enhance the model's regional focus, we employ an ensemble of domain-specific segmentation mod- els and grounding models to generate regions of interest (ROIs). MLLMs are then prompted to produce multigranular visual and textual annotations, enriched by the retrieved medical knowledge and ROIs. Our proposed pipeline enables the transformation of large-scale images without paired ROIs or text into image- RO1-description triplets. These triplets provide multigranular annotations that encompass both global textual information, such as disease/lesion type, modality, and inter-regional relationships, as well as detailed local annotations for ROIs, including bounding boxes, segmentation masks, and region-specific textual descrip- tions. Using the proposed pipeline, we create a large-scale multimodal multigranular medical dataset con- taining over 25 million triplets, namely MedTrinnit--25M. To the best of our knowledge, this is the largest multimodal dataset in medicine to date.
To demonstrate the effectiveness of our dataset, propose LLaVA-Tri by pretraining LLaVA on ME5 We conduct extensive evaluations across three external medical visual QA datasets representing different sub-pathologies. LLaVA-Tri achieved otate-of-the-art results in all three VQA bench- marks, with 81.6% accuracy on VQA-RAD (Lau et al.) (0188)), 87.8% on SLAKE (Liu et al., 2021). and 82.8% on VathVQA (He al.) 2020a). Moreover, consistent performance improvements are observed when pretraining other multimodal models on MedTrrinity-5MM These findings emphasize the potential of MedTrinity-2555 255M as a foundational dataset that can improve the medical performance of diverse multimodal models.
2 RELATED WORK
Medical Multimodal Foundation Models. Due to the success of multimodal foundation models in com- prehending visual features, their adaptation for medical vision-language tasks has garnered increasing atten- tion (Moor et al. 2023] Tu et al. 2024a Li et al. 2024a Zhou et al. 2024). Several works adapt general multimodal models the medical domain via end mnd-to-end training on medical datasets. For instance, Med- Flamingo (Moor et al., 2023) fine-tunes OpenFlamingo-98 (Awadalla et al. 2023) using 0.8M interleaved and 1.6M paired medical image-text data. LLLVA-MMed (Li et al.) (024a) uses two-stage end-to-end visual instruction tuning (Liu et al. 2024), excelling in medical visual question answering (VQA) tasks. Med- Gemini (Saab et al. (024) adapts Gemini (Team et al., (223aa) using a long-form question-answer dataset to enhance multimodal and long-context capabilities. Despite these achievements, the limited scale of training data remains a challenge. Prior research (Kaplan et al., 2020) shows that increasing training data size im- proves large multimodal model performance. This paper aims to build a large-scale medical dataset to drive the development of stronger medical multimodal foundation models.
Multimodal Datasets for Medicine. The importance of constructing medical multimodal datasets has drawn significant attention (Li et al. [224aa] Pelka et al.) 2018 Zhang et al. 22024] Irvin et al.) 2019). Sev-
eral works focus on collecting images paired with clinical reports from specialists (Zhang al.
2024 Irvin
2

---

Published as a conference paper ICLR 2025
ct al., 2019 Johnson al. 2019a). providing detailed descriptions, including disease types and reasoning. For instance, MIMIC-CXR (Johnson et al., 2019a) contains 227.835 images for 65,379 patients, with cor- responding reports. However, constructing such reports manually is time-consuming and costly, limiting dataset size. PMC-OA (Lin et al. 2023) includes up to 1.65 million image-caption pairs from medical pa- pers but lacks detailed clinical reports. RadGenome-Ches CT (Zhang et al., 2024) offers richer annotations but remains dependent paired image-text data, limiting its scale. In comparison, we introduce the first au- tomated pipeline to generate multigranular annotations for independent images, generating a comprehensive dataset containing 25 million samples.
3
MEDTRINITY-2SM DATASET
3.1 DATA TRIPLET
In this section, we provide details about the data format within MedTrinity-25M Our dataset comprises triplets of {image, ROI, taining both textual description and visual ROI.
For each image, we provide multigranular annotations con-
Images. We gather 25,016,845 samples across 10 medical image modalities and over 65 diseases. Specif- ically, we utilize original medical images from various datasets, extensively collecting from online sources such TCIA Kaggle Zenodo Synapsel HHugging Face, Grand Challenge, GitHub and medical datasets, including CheXpert (Irvin et al., 2019) and DeepLesion Yan et al. 2017a). 3D volumetric images in DI- COM or NIFTI formats are converted to 2D slices in PNG format. The detailed data sources are illustrated in Appendix
We use ROIs to provide visual annotations for each image, primarily focusing on pathological findings such as lesions, inflammation, neoplasms, infections, and other abnormalities. In cases without such abnormalities, the ROIs generally mark the primary object or organ in the image, as illustrated in Figure TO When multiple organs are relevant for disease diagnosis, the ROIs aim cover several regions associated with the disease, providing detailed analysis of each affected area, as shown in Figure
ROIs
The textual descriptions for each image are composed of detailed information across various attributes. In contrast the unstructured medical reports short captions in previous med- ical datasets (Irvin et al. 2019 Johnson et al. 2019a Bustos et al.) 2020 /Zhng et al., [023b] Liu et al.) 2021 Pelka et al.] [2018 Lin 2023), our textual descriptions are structured and contain multigranular in- formation for five attributes. As illustrated in Figure [ the general attributes of the image are described initially, covering aspects such as modality, the detection of specific organs, and their depiction. Following this, the attributes related to ROI are detailed, including the ROI analysis, locations and texture of the lesions, which encompass the disease type and relevant pathological features. Furthermore, region- wise correlations are highlighted to showcase relationships between the ROIs and surrounding regions, providing insight into differences features and the extent of disease progression.
Textual Descriptions.
3.2 DATA CONSTRUCTION PIPELINE
Given a medical image, we aim to generate corresponding multigranular visual and textual annotations. Specifically, as shown in Figure 2 our pipeline can be decomposed into two stages: 1) Data Processing, and 2) Multigranular Textual Description Generation. Firstly, our data processing stage includes three key steps: a) Metadata Integration produce coarse captions encapsulating fundamental image information such modality and disease types; b) ROI Locating to identify regions of abnormalities; and c) Medical Knowledge Retrieval to extract relevant fine- -rainnd medical details. All processing steps further detailed in Sec-
3

---

Published as conference paper ICLR 2025
CHES"
This chest Ray image, taken from the Pk plltors the thoracic incluctig struruues. unge pooply majoriy Image, hear situated cenirally The nepton of interaa laaated for cortally image suggorsing a ploural affusion. The contant this area inciaates fluid accumutaiin skkly attecding adjecantt petentatlyy influrninng posttio functio suroonddnn structures.
your
heart, and surmundinn
TECHACQUE: Cheel Rkand bewral..
COMPARISOON: Chest radigraph hoo
FINDINGS:
There been mecoumilatin moderate lett pleural
eftision common new dince the most resent provicus soucy. There right
approdmateyy opacily
reglon,
lower IMPRESSION:
lortacus unchanged confipuaation
MIMIC- CHR
Med: Trinily 25M (Ours)
Restoumndatton offodeeat lel, pleared Witiion.
Asaypl
Modalllo
DeleSiie
Assipoll
Carmediion
(a) Qualitative Comparison with sample in radiology report of chest x-rays dataset MIMIC-CXR JJohnson et al.
2019h
"What modillly used to als this innege?? "OTT
lathornter honzantally mitdle
C: "Witich part the body deen the imapp baloon tof", Ac "Cheet"" imaeaa" "Luang"
vertinily sheerd rigion of insoest. cnvering m' of the ama Iestuee denetty that ditere the sumounding long taaum poss byy indicating lung canoer: The affected area might be Infucnding fosues, suggosting progression implical
"Wall contain kingn" Voe"
SLAKE
"Where wlara the abnormallyy bostecc"" githt Lung Laft
Med- TorndTTiiilyy--ae 25M (Ours)
parts lung.
Organ
Analyytt
Lidladion
Anadyran
locaae
(b) Qualitative Comparison with sample in visual QA dataset SLAKE (Liu et al. 2021
Figure 1: Qualitative comparison with different types of dataset.
(1) Data Processing
Multigranular
Image
Description Generation
ROI Locating
Prompt
501
ROI
detailed deseription
based
Classiiiation
Metadata Integration
Multirannarr Descriptin
Coamse Caption
A Achost X X-ray. Showing lunge contrally located within the thoracic cavity.
MLLM
Report or QA
ROI is positioned horizontslly canter and veriiclly Ragion echibis incresseed opacilyy and imaguiar sexture, ind andiiaiin.. Showing pettern rightt lung involvement typically COVID- cases.
Knowladdge
Muligronuler
lef-
Medical Knowledge
Dascription
Modallly
Organ
Music B-box
PublRed
Detection
Knowledge Base
Analysis Lesion Texhura
Var STATPEABLS
Region-stin
Corralations
Data Triplet
Figure Data construction pipeline. /) Data processing, including metadata integration to generate coarse caption, ROI locating, and medical knowledge collection. 2) Mulrigranular Textual Description Generation based on processed data.
Subsequently, we prompt MLLMs to integrate information within processed data and generate multigranular textual descriptions. Corresponding details are provided in Section 3.2.2 The original image, generated visual ROIs, and textual descriptions combined into data triplet in MedTrinity-2555
tion 3.2.
3.2.1
DATA PROCESSING
Coarse Caption Generation via Metadata Integration. We aim to generate coarse captions that provide fundamental information for given image, including modality, organ labels, disease types, and optionally,
4

---

Published as conference paper ICLR 2025
Caption
cAchest Image with COVID- 19 the kings*
Without coarse caption:
X-ray showing views lungs heart. lunga nccupy the majorily of the thoracic cavily, which the extending from disphragm.
image
caption:
image is cheal X-ray showing both lunga,
contrally posilioned the thoradi cavily, flanked by the rbs and the despragm visibre at bottom. lungs. There no..... The lungs show patihyy opacities suggesting an infectiaus process, consistent with pulmonary invoivement COVID-19.
encased by the
collarbone
located centrally alll just banseth ribs, alightly tited to the loft. No medical devices are visiblo the
Specify Disease
Figure 3: A qualitative comparison example generated textual description with and without coarse caption. Without a coarse caption, MLLMs fails to detect diseases. On the contrary, providing caption mentioning "COVID-19" allows MILMS identify and categorize the disease, facilitating further analysis.
ROIs
Without ROIs:
With ROls:
mage X-ma showing both lunga, cantrally posilianed the tharacic cavity, flankee by the ribs and the diaprragm visible the heart visible center between lungs.
analysis
centrally positioned belween
two spacific regions interest located at the left-center and right-center of the middle the lungs, there are unusual findings suggestive of COVID-19 occupying
the image respactivelly display changes lung fexture may as increased opacily. The left center region is slightly larger and potentially indicates more extensive involvement the luna tssue the righticenter
no..... The lunas show patchy opasities suggesting infectious consistent with pulmonary nvelvement
Region-wise Correlation
These Ussue orttical understanding spread impact COVID 18, arfecting surrounding lung areas.
ateration
Figure 4: A qualitative comparison example of generated textual description with and without locating ROIs. Without ROIs, the caption offers only a brief global analysis; with ROIs, MLLMs conducts detailed local analysis and assesses the impact of lesion ROIs adjacent normal regions.
camera views and equipment information. Instead of extracting features directly from the images, gen- erate these captions by integrating dataset metadata. We first extract metadata from the datasets and then apply a fixed rule integrate this information into coarse captions. For example, for image in the QaTa- COV19 d we derive metadata from the dataset's 's accompanying paper or documentation, indicating that it consists COVID-19 chest X-ray images. Next, we construct coarse captions like "A chest X-ray image with COVID-19 in the lungs" highlighting the modality, organ types, and disease labels. We also integrate additional paired textual information (if any), such as radiological findings into coarse captions.
The effectiveness of applying coarse captions when generating multigranular textual descriptions is illus- trated F contrast to the scenario without a coarse caption, where MLLMs fails to recognize the disease, providing MLLMs with a coarse caption that includes the disease type "COVID-19" enables it to identify and categorize the disease, thereby laying the foundation for further analysis.
ROI Locating. We employ appropriate strategies locate ROIs for images paired with different annota- tions. For datasets that already include localization annotations, such as segmentation masks or bounding boxes, we derive the ROIs from these paired annotations. Specifically, bounding boxes are directly used as the ROIs, while segmentation masks are converted to ROIs by creating the smallest bounding box that covers
httpps:/ htpp:://www.tt. www.aaall......sseeaae..eee
9-datastt.
5

---

Published as conference paper ICLR 2025
external medical knowledge
new opecilioe. "coracliaaion"
Without medical knowledge:
With medical knowledge:
The imaaee chost X-ray showing both lungs and regions
The image is chest X-ray showing the thoracic
postiooed between
Visble argans
Standardize Terminology
lunga
mage respechvely, dispbay changes lung texture increased opacity. The left center region slighdy larger and palenisilly indioaise more exdettire insovammnnt the lung tesue companed the right-center region. These ereze of alteration the lang lissue criiccal COMID
stemum
lungs.
that may indioate infection,
regions Interest, locatoc..... These regiona
cuhibit ground-glass opasities and consolidation
typical indicators of COVID-19 preumonis, which
suggnet the presence of infirmmatory processes
Revise
These affected areas aignificntt as they indicato
understanding the affecting
Diagnosis
primary Intection progresses.
COVID-19,
mullifocal
(ung
Figure 5: A qualitative comparison example of generated textual description with and without external medical knowledge. MILMS can standardize medical terminology in its expressions and refine its diagnosis based on disease progressions detailed in medical literature.
ROI
Textual description of ROI horizontally; center vertically: middle area ratio:21.2%
ROI
Textual deseription ROI borizontally: left-center
ROI
Textual deseription of ROI horizontally: left vertically: lower-middlo area ratior8.5%
vertically: lower-middle
area 1.2%
(b) Example of locating ROI via
(a) Example of locating ROI via
BA Transformer
(Wang et al.
(c) Example of locating ROI via Chexmask (Gaggion al. 2022
SAT (Zhao et al. 2023
2021)
Figure 6: Example of ROIs and their corresponding textual descriptions.
the mask. When such localization annotations are not available, we apply corresponding pretrained expert models to generate ROIs. More details about the selection of expert models are provided in Appendix D Examples of generated ROIs from various modalities using corresponding models are demonstrated in Fig- 6 For modalities such as X-ray and MRI scans viewed from the z-axis, our ROI localization employs a coordinate system relative to the human body, resulting a left-right reversal in the image representation.
Incorporating ROIs as the guidance facilitates MLLMs conduct detailed analysis and generate multi- granular textual descriptions. demonstrated Figure 4. description generated without guidance of ROIs limited to a brief global overview of the image. In comparison, with ROIs, generated description contains local analysis regarding the abnormal region and its correlations to other regions.
Medical Knowledge Retrieval. General-purpose MILMS often lack medical terminology and expertise. To address this issue, we build a medical knowledge database following MedRAG (Xiong et al. 2024). We collect three main corpora: Pub 2 for biomedical knowledge, StatPearl for clinical decision support, and medical textbooks (Tin et 202T) I for domain-specifer knowledge. We segment these corpora into short snippets and encode them into high-dimensional vectors using the text encoder from Med-CPT (Tin et al. 2023). These vectors are then indexed into a specialized vector knowledge base using TJhnsooonnson et al. 2019c), optimized for efficient retrieval. For given image, we retrieve relevant medical knowledge by us- ing its coarse caption, which generated through metadata integration. Specifically, we encode the coarse captions, including disease and organ classifications, into vectors using the Med-CPT text encoder. We then perform a vector similarity search in the medical vector database, retrieving the top eight medical knowledge
nlm www..nn....iiinno......mm.mm
6

---

Published as a conference paper at ICLR 2025
Knowledge 1:
Title: Mobile chest X-ray manifestations of 54 deceased patients with coronavirus disease 2019: Retrospective study,
Contant: ... found that (93%) patients with lesions occurred the bilateral lung. (7%) patients occurred the right 54 (100%) petients were multifocal involvement.
number of lung fields involved was 42 (78%) palients fields, (8%) patients in lung fields, (7%) patients in 4 lung fiiids, and 5 (9%) pationts in lung flalss. Fifty-three (98%) pationts had patchy opacities. 3 (6%) patients had round or oval solid nodules, 9 (17%) palients had fibrous stripes, 13 pelients had pleural effusion, 8 (15%) pelients had pleural thickening, (11%) patients had pneumothorax, (6%) palients 16 (87%) palients had the progreesion of the leeions, 8 (33%) palients had no significant change of the lesions, and there was no case of reduction of the lesions, The mobile cheat X-ray manifettation of decessed palients with COVID-19 19 were mostly bilateral lung, multifocal involvement, and extensive lung field, and pleural effusion, pleural thickening, and pneumothorax probably could be observed. The serial mobil chest X-ray showed the chest leeione were progressive with high probabiity.
subcutaneous emphysema. Among 24 palients
had
moblle chest X-rays,
An example of the Top-8 retrieval results. By leveraging COVID- 10-relatee medical knowledge, MLLMs can standardize medical terminology and enhance diagnoses according to the disease progressions described in medical literature.
Figure 7:
snippets that semantically match the query. These snippets provide the external medical knowledge paired with the image for generating textual descriptions. A qualitative example demonstrating the effectiveness of incorporating external medical knowledge is shown in Figure 7 With access to COVID-19-relate med- ical knowledge, MLLMs can standardize medical terminology and refine diagnoses based on the disease progressions outlined in medical literature.
A qualitative comparison of generated text descriptions, both with and without external medical knowl- edge, is presented in Figure MLLMs are capable of standardizing medical terminology and enhancing diagnostic accuracy by incorporating insights from disease progressions documented in medical literature.
3.2.2 GENERATION OF MULTIGRANULAR TEXT DESCRIPTION
Generation Prompt. After data processing, a comprehensive prompt is utilized to guide MLLMs to in- tegrate all information and generate multi-granular descriptions. We incorporate the processed data (coarse captions, ROIs, and retrieved medical knowledge) into the prompts. Specifically, textual information such coarse captions and retrieved medical knowledge are directly integrated into the prompt. While ROIs on images are converted into textual information based on their coordinates and area ratio within the images, using terms such "left-center" and "area ratio: 1.2%" Examples of textual information converted from ROIS are shown in Figure 6 Instead of merely inserting retrieved knowledge, we instruct MLLM to identify and align the relevant knowledge with ROIs to provide diagnostic insights. The prompt template consists of a three-level hierarchical framework with questions to instruct MLLMS to generate: (1) global descrip- tion that captures all details of the image, (2) a local-focused analysis of specific ROIs that potentially are diseased; and (3) an inference of the correlations between region- wise attributes to understand the impact of local abnormalities on the surrounding regions and extent of disease progression. Detailed prompt template
is presented in Appendix F
Choice of MLLM. All textual description in MedTrinity-2255 25M are generated using LLaVA-Medcap which is a specifically fine-tuned LLaVA to generate high-quality textual descriptions. To obtain the fine-tuning data, we first generate 200,000 multigranular textual descriptions using our generation pipeline and GPT- 4V (Achiam et al. (222). Subsequently, we pretrain our LLLVA--Medi strategy in LLaVA-Mer (Li et al.] 2024a), then these generated data are used to to finetune the LLaVA- Medcap. LLa LLdVA- is then used in our pipeline to generate text descriptions for whole 25 million images in MedTrinity-25555 25M. As shown in Appendix(B] LLaVA-Medcap is capable of generating high-quality descriptions with more details compared to GPT-4V.
following the two-stage fine-tuning
7

---

Published as conference paper ICLR 2025
(Itrasound(O.7K)
Brain
MRI 1 12M
(icroscopy(10K)
Head&Neck
Dermoscopy(10K)
CT 5.1M
PET(44K)
Histo- pathology 6.2M
Skin
Cutton
Methony
Liver
Lung
(ndoscopy(64K)
(a) Modality distribution.
(h) Biological structures.
20 ROCO
MedTrrintt-25MM
208K
PMC-15M
337K
PMC--A
SLAKE 68
MIMIC-CXR-JPG
Dataset
1.6M
OpenPath
MIMIC-CXR 85
Ours
120
25M
Number of Samples
(c) Data size comparison.
(d) Average word count of descriptions comparison.
Naa Partaaatn (9M) tubular adonosarcinoma
Fibrosis
melanema
covid-19 19
Edema
cancer
birat
Brain Hemorrhage
Mbslectadt
liver Euror
prain
Loskersia
tumor
broast canoer
wepdtedd csforeccal Tonoer
pleumonia
Infiltration
Opacity
Pepertorates
Emphysema
breast cancer
Fneeh
Hernia
cancer
lang cascer
(c) Wordeloud of discase statistic.
Figure 8: Statistical overview of MedTrinity-2555
3.3 DATASET ANALYSIS
Scale. Figure Sc compares the amount of data samples in McdTrinity-25M and other medical multimodal datasets. To the best of our knowledge, MedTrinnty-25M is the largest open-source, multi-modal multigran- ular medical dataset to date.
Diversity. Our dataset encompasses 10 imaging modalties, with more than 65 diseases across various anatomical structures in human. The number of the samples within each modality in MedTrinity-25M are shown in Figure Sa and the distribution of each anatomical and biological structures is provided in Figure (b Meanwhile, Figure Se illustrates the frequently used words related diseases in our dataset.
Richness. We provide both qualitative examples and quantitative analysis demonstrate the richness of annotations in MedTrinity-25M. As shown in Table [ we compare the types of annotations in our dataset with those of other multimodal datasets. Our dataset provides multigranular and richer annotation informa- tion. surpassing other multimodal datasets. Qualitative examples tions provide more comprehensive information compared to the chest X-rays dataset MIMIC-CXE (Johnson et al. 2019b) and the visual QA dataset SLAKE (Liu al. 2021). Figure 8d compares the average word count of text descriptions in multiple medical multimodal datasets. The word count in our dataset signifi- cantly larger, indicating greater richness.
shown in Figure T Our textual descrip-
8

---

Published as conference paper ICLR 2025
Table 1: Comparison of types of annotations in MedTrinity-25M with other multimodal datasets.
Lesion Type
Lesion BBox/Mask
Lesion Description
Region-wise
Modality
Correlations
MedMNIST Yang cl
2023
X
v
K
X
X
DeepLesion
BraTS 2024 de Verder at
2024g
MIMIC-CXE Dolmson al.
20196
Quilt 1Ml [eezogwo
"QA-RADTL au ni
20180
CRC100K (Kather
2018
SA-Med2D-JUM Yc et 2023
X
X
Med (5.((O
Table 2: Comparison of alignment scores between LLM and Expert.
Attributes
Evaluator
Organ Detection
ROI Analysis
Lesion Texture
Region-wise Correlations
Modality
Avg.
LLM 1.00/1.00
0.90/1.00
0.90/1.00
0.80/1.00
0.70/1.00
0.86/1.00
Expert
1.00/1.00
0.90/1.00
0.90/1.00
0.70/1.00
0.80/1.00 0.85/1.00
Quality. We conduct expert and LLM evaluations to verify the quality the generated multigranular descriptions. Each description in MedTrinity-25M contains five key attributes of medical images: modality, organ detection, ROI analysis, lesion texture, and region-wise correlations. A random subset of 200 samples is selected for evaluation. attribute by comparing the generated descriptions with ground-truti annotations. Scores are averaged across all samples obtain an overall score. For LLM--ased evaluation, we GPT-4V assess the accuracy of medical facts and diagnoses based on the same five attributes. GPT-4V scores each attribute on scale of O points. All scores are normalized 0-1 scale for comparison.
expert-basee evaluation, medical professionals assess the accuracy of each
Table shows that MedTrinity-2555 25M achieves 0.85 and 0.86 in expert and LLM evaluations, with modality, organ detection, and ROI analysis nearing perfect scores. To illustrate, Figure 12 shows sample that achieved a perfect score from GPT-4V.
4 EXPERIMENT
4.1 LLAVA-TRI: ALIGNING MULTISCALE MLLM WITH MEDTRINITY-25M
To fully exploit the multigranular annotations, we propose LLaVA-Tri, which is based on LLaVA Liu
ctt al. (024) and incorporates MedTrinity-25M to align into the medical domain. LLaVA-Tri integrates LLaMA3 (Team 2024) to enhance linguistic capabilities and incorporates multiscale feature extraction (Shi al. 2024 to boost visual performance. Specifically, we first pretrain LLaVA-Tri on 600K image-text pairs from PMC-15M (Zhang et al. (023a)), following the training settings from Li et al. then trained on MedTrinity-25MMr multigranular alignment.
2024a). The model is
We benchmark VLa on three biomedical Visual Question Answering (VQA) datasets, VQA- RAD (Lau et al.) 2018a). SLAKE (Liu et al.) (02T), and PathVQA (He et al., 2020a to assess the efficacy of aligning the model using MedTrimity-225 25M.. The model is fine-tuned for three epochs each of the three VQA datasets and evaluated accordingly. As shown in Table LLaVA-Tri achieved state-of-the-ae oh-the-rrt results in all of the three VQA benchmarks, with 81.6% accuracy VQA-RAD, 87.8% SLAKE, and 82.8% on VQA. These results highlight the significant advantages incorporating multiscale LLa LLaVA-Ti with multigranular alignment.
9

---

Published as a conference paper ICLR 2025
Table 3: Comparison LLaVA-Tri with existing SOTA methods. Following our multigranular alignment pretraining on MeddTrinty-25MM. LLaVA-Tri achieves SOTA in all three VQA benchmarks. The asterisk (*) indicates that, for open-ended questions, prior methods still formulate the problem as classification among distinct answers in the training set. This approach may potentially overestimate generalizability, test answers are often seen during training.
VQA-RAD
SLAKE Closed
PathVQA Closed
Method
Open
Closed
Average
Open
Average
Open
Average
GPT-4V (Achiam al.
2023
39.5
78.9
59.2
33.6
43.6
38.6
VL Encoder-
2023
71.5*
82.5
77.0
71.5
85.6
78.6
Q2ATransformr (Liu
2025
79.2'
81.2
80.2
54.9*
88.9
71.9
Prefix T. Medical LM (Sonsheck ct
2023
84.3"
82.0
83.2
40.0*
87.0
63.5
PubMedCLIP [Eslami ct 2025
60.1
80.0
70.1
78.4
80.5
BiomedCLIP [Zhang
67.6'
79.8
73.7
82.1"
89.7
85.9
M212 [L 20245
66.5'
83.5
75.0
74.7"
91.1
82.9
36.3"
88.0
62.2
(Lu ar. 2024
LLaVA (Liu
50.0
65.1
57.6
78.2
63.2
70.7
7.7
63.2
35.5
LLaVA-Med ara 2024b
84.2
72.8
83.1
84.1
37.9
91.2
64.5
LLaVA-THi
86.0
87.8
66.5
82.8
Table 4: Comparison different models with or without alignment pretraining with MedTrinniy--5MM. The notation w/ and w/o indicate models with and without pretraining MedTrinity-25M, respectively.
Mode
Dataaet Use
VQA-RAD close
SLAKE close
PatiVQA close
average
LLaVA-THi
Wlo
w/
77.1 ((2.5)
64.6
86.0 (9.0))9))
77,0
816 (10.5)
70,8
86.2 (16.9)
79.3
89.3 (1.3)
$4.0
87.8 (6.1)
S1.7
66.5 (11.5)
55.0
99.0 -c5.0)
94.0
82.8 (ck.1)
74.5
MiICCMM-V-2AA Yao et al. 2024
wio
wi
50.5 (2.0)
48.5
87.6-
86.4
60.1
67.5
65.3 x0.1)
57.2
80.6-000)
50.0
73.0-(44))
68.5
34.2-(5.0)
31.2
94.8
90.5
64.5x(1.)
60.9
IL1ernVL2--BB Chen
2024
wil
wf
40.7 (2.55
38.2
(3.0
76.2
60.4
57.2
x6.4
61.7
00.0000
77.8
+2.6-
60.8
23.6-(688
16.8
67.4
86.4
$5.5
51.6
PubMedCLIP Eshami al. 2023
wil
55.6
79.3
67.5
19.7-104)
70.2x
4.2 ENHANCING MODEL PERFORMANCE THROUGH MULTIGRANULAR ALIGNMENT
To further demonstrate the effectiveness of multigranular alignment, we conducted ablation studies by train- ing and evaluating the model with or without aligning using Med'Trinity-255 respectively. We conduct experiments on various multimodal models, including both multimodal language models and CLIP models:
LLaVA-Tri VL2-8B (Chen et al. 2024). MiniCPM-V-2.66 (Yao et al. 2024), and PubMedCLIP (Eslami al. 2023). As shown in Table incorporating multigranular alignment significantly enhances performance across all tested multimodal models. Notably, LLaVA-Tri exhibited improvements of 10.8%, 6.1%, and 8.3% VQA-RAD, SLAKE, and Path VQA, respectively, compared to its counterpart with- out alignment. These findings underscore the potential of LLaVA as a foundational dataset capable of enhancing the medical performance of various multimodal models.
5 CONCLUSION
This paper introduces MedTrinity-25M, large-scale multimodal medical dataset comprising over 25 mil- lion Image-RO1-ddecriple triplets sourced from more than 30 online resources, spanning 10 modalities and covering over 65 diseases. We have developed the first automated pipeline to scale up multimodal data by generating multigranular visual and textual annotations from unpaired images. We believe that McdTriintty- 25M's enriched annotations have the potential support wide range of multimodal tasks, such as caption- ing, report generation, classification, and segmentation, as well as facilitate the large-scale pre-training of multimodal medical Al models.
10

---

Published as a conference paper ICLR 2025
ACKNOWLEDGEMENT
We thank the Microsoft Accelerate Foundation Models Research Program, the OpenAl Researcher Access Program, TPU Research Cloud (TRC) program, Google Cloud Research Credits program, AWS Cloud Credit for Research program, and Lambda Cloud for supporting computing needs.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint w031v22303.08774, 2023.
Amir Akharnejad, Nilanjan Ray, Penny Barnes, and Gilbert Bigras. Predicting ki67, er, pr, and her2 statuses from h&c-stained breast cancer images. arXiv preprint arXiv:2308. 019882 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openftamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint wrXiv:2308.01390 2023.
Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Laila Bashmal, and Mansour Zuair. Vision language model for visual question answering in medical imagery. Bioengineering, 10(3):380, 2023.
Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria De La Laaeesia-aaaa Padchest: A large chest X-ray image dataset with multi-label annotated reports. Medical image analysis, 66: 101797, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint (tti::2444.168221 2024.
Angel Cruz-Roa, Ajay Basavanhally, Fabio González, Hannah Gilmore, Michael Feldman, Shridar Gane- san, Natalie Shih, John Tomaszewski, and Anant Madabhushi. Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In Medical Imaging 2014: Digital Pathology. SPIE, March 2014.
Maria Correia de Verdier, Rachit Saluja, Louis Gagnon, Dominic LaBella, Ujjwall Baid, Nourel Hoda Tahon, Martha Foltyn-Dumitru, Jikai Zhang, Maram Alafif, Saif Baig, et al. The 2024 brain tumor segmentation (brats) challenge: Glioma segmentation on post-treatment mri. arXiv preprint arXiv:2405. 2024a.
Maria Correia de Verdier, Rachit Saluja, Louis Gagnon, Dominic LaBella, Ujjwall Baid, Nourel Hoda Tahon, Martha foltyn-Dumitry Jikai Zhang, Maram Alafif, Saif Baig, et al. The 2024 brain tumor segmentation (brats) challenge: Glioma segmentation on post-treatment mri. arXiv preprint 2.Ki:::200..33666 2024b.
Kexin Ding, Mu Zhou, He Wang, Olivier Gevaert, Dimitris Metaxas, and Shaoting Zhang. A large-scale synthetic pathological dataset for deep learning-enabled segmentation of breast cancer. Scientific Data, 10(1):231, 2023.
Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: 2023.
EACL 2023, pp.
11

---

Published as a conference paper ICLR 2025
N. Gaggion, C. Mosquera, M. Aineseder, L. Mansilla, D. Milone, and E. Ferrante. CheXmask Database: a large-scale dataset of anatomical segmentation masks for chest x-ray images (version 0.1). https: 10. 1r0.:////90.130220///----333333 2023.
Nicolas Gaggion, Lucas Mansilla, Candelaria Mosquera, Diego H. Milone, and Enzo Ferrante. Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis. IEEE Transactions Medical Imaging, 2022.
Jevgenij Gamper, Navid Alemi Koohbanani, Simon Graham, Mostafa Jahanifar, Ksenija Benet, Syed Ali Khurram, Ayesha Azam, Katherine Hewitt, and Nasir Rajpoot. Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003. 10778, 2020.
Lidia Garrucho, Claire-Ann Reidel, Kaisar Kushibar, Smriti Joshi, Richard Osuala, Apostolia Tsirikoglou, Maciej Bobowiez, Javier del Riego, Alessandro Catanese, Katarzyna Gwożdziewicz, al. Mama-mia: large-scale multi-center breast cancer dce-mri henchmark dataset with expert segmentations. arXiv preprint arXiv:2406. 13844, 2024.
Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Do- gan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, et al. A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-shot detection of abnor- malities. arXiv preprint arXiv:2403. 2024.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering, arXiv preprint arXiv:2003. 102200000...288, 2020a.
He, G Yang, Yang. Chen, Kong, J Wu, L Tang, X Zhu, JL Dillenseger, P Shao, Zhang, Shu, JL Coatrieux, and S Li. Dense biased networks with deep priori anatomy and hard region adaptation: Semisupervised learning for fine renal artery segmentation. Medical Image Analysis, 63, 2020b.
Yuting He, Guanyu Yang, Jian Yang, Rongjun Ge, Youyong Kong, Xiaomei Zhu, Shaobo Zhang, Pengfei Shao, Huazhong Shu, Jean-Louis Dillenseger, Jean-Louis Coatricux, and Shuo Li. Meta grayscale adap- tive network for 3D integrated renal structures segmentation. Med. Image Anal., 71:102055, July 2021.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Ku- mar Anand, Ranjay Krishna, and Linda Shapiro. Quilt- Im: One million image-text pairs for histopathol- ogy. Advances in Neural Information Processing Systems, 36, 2024.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intel- ligence, volume 33, pp. 590-597, 2019.
Andrew Janowezyk and Anant Madabhushi. Deep learning for digital pathology image analysis: A compre- hensive tutorial with selected use cases. Journal of pathology informatics, 7(1):29, 2016.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, (144)264,,, 2021.
Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomed- ical information retrieval. Bioinformatics, (9 I 1)btdd651, 2023.
12

---

Published as a conference paper ICLR 2025
Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpas a large publicly available database of labeled chest radiographs. arXiv preprint XXXi:/900.000000 07042, 2019a.
AlistairEW Johnson, TomJ Pollard, SethJ Berkowitz, NathanielR Greenbaum, MatthewP Lungren, Chih- ying Deng, RogerG Mark, and Steven Horng. Mimic-exr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019b.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547, 2019c.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836. 2020.
Alexandros Karargyris. Renato Umeton, Micah J Sheller, Alejandro Aristizabal, Johnu George, Anna Wuest, Sarthak Pati, Hasan Kassem, Maximilian Zenk, Ujjwal Baid, et al. Federated benchmarking of medical artificial intelligence with medperf. Nature Machine Intelligence, 5(7):799-8) -10, 2023.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. https://doi.org// httpg//1..221/eeeeee 1214456, 2018
Masakata Kawai, Noriaki Ota, and Shinsuke Yamaoka. Large-scale pretraining pathological images for fine-tuning of small pathological benchmarks. In Workshop on Medical Image Learning with Limited and Noisy Data, pp. 257-267. Springer, 2023.
Jason Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. dataset of clinically gener- ated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018a.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. dataset of clinically gener- ated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018b.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vviion assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024a.
Chunyuan Li, Cliff Wong. Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language- language-and-vision and-visoon assistant for biomedicine one day. Advances in Neural Information Processing Systems, 36, 2024b.
Pengfei Li, Gang Liu, Lin Tan, Jinying Liao, and Shenjun Zhong. Self-supervised vision-language pretrain- ing for medial visual question answering. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1-5. IEEE, 2023.
Weixiong Lin. axiong/pmc_oo datasets at hugging face. https://ugggingaac..oo/ataeee axiong/pmor paa_oa_pc_aa__a_eeooo 2023.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- clip: Contrastive language-image image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 525-536, Springer, 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- coowledge-enhaneee dataset for medical visual question answering. In 202/ IEEE 18th International Symposium Biomedical Imaging (ISBI), pp. (650-1654, IEEE, 2021.
13

---

Published as a conference paper ICLR 2025
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yunyi Liu, Zhanyu Wang, Dong Xu, and Luping Zhou. QZatransformer: Improving medical vqa via an answer querying decoder. In International Conference Information Processing in Medical Imaging, pp. 445 456. Springer, 2023.
Meng Lou, Hanning Ying. Xiaoqing Liu, Hong- Yu Zhou, Yuqing Zhang, and Yizhou Yu. Sdr-former: siamese dual-resolution transformer for liver lesion classification using 3d multi-phase imaging, arXiv preprint arXiv:2402.17246 2024.
Jun Ma and Bo Wang. Miccai flare23: Fast, low-resource, and accurate organ and pan-cancer segmenta- tion in abdomen ct, Apr 2023. URL http:::://coaaaaaaaaaaa lisn. upsaclay fr/ wwwcompttiionnsss
12239
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp- 353-367. PMLR, 2023.
Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M Friedrich. Radiology objects in context (roco): multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Hell in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pp. 1880-19. Springer, 2018.
Eduardo Pontes Reis, Felipe Nascimento, Mateus Aranha, Fernando Mainetti Secol, Birajara Machado, Marcelo Felix, Anouk Stein, and Edson Amaro. Brain hemorrhage extended (bhx): Bounding box extrap- olation from thick to thin slice et images. PhysioNet, 101(23)c211-200, 2020.
Khaled Saab. Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulezyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.188 18416, 2024.
Pengfei Shao, Chao Qin, Changjun Yin, Xiaoxin Meng, Xiaobing Ju, Jie Li, Qiang Lv, Wei Zhang, and Zhengquan Xu. Laparoscopic partial nephrectomy with segmental renal artery clamping: technique and clinical outcomes. Eur: Urol., 59(5):849 855, May 2011.
Pengfei Shao, Lijun Tang, Pu Li, Yi Xu, Chao Qin, Qiang Cao, Xiaobing Ju, Xiaoxin Meng, Qiang Lv, Jie Li, Wei Zhang, and Changjun Yin. Precise segmental renal artery clamping under the guidance of dual-source computed tomography angiography during laparoscopic partial nephrectomy. Eur. Urol., 62 06):1001-1008, December 2012."
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do not need larger vision models? arXiv preprint arXiv:2403. 13043, 2024.
Tom Van Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees GM Snoek, and Marcel Worring. Open-ended medical visual question answering through prefix tuning of language models. In International Conference on Medical Image Computing and Computer-sssisted Intervention, pp. 726- 736. Springer, 2023.
Carsen Stringer and Marius Pachitariu. Cellpose3: one-click image restoration for improved cellular seg- mentation. bioRxiv, pp. 2024-02222 2024.
14

---

Published as a conference paper ICLR 2025
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 2023a.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023b.
Meta LLaMA Team. Introducing meta Ilama 3: The most capable openly available Ilm to date. https:
7/ai .eeta. comm/looggmeta- 1atr://mttaallmma--0 2024.
Yuri Tolkach, Lisa Marie Wolgast, Alexander Damanakis, Alexey Pryalukhin, Simon Schallenberg. Wolf- gang Hulla, Marie- Eich, Wolfgang Schroeder, Anirban Mukhopadhyay. Moritz Fuchs, et al. Artificial intelligence for tumour tissuc detection and histological regression grading in oesophageal adenocarci- nomas: a retrospective algorithm development and validation study. The Lancet Digital Health, 5(5): e265-e275. 2023.
Masayuki Tsuneki and Fahdi Kanavati. Inference of captions from histopathological patches. In Interna- tional Conference on Medical Imaging with Deep Learning pp. 1235-1250. PMLR, 2022.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. NEJM AI, 1(3): Aloa2300138, 2024a.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schackermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, al. Towards generalist biomedical ai. NEJM AI, 1()): Aloa2300138, 2024b.
Patrick Wagner, Maximilian Springenberg, Marius Kröger, Rose KC Moritz, Johannes Schleusener, Mar- tina C Meinke, and Jackie Ma. Semantic modeling of cell damage prediction: a machine learning ap- proach at human-level performance in dermatology. Scientific Reports, 13(1):8336, 2023.
Jiacheng Wang, Lan Wei, Liansheng Wang, Qichao Zhou, Lei Zhu, and Jing Qin. Boundary-aware trans- formers for skin lesion segmentation. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 2021, Pro- ceedings, Part I 24, pp. 206-216. Springer, 2021.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. ChestX-Ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017a.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. ChestX-Ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017b.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. ChestX-ray: Hospital-scalc chest x-ray database and benchmarks on weakly supervised classification and localization of common thorax diseases. In Deep Learning and Convolutional Neural Networks for Medi- cal Imaging and Clinical Informatics, Advances in computer vision and pattern recognition, pp. 369-392. Springer International Publishing, Cham, 2019.
15

---

Published as a conference paper ICLR 2025
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402. 2024.
Ke Yan, Xiaosong Wang, Le Lu, and Ronald M Summers. Deeplesion: Automated deep mining, catego- rization and detection of significant radiology image findings using large-scale clinical lesion annotations. arXiv preprint arXhv:1710.01766 2017a.
Ke Yan, Xiaosong Wang, Le Lu, and Ronald M Summers. Deeplesion: Automated deep mining, catego- rization and detection of significant radiology image findings using large-scale clinical lesion annotations. arXiv preprint 1rXivv1710.01 2017b.
Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai. Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408..1 01800, 2024.
Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng. Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, al. Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks. arXiv preprint arXiv:2311.11900 2023.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprini arXiv:2303.009 2023a.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.104 10415, 2023b.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei. Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest et: A grounded vision-language dataset for chest ct analysis. arXiv preprint arXiv:2404. 16754, 2024.
Ziheng Zhao, Yao Zhang, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. One model to rule them all: Towards universal segmentation for medical images with text prompts. arXiv preprint arXiv:2312.17183 2023.
Hong- Yu Zhou, Subathra Adithan, Julián Nicolás Acosta, Eric J Topol, and Pranav Rajpurkar. A generalist learner for multifaceted medical image interpretation. arXiv preprint arXiv:2405.07980 079888 2024.
16

---

Published as conference paper ICLR 2025
APPENDIX
We present the following items in the Appendix:
1. Data source about MedTrinity-25M. (Section
2. Quantitative comparison between GPT-4V and LLaVA-Medcap (Section B
3. Examples of ROI for normal regions and multiple (Secton. Cp.
4. The list of expert ROI models (Section D)
5. Details of LLM Evaluation of Alignment (Section
6. Prompt for generating MedTrinity-2555 (Section
A DATA SOURCE
Table 5: Data sources for MedTrinity-222 255M from various medical image datasets, detailing their modalities,
biological structures, quantities, and annotations.
Biological Structures
Disease
Dataset Name
Modality
Quantity
Text
BBox Mask
Type
Reis B
BHX
al. 2020)
MRI
brain
973908
X
x
y
BRAIS24-MICCO CA Verdier 2024b)
MRI
brain
2535132
X
BRATS-ISBI Karargyris et al. 20233)
MRI
hrain
987340
x
x
d
x
breast
histopathology (anowccyy & Madab Histopathology breast
547403
x
x
hushi 2016 Cruz-Roa
(2014)
Breas(Cancer( Ding el al. 2023)
Histopathology breast
1824
x
[heXpent(rvinn et
2019)
X X-Ray
lung
183242
x
X
Adrenal,
Bile duct, Bladder,
Breast,
CISC CISCGammpr al. 2020
Histopathology Colon,
16285
x
Cervix,
Esophagus
Kidney.
Liver,etc
CPD Wagner 2023
Histopathology skin
204
x
lung.
liver.
CT-RATE Hamamei et al.
2024
mediastinum, 3869640 kidncy,
X
heart, etc. hone, ahdomen, mediastinum, liver. lung. kidney, soft tissue, pelvis
DeepLesion
2017b 7b)
2889672
17

---

Published as conference paper ICLR 2025
Table Continued from previous page
Biological Structures
Discase
Dataset Name
Modality
Quantity
Text Type
BBox Mask
Liver,
kidncy,
spleen,
pancreas, Aorta, adrenal gland, Gallbladder, esophagus, stomach, duodenum,ete.
FLARE23 Ma & Wang 2023
CT
13770
X
ihe4be Akbarnejad et al. (023)
Microscopy
cell kidney, cervix
102535
X
X
X
KIPA22 Shao et al.
2012
2011
He et al.
CT
26878
x
X
x
20206
2021)
CT;
MR, cell, rib,
Endoscopy, tissue, X-Ray. face.
Ultrasound, brain,
LLaVA-Med(L al. 2024b
22550
X
X
x
Histopathology. vascular,
Dermoccopy, liver,
Microscopy, bonc,
Fundus, lymph, etc.
PET liver
LLD-MMRI(LOu et al
2024
MRI
21523
X
x
MAMA-MIA/aamuee
2024
MRI
breast
316113
x
II
et al.
2019a
X-Ray lung
240506
/
NCT-CRC- HE- T00K K aather et all
2018)
Histopathology colon
100361
x
x
x
NIH-CXR Wang al.
201
2019)
X-Ray
lung
986
y
PadChest Bustos et al.
2020)
CT
lung
96284
x
x
PacchGasinAAD222
(suneki
& Kanavati
MRI
brain
98399
X
8
x
2022
gastrointestinal,
colon, appendix. pinworm,etc.
Path VQA training He
2020a
Pathology
13375
MR. cell,
tissue,
Endoscopy, vascular,
X-Ray. brain, Ultrasound, bone, Histopathology, liver.
PMC-OA Lin 2023
856999
X
X
x
Demoscopy, lymph,
Microscopy,
Fundus, eyc,
PET epithelium, etc.
18

---

Published as conference paper ICLR 2025
Table Continued from previous page
Biological Structures
Discase
Dataset Name
Modality
Quantity
Text
BBox Mask
Type
CT,
cell.
MR,
brain,
Endoscopy, tissuc,
X-Ray. artery, Ultrasound, bone, Histopathology. face,
PMC-VQA Zhan al. 2023b)
144999
X
X
Dermoscopy, rib,
Microscopy, vascular,
Fundus, liver,
PET eyc, etc.
brain,
breast,
PTCGA Kawai et al.
2023
Histopathology
uterine corpus, 3293965 kidney,
X
/
v
x
lung,
thyroid
skin,
lung, soft tissue, blood, kidney,
Quilt- M Ikezogwo 2024
Histopathology
643819
x
bone, etc. brain, kidney, liver,
X-Ray,
PET,
CT.
lung.
20M et al. 2023)
5491274
x
MR. Endoscopy, dermoscopy
pancreas, pulmonary. hepatic, skin, etc. brain,
CT.
liver.
SLAKE training Liu et al.
2021
MRI,
kidney, pelvic,
646
x
X- X-Ray
lung
TCGA Kawai et al.
2023
Histopathology
tissue
1142221
X
X
X
lung,
lymph nodes, bladder, brain, colon.
UL.223
CT
105669
x
X
x
kidney. lung.
oesophagus, stomach brain, lung.
(ALSET/Tllacch et al.
2023
Histopathology
277565
x
x
X-Ray, MRI
VQA-RAD training au et al.
201 8b
1758
X
x
abdomen.ete.
Total
25016845
19

---

Published as conference paper ICLR 2025
GPT-4V
LLaVA-MedcappOurs)
sice brain,
The image is transverso siice of magnetic resonance imaging (MR)) scan the brain, showing the cerebral hemissheres with cerebrum visible region interest located center part image horirontaly and upoer-midlle part image vertictty-hhccupyin approximately 3.9% of the total area. The region of interest located
the cerebral hemispheres are visible, with the region
located centrally towards uoper-middle
Mare octailed
portion of the mage syongying a small area retio This rigion
arees noloo
axhibits abnormalliy liy characten by allered signal
Moro detailed
right corebral hemisphare and appears have abnormal texture and sice, which is indicative of brain tumor. This abrormal area is likey affecting surrounding
intensily, contrasts
surrounding
Uissue,
RII andlysle
suggesting the presence a brain tumor. The abnormal
postion, relatve effect on from adjecint structures, polantially influancing
imply
Ieeton impord
aralynis
brain functin vicinty, The provides dotalled onatomy, a allowing for the visualization of sott tasue differences that may not be poss ble with other modailles.
which can cause a shitt the positio of adjacett atructures or polantially disnupt the brakt's
tissue pathological process within the brain.
effect being larger
Conto imago
analysis
Figure 9: Qualitative Comparison with sample generated by GPT-4V. Compared to GPT-4V, our model generate more detailed caption.
(a) infection sample from MIMIC-XXR. ROIS highlight the and right lungs.
(h) A healthy sample from SLAKE, The ROI points out the liver.
Figure 10: Examples of ROIs for normal regions.
B QUANTITATIVE COMPARISON OF LLAVA-MEDCAP WITH GPT-4V
As detailed in Scction 3.2.2 of the main paper, we developed an enhanced version of LLaVA (Li ct al. 2024a). called LLaVA-Medcap. This enhancement leverages the latest LLaMA3 (Team 2024 boost linguistic capabilities and incorporates multi-scat feature extraction (Shi
2024 to improve vision capabilities.
To justify the selection our specialized medical model, LLaVA-Tri, over GPT-4V for generating textual descriptions, we conducted quantitative comparison of the outputs generated by both models. We assessed the level of detail by com- paring the average word count of text descriptions generated for the same sample. LLaVA Tri, after task- specific fine- tuning, outperformed GPT-4V by 3.6% in word count, indicating that the descriptions generated by LLaVA-Medcap are more detailed, also provide a qualitative comparison u Based on these findings, we selected Medeap to generate multigranular textual descriptions for our entire 254
sample generated by LLaVA- Tri and GPT-4V Fig-
C EXAMPLES OF ROIS
As described Section 3.1 the main paper, the ROIs identified by expert grounding models predominantly capture pathological features such as lesions, inflammation, ncoplasms, infections, or other potential abnormalities. In rare cases where abnormalities are found, the ROIs typically focus the primary object organ in the image. Examples of such normal ROIs presented in Figure To
20

---

Published as conference paper ICLR 2025
image radlograph showing thacc cavity viible. The encotracheal tube is posiltoned 3.7 from the carina, and enteric tube is seen passing diaphragm
ungs, disphragm
terminates The left hemithorax exhibis opacificaiin, likey due to combinetin of efuusion consofidatin, shows pulmonary edema slighty moderate pleural effusion with assaciated compressive ste ectosi. The region interest located hortrontllly figh-ccnter vertically the middle, accupying 11.2% area, conespones the right ung and shows signs of mild pulmonary edemo. Another region of interest, horizontally center vertically and inoicates opacification consistent with ellusion, consolidaiio, and collapse. The abnomalties within these regions suggest possible relationship with the sleural effus lons and atelectasis, as fluid accumulation lung with one potentially exacerbating the other.
small
middle, taking 24.4% area, comessonds
effusions may internlaaedd
lead
Med-Trinity255M
Goal Description
Local Descripton
Local-giotel
Figure 11: A chest radiography example where global information matters. The diagnosis in this case requires comprehensive analysis of the entire image, encompassing both the left and right lungs. Here, ROIs encompass the large lesion areas of left and right lungs. Detailed local texture analysis of each region contributes the overall global diagnosis
In instances where global context is also critical for disease identification, the ROIs encompass multiple lesion areas, integrating both global and local information. For example, in chest radiography, analyzing both lungs and their overall structure often essential for accurate diagnosis, as shown in Figure []] By providing multigranular annotations that incorporate both local and global perspectives, our dataset helps multigranular alignment for medical foundation models.
D LIST OF EXPERT MODELS TO LOCATE ROIS
As detailed in Section 3.2.1 of the main paper, for datasets lacking localization information such segmentation masks and bounding boxes, employ various pretrained expert models to identify the ROIs. The specific expert models used for each dataset are listed Table
E DETAILS OF LLM EVALUATION OF ALIGNMENT
An example of perfect alignment score results evaluated by GPT-4V is shown in Figure In these examples, GPT-4V fully aligned with human annotations across all five criteria, resulting in perfect alignment scores. The prompt used to query GPT-4V for evaluating the aligmment score is shown in Figure 13 of supplementary.
The prompt used to query GPT-4V for evaluating the alignment score is shown in Figure 13
F PROMPT TEMPLATE FOR GENERATION OF MULTIGRANULAR TEXT DESCRIPTION
generate multigranular textual descriptions, we design multi-task prompting approach, breaking down this task into several smaller descriptive tasks. The model's responses to these different tasks collectively form the final fine-grained text description.
Figure 14 illustrates prompt template consisting of a three-level hierarchical framework with questions to instruct MLLMM::
21

---

Published as conference paper ICLR 2025
Table 6: List of expert models used to generate ROIs for different datasets.
ID
Dataset Name
Model
breast histopathology
2
BreastCancer
CISC
4
CPD
NCT-CRC-HEE-10K
HoverNet (Stringer & Pachitariu 2024
PTCGA
TCGA
8
VALSET
ihe4be
10
Quilt-IM 1M
11
CT-RATE
SAT (Zhao al.
2023)
PMC-OA
13
PMC-VQA
DINO
Caron et al.
2021
LLaVA-Med
15
Path-VQA training
16
PadChest
17
MIMIC-CXR-IPG
CheXmask Gaggion et al.
2023) Gaggion et al. 2022
18
CheXpert
Step 1- Global Understanding: Instruct MLLMs to provide comprehensive description of the image, detailing all modalities, identified anatomical structures, and their approximate locations. This step ensures that MLLMs gains overarching understanding and basic information about the image.
Step Local Analysis: Instruct MILMs to conduct detailed analysis of regions interest (ROI), including their locations, abnormalities, and textures. This step guides MI.LMs to facus on specific lesions for thorough assessment.
Step 3 Region-wise Correlations: Instruct MLLMs to examine the relationship between different regions and predict how the surrounding areas will be affected by the lesions in the ROL. This step aims to understand the interaction between local and global attributes, assessing the impact of local abnormalities the entire organ for accurate disease diagnosis.
22

---

Published as a conference paper ICLR 2025
Human Annotations
Multigranular Annotations
The Image i chest radiograph showing the thoracic cavity with the head. lungs, disphragm visible endstracheal
passing positioned diaphragm
from the carina, end enteric into the stomach. left-sided PICC line terminates in the low SVC. The left hemithraa exhiblss opacification, likely due to a combination of effusion, consolidation, and collapse, while the right lung shows mild pulmonary edema and a slightly increased small to moderate pleural effusion with associated compreesive
Modality
Organ
"INDIIGS" The ondetrachoal tube is unchanged, ending 3.7 cm from the carina. The enteric tube
region interest located
ROI Analysis
horizontaly right-center and vertically in the middle, occupying 11.2% of the area, corresponds to the right lung and shows signs of mild
passes below the diaphragm outside of the field of
view within desompressed stomach.
line ends the SVC. Opacification of the left hemithorax is unchanged, likely coombnattin of effusion, corisolidation, and collapse. hight-sided/puimonaa edema mild small to moderate right pleural effusion and associated compressive stelectasis
Lesion
pulmonary edema. Another region of interest, horizonially left-center and vertically in the middle,, taking up 24.4% of the area, corresponds to lung indicates opacification consistent consolidation,
Region-wise Correlation
The abnomalities within regions suggest possible relationship with the pleural effusions and stelectasis, us fluid accumulation and lung collapse can lead to changes in lung opacity and may be interrolated, with one potentialy exacerbating the other.
GPT-4V Rating
increased slightly. There pneumotharax.
GPT-4V Rating Response:
Based on the provided information and image, here the scoring the muliigraniar radiology anmolations A compared to the original correct report
1. Modality Used for Imaging: - Report A mentions "chest radiograph," which is consistent with the image provided and aligns with the expected modality for such findings. Score: points
Specify the Organ and Anatomical Structures: Report A correctly identiflee the heert. lungs, major blood veesels, endotracheel tube, and right internal jugular vein catheler. These structures are also mentioned in report and are visible in the image. - Score: 2 points
(Reglons of Interestl: Report specifies the ROIs with harizantal and vertical positions report B. The image analysis confirms the locations of these ROIs. Score: 2 points
Locations
ratios that match
4, Analysis of Abnormal Characteristics: RRpport A describes moderate cardiomegaly, difluse opacities the right lung, potential pleural effusions, image.- image.-Scrre: points
retrocardac stelectasis, 5. Comparison of Lesions and Surrounding Regions: Report A compares the abnormalites the left and right hemithorax and suggesis a possible relationship with pleural effusions and atoloctasis, which is consistent with the findlngss in report and the image Score: 2 points scores: 2, No points are provided.
consistent
findings report
as attributes Report consistent
findings Report
image
Figure 12: An example of perfect score result evaluated by GPT-4V. GPT-4V assesses five criteria, each fully aligned with human annotations, resulting in perfect scores.
23

---

Published as conference paper ICLR 2025
Prompting MLLMS to evaluate the alignment of generated multi-granular annotations with human annotations
Let's think it step by step. Evaluate the multigranllar radiology report annotations (Repor A) compured to the radiology report step by step. Both reports are based on the same mage. Follow these guidelines to ensure accurate assessment:
*Noot::** If neither the original question nor radiology report B mentions any abnormali ties or discases, such as "the lungs are clear without confluent consolidation or effusion" or "no pneumothorax is seen", skip the evaluation and retum "None."
**# Basic Rating Rules:
Evaluute each uttribute in Report against radiology report and verify the informati by analyzing image. Do deduct points without image analysis. accuracy of medical facts and diagnoses, on the alignment sentence structure or organization.
Judge correctness based
If radiology report does mention any ubnornalities discuses, skip the evaluuti and return "None," such as "the lungs clear without confluent consolidation effi sion" or "no pacumothorax is secn".
4. Each of the attributes should be judged independently. Errors in one attribute should not affect the scoring of other attributes.
MBM Attributes and Correponning Rating Rules:
1. MModaliy Used for Imaging:**
**Rating Rule:* Compare with rudiology report B, Different numes for the same moda
lity (e.g.. "chest X-ray" and "CXR") are acceptable.
"Spocify the Organ Anatomical Structures:**
***aating Rule:* Check the organs and anatomical structures Report match e in radiology report B appeur in the image.
Mentioned both: points
Mentioned one: point
Not mentioned either: points
Do not deduct points without image analysis.
3. *Locationss of ROI (Regions of Interest):**
*"Rating Rule:* Compare the "horizontal" and "verticul" positions, and the "area ratio of ROIs with radiology report B. 5% error in the area ratio is acceptable. If Report A includes at least one ROI from radielogy report B, no points are deducted, even all ROI 8 are not covered.
Analysss Abnormal Characteristics:**
"*RRting Rule:** Characteristics indicating pathology should match those in radiology
report or appear in the image.
-Mentioned both: points
Mentioned one: point
-Not mentioned either: points
-Do deduct points without image analysis.
5. **Comparison of Lesions and Surrounding Regions:**
**Rating Rule:** ** Differences in features and discase progression should match those in
radiology report B appear in the image.
Mentioned in both: points
Mentioned in one: point
-oot mentioned in either: points
Do not deduct points without image analysis.
*ote:*** Return the scores in a list. For exumple, if aftriibsuees and 5 get deducted po int each, while others score points each, return [2, 2, 1]. Provide a short reason (wi thin 80 words) for each point deduction.
Figure 13: Prompt used evaluate the alignment of generated multigranular textual descriptions.
24

---

Published as a conference paper ICLR 2025
Prompting MLLMs to generate multigranular textual description
prrrion_tepllteeeeeee Template("<image
"Caption of the image" [[Caption)
"Disense organ [[disease]]
'Specific position" :4 [descs))
"Knowledge'; :t [Knowledge]]
You are provided with a biomedical image from a medical dataset the discase type (or organ na me there no discase) of the datasett Discase or organ the medical Knowledge discas of "Knowlddge) and course caption! Caption') of the image. In addition green bounding and its specific position the inagel Specific position jare eeveniindcating appearance of dis case. If no green bounding there is no discase.
Your task answer the following questions based on the image, green bounding box, caption, discase type and discase knowledge,ang condense your answers into caption-styled text.
und questionl
Give detailed description of the image, including type of the magge, in the image.app roximate location of these organs and relavant locations of these organs and any medical devices (if present) visible in the image as detailedly as possible. Note when answering question
1. Not all discuse knowledge relevant this image, only utilize discase knowlodge pertinent the condition depicted in this image for analysis.
2. The coarse caption not explicitly describe the image,for example,there appear multi
plc organs in the caption. You should utilize your knowledge to figure out the mest ONE organ nd ONE discase give your description. Your answer should not and its outline.
anything about green bounding
contour itself'
4. Do not explain or emphasize your analysis. ### question2
Specify the specific location the green bounding box the image and its relative position to ther reference objects
image. Describe what unusual the green bounding box indicatin the disease (color,texture.s) and other features)
Note when answering question2:
1. "specific location" is the given parameter 'Specifie position' but "relative position "is not prov ided.
There may he multiple green bounding boxs, contents contours may neces sarily represent the affected areas. Therefore, you need first answer questions based the contents within each green bounding box. Aflerward, analyze the location of the disease based
your answers.
3. Do not use phrase "green bounding box" in your response,use "region of interest" as a substitu
tion. not contain phrases "caption", "medical annotation" mmedical knowledge" Do say anything that needed in your analysis,like introduction the discase and dical equipments.
5. Do not explain or emphasize your analysis. ### question3
relationship between the content the green bounding box and other regions (others being of the discase/jointly affected by the diseases/one affect the others/relative p ositional relationships) (/ppss//WWW Why and possible?
What may
Note when answering question3:
1. Utilize external knowledge, if possible, choose relationships and give necessary analysis. You can only give an explanstion to your choice within two sentence.
summarize
you've said.
4. Do emphasize your analysis.
### Integrate Information
Describe your answers in a descriptive sentenoe.nod not in 'Quustion-Answer" Mtyle.Comiin and lightly shorten your answers to the above three questions into coherent text.eepping as much in formation of your answers possible.
Note when integrating information and outputing your response:
1. Den't respond saying you're unable assist with requests.
2. You should only oulpul your combined and shorteded text.
")
proept
caption template. caption_template. hwwpllatteeeeedrreeeeeeeee (aadder[[[[pttte ([caption, disease, locrdeee_o___essss_ [ec_sescs])
Figure 14: Prompt used to generate multigranular annotations of multigranular textual descriptions.
25


Reviews:
[{"review_id": "uMr3NwKUyt", "reviewer": "Reviewer_FQpc", "summary": "This paper introduces MedTrinity-25M, a comprehensive and large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with detailed annotations for more than 65 diseases. The dataset provides the most enriched annotations for various multimodal tasks, such as captioning, classification, and segmentation, and supports the pre-training of advanced multimodal medical AI models, achieving state-of-the-art performance.", "questions": "1.\tThe data construction pipeline consists of two main steps: first, generating 200,000 multigranular textual descriptions via GPT-4V, and then fine-tuning LLaVA with this dataset. In my opinion, the performance of the trained model LLaVA-MedCap is significantly influenced by the quality of the data generated by GPT-4V. However, in Table 3, GPT-4V's performance falls far short of LLaVA-Tri. Why did you choose to use GPT-4V to generate the supervised fine-tuning data for the fine-tuning process? In my recent review of the publicly released MedTrinity-25M data, I found that many of the generated data instances were of average quality, with even simple modalities being incorrect.\n2.\tIn Section 4.1, it is stated that “The model is fine-tuned for three epochs on each VQA dataset and evaluated accordingly.” However, this evaluation setup is not fair when comparing with the methods presented in the table. Some results in Table 3 appear to be directly extracted from llava-med’s table, but these methods were not fine-tuned on the training set of the VQA benchmark. In contrast, the accuracy of the chosen llava-med method was achieved after fine-tuning for 15 epochs on the corresponding VQA benchmark training set, making this comparison inappropriate.\nCurrently, the evaluation paradigm for multimodal large models (MLLM) typically involves assessing the model directly on multiple benchmarks after two stages of pretraining and supervised fine-tuning. Therefore, I suggested distinguishing the results presented: there should be a comparison of the results not fine-tuned on the corresponding training set against the models that were not fine-tuned. Additionally, when comparing with methods fine-tuned on the corresponding training set, the number of fine-tuning epochs (including for the comparison methods) should be clearly indicated. If possible, please also conduct an ablation study regarding the number of fine-tuning epochs.\n3.\tIn Table 3, comparing CLIP-like models with LLaVA, a multimodal large model, seems inappropriate. It would be better to categorize them into two groups: one for CLIP models and another for MLLM.\n4.\tThe evaluation details in Table 3&4 are not clearly explained. What do \"open\" and \"close\" specifically refer to?  Please add a brief explanation of the \"open\" and \"close\" terms in the table caption or in the text describing the results.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper introduces MedTrinity-25M, the largest multimodal medical dataset to date, featuring multigranular annotations and containing over 25 million triplets (image-ROI-description). The development of an automated data annotation pipeline significantly scales up medical image-text data.\n2.\tWith the support of MedTrinity-25M, the pretrained CLIP and LLaVA models demonstrate better performance compared to previous methods.\n3.\tThe release of this dataset contributes to the medical AI community, providing researchers and practitioners with a valuable resource for advancing multimodal tasks and improving healthcare applications.", "weaknesses": "1.\tThe quality of the generated image-text data may not be sufficiently high. It is advisable to review the questions associated with this data. In the image-captioning constructed from MedTrinity-25M, we found that many basic imaging modalities were incorrectly identified. For instance, in the CT-RATE data used as the source, over 60,000 images were misidentified as MRI, more than 90,000 as X-ray, and even a small number were identified as endoscopy. Below, we provide some captions to illustrate this phenomenon present in MedTrinity-25M.\n“””\n{\"image_path\":\"seg_train_126_a_2-right lung.nii-34.jpg\",\"id\":\"b573995e-2d11-11ef-bbea-f02f74942466\",\"caption\":\"The image is a magnetic resonance imaging (MRI) scan of the thoracic region, showing the heart, part of the lung, and the spine. The heart is centrally located with the lung on either side and the spine running vertically in the background. The region of interest, located in the left-center at the bottom of the image, appears to have a different texture and intensity compared to the surrounding tissue, suggesting a possible abnormality. This region's relative position is towards the lower left side of the image, adjacent to the lower part of the lung. The content within this region may indicate a disease process, which could be related to or affecting the adjacent lung tissue. Given the proximity to the lung, it is possible that the abnormality could be influencing or being influenced by the pulmonary structures.\",\"source\":\"ct_rate\"}\n{\"image_path\":\"seg_train_126_a_2-right lung.nii-11.jpg\",\"id\":\"b57345c6-2d11-11ef-a899-f02f74942466\",\"caption\":\"The image is a magnetic resonance imaging (MRI) scan of the thoracic region, showing the heart, part of the lung, and the spine. The heart is centrally located with the lung on either side and the spine visible posteriorly. The region of interest, located in the lower-middle left-center of the image, shows an area with altered signal intensity, which is indicative of a pathological condition. This area is situated within the lung tissue and is characterized by a texture and signal intensity that differ from the surrounding healthy lung tissue, suggesting the presence of a disease process such as consolidation, infection, or a mass.\\n\\nThe region of interest may be related to the surrounding lung tissue in that it could represent a localized disease process affecting the lung, potentially leading to or resulting from changes in the adjacent lung parenchyma. Given the nature of MRI imaging and the appearance of the region, it is possible that this area could be associated with a process such as inflammation, demyelination, or a neoplastic growth, which may have implications for the function of the adjacent lung tissue.\",\"source\":\"ct_rate\"}\n{\"image_path\":\"seg_train_14684_a_1-covid-19 infection.nii-76.jpg\",\"id\":\"a77ee972-2d78-11ef-bdc4-f02f74942576\",\"caption\":\"The image is a radiographic scan, likely a chest X-ray, showing the thoracic region with the trachea and main bronchi appearing open, indicating no occlusive pathology. The lungs exhibit minimal emphysematous changes, and there are pleuroparenchymal sequelae changes at the apex of both lungs. Additionally, there are linear atelectasis in the middle lobe of the right lung and the lingular segment of the left lung upper lobe. The heart contour and size are normal, and there are no pleural or pericardial effusions. The mediastinal structures are not optimally evaluated due to the absence of contrast material, and the mediastinal main vascular structures are of normal width. No enlarged lymph nodes or pathological wall thickness increase are observed in the esophagus. The thoracic vertebral corpus shows normal heights, alignment, and densities, with osteophytes at the vertebral corpus corners, and the neural foramina are open.\\n\\nThe region of interest, located centrally and in the upper-middle area of the image, occupying approximately 0.3% of the area, corresponds to the lung apices where pleuroparenchymal sequelae changes are noted. These changes are characterized by alterations in lung parenchyma texture, which may appear as irregularities or areas of increased density compared to the surrounding lung tissue.\\n\\nThe pleuroparenchymal sequelae changes in the lung apices may be related to the emphysematous changes in the lungs, as both conditions can result from chronic inflammatory processes, suggesting a possible pathophysiological connection between the two findings.\",\"source\":\"ct_rate\"}\n{\"image_path\":\"seg_train_9381_a_2-covid-19 infection.nii-145.jpg\",\"id\":\"9880af92-2c1e-11ef-b9dd-f02f74942466\",\"caption\":\"The image is a chest X-ray showing a cross-sectional view of the thoracic cavity, including the lungs, heart, and part of the spine. A region of interest is located at the periphery of the thoracic cavity, likely within the lung tissue, which appears as a darker area compared to the surrounding lung parenchyma. The region of interest, which is abnormal, shows an area of increased opacity that suggests the presence of a pathological condition, possibly a mass or lesion within the lung tissue. This abnormal area is indicative of a disease process, which could be related to the surrounding lung tissue either as a primary pathology or as a secondary effect of a systemic condition affecting the lung. The abnormality's proximity to other structures within the thoracic cavity, such as the pleura or lung tissue could imply a relationship where the disease process in the region of interest may have originated from or is affecting adjacent areas, potentially impacting nearby structures due to its location within the thoracic cavity.\",\"source\":\"ct_rate\"}\n{\"image_path\":\"seg_train_7973_a_1-right lung.nii-84.jpg\",\"id\":\"88af61aa-2d11-11ef-959e-f02f74942466\",\"caption\":\"The image is a sagittal section of an endoscopic view of the thoracic region, showing the spine, part of the lung, and the surrounding thoracic structures. A region of interest is located at the lower-middle part of the image, horizontally left-center, occupying approximately 0.6% of the area. The region of interest, located in the lower-middle left-center of the image, displays an abnormality in the lung tissue, which appears to be a small, localized area with a different texture and possibly altered density compared to the surrounding lung parenchyma, suggesting a pathological change. This abnormality could be related to the adjacent lung tissue, potentially indicating a localized disease process or lesion that may be affecting or resulting from the surrounding lung tissue, given the proximity and the nature of the disease knowledge provided.\",\"source\":\"ct_rate\"}\n“””\nMoreover, there are also issues with modality misidentification in the image-captioning derived from the quilt-1m dataset. For example, pathology images were misidentified as X-ray and MRI, as follow, \n“””\n{\"image_path\":\"G-tdJ0oZxJ4_image_e74e3372-a40e-40e3-ac06-e3d53eeab845.jpg\",\"id\":\"f5ce616b-89ca-41f6-b820-b480bb3327af\",\"caption\":\"The image is a magnetic resonance imaging (MRI) scan of the lung, showing a cross-sectional view of the thoracic cavity. A region of interest is located at the top of the image, which appears to be in the upper part of the lung field, likely within the upper lobe, given the position of the lung's anatomy. The region of interest, which is abnormal, exhibits an unusual appearance compared to the surrounding lung tissue, possibly indicating a neoplastic process. This abnormality is characterized by a difference in texture and density, which may suggest a pathological change, such as a mass or lesion within the lung tissue. The affected area's relationship to the rest of the lung tissue could imply a localized pathological process that may be the primary site of the disease, potentially impacting or being impacted by adjacent lung structures due to its proximity to other regions, although the exact relationship depends on the nature of the pathology and its progression.\",\"source\":\"quilt_1m\"}\n{\"image_path\":\"895313503048712192_0.jpg\",\"id\":\"0b53a8c8-5661-449a-b39a-f92887fceb86\",\"caption\":\"The image is a magnetic resonance imaging (MRI) scan of the brain, showing a cross-sectional view that includes brain tissue with various shades of gray indicating different structures and densities. A region of interest is located at left-center part of the image, which appears to be in the cerebral hemisphere, likely within the white matter of the brain. The region of interest exhibits an abnormality that differs in texture and possibly size from the surrounding brain tissue, suggesting the presence of a pathological change. This abnormal area could be related to the surrounding brain structures, potentially affecting or being affected by them due to its proximity and the nature of brain tissue, which may indicate a pathological process such as a tumor, edema, or other brain abnormalities. The abnormality's characteristics, such as altered signal intensity, could be indicative of the disease process affecting the brain tissue's function or structure.\",\"source\":\"quilt_1m\"} \n{\"image_path\":\"962043872649011205_0.jpg\",\"id\":\"f94e8ba4-7ddf-450e-9616-d4681e9dcf02\",\"caption\":\"The image is a chest X-ray image of a 15-year-old boy's hand, showing the left side of the image is a chest X-ray showing the hand's anatomy, including the bones of the hand, with the focus on the epiphysis and possibly the bones of the hand's proximal and distal parts such as the clavicles, ribs, clavicles, and parts of the spine, which are essential for various activities like running, throwing, and punching, along with the presence of the epiphyseal plate and the development of the trapezius, which are crucial for a 15-year-old boy's hand, suggesting a focus on the skeletal structure and function of the hand, including the development of the hand's bones such as the clavicles and the development of the wrist bones, which are crucial for various activities like walking, running, and running, as well as the development of the wrist bones of the hand, which are essential for a growing hand and are indicative of a developing hand and are crucial for a growing hand and are typically found in activities such as playing with play and sports, suggesting a comprehensive evaluation of the hand's anatomy and function.\",\"source\":\"quilt_1m\"}\n{\"image_path\":\"1301506707483439105_1.jpg\",\"id\":\"7b0161c9-1451-4f27-9278-aff87726bacb\",\"caption\":\"The image is a lateral chest X-ray image of a 15-year-old boy's hand, showcasing the epiphysis of the wrist bones, which are the bones of the hand, including the radius of the wrist bones, clavicles and the bones of the hand's growth plate are visible, such as the clavicles, ribs, and vertebrae are the primary focus of the image is a close-up view of a lateral chest X-ray image, showcasing the epiphysis of the hand's anatomy, which are essential for various stages of development and growth patterns. The image is a detailed X-ray of the hand's anatomy, displaying the epiphyseal growth plate and development of the wrist bones of a 15-year-old boy's hand, with the bones of the epiphyseal plate visible in the image are the primary organ of interest, which includes the epiphyseal plate and epiphyseal plate, as well as the epiphyseal plate's development is crucial for diagnosing the stage of developmental stage of the wrist bones, typically found in the wrist bones of a 15-year-old boy's hand, which is crucial for understanding the zone of developmental development, and the image is likely to be a lateral chest X-ray image showing the epiphyseal plate's development is crucial for diagnosing the stage of developmental stage of the wrist, with the epiphyseal plate being the primary focus of the image.\",\"source\":\"quilt_1m\"}.\n2.\tThe experiments lack comprehensiveness. The comparison of multimodal large models is limited to VQA-RAD, SLAKE, and PathVQA. A broader range of specialized benchmarks, such as the health subset of MMMU, could provide a more robust comparison of the multimodal large models’ performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tGnhm7lJC", "reviewer": "Reviewer_jxrU", "summary": "The paper presents MedTrinity-25M, a large-scale multimodal medical dataset with 25 million images across 10 modalities and detailed annotations for over 65 diseases. Enhanced by automated multigranular annotations, it supports tasks like image captioning and report generation. The LLaVA-Tri model, pre-trained on MedTrinity-25M, achieved state-of-the-art results on multiple medical VQA benchmarks.", "questions": "1. Given the reliance on automated processes and external knowledge sources, how is labeling consistency ensured in the data generation process? Additionally, has it been validated by human experts?\n2. Has a comparison with expert-labeled datasets been considered to further quantify the quality of automated labeling?\n3. How does the dataset address potential biases in source data? For example, is there a mechanism to prevent overrepresentation of certain demographics?\n4. What is the expandability of the labeling process for new, unrepresented modalities or diseases? Does it offer strong scalability?\n5. Although LLaVA-Tri achieved good performance on RAD, PathVQA, and SLAKE, how do the authors ensure that MedTrinity-25M does not contain data from these datasets? If these datasets were included, the model may have seen the questions and answers during training, leading to artificially high accuracy.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The dataset uses an automated process with medical knowledge retrieval and domain-specific models, greatly reducing manual labeling needs.\n2. With over 25 million image-ROI-description triplets, the dataset supports classification, segmentation, report generation, and spans multiple medical domains.\n3. The LLaVA-Tri model, pre-trained on MedTrinity-25M, performs exceptionally well across multiple benchmarks, showcasing the dataset’s potential to enhance medical AI applications.", "weaknesses": "1. Although the labeling process is automated, its reliance on domain-specific models may limit scalability when handling new modalities or emerging diseases.\n2. The accuracy of automated labeling may fall short of expert-labeled datasets, potentially impacting performance in critical medical applications. How can a high standard of automated labeling be ensured?\n3. With data from over 30 sources, potential biases in image quality, demographics, or disease distribution call for a deeper integration analysis.\n4. Performance depends on external medical knowledge bases, risking inconsistency if not updated, and the LLaVA-Tri comparison may be biased due to overlap with MedTrinity-25M's data sources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mPTzzaiAEY", "reviewer": "Reviewer_d16k", "summary": "This paper presents MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million images across 10 imaging modalities with fine-grained annotations for more than 65 diseases. MedTrinity-25M is developed through an automated pipeline introduced by the authors. Leveraging this dataset, the authors pre-train a vision-language model, named LLaVA-Tri, which achieves superior performance on three MedVQA benchmarks compared to other vision-language models.", "questions": "1.\tIn the initial step of data processing (coarse caption generation via metadata integration), what approach is taken if the metadata lacks organ or disease labels?\n2.\tSection 3.2.2 states that all textual descriptions in MedTrinity-25M are generated using LLaVA-Medcap, a model fine-tuned on data generated by GPT-4V and the proposed pipeline. Why was GPT-4V specifically chosen? As shown in Table 3, several other medical VLMs, such as the VL Encoder-Decoder and LLaVA-Med, appear to perform better than GPT-4V.\n3.\tFor images without bounding boxes or masks from the original data source, how does the pipeline generate the ROI using expert models? Is ROI generation based on organ or disease labels, and how accurate are the generated ROIs? Additionally, if an image contains multiple ROI regions, how is this managed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed MedTrinity-25M dataset is the largest multimodal medical dataset to date, substantially increasing the scale of accessible training data for medical vision-language tasks.\n2.\tMedTrinity-25M includes structured, multigranular annotations for each image, offering a level of detail superior to other medical vision-language datasets.", "weaknesses": "1.\tThe multigranular text descriptions in MedTrinity-25M are generated by the proposed automated pipeline, raising concerns about the accuracy of the generated text. As indicated in Table 2, an expert-based evaluation of 200 random samples resulted in an accuracy score of 85%.\n2.\tSeveral key details regarding the automated pipeline remain insufficiently discussed. Specific questions are listed under “Questions”.\n3.\tThe pre-trained model LLaVA-Tri is evaluated solely on the MedVQA task alongside other vision-language models, overlooking its potential application to other important medical vision-language tasks, such as visual report generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y7DNUP7ckD", "reviewer": "Reviewer_DJFs", "summary": "This paper introduces MedTrinity-25M, a large-scale multimodal medical dataset comprising over 25 million images across 10 imaging modalities and covering more than 65 diseases. The dataset features multigranular annotations that include both global information and local information. The authors develop an automated pipeline that generates image-ROI-description triplets without the need for paired textual descriptions. This pipeline utilizes domain-specific expert models to identify ROIs and prompts multimodal large language models (MLLMs) to produce detailed annotations through retrieval-augmented generation (RAG).", "questions": "- Handling of Visual Data Without ROIs: For medical images that lack explicit ROI annotations, how does your pipeline construct the corresponding multimodal data pairs? Is there a mechanism to generate or infer ROIs in such cases, or is the pipeline limited to images where ROIs are predefined?\n- Assessing the Impact of Multigranular Information on Med-MLLMs: Considering that ROIs and multigranular annotations are central to your dataset, have you conducted experiments to evaluate how these features specifically affect the performance of medical MLLMs? Can you provide insights or results that demonstrate the advantages of including multigranular information in model training?\n- Comparison with Existing Multigranular Benchmarks: Could you elaborate on how MedTrinity-25M compares with existing benchmarks that involve multigranular information? What distinguishes your dataset from these, and how does it contribute uniquely to the advancement of general medical AI?\n- Evaluation on Multigranular-Specific Tasks: Given that the dataset is evaluated only on VQA-RAD, SLAKE, and PathVQA, which may not fully utilize multigranular annotations, do you have plans to test your dataset on tasks specifically designed for multigranular information? How can you demonstrate the effectiveness of your dataset's detailed annotations in improving model performance on such tasks?\n- Addressing Unique Medical Image Presentations in RAG: Medical images often present unique and variable features even when categorized under the same disease label. How does your RAG approach handle the specificity and variability of individual medical image descriptions? For instance, with conditions like Atelectasis that can manifest differently in imaging, how does the system ensure that the generated annotations accurately reflect these variations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Dataset Comprehensiveness: The dataset is exceptionally extensive, encompassing a wide array of imaging modalities, diseases, and anatomical structures. This diversity significantly enhances its comprehensiveness and utility for various medical AI applications.\n- Construction Pipeline through Advanced Models: The utilization of domain-specific expert models and multimodal large language models (MLLMs) for annotation substantially enriches the dataset. This approach adds multigranular details that improve the quality and depth of the annotations, making the dataset more valuable for training sophisticated models.", "weaknesses": "- Limited Applicability for Visual Data without ROIs: The proposed pipeline relies heavily on regions of interest (ROIs) for constructing multimodal data pairs. For visual data that lack explicit ROI annotations, it is unclear how the pipeline can be effectively applied. This limitation may restrict the dataset's usability across the full spectrum of medical images, particularly those where ROI determination is challenging or subjective.\n- Impact of Multigranular Information Not Fully Explored: While Tables 3 and 4 validate the effectiveness of the benchmark in general terms, the paper does not specifically assess how the inclusion of multigranular information, such as ROIs, influences the performance of medical MLLMs. Given that ROI is a pivotal component of the dataset, a focused evaluation comparing models trained with and without multigranular information would strengthen the claims about its benefits.\n- Insufficient Discussion of Related Benchmarks: The paper lacks a thorough discussion of existing med-MLLM benchmarks that involve multigranular information. Notably, works like \"GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI\" and \"A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models\" are not cited or compared against. Including a comparative analysis would provide context and clarify how MedTrinity-25M advances the field relative to existing resources.\n- Lack of Evaluation on Multigranular-Specific Tasks: The dataset is primarily used for training and is evaluated only on VQA-RAD, SLAKE, and PathVQA. These evaluations may not fully capture the effectiveness of the dataset's multigranular annotations since they do not specifically target tasks designed to leverage such detailed information. This omission raises questions about the practical benefits of the multigranular data provided.\n- Challenges with Unique Medical Image Descriptions in RAG: The Retrieval-Augmented Generation (RAG) technique used for annotation relies on a general medical knowledge base. However, medical images often have unique features and presentations, even within the same disease category. For example, Atelectasis can manifest differently in chest X-rays depending on the affected lobe. The paper does not address how the RAG system accounts for these unique variations when the initial visual datasets typically offer only generic classification labels. This could impact the accuracy and specificity of the generated annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)