MiniGPT-Med: A Unified Vision-Language Model for Radiology Image Understanding
Authors: Asma Alkhaldi
Raneem Alnajim
Layan Alabdullatef
Rawan Alyahya
Jun Chen
Deyao Zhu
Ahmed Z. Alsinan
Mohamed Elhoseiny
Venue: Accepted by TMLR ()

Abstract: Recent advances in artificial intelligence (AI) have precipitated significant breakthroughs in healthcare, particularly in the refinement of diagnostic procedures. However, existing studies have been limited in terms of functional coverage. This study introduces MiniGPT-Med, a vision-language model adapted from MiniGPT-v2 for medical applications through domain-specific fine-tuning on medical datasets. MiniGPT-Med demonstrates remarkable versatility across various imaging modalities, including X-rays, CT scans, and MRIs, enhancing its utility. The model is capable of performing tasks such as medical report generation, visual question answering (VQA), and disease identification within medical imagery. Its integrated processing of both image and textual clinical data markedly improves diagnostic accuracy. Our empirical assessments confirm the superior performance of MiniGPT-Med in disease detection, medical report generation, and VQA benchmarks, representing a significant step towards reducing the gap in assisting radiology practice. Furthermore, it achieves state-of-the-art performance in medical report generation, with substantial gains in BERT-Sim over both specialist and generalist baselines, improving by 17 and 12 points, respectively. MiniGPT-Med promises to become a unified Vision-Language model for radiology diagnoses, enhancing diagnostic efficiency across a wide range of medical imaging applications.

Published in Transactions on Machine Learning Research (02/2026)
MiniGPT-Med: A Unified Vision-Language Model for Radi- ology Image Understanding
Asma Alkhaldi Raneem Alnajim Layan Alabdullatef Rawan Alyahya', Jun Chen', Deyao Zhu?, Ahmed Alsinan Mohamed Elhosciny
Johns Hopkins University (JHU)
King Abdullah University of Science and Technology (KAUST)
Saudi Data and Artificial Intelligence Authority (SDAIA)
aalkhal6@jhuu.co edu, mohamed. Cathaamee.lhooscnyy@@aaa..e whhoseiny@kassttt..mmmmmomoom... edu.sa
Reviewed on OpenReview: httpp://pppaeeri....eeeoo//oooop
Abstract
Recent advances in artificial intelligence (AI) have precipitated significant breakthroughs in healthcare, particularly in the refinement of diagnostic procedures. However, existing studies have been limited in terms of functional coverage. This study introduces MiniGPT-Med, vision-language model adapted from MiniGPT-v2 for medical applications through domain- specific fine-tuning on medical datasets. MiniGPT-Med demonstrates remarkable versability across various imaging modalities, including X-rays, CT scans, and MRIs, enhancing its utility. The model capable of performing tasks such as medical report generation, visual question answering (VQA), and disease identification within medical imagery. Its integrated processing of both image and textual clinical data markedly improves diagnostic accuracy. Our empirical assessments confirm the superior performance of MiniGPT-Med in disease detection, medical report generation, and VQA benchmarks, representing significant step towards reducing the gap in assisting radiology practice. Furthermore, it achieves state- of-the-art performance in medical report generation, with substantial gains in BERT-Sim over both specialist and generalist basclines, improving by 17 and 12 points, respectively. MiniGPT-Med promises to become a unified Vision-Languaa model for radiology diagnoses, enhancing diagnostic efficiency across a wide range of medical imaging applications. Our model and code have been made publicly available on GitHub https://github.comm/ /Vision- CAIR/MiiiGPT-Med
1 Introduction
The unprocedented surge in both the quantity of image-text data across diverse fields and the strides made in vision-language modeling have paved the way for groundbreaking research in Generalive Pretraining, This era of innovation is marked by the emergence of multimodal models such as GPT-4 Achiam et al. (2023) and Gemini Team et al. (2023). These advancements signify a leap forward in our ability to process and understand complex data. Despite this progress, the adoption of Multi-modal Large Language Mod- els (LLMs) within the medical sector remains limited. The medical field's unique requirements for data complexity, sensitivity, and specificity highlight the need for tailored approaches to harness the potential of LLMs in transforming healthcare rescarch and practice. Numerous models designed for medical applica- tions have been introduced, yet. they often exhibit a high degree of specialization for specific tasks. This specialization limits their versatility, particularly in performing diverse medical applications. For instance,
"Corresponding author.

---

Published in Transactions on Machine Learning Research (02/2026)
models like Med-Flamingo Moor et al. (2023) and XrayGPT Thawkar et al. (203b) are primarily tailored for lasks such as medical report generation and medical visual question answering, respectively. However, they lack capabilities in essential areas like disease detection, which requires visual grounding skills a crucial component in the medical field. To address this deficiency, we introduce MiniGPT-Med, a unified model capable of adeptly handling both grounding and non-grounding tasks. We introduce MiniGPT-Med, a versatile model designed for various tasks in the medical domain, including but not limited medical report generation, medical visual question answering, and disease identification. MiniGPT-Med builds upon the architecture of large language models (LLMs), which have demonstrated exceptional generative capabil- ities and extensive linguistics, including medical knowledge. Drawing on the successes of LLMs in wide range of vision-language applications, as evidenced in recent studies Zhu et al. (2023); Chen et al. (2023): ) Li
et al. (2024), our model adopts a design similar to MiniGPT-v2 Chen et al. (2023). utilizing the LLaMA-2 language model as a universal interface. Additionally, we incorporate distinct task identifiers to enhance the model's ability to accurately perform various medical vision-language skills. Through extensive exper- imentation, we have retonstrated that our model exhibits strong performance across a range of medical rision-language tasks, including medical report generation, medical visual question answering, and disease detection. We benchmarked our model against both specialized and generalized baseline models, revealing that our approach achieves strong results across all evaluated tasks. Notably, in the domain of medical report generation, our model attained state-of-the-arl performance, surpassing the best baseline models by 19% in BERT-Sim and 5.2% in CheXbert- Sim. This indicates our model has strong generation capabilities on diverse medical vision-language tasks.
Our contributions are as follows:
1. We introduce MiniGPT-Med, medical-domain adaptation of MiniGPT-v2 that retains the original architecture while extending it to heterogeneous radiological modalities, including X-rays, CT scans, and MRIs. MiniGPT-Med is fine-tuned on diverse collection of medical datasets and supports multiple vision-language tasks such as disease identification, medical visual question answering, and radiology report generation within a unified task identifier framework.
We conduct a comprehensive evaluation across grounding and non-grounding tasks, complemented by expert. manual assessment. On medical report generation, MiniGPT-Med achieves substantial improvements in terms of BERT-Sim, outperforming the strongest specialist baseline by 17 points and exceeding the strongest generalist baseline (Gemini) by 12 points (72 vs. 60) under the same evaluation protocol.
2 Related Work
Aligning Visual Data with Large Language Models: Recent advancements in the domain of large language models, such the release of GPT-4, have enhanced the interpretative and generative capabilities of LLMs. This progress is exemplified by models such LLaVA Liu et al. (022)) Flamingo Alayrac et a.. [2022), and MiniGPT-v2 Chen et al. (2023). LLaVA is designed to augment the understanding of visual content in large language models through diverse multimodal instructions. This enhancement in comprehension is critical for integrating different forms of data input. In contrast, Flamingo demonstrates remarkable proficiency in quick adaption to novel tasks with minimal data. This model effectively manages sequences that incorporate both visual and textual elements. MiniGPT-v2, on the other hand, displays enhanced multimodal capabilities within singular model framework. This is achieved through task-specific training and specialized architecture that combines visual tokens with large language model, aligning well with the objectives of LLa and Flamingo.
While the aforementioned models primarily rely on supervised fine-tuning to align visual representations with LLMs, recent work has explored reinforcement learning (RL) as an alternative mechanism to encourage explicit reasoning. MedVLM-R1 Pan al.) (2025) introduces an RL-basod medical vasion-language model that generates natural language reasoning alongside final predictions for radiology visual ques- tion answering. Instead of relying on large-scale supervised datasets chain-of-thought annotations,
2

---

Published in Transactions on Machine Learning Research (02/2026)
Detect more than 14 ciseases like: Pneumonia, Edema, Brain tumor, Lung cancer... ... etc
Accept all medical modalities
Trained on 5 medical datasets
X-ray
MIMIC
RSNA
miniGPT-Med
NLST
RadvQA
MRI
SLAKE
CT
Provides different vision-language tasks
Benchmark
Detection
Captioning
VQA
Grounding
Refer
Identify
miniGPT-Med (ours)
VA-Med
X
X
X
X
RadFM
X
X
X
X
X
XrayGPT
X
/
x
x
x
X
CheXagent
X
/
x
x
X
MedKLIP
X
X
X
X
X
BioViL
X
X
X
X
Figure 1: MiniGPT-Med demonstrates diverse capabilities across multiple clinical tasks, including discase detection, medical visual question answering, and medical report generation. It operates effectively on various radiological modalities such as X-rays, CT scans, and MRIs, and is capable of recognizing and diagnosing a broad range of conditions.
McdVLM-R1 employs Group Relative Policy Optimization (GRPO) incentivize emergent reasoning using only fial-answer supervision. Despite being trained on a limited number of samples and a compact backbone, the model demonstrates strong generalization across imaging modalities such MRI, CT, and X-ray, outperforming larger supervised counterparts. This line of work highlights the potential of RL to enhance robustness in medical vision vision-language language models beyond traditional supervised alignment strategies.
Integration Vision Language Models for Enhanced Medical Diagnostics: Recent work vision- language models has led to significant. improvements in healthcare applications, especially medical image analysis and diagnostic report generation. Utilizing VLMs in medical diagnostics marks a significant progres- sion in the healthcare industry. Models combine computer vision and language processing to better analyze medical images like X-rays, computed tomography (CT), and MRTs. More specialized applications ions in the medical field such as LLaVA-Med Li et al. (2024) and Med- BERT Rasmy et al. (2020) have shown promise in incorporating structured electronic health records for improvements in disease prediction tasks. MedVQA Canepa et al. 2023 has demonstrated medical visual question annweerig and image analysis capabilities. Furthermore, for classification and interpretation tasks, Med-Flamin Flammnlaonngo Moor et al. (2023) MedVis Shen et al. (2008), and MedMCQA Pal et al. (2022) have shown the importance of few-shot learning, visual inter- pretation, and domain-specific question-answeringg in medical AI. Both LLaVA-Med and Med-Flamingo focus multimodal conversational Al and few-shot learning in medical contexts, utilizing large-scale datasets and showcasing proficiency in visual question answering. Vio ban 2023). BioBERT Lee et al. (2019). and BioGPT Luo et. al. (2022) all have tackled a more domain-specifie language model pretraining. BioViL em- phasizes text semantics for enhanced biomedical lasion-language processing. Emphasis on specialized models for radiology applications has also been presented in MedKLIP Wu et al. (2023a), XrayGPT Thawkar et al.) (2023a), and BERTHop Monajatipoor al. (2021) all demonstrating the challenge of achieving high diag- nostic accuracy. MedKL.IP in particular innovates by integrating medical knowledge into vision-language
3

---

Published in Transactions on Machine Learning Research (02/2026)
pre-training for improved disease classification. XrayGPT integrated medical visual encoder with large language model to combine visual and textual analysis to generate precise summaries from radiological data, while HERT showed diagnostic performance with smaller datasets on chest X-rays. Moreover, the con- tributions of CheXagent Chen et al. 20024, CheXNeXt Rajpurkar et al. (2018), and CheXpert [rvin et al. (2019) have set benchmarks in chest pathologies detection. While each work presents unique approaches, their common goal to enhance radiological analysis through sophisticated AT models.
3 Method
3.1
Model architecture
is composed of three key components: visual backbone, a linear projection layer, and extensive language model. The details of each component described follows.
Our model architecture, illustrated in Fig 2
[NST] cimages
concat
Large Language Model
Vision Encoder
nodule (<29> <43> <42><56>>
<Almn> [refer] What image indicates cancer? [ANST]
Figure 2: Overview of MiniGPT-Med Architecture: The model integrates vision encoder, linear projection layer, and large language model (LLM). A single medical image is processed into visual semantic features via the pre-trained vision encoder, which is then transformed into a single visual token. This token is mapped to the LLM's space using the linear projection layer. During training, the vision encoder remains frozen, while the LLM and projection layer are fine-tuned for optimal performance.
Vision Encoder. We use EVA Fang et al. 2022 as our vision backbone, which has been pre-trained on large-scale, publicly available datasets that capture essential visual representations such as shapes, patterns, and textures. To preserve the benefits of this pretraining and mitigate the risk of overfitting during fine- tuning, initially freeze the vision encoder and fine-tune only the projection layer to ensure effective alignment with the textual modality. In cases where overfitting is observed, we explore unfreczing the encoder for further adaptation.
Large Language Model (LLM). Our model integrates LLaMA2-chat (7B) [Tuvron et al. 2023) an open- source language model, as a unified interface for broad range medical vision languag tasks. Leveraging flexible architecture, our system is capable of generating comprehensive medical reports and accurately localizing tumors in imaging data.
4

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: sask-speciiic instruction format. < FmaurFe > denote the image features. During our model training, we used six different types of task identifiers for diverse grounding and non-grounding tasks.
Task Types
[dentifier] Instruction
Caption
caption Could you describe the contents of this image for me?
VQA
[vqa] What plane is the image in? [detection] pneumonia [refer] the nodule in the left lung [grounding] describe this image detail [identify] what is this <50><16><84><65>
Detection
Refer
Grounding
Identify
Vision-Language Alignment. We enhance the model's efficiency by reducing the sequence length of visual tokens from the vision encoder, merging lour adjacent tokens into a single embedding. This stralegy helps maintain manageable sequence length, enabling faster processing and more efficient computation. The resulting embeddings are then mapped into the language model's feature space through a linear projection layer. Importantly, this concatenation strategy does not sacrifice critical contextual information. Instead, it preserves spatial relationships and interactions among visual elements, which are essential for tasks like VQA.
3.2 Prompt Template
Our prompt template enables the model to handle diverse medical vision language tasks, including visual question answering, image captioning, referring expression comprehension (REC), referring expression gen- eration (REG), disease detection, and grounded image captioning. To mitigate hallucinations and confusion common in handling various tasks, such as misidentifying lung tumors as calcifications in blood vessels or the heart, we incorporate task-specific tokens into the training framework. This ensures clarity and precision in multi-task environments. Our instruction design closely follows the template of MiniGPT-v2:

 < <Tmg>> < Teature
</Img>
(Task Identifier/ Instruction I/INST)
We present diverse prompt templates in Table to demonstrate how our model effectively deals with the different tasks through task identifiers.
3.3 Region grounding representation.
For tasks like discase detection and grounded image captioning, we represent bounding boxes as text, inte- grating spatial data into the language model. Coordinates are normalized to the [0, 1000 range and formatted as text, Each spatial location expressed in the format:
{< Xteft >< Moo >< Xright >< Ybottom >}
4 Experiments
4.1 Dataset Setup
The scarcity of quality medical datasets is significant challenge in deep learning for medical imaging, To address this, we focused on curating comprehensive radiology dataset, particularly for lung discases and general medical information. Our collection includes diverse medical images (X-rays, CT scans, MRIs) and enriched datasets with bounding boxes, question and-answer formats, and report generation to enhance Johnson (2023), SLAKE Medical Visual Question Answering (Med- VQA)
model training. The datascts used include Johnsoon al. (2019), NL.T The Cancer Imaging Archive
2023). RSNA Radiological Society of North America (2018). and RadVQA OSF 2023.). detailed as follows:

---

Published in Transactions on Machine Learning Research (02/2026)
MIMIC MIMIC consists of 377,110 images and 883 medical reports. We used the XrayGPT Thawkar et al. (023b) preprocessed MIMIC dataset, which includes 114,690 reports associated with 241, ,00 training images in JPG format. For our study, we split the data into 80% for training and 20% for testing, primarily for report generation.
NLST used for lung cancer detection, containing 7,625 annotated low-dose CT scans with marked nodule locations. We extracted 2D CT slices from 3D volumes to highlight nodules. The annotations for training were sourced from Sybil Mikhael et al. (2023).
SLAKE used for grounding and VQA tasks, containing 579 radiology images of various organs and 3,543 question-answer pairs for training.
RSNA. We use the RSNA for evaluating pneumonia detection, consisting of 1,218 patients. We conduct a zero-shot evaluation on this dataset for disease detection.
RadVQA contains 315 radiology images across the head, chest, and abdomen, paired with 2,248 question- answer pairs spanning 11 categories. Responses split between closed-ended (yes/no) and open-ended (short answers). We conduct zero-shot evaluation this dataset.
4.2 Training Details
We initialized our model with pre-trained MiniGPT-v2 weights. During fine-tuning, we updated only the linear projection layer and applied LoRA Hu et al. (2021) to optimize the language model. The model was trained using cross-entropy loss and the AdamW optimizer. The dataset comprises 124,276 medical images with a resolution of 448x 448 pixels, and no data augmentation was applied. Training was performed on single NVIDIA A100 GPU for 100 epochs with a maximum learning rate of 1c-5. For hyperparameter tuning, experimented with different epoch counts, learning rates, and optimization strategies. Evaluation conducted every 20 epochs to select the best-performing checkpoint.
5 Evaluation
5.1
Baseline models
We evaluated MiniGPT-Med's performance on medical report generation, discase detection, and medical visual question answering (VQA), comparing to specialist and generalist models. Specialist models handle either grounding or non-grounding tasks, while generalist models perform both. For the medical report generation task, we compared MiniGPT-Med with specialist models, including Med-Flamingo Moor et al.) (2023) and LLa Li et al. (2024). known for their prowess in vision language tasks and contextual learming abilities. Additionally, we compared MiniGPT-Med with RadFM Wu et al. 2023b), which is specifi- cally tailored for radiology, and Trawkar chest radiograph analysis. Furthermore, we evaluated MiniGPT-Med against CheXagent Chen et al. (2024). a foundation model focused on improving chest X-ray interpretation. Moreover, comparisons were made with generalist models like MiniGPT-v2 and Qwen-VT. Bai et al. (2023), trained on the general vision-language data, showcasing exceptional performance across various dision-focused comprehension benchmarks. For the discase detection task, MiniGPT-Med was compared against specialist models including Bio Vil ban vision-language medical datascts, as well as generalist models including MiniGPT-v2 and Qwen-VI. WL. In the medical VQA lask, we compared MiniGPT-Med with specialized models like MedVINT Zhhnn et al. (2023). OpenFlamingo Awadalla et al. (2023), and Med-Flammn Alem-ila Moor et al. (2023) tailored to address the challenges of medical VQA, particularly in zero-shot scenarios, utilizing the RadVQA datasct. Additionally, our work was com- pared with generalist models such as MiniGPT-v2 and Qwen- VL provide comprehensive evaluation of MiniGPT-Med performance.
Thawkar et al. (023bb). a novel
model designed for
(2023), MedKLIP Wu et al. (2023a), and GLoRIA Huang et al. (2021), all pre-trained
6

---

Published in Transactions on Machine Learning Research (02/2026)
5.2 Evaluation Metrics
In our study, we adapted our evaluation approach align with the distinct skills required for interpreting radiology images using MiniGPT-Med. To assess the model's ability to generate radiological reports, we used two metrics: BERT Similarity (BERTsim) and CheXbert Similarity (CheXbert-Sim)). BERTsim utilized to evaluate the semantic similarity between the model-generated descriptions of radiological images and the expert-provided ground truth annotations. This involved using a BERT model to embed both the ground truth and generated sentences, followed by computing the cosine similarity between these embeddings. CheXbert-Sim, conversely, was selected for its relevance assessing the model's accuracy replicating professional medical report standards. It is specialized version of the BERT model, fine-tuned on clinical texts, which computes the cosine similarity between embeddings for each corresponding sentence pair after encoding. For the Visual Question Answering (VQA) aspect, we exclusively used BERTsim measure the semantic accuracy of the model responses. Additionally, we employed Intersection over Union (ToU) for grounding, a metric that quantitatively measures the model's precision in localizing and identifying specific features or abnormalities within the radiology images, such as pneumonia in the RSNA dataset.
5.3 Medical Report Generation
In our comprehensive study, evaluate the efficacy of MiniGPT-Med for medical report generation the MIMIC dataset Johnson et al. (011)). As summarized in Table 2 MiniGPT-Med consistently outperforms all specialist models, including the strongest domain-specific baseline, CheXagent, achieving improvements of 21.6 and 5.2 points on BERT-Sim and CheXbert-Sim, respectively. Among genoralist models, MiniGPT- Med attains the highest BERT-Sin score, exceeding Gemini by 12 points, indicating stronger linguistic and semantic alignment with reference reports. In contrast, Gemini achieves the best performance on CheXbert- Sim, reflecting superior clinical concept consistency as measured by entity-based evaluation. These results highlight inherent tradeoff between optimizing for linguistic fidelity (BERT-Sim) and clinical accuracy (CheXbert-Sim), suggesting that models excelling in natural language generation may not always achieve the highest scores on clinically grounded metrics, and vice versa.
Table 2: Evaluation of medical report generation the MIMIC-CXR dataset. MiniGPT-Med is compared to generalist models capable of both grounding and non- non-grounding tasks, and specialist models limited to non-grounding tasks. The best scores for each model category are highlighted in bold.
Method
Type
BERT-Sim
CheXbert-Sim
MedFlamingo
10.4
3.2
T.LaVA-Med
6.2
17.5
RadFM
Specialist
45.7
17.5
XrayGPT
11.0
24.2
CheXagent
50.4
24.9
MedVLM-RI
55.0
27.0
Gemini
60.0
57.4
GPT-4o
58.0
55.0
MiniGPT-v2
Generalist
53.0
21.1
QW
51.9
20.3
Ours
72.0
30.1
5.4 Disease Detection
The data showcased in Table 3 reveal that MiniGPT-Med stands out for its competitive performance when compared against a comprehensive range of baseline models. With an Intersection over Union (loU) score of 0.26, MiniGPT-Mod not only exccods the capabilitics generalist models by a margin of 16% but also attains performance metrics on par with specialist models. The peak IoU score among these specialist models is noted to be 0.31. Our MiniGPT-Mer achieves competitive results and it demonstrates good disease detection

---

Published in Transactions on Machine Learning Research (02/2026)
performance among all the baseline models, highlighting its potential versatile and effective tool in the medical domain.
Table 3: Evaluation of disease detection on the RSNA benchmark in zero-shot setting: comparison of our model with generalist and specialist models. The best performance is highlighted in bold.
Method
Model Type
RSNA IoU
BioViL
0.30
MedKLIP
Specialist
0.31
GLoRIA
Qwen- VL
0.10
MiniGPT-v2
Generalist
0.13
Ours
0.26
5.5
Medical Visual Question Answering
This study evaluates our model, MiniGPT-Med, against various baseline models using the RadVQA OSF (023s)) benchmark, as presented in Table ! MiniGPT-Med achieves notable performance metric of 0.58, surpassing both generalist models such as MiniGPT-v2 Chen et al. (2023) and specialist models like OpenFlamingo Awadalla et al. (2023) and Med-Flamingo Moor al. (2023). This performance not only demonstrates MiniGPT-Med's superiority over a broad range of models but also shows it achieve results comparable to those of the leading specialist model, MedVIN Zhang et al. (2023), which has accuracy of 0.62. The ability of MiniGPT-Med to outperform or match the performance of several specialized and generalist models underscores its significant potential as a foundation for the developmentt of advanced medical visual cunestion-answeii models.
Table 4: Evaluation visual question answering on the RadVQA benchmark in a zero-shot setting, us- ing BERT-Sim as the evaluation metric. Comparison includes generalist and specialist models, with top performance in each category highlighted in bold.
Method
Model Type
RadVQA BERT-Sim
MedVTN
0.62
OpenFlamingo
Specialist
0.49
Med-Flamingo
0.48
Qwen-VL
0.13
MiniGPT-v2
Generalist.
0.55
Ours
0.58
5.6 Assessing Hallucinations
Hallucination is a common issue for large language models such as ChatGPT and Gemini, which sometimes generate inaccurate content. Our model, equipped with the LLaMA language model, may also occasionally produce hallucinated medical reports. As there no established automatic method for detecting hallucina- tions in the biomedical domain, rely expert radiologists to review the generated reports and quantify the level of hallucination in the model outputs. We asked three senior radiologists to assess 50 randomly sam- pled cases from the MIMIC dataset, focusing on the robustness, granularity, and accuracy of the generated reports. The evaluation was conducted using three questions: Q1) how closely the generated report aligns with the radiologist's 's expert judgment, Q2) the level clinical detail in the report, and Q3) the accuracy of the report for diagnosing pathologics. Each response was labeled as Good, Medium, or Poor. For each model, we aggregate all radiologist judgments across the three questions and normalize the counts of each label by the total number of evaluations, yielding the average radiologist evaluation percentages reported in
8

---

XrayGPT Published in Transactions on Machine Learning Research (02/2026)
Table 5
We apply the same evaluation protocol to competing methods, including XrayGPT and CheXagent, to ensure fair comparison.
Table 5: Expert evaluation of medical report generation. The table shows the percentage of expert votes for each quality category (Good, Medium, Poor) across three models.
Quality
MiniGPT-MMdd
CheXagent
Good
65.48%
61.78%
49.00%
Medium
29.33%
23.33%
21.79%
Poor
5.19%
16.43%
27.67%
The analysis shows that 65.48% of the medical reports generated by our model are of high quality and better than CheXagent and XrayGPT, Furthermore, 29.33% of the reports are classified as medium quality, and only 5.19% are poor, which is the lowest rate among the compared modcls. This distribution underscores the model's ability to produce reports that meet professional standards by offering detailed and accurate diagnostic information. These findings highlight MiniGPT-Med''s considerable potential a reliable and effective tool for advancing medical reporting processes.
6 Qualitative Examples
To illustrate the versatility of our proposed model, present qualitative examples in multiple clinical vision- language tasks. These include detection, anatomical grounding, region identification, referring expressions, VQA, and medical report generation. As shown Figure 3 the model accurately interprets diverse imput modalities and responds to task specific prompts. These examples highlight the model's strong generalization capabilities and its potential support various real world diagnostic.
Grounding
Idantity
(groundingl desoribe this
prompt (dutection] Preumonia Preummrie
detail
prompe lddeniiyl what Is tW: 1-55-16-86--591
response The chatt X portnays preumoni wis the lungs" alreali rilted with 1l.dd
(a) Detection
(b) Grounding
(c) Identify
prompt: trcfer which part the Imageindicnn indicanes cancer? response: nodule
prompt Could you contents image response: serpy comredidettio,, or preumothoras, The cardtomedisst nat sinsuette is normal, and the imaged osseaus structure rcs are intact. There is free oir below the right handdaaphaag.. The imrp ussion is that there is no acuti intrathorec procees present.
prompt: (oqa) Which region the brais is impacted response: Anterior Corebrum
(d) Refer
Report Generation
(f) VQA
Figure Examples of MiniGPT-Med multi-task abilities, including: (a) discase detection, (b) anatomical grounding, (c) region identification, (d) referring expressions, (e) medical report generation, and (f) visual question answering.

---

Published in Transactions on Machine Learning Research (02/2026)
Task: Detection
prompt: [detection] Pneumonia
response:
groundtruth
Figure 4: False positive example. The red bounding box indicates a falsely detected disease, while the green bounding box shows the ground truth.
7 Limitation
Despite its notable capabilities, MiniGPT-Med faces challenges stemming from a lack of diverse and high- quality training datasets. This limitation restricts the model's coverage narrow range of discases. To overcome this, the expansion and diversification of training datasets are crucial, enabling the integration of extensive medical knowledge into the model. Additionally, MiniGPT-Med occasionally experiences issues with generating inaccurate medical reports and improperly connecting symptoms to diseases, phenomenon known as hallucination. These problems are partly due to the inadequate quality annotated data available for training. Addressing these issues requires not only richer dalasets but also the incorporation of advance vision backbones and enhancements in the underlying large language model ensure more accurate and reliable outputs. Additionally, the deployment of deep learning models in the health sector has encountered several notable failure cases. One such example for MiniGPT-Med is their difficulty distinguishing between the abnormality and the medical images that include device implants in the human body. Fig. |demonstrates sample of the data that MiniGPT-Med failed in correctly identifying the pneumonia location. The object under the green bounding box is the ground truth and the object under the red bounding box is the false detection. The model easily confuses the device implants as an abnormality. This shortcoming often results in misdiagnosed conditions. Specifically, when Al encounters X-rays or MRIs featuring implants, may incorrectly identify these abnormalities.
8 Conclusions
In this study, we introduce MiniGPT-Med, a specialized multi-modal designed for radiology diagnosis applications. It handles various medical vision-language tasks such as generating medical reports, detecting diseases, and answering visually-based medical questions, using distinct task identifiers to navigate these tasks efficiently. MiniGPT-Med outperforms baseline models in both grounding and non-grounding tasks, achieving statc-of-the-art performance in the MIMIC-CXR medical report generation task. Radiologist evaluations show that approximately 65% of the generated reports are of preferred quality, highlighting the model's superiority. Future plans include incorporating more diverse medical datasets, improving the understanding of complex medical terminology, enhancing interpretability and dependability, and con- ducting extensive clinical validation studies to ensure effectiveness and safety in real healthcare environments.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
A Patient-Level Data Splits
All datasets used in our experiments were partitioned at the patient ID level to prevent any overlap of subjects between the training, validation, and test sets. This ensures that no patient appears in more than one split, thereby eliminating the risk of information leakage across splits.
For each dataset, patient IDs were randomly assigned the training, validation, and test subsets according to predefined split ratios. Once assigned, all samples associated with given patient (e.g., images, volumes, reports) were included exclusively in the corresponding split. This protocol was applied consistently across all datasets and tasks considered in this work.
B NLST Slice Selection Protocol
The NLST dataset consists of chest CT scans provided as 3D volumetric images. To enable 2D-based model training and evaluation, consistent slice selection protocol was applied. Each NLST scan includes the annotated nodule location specified in 3D coordinates (x,yz). For each scan, we extracted the axial slice corresponding to the z-coordinate of the annotated nodule center. This slice typically provides the highest visual prominence of the target nodule and aligns with standard clinical practice for nodule assessment, The selected slice was used as the representative 2D imput for both training and evaluation.
C Preprocessing Pipeline Specifications
To ensure fair and controlled comparisons, we standardized the preprocessing pipeline across all models, tasks, and datascts, including all basclines.
Image Preprocessing
All images were converted to grayscale.
Images were resized to 448 x 448 using bilinear interpolation with antialiasing.
Pixel intensities were normalized using shared mean and standard deviation.
Data Augmentation
During training, RandAugment was applied.
During evaluation, data augmentation used.
D Ablation Studies
We conduct ablation experiments to analyze the impact of key architectural and training design choices on performance and efficiency in the medical report generation, disease detection, and VQA tasks.
D.1 Effect of Model Components.
Table i reports the effect of removing or modifying individual components of our model. We evaluate performance MIMIC (report generation), RSNA (disease detection), and RadVQA (VQA).
11

---

Published in Transactions on Machine Learning Research (02/2026)
Experiment
MIMIC (BERT-Sine
RSNA
RadVQA
(loU)
(BERT-Sim)
Our Model
0.72
0.26
0.58
Removing instruction token
0.72
0.02
0.58
With Q-Former
0.72
0.22
0.58
Token-mergine (2-+1)
0.66
0.18
0.54
Token-merging (None)
0,66
0,18
0.54
Unfreezing vision encoder
0.57
0.15
0.47
Table 6: Ablation study evaluating the impact of architectural and training design choices on medical report generation (MIMIC), discase detection (RSNA), and visual question answering (RadVQA).
Removing the instruction token leads severe degradation detection performance on RSNA, indicating that explicit task identifiers critical for structured prediction tasks. Token merging and unfreezing the vision encoder consistently degrade performance across all lasks, suggesting that aggressive compression full visual fine-tuning may harm multimodal alignment under limited data. Incorporating a Q-Former does not yield consistent gains, highlighting that additional architectural complexity does not necessarily translate to improved performance this setting.
D.2 Efficiency and Resource Trade-offs.
Table 7 Janalyzes the computational cost of different design choices in terms of training time, memory usage, and inference latency.
Experiment
GPUs
Train Time
FLOPs
VRAM
Latency
Speedup
(hrs)
(rel.)
(GB)
(ms)
(vs. Base)
Baseline
1xA100
20
1.00
22
85
1.00x
Remove instruction token
1xA100
20
1.00x
22
85
1.00x
With Q-Former
AxA100
26
1.15x
25
100
0.77x
Token-merging 4-+2
1xA100
20
1.00x
22
85
1.00x
Token-merging 2-+1
2xA100
40
1.60x
28
125
0.50x
Token-merging (None)
AxA100
60
2.20x
36
170
0.33x
Unfreeze vision encoder
A100
27
1.00x
22
85
0.74x
Table 7: Computational cost and efficiency trade-ofis of different model variants in terms of training time, memory usage, and inference latency.
Some variants increase computational cost, requiring more GPUs, longer training time, and higher inference latency. contrast, the baseline provides good balance between performance and efficiency.
E Prompt Templates
We report below the full set of prompt templates used for each task. We use the same prompts across all datasets and evaluation splits, and show them exactly as used. Curly braces O denote variable placeholders (e.g., spatial coordinates or regions).
E.1 Report Generation
Describe this image in detail.
Take look at this image and describe what you notice.
Please provide a detailed description of the picture.
Could you describe the contents of this image for me?
12

---

Published in Transactions on Machine Learning Research (02/2026)
E.2 Grounding
[grounding] please describe this image in detail
[grounding] describe this image as detailed as possible
[grounding] summarize this image in detail
[grounding] give a thorough description what you see in this image
E.3 Detection
[detection] pneumonia
[detection] nodules
[detection]
abnormality
[detection] tumor
E.4 Referring Expression (Refer)
[refer] tumor
[refer] abnormality
[refer] where is tumor?
[refer] where pneumonia?
[refer] where the nodule?
[refer] give the location of the tumor
[refer] give the location of the abnormality
[refer] could you tell the location for pneumonia?
[refer] from this image, tell the location of the tumor
[refer] from this image, tell the location of the abnormality
[refer] from this image, tell the location of the nodules E.5 Object Identification [identify] 0
[identify] what object is in this location 0
[identify] identify the object present at this location 0 [identify] what is it [identify] describe this object in 0 [identify] this 0 is
[identify] the object in 0 is
13

---

Published in Transactions on Machine Learning Research (02/2026)
E.6 Visual Question Answering (VQA)
For the VQA task, the model accepts both closed-form and open-ended natural language questions. No fixed prompt template is enforced; questions are provided directly in free-form text during both training and evaluation.
References
Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognation, pp. 15016-15027, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint arXie:2303.088 08774, 2023.
Jean-Baptias Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Monsch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Newral Information Processing Systems, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanaly, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 01390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Pong Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprinl arXiv:2308. 129,: 2023.
Louisa Canepa, Sonit Singh, and Arcot Sowmya. Visual question answering in the medical domain, 2023.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pongchuan Zhang, Raghuraman Krishnamoor- thi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. 2023.
Zhihong Chen, Maya Varma, Jean- Delbrouck, Magdalini Paschali, Louis Blankemeier, al. Chexa- gent: Towards a foundation model for chest x-ray interpretation. arXiv preprint orXiv:2401.1220, 2024.
URL httts://aaxii http://axxivo..g/pttt.. wwt/pof/////o//oto/ottpppto..ppt http:ppff2241111 12201 12208. pdf
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint
vrXiv:2211.07633 2022. URL /ttpss//arxiv... oww//ogg///...o.oo...oo.o.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106. 09685, 2021.
Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung, Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3942 3951, 2021.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Tleus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lun- gren, and Andrew Y. Ng. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison, 2019.
Alistair Johnson, Matt Lungren, Yifan Peng. Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxx-ippe-cestt radiographs with structured labels. PhysioNNe,, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinfor- matics, 36(4):1234 1240, September 2019. ISSN 1367-4811. doi: h btz662.. URL
http: http://dxx.o...... wtt.:/o...ooo..10311111 http:///00/////oooooooaaaa//////
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Naumann, Hoi- fung Poon, and Jianfeng Gao. Llava-med: Training large auguuge-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognátion, pp. 36296-26306 2024.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tic- Liu. Biogpt:
generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23, 09 2022. doi: 10. 1093/bib/// /bbbbbb4449
Medical Visual Question Answering (Med-Ve (QA). The slake dataset. betpp://www.med-vve cww/ slake/ 2023.
Peter G Mikhael, Jeremy Wohlwend, Adam Yala, Ludvig Karstens, Justin Xiang, Angelo K Takigami, Patrick P Bourgouin, PuiYee Chan, Sofiane Mrah, Wael Amayri, et al. Sybil: A validated deep learning model to predict future lung cancer risk from a single low-dose chest computed tomography. Journal of Clinical Oncology, 41 (12):2191 2200, 2023.
Masoud Monajalipoor, Mozhdeh Rouhsedaghat, Liunian Harold Li, Aichi Chien, C. C. Jay Kuo, Fabien Scalzo, and Kai- Wei Chang. Berthop: An effective vision-and and-lanuuage model for chest x-ray disease diagnosis, 2021.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasınaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
OSF. Osf project: Radvqa. https: http:////off.//8999// 10/89kps/
2023s.
Ankit Pal, Logesh Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi- choice dataset for medical domain question answering, 03 2022.
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision-language mod- els (vims) via reinforcement learning. In International Conference on Medical Image Computing and Computer- Intervention, pp. 337-347. Springer, 2025.
Radiological Society of North America. Rsna pneumonia detection challenge 2018. https://www.res org/ 2018.
rsnai haapa///-taiiiggglleeoeeeeeereee
Pranav Rajpurkar, Jeremy Irvin, Robyn Ball, Kaylie Zhu, Brandon Yang. Hershel Mchta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Bhavik Patel, Kristen Yeom, Katie Shpanskaya, Francis Blankenberg, Jayne Seckins, Timothy Amrhein, David Mong, Safwan Halabi, Evan Zucker, and Matthew Lungren. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLOS Medicine, 15:e1002686, 11 2018. doi: (0.331/jjounal)
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pre-trained contextualized em- beddings on largo-scale structured electronic health records for discase prediction. CoRR, abs/ 2ts//2055.28333 2020. URL
/ 12833
Rui Shen, Pierre Boulanger, and Michelle Noga. Medvis: real-time immersive visualization environment for the exploration of medical volumetric data. pp. 68, 08 2008. ISBN 078-0-7695-9284-4. doi:
10.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Caan-BBptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXio:2312.11805, 2023.
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models, 2023a.
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs sunmarization using medical vision-language models. arXiv preprint arXi::2366.0 2023b.
The Cancer Imaging Archive. The cancer imaging archive (tcia) national lung screening trial (nlst) wiki page. htttss//wikk we /t 2023.
Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. 09288
arXiv preprint arXiv:2307.09288, 2023. URL
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xic. Mcdklip: Medical knowledge
enhanced language-inage pre-training. arXiv preprinl ar arXiv:2301.02 022228 2023a. URL https://arxiv. org/ ht//abbs2331... 02228
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint 2r30: 02466, 2023b.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pme- vqa: Visual instruction tuning for medical visual question answering. arXiv preprint 2rXi:22305.04411 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXic:2344.10922 2023.
16


Reviews:
[{"review_id": "fvhG1KU50H", "reviewer": "Reviewer_D4zv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Unify multiple radiology tasks (report generation, VQA, detection/grounding, referring) behind one LLM interface by wiring a frozen EVA vision encoder → linear projector → LLaMA-2-chat 7B (fine-tuned with LoRA). Visual tokens are aggressively merged (4→1) to keep sequences short; task identifiers steer the LLM (
 … [Task Identifier] … 
). \n\nTrains on a curated mix (MIMIC-CXR for reports; NLST slices for nodules; SLAKE/RadVQA/RSNA for VQA/grounding/detection) and evaluates on three axes: report generation (BERT-Sim & CheXbert-Sim), pneumonia localization on RSNA (IoU), and VQA on RadVQA (BERT-Sim).\n\nReports strong numbers on MIMIC report generation and competitive results on VQA and RSNA detection; adds a small radiologist study on hallucination.\n\n**Key strengths**\n\n- Single interface covering grounding + non-grounding tasks with simple, reproducible scaffolding (frozen EVA + LoRA on LLaMA-2).\n\n- Practical token-compression trick to keep multimodal sequences tractable in a 7B-class model.\n\n- Broad modality coverage (X-ray, CT, MRI) and task breadth with one system.\n\n- Human study acknowledges hallucination/clinical quality concerns (good to see this included).\n\n**Key weaknesses**\n\n- Data curation/splits are under-specified (risk of leakage between training on MIMIC and evaluation formats; CT slice extraction details for NLST; zero-shot protocols on RSNA/RadVQA).\n\n- Ablations minimal: no study of token-merging factor, projector capacity, frozen-vs-unfrozen EVA, instruction format, or LoRA ranks.", "claims_and_evidence_yn": "No", "claims_explanation": "**Split and protocol clarity**: The paper does not convincingly rule out data leakage (e.g., patient/report overlap across splits or between pretraining and evaluation). RSNA and RadVQA are “zero-shot”, but provenance and prompt templates vs. training distributions are only briefly described.", "audience_yn": "Yes", "audience_explanation": "There’s clear community interest in unified VLM/LLM interfaces for medical imaging with lightweight adaptation (LoRA) and token-efficient vision alignment.\n\nThe task breadth (reporting, VQA, grounding/detection) from one model is practically appealing and relevant to multimodal LLM research, even outside medicine.\n\nThe negative insight (implant-related failures) is also valuable: it highlights open problems in clinical robustness.", "requested_changes": "Protocol clarity (Critical)\n\n- Provide explicit data splits (e.g., patient-level) and leakage safeguards; detail NLST 3D→2D slice selection.\n\n- Precisely define “zero-shot” (no image/task leakage, no instruction leakage); share prompts.\n\n- Normalize preprocessing across baselines (or ablate preprocessing impact) to validate comparisons.\n\nAblations (Critical)\n\n- Token-merging factor (4→1 vs 2→1 vs none), projector size, LoRA rank, freezing vs partial unfreeze of EVA, instruction token variants.\n\n- Show compute/latency vs accuracy trade-offs.\n\nRelease (Critical for camera-ready)\n\n- Provide code, configs, and exact splits; include inference scripts for all tasks, plus the CT slice extraction script/protocol.", "broader_impact_concerns": "Clinical risk: Unified LLM interfaces can generate confident but incorrect reports or boxes; the paper itself shows a failure with implants. Without calibrated uncertainty + human-in-the-loop safeguards, risk of harm is non-trivial.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X04mDlhaBt", "reviewer": "Reviewer_xBKE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes MiniGPT-Med, a model that takes a generalist approach to training in terms of tasks within the medical domain. It builds of a general domain model MiniGPT-v2 that attempts to train on a wider variety of vision-language tasks, including the detection, captioning, VQA, grounding, refer (referring expression comprehension ), and identity (referring expression generation) tasks by using task identifier tokens in the prompt along with instructions. The model achieves competitive performance across the tested tasks, and in particular achieves SOTA performance on medical report generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The narrow claim of SOTA performance is substantiated in the first table. However, I still believe more analysis is required to understand why this model does better. In particular the requested change #9 below about an ablation study would substantially strengthen the paper by demonstrating whether their generalist approach is the cause of the performance gain.", "audience_yn": "Yes", "audience_explanation": "Certainly, it is useful for the community to know how the generalist approaches taken in the general domain extend to the medical domain. It is also useful to have a new SOTA model for report generation that can do a variety of tasks. A model that is good at this variety of tasks is definitely a useful building block for future work with many possible applications.", "requested_changes": "1. Though recent, this model:\nhttps://aclanthology.org/2025.naacl-long.89/\nmight be a reasonable additional baseline as it does grounding, unlike other baselines you have. It would also be interesting to note the differences between the grounding style in that paper (identifying grounding phrases and doing a bbox regression loss). I think at least it should belong in your related work.\n\n2. Is there any reason you didn’t use LATTE-CXR (https://physionet.org/content/latte-cxr/1.0.0/) or the Chest ImaGenome (https://physionet.org/content/chest-imagenome/1.0.0/) datasets?\n\n3. Cites in the first paragraph of the related work do not show up correctly.\n\n4. Could you add a brief explanation as to how BERTSim works with the multi-sentence outputs in a produced report?\n\n5. Despite the limitations, I still think it’s important to include the rouge metrics for pure generation tasks like report generation, even if it just means putting it in the appendix. This helps compare with other papers that use rouge.\n\n6. Are there any multi-object grounding tasks? If so, how do the IoU metrics handle this. If not, could you talk about this and why you excluded this type of task?\n\n7. Why do the generalist model scores tell a different story when looking at the BERT-Sim vs CheXbert-Sim scores?\n\n8. Can you comment on how the generalist models outperform the specialist models on the report generation task? Is this because of the size/scale or RAG aspects of those models or a different reason?\n\n9. The paper claims that the generalist aspect of the approach is what allows for the performance gains. Can the authors do some ablation experiments to show this in more detail? For example, what would happen if you trained dropped some of the tasks from training? What would happen if you omitted the task token during training and inference? Or just during inference? I understand some of these ablations are not easy if you need to repeat training, but it would make for a much stronger paper.\n\n10. One of the noted limitations is that the lack of diverse and high-quality training data restricts the model’s coverage to a narrow range of diseases. It would be nice to show this a little more quantitatively. Stress-tests like this still strengthen the paper even if they show the limitations of the model.\n\n11. It might be nice to understand how varied the instructions can be even within a task and how that effects performance.", "broader_impact_concerns": "I have no concerns about the ethical implications of this paper.", "additional_comments": "This paper shows significant potential, but I would highly recommend making at least some of the requested changes to strengthen the analysis before publishing this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XQilPUP4Bc", "reviewer": "Reviewer_cy1u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces MiniGPT-Med, a vision-language model designed for medical applications that can handle 6 radiology tasks including medical report generation, visual question answering (VQA), and disease detection. The model is developed based on LLaMA2-chat, employing EVA as the vision encoder and adopting the MiniGPT-v2 style template as instructions.. Key contributions include:\n\n1. **Unified Multi-task Model**: A single model capable of handling both grounding (disease detection) and non-grounding (report generation, VQA) tasks across multiple imaging modalities (X-rays, CT scans, MRIs).\n\n2. **Strong Empirical Results**: State-of-the-art performance on medical report generation (19.0% improvement over previous best), competitive performance on disease detection and VQA tasks.\n\n**Key Strengths:**\n\n- Comprehensive evaluation across multiple medical vision-language tasks.\n- Strong quantitative results, particularly in medical report generation.\n- Practical relevance to clinical applications.\n\n**Key Weaknesses:**\n\n- Insufficient technical novelty - primarily an adaptation of existing  vision-language architecture.\n- Lack of clinical validation in real healthcare settings.", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper presents extensive experimental results, several claims lack sufficient evidence or are overstated:\n\n1. The proposition that it functions as a “general interface for radiology diagnosis” is insufficiently substantiated. When considering the limited scope of disease coverage and the identified hallucination problems, it is advisable to revise this assertion.\n\n2. While the paper compares against several models, some key recent (published in 2024 or 2025) medical vision-language models are missing from the comparison, making it difficult to assess true state-of-the-art performance.", "audience_yn": "Yes", "audience_explanation": "Despite the limitations, this work addresses an important and timely problem in medical AI. The TMLR's audience would be interested in :\n\n1. **Medical AI Applications**: The intersection of large language models and medical imaging is a rapidly growing area of interest\n2. **Multi-task Learning**: The approach to handling multiple medical vision-language tasks in a unified framework\n3. **Practical Relevance**: Medical report generation and diagnostic assistance are high-impact applications", "requested_changes": "### Critical Changes:\n\n- Provide more detailed analysis of why task identifiers are effective\n- Include ablation studies on architectural choices\n- Compare against more recent medical vision-language models\n\n### Changes to Strengthen the Work:\n\n1. It is recommended to enhance the aesthetics of Figure 1. Note that the simultaneous appearance of \"...\" and \"etc.\" is redundant and should be avoided.\n2. Figure 2 appears rather simplistic, as it does not present the details of crucial structures.\n3. Enhance the clarity within the methodology section and minimize redundancy in the results presentation. For instance, presenting radar charts could effectively illustrate the method's superiority across six tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)