MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
  Models
Authors: name: Peng Xia, name: Kangyu Zhu, name: Haoran Li, name: Tianze Wang, name: Weijia Shi, name: Sheng Wang, name: Linjun Zhang, name: James Zou, name: Huaxiu Yao

Artificial Intelligence (AI) has demonstrated significant potential in
healthcare, particularly in disease diagnosis and treatment planning. Recent
progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
possibilities for interactive diagnostic tools. However, these models often
suffer from factual hallucination, which can lead to incorrect diagnoses.
Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
address these issues. However, the amount of high-quality data and distribution
shifts between training data and deployment data limit the application of
fine-tuning methods. Although RAG is lightweight and effective, existing
RAG-based approaches are not sufficiently general to different medical domains
and can potentially cause misalignment issues, both between modalities and
between the model and the ground truth. In this paper, we propose a versatile
multimodal RAG system, MMed-RAG, designed to enhance the factuality of
Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
adaptive retrieved contexts selection method, and a provable RAG-based
preference fine-tuning strategy. These innovations make the RAG process
sufficiently general and reliable, significantly improving alignment when
introducing retrieved contexts. Experimental results across five medical
datasets (involving radiology, ophthalmology, pathology) on medical VQA and
report generation demonstrate that MMed-RAG can achieve an average improvement
of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
in https://github.com/richard-peng-xia/MMed-RAG.