RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language
  Models
Authors: name: Peng Xia, name: Kangyu Zhu, name: Haoran Li, name: Hongtu Zhu, name: Yun Li, name: Gang Li, name: Linjun Zhang, name: Huaxiu Yao

The recent emergence of Medical Large Vision Language Models (Med-LVLMs) has
enhanced medical diagnosis. However, current Med-LVLMs frequently encounter
factual issues, often generating responses that do not align with established
medical facts. Retrieval-Augmented Generation (RAG), which utilizes external
knowledge, can improve the factual accuracy of these models but introduces two
major challenges. First, limited retrieved contexts might not cover all
necessary information, while excessive retrieval can introduce irrelevant and
inaccurate references, interfering with the model's generation. Second, in
cases where the model originally responds correctly, applying RAG can lead to
an over-reliance on retrieved contexts, resulting in incorrect answers. To
address these issues, we propose RULE, which consists of two components. First,
we introduce a provably effective strategy for controlling factuality risk
through the calibrated selection of the number of retrieved contexts. Second,
based on samples where over-reliance on retrieved contexts led to errors, we
curate a preference dataset to fine-tune the model, balancing its dependence on
inherent knowledge and retrieved contexts for generation. We demonstrate the
effectiveness of RULE on three medical VQA datasets, achieving an average
improvement of 20.8% in factual accuracy. We publicly release our benchmark and
code in https://github.com/richard-peng-xia/RULE.