Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
  Systems for Medical Question Answering
Authors: name: Nghia Trung Ngo, name: Chien Van Nguyen, name: Franck Dernoncourt, name: Thien Huu Nguyen

Retrieval-augmented generation (RAG) has emerged as a promising approach to
enhance the performance of large language models (LLMs) in knowledge-intensive
tasks such as those from medical domain. However, the sensitive nature of the
medical domain necessitates a completely accurate and trustworthy system. While
existing RAG benchmarks primarily focus on the standard retrieve-answer
setting, they overlook many practical scenarios that measure crucial aspects of
a reliable medical system. This paper addresses this gap by providing a
comprehensive evaluation framework for medical question-answering (QA) systems
in a RAG setting for these situations, including sufficiency, integration, and
robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
(MedRGB) that provides various supplementary elements to four medical QA
datasets for testing LLMs' ability to handle these specific scenarios.
Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
commercial LLMs and open-source models across multiple retrieval conditions.
Our experimental results reveals current models' limited ability to handle
noise and misinformation in the retrieved documents. We further analyze the
LLMs' reasoning processes to provides valuable insights and future directions
for developing RAG systems in this critical medical domain.