GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
  General Medical AI
Authors: name: Pengcheng Chen; name: Jin Ye; name: Guoan Wang; name: Yanjun Li; name: Zhongying Deng; name: Wei Li; name: Tianbin Li; name: Haodong Duan; name: Ziyan Huang; name: Yanzhou Su; name: Benyou Wang; name: Shaoting Zhang; name: Bin Fu; name: Jianfei Cai; name: Bohan Zhuang; name: Eric J Seibel; name: Junjun He; name: Yu Qiao
Published: 2024-08-06 17:59:21

Abstract: Large Vision-Language Models (LVLMs) are capable of handling diverse data
types such as imaging, text, and physiological signals, and can be applied in
various fields. In the medical field, LVLMs have a high potential to offer
substantial assistance for diagnosis and treatment. Before that, it is crucial
to develop benchmarks to evaluate LVLMs' effectiveness in various medical
applications. Current benchmarks are often built upon specific academic
literature, mainly focusing on a single domain, and lacking varying perceptual
granularities. Thus, they face specific challenges, including limited clinical
relevance, incomplete evaluations, and insufficient guidance for interactive
LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
comprehensive general medical AI benchmark with well-categorized data structure
and multi-perceptual granularity to date. It is constructed from 285 datasets
across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
and 4 perceptual granularities in a Visual Question Answering (VQA) format.
Additionally, we implemented a lexical tree structure that allows users to
customize evaluation tasks, accommodating various assessment needs and
substantially supporting medical AI research and applications. We evaluated 50
LVLMs, and the results show that even the advanced GPT-4o only achieves an
accuracy of 52%, indicating significant room for improvement. Moreover, we
identified five key insufficiencies in current cutting-edge LVLMs that need to
be addressed to advance the development of better medical applications. We
believe that GMAI-MMBench will stimulate the community to build the next
generation of LVLMs toward GMAI.
  Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/