WorldMedQA-V: a multilingual, multimodal medical examination dataset for
  multimodal language models evaluation
Authors: name: João Matos; name: Shan Chen; name: Siena Placino; name: Yingya Li; name: Juan Carlos Climent Pardo; name: Daphna Idan; name: Takeshi Tohyama; name: David Restrepo; name: Luis F. Nakayama; name: Jose M. M. Pascual-Leone; name: Guergana Savova; name: Hugo Aerts; name: Leo A. Celi; name: A. Ian Wong; name: Danielle S. Bitterman; name: Jack Gallifant
Published: 2024-10-16 16:31:24

Abstract: Multimodal/vision language models (VLMs) are increasingly being deployed in
healthcare settings worldwide, necessitating robust benchmarks to ensure their
safety, efficacy, and fairness. Multiple-choice question and answer (QA)
datasets derived from national medical examinations have long served as
valuable evaluation tools, but existing datasets are largely text-only and
available in a limited subset of languages and countries. To address these
challenges, we present WorldMedQA-V, an updated multilingual, multimodal
benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
includes 568 labeled multiple-choice QAs paired with 568 medical images from
four countries (Brazil, Israel, Japan, and Spain), covering original languages
and validated English translations by native clinicians, respectively. Baseline
performance for common open- and closed-source models are provided in the local
language and English translations, and with and without images provided to the
model. The WorldMedQA-V benchmark aims to better match AI systems to the
diverse healthcare environments in which they are deployed, fostering more
equitable, effective, and representative applications.