AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
  Benchmark Dataset
Authors: name: Tobi Olatunji, name: Charles Nimo, name: Abraham Owodunni, name: Tassallah Abdullahi, name: Emmanuel Ayodele, name: Mardhiyah Sanni, name: Chinemelu Aka, name: Folafunmi Omofoye, name: Foutse Yuehgoh, name: Timothy Faniran, name: Bonaventure F. P. Dossou, name: Moshood Yekini, name: Jonas Kemp, name: Katherine Heller, name: Jude Chidubem Omeke, name: Chidi Asuzu MD, name: Naome A. Etori, name: Aimérou Ndiaye, name: Ifeoma Okoh, name: Evans Doe Ocansey, name: Wendy Kinara, name: Michael Best, name: Irfan Essa, name: Stephen Edward Moore, name: Chris Fourie, name: Mercy Nyamewaa Asiedu

Recent advancements in large language model(LLM) performance on medical
multiple choice question (MCQ) benchmarks have stimulated interest from
healthcare providers and patients globally. Particularly in low-and
middle-income countries (LMICs) facing acute physician shortages and lack of
specialists, LLMs offer a potentially scalable pathway to enhance healthcare
access and reduce costs. However, their effectiveness in the Global South,
especially across the African continent, remains to be established. In this
work, we introduce AfriMed-QA, the first large scale Pan-African English
multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
and closed-ended) sourced from over 60 medical schools across 16 countries,
covering 32 medical specialties. We further evaluate 30 LLMs across multiple
axes including correctness and demographic bias. Our findings show significant
performance variation across specialties and geographies, MCQ performance
clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
models and smaller edge-friendly LLMs struggle to achieve a passing score.
Interestingly, human evaluations show a consistent consumer preference for LLM
answers and explanations when compared with clinician answers.