MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
  Applications
Authors: name: Praveen K Kanithi; name: Clément Christophe; name: Marco AF Pimentel; name: Tathagata Raha; name: Nada Saadi; name: Hamza Javed; name: Svetlana Maslenkova; name: Nasir Hayat; name: Ronnie Rajan; name: Shadab Khan
Published: 2024-09-11 14:44:51

Abstract: The rapid development of Large Language Models (LLMs) for healthcare
applications has spurred calls for holistic evaluation beyond frequently-cited
benchmarks like USMLE, to better reflect real-world performance. While
real-world assessments are valuable indicators of utility, they often lag
behind the pace of LLM evolution, likely rendering findings obsolete upon
deployment. This temporal disconnect necessitates a comprehensive upfront
evaluation that can guide model selection for specific clinical applications.
We introduce MEDIC, a framework assessing LLMs across five critical dimensions
of clinical competence: medical reasoning, ethics and bias, data and language
understanding, in-context learning, and clinical safety. MEDIC features a novel
cross-examination framework quantifying LLM performance across areas like
coverage and hallucination detection, without requiring reference outputs. We
apply MEDIC to evaluate LLMs on medical question-answering, safety,
summarization, note generation, and other tasks. Our results show performance
disparities across model sizes, baseline vs medically finetuned models, and
have implications on model selection for applications requiring specific model
strengths, such as low hallucination or lower cost of inference. MEDIC's
multifaceted evaluation reveals these performance trade-offs, bridging the gap
between theoretical capabilities and practical implementation in healthcare
settings, ensuring that the most promising models are identified and adapted
for diverse healthcare applications.