Boosting Healthcare LLMs Through Retrieved Context
Authors: name: Jordi Bayarri-Planas; name: Ashwin Kumar Gururajan; name: Dario Garcia-Gasulla
Published: 2024-09-23 15:33:38

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in
natural language processing, and yet, their factual inaccuracies and
hallucinations limits their application, particularly in critical domains like
healthcare. Context retrieval methods, by introducing relevant information as
input, have emerged as a crucial approach for enhancing LLM factuality and
reliability. This study explores the boundaries of context retrieval methods
within the healthcare domain, optimizing their components and benchmarking
their performance against open and closed alternatives. Our findings reveal how
open LLMs, when augmented with an optimized retrieval system, can achieve
performance comparable to the biggest private solutions on established
healthcare benchmarks (multiple-choice question answering). Recognizing the
lack of realism of including the possible answers within the question (a setup
only found in medical exams), and after assessing a strong LLM performance
degradation in the absence of those options, we extend the context retrieval
system in that direction. In particular, we propose OpenMedPrompt a pipeline
that improves the generation of more reliable open-ended answers, moving this
technology closer to practical application.