MedFuzz: Exploring the Robustness of Large Language Models in Medical
  Question Answering
Authors: name: Robert Osazuwa Ness, name: Katie Matton, name: Hayden Helm, name: Sheng Zhang, name: Junaid Bajwa, name: Carey E. Priebe, name: Eric Horvitz

Large language models (LLM) have achieved impressive performance on medical
question-answering benchmarks. However, high benchmark accuracy does not imply
that the performance generalizes to real-world clinical settings. Medical
question-answering benchmarks rely on assumptions consistent with quantifying
LLM performance but that may not hold in the open world of the clinic. Yet LLMs
learn broad knowledge that can help the LLM generalize to practical conditions
regardless of unrealistic assumptions in celebrated benchmarks. We seek to
quantify how well LLM medical question-answering benchmark performance
generalizes when benchmark assumptions are violated. Specifically, we present
an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
attempts to modify benchmark questions in ways aimed at confounding the LLM. We
demonstrate the approach by targeting strong assumptions about patient
characteristics presented in the MedQA benchmark. Successful "attacks" modify a
benchmark item in ways that would be unlikely to fool a medical expert but
nonetheless "trick" the LLM into changing from a correct to an incorrect
answer. Further, we present a permutation test technique that can ensure a
successful attack is statistically significant. We show how to use performance
on a "MedFuzzed" benchmark, as well as individual successful attacks. The
methods show promise at providing insights into the ability of an LLM to
operate robustly in more realistic settings.