Towards Accurate Differential Diagnosis with Large Language Models
Authors: name: Daniel McDuff; name: Mike Schaekermann; name: Tao Tu; name: Anil Palepu; name: Amy Wang; name: Jake Garrison; name: Karan Singhal; name: Yash Sharma; name: Shekoofeh Azizi; name: Kavita Kulkarni; name: Le Hou; name: Yong Cheng; name: Yun Liu; name: S Sara Mahdavi; name: Sushant Prakash; name: Anupam Pathak; name: Christopher Semturs; name: Shwetak Patel; name: Dale R Webster; name: Ewa Dominowska; name: Juraj Gottweis; name: Joelle Barral; name: Katherine Chou; name: Greg S Corrado; name: Yossi Matias; name: Jake Sunshine; name: Alan Karthikesalingam; name: Vivek Natarajan
Published: 2023-11-30 19:55:51

Abstract: An accurate differential diagnosis (DDx) is a cornerstone of medical care,
often reached through an iterative process of interpretation that combines
clinical history, physical examination, investigations and procedures.
Interactive interfaces powered by Large Language Models (LLMs) present new
opportunities to both assist and automate aspects of this process. In this
study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
evaluated 302 challenging, real-world medical cases sourced from the New
England Journal of Medicine (NEJM) case reports. Each case report was read by
two clinicians, who were randomized to one of two assistive conditions: either
assistance from search engines and standard medical resources, or LLM
assistance in addition to these tools. All clinicians provided a baseline,
unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
exhibited standalone performance that exceeded that of unassisted clinicians
(top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
arms, the DDx quality score was higher for clinicians assisted by our LLM
(top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
(McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
differential lists than those without its assistance. Our study suggests that
our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
accuracy in challenging cases, meriting further real-world evaluation for its
ability to empower physicians and widen patients' access to specialist-level
expertise.