Clinical trial NCT06157944: Physician Reasoning on Diagnostic Cases With Large Language Models
Official title: Diagnostic Reasoning With Large Language Model Chat Bots
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Diagnosis
Keywords: Computer-assisted diagnosis
Large language models
clinical reasoning
Lead sponsor: Stanford University
Sponsor class: OTHER
Collaborators: name: Beth Israel Deaconess Medical Center
name: University of Minnesota
First posted: 2023-12-06
Enrollment: 50
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: yes
Allocation: RANDOMIZED
Intervention model: PARALLEL
Intervention model description: The trial will be designed as a randomized, two-arm, single-blind parallel group study.
Primary purpose: DIAGNOSTIC

Brief summary: This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional diagnostic decision support tools on performance on case-based diagnostic reasoning tasks.

Detailed description: Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.

Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was not developed for the purpose of clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve diagnostic reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.

In this study, we will randomize participants to answer diagnostic cases with or without access to ChatGPT-4. The participants will be asked to give three differential diagnoses for each case, with supporting and opposing findings for each diagnosis. Additionally they will be asked to provide their top diagnosis along with next diagnostic steps. Answers will be graded by independent reviewers blinded to treatment assignment.

Eligibility criteria: Inclusion Criteria:

* Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
* Training in Internal medicine, family medicine, or emergency medicine.

Exclusion Criteria:

* Not currently practicing clinically.

Primary outcomes:
measure: Diagnostic reasoning
timeFrame: During evaluation
Secondary outcomes:
measure: Time Spent on Diagnosis
timeFrame: During evaluation
Other outcomes:
Locations:
city: Palo Alto
country: United States
facility: Stanford University
state: California
Study officials:
affiliation: Stanford University
name: Jonathan H Chen, MD, PhD
role: PRINCIPAL_INVESTIGATOR
affiliation: Beth Israel Deaconess Medical Center
name: Adam Rodman, MD
role: PRINCIPAL_INVESTIGATOR
affiliation: University of Minnesota
name: Andrew Olson, MD
role: PRINCIPAL_INVESTIGATOR
References: