Clinical trial NCT06911645: Diagnostic Reasoning With Customized GPT-4 Model
Official title: Evaluating the Performance of LLMs and Clinicians in Complex Diagnostic Cases: A Randomized Controlled Trial
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Pathologic Processes
Disease
Keywords: Computer-assisted diagnosis
Large language models
Clinical reasoning
Lead sponsor: Stanford University
Sponsor class: OTHER
Collaborators: name: Beth Israel Deaconess Medical Center
First posted: 2025-04-04
Enrollment: 70
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: yes
Allocation: RANDOMIZED
Intervention model: PARALLEL
Intervention model description: The trial will be designed as a randomized, two-arm, single-blind parallel group study.
Primary purpose: DIAGNOSTIC

Brief summary: This study will assess the impact of immediate access to a customized version of GPT-4, a large language model, on performance in case-based diagnostic reasoning tasks. Specifically, it will compare this approach to a two-step process where participants first use traditional diagnostic decision support tools to support their diagnostic reasoning before gaining access to the customized GPT-4 model.

Detailed description: Artificial intelligence (AI) technologies, particularly advanced large language models like OpenAI's ChatGPT, have the potential to enhance medical decision-making. While ChatGPT-4 was not specifically designed for medical applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultations. However, its impact on clinicians' diagnostic reasoning remains largely unknown.

Clinical reasoning is a complex process that involves pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could help reduce clinician workload and decrease the likelihood of missed diagnoses. However, ChatGPT-4 was neither developed nor validated for diagnostic reasoning, and it may produce misleading information, including plausible but incorrect conclusions that could misguide clinicians. If not used appropriately, it may fail to improve-and could even hinder-clinical decision-making. Therefore, it is essential to study how clinicians use large language models to support clinical reasoning before integrating them into routine patient care.

This study will examine how immediate access to a customized version of ChatGPT-4 impacts performance on case-based diagnostic reasoning tasks, compared to a stepwise approach. In the stepwise approach, participants will first use traditional diagnostic decision support tools to support their case reasoning before interacting with a customized ChatGPT-4 model, at which point they will have the opportunity to revise their initial answers.

Participants will be randomized into different study arms and will respond to diagnostic cases by providing three differential diagnoses, along with supporting and opposing findings for each. They will also identify their top diagnosis and propose next diagnostic steps. Independent reviewers, blinded to treatment assignment, will evaluate their responses.

Eligibility criteria: Inclusion Criteria:

* Participants must be licensed physicians and have completed at least post-graduate year 1 (PGY1) of medical training.
* Training in Internal medicine, family medicine, or emergency medicine.

Exclusion Criteria:

* Not currently practicing clinically.
* Participated in one of our previous studies that used the same six diagnostic cases.

Primary outcomes:
measure: Diagnostic reasoning
timeFrame: Through study completion, an average of 6 months
Secondary outcomes:
measure: Time Spent Per Case
timeFrame: Through study completion, an average of 6 months
measure: Prompt frequency
timeFrame: Through study completion, an average of 6 months
measure: Sentiment
timeFrame: Through study completion, an average of 6 months
measure: Participant Perceptions of AI in Clinical Reasoning
timeFrame: Through study completion, an average of 6 months
measure: Customized GPT-4's diagnostic reasoning
timeFrame: Through study completion, an average of 6 months
Other outcomes:
Locations:
city: Palo Alto
country: United States
facility: Stanford University
state: California
Study officials:
affiliation: Stanford University
name: Jonathan H Chen, MD, PhD
role: PRINCIPAL_INVESTIGATOR
References: