Clinical trial NCT06774612: The Impact of Large Language Models on Diagnostic Reasoning Among LLM-Trained Medical Doctors
Official title: Diagnostic Reasoning With and Without AI Support: A Randomized Controlled Trial of LLM-Trained Medical Doctors
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Diagnosis
Keywords: clinical reasoning
large language models
computer-assisted diagnosis
Lead sponsor: Lahore University of Management Sciences
Sponsor class: OTHER
Collaborators: name: King Edward Medical University
First posted: 2025-01-14
Enrollment: 60
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: yes
Allocation: RANDOMIZED
Intervention model: PARALLEL
Intervention model description: The trial will be designed as a randomized, two-arm, single-blind parallel group study.
Primary purpose: DIAGNOSTIC

Brief summary: This study aims to evaluate whether large language model-trained medical doctors demonstrate enhanced diagnostic reasoning performance when utilizing ChatGPT-4o alongside conventional resources compared to using conventional resources alone.

Detailed description: Diagnostic errors are a major source of preventable patient harm. Recent advances in Large Language Models (LLM), particularly ChatGPT-4o, have shown promise in enhancing medical decision-making. However, little is known about their impact on medical doctors' (e.g., physicians' and surgeons') diagnostic reasoning.

Diagnostic accuracy relies on complex clinical reasoning and careful evaluation of patient data. While AI assistance could potentially reduce errors and improve efficiency, ChatGPT-4o lacks medical validation and could introduce new risks through incorrect information generation (also known as hallucinations). To mitigate these risks, doctors need adequate training in understanding ChatGPT-4o's capabilities, limitations, and proper usage. Given these uncertainties and the importance of proper AI training, systematic evaluation is essential before clinical implementation.

This randomized study will assess whether ChatGPT-4o access improves LLM-trained medical doctors' diagnostic performance compared to conventional resources (e.g., textbooks, online medical databases) alone. All participating doctors will have completed at least a 10-hour training program covering ChatGPT-4o usage, prompt engineering techniques, and output evaluation strategies. Participants will provide differential diagnoses with supporting evidence and recommended next steps for clinical cases, with responses evaluated by blinded reviewers.

Eligibility criteria: Inclusion Criteria:

* Full or Provisionally Registered Medical Practitioners with the Pakistan Medical and Dental Council (PMDC).
* Completed Bachelor of Medicine, Bachelor of Surgery (MBBS) Exam. The equivalent degree of MBBS in US and Canada is called Doctor of Medicine (MD).
* Participants must have completed a structured training program on the use of ChatGPT (or a comparable large language model), totaling at least 10 hours of instruction. The program must include hands-on practice related to LLM's aspects, specifically prompt engineering and content evaluation.

Exclusion Criteria:

* Any other Registered Medical Practitioners (Full or Provisional) with PMDC (e.g., Professionals with Bachelor of Dental Surgery or BDS).

Primary outcomes:
measure: Diagnostic reasoning
timeFrame: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
Secondary outcomes:
measure: Time Spent on Diagnosis
timeFrame: Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.
Other outcomes:
Locations:
city: Lahore
country: Pakistan
facility: Lahore University of Management Sciences
state: Punjab
Study officials:
affiliation: Lahore University of Management Sciences
name: Ihsan Ayyub Qazi, PhD
role: PRINCIPAL_INVESTIGATOR
affiliation: King Edward Medical University
name: Muhammad Asadullah Khawaja, MBBS
role: PRINCIPAL_INVESTIGATOR
affiliation: Lahore University of Management Sciences
name: Ayesha Ali, PhD
role: PRINCIPAL_INVESTIGATOR
References: