Clinical trial NCT06457269: Evaluating the Potential of Large Language Models for Respiratory Disease Consultations
Official title: Evaluating the Potential of Large Language Models for Respiratory Disease Consultations: A Randomized Crossover Trial
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Hay Fever
Asthma
Acute Upper Respiratory Infection
Pulmonary Embolism
Pneumonia
Tuberculosis
Acute Bronchitis
Pulmonary Fibrosis
Lung Cancer
Bronchiectasis
Lead sponsor: North Sichuan Medical College
Sponsor class: OTHER
Collaborators: name: University of Glasgow
name: Nanchong Central Hospital
name: Afﬁliated Hospital of North Sichuan Medical College
First posted: 2024-06-13
Enrollment: 703
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: no
Allocation: RANDOMIZED
Intervention model: CROSSOVER
Primary purpose: DIAGNOSTIC

Brief summary: The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios.

The main question aims to answer are:

* How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios?

In three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.


Eligibility criteria: Inclusion Criteria:

1. Self-reported symptoms of common respiratory diseases, such as cough, chest tightness, fever, and wheezing
2. Ability to engage in LLM dialog operations independently or with minimal peer training
3. A health status deemed suitable for study participation by the pulmonology experts

Exclusion Criteria:

1) Excessively poor health status

Primary outcomes:
measure: Expert indicators-Accuracy
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
measure: Expert indicators-Comprehensiveness
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
measure: Expert indicators-Correctness
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
measure: Expert indicators-Ethical compliance
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
measure: Empathy indicators
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective empathy indicators, the evaluation will be conducted within two months.
Secondary outcomes:
measure: Regular indicators-Total number of questions
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Follow-up words
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Total number of conversations
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Total conversation cost ($)
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Total conversation time (min)
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Number of output statements
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
measure: Regular indicators-Number of input statements
timeFrame: For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Other outcomes:
Locations:
city: Nanchong
country: China
facility: The Affiliated Hospital of North Sichuan Medical College
state: Sichuan
Study officials:
affiliation: North Sichuan Medical College
name: Jiebin Xie, Doctor
role: PRINCIPAL_INVESTIGATOR
References:
citation: Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004 Dec;21(6):699-705. doi: 10.1093/fampra/cmh621. Epub 2004 Nov 4.