Clinical trial NCT06098950: Human Algorithm Interactions for Acute Respiratory Failure Diagnosis
Official title: Measuring the Impact of AI in the Diagnosis of Hospitalized Patients: A Randomized Survey Vignette Multicenter Study
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Acute Respiratory Failure
Keywords: Artificial Intelligence
Diagnostic Accuracy
Computer Assisted Diagnosis
Biased Model
Lead sponsor: University of Michigan
Sponsor class: OTHER
Collaborators: name: National Heart, Lung, and Blood Institute (NHLBI)
First posted: 2023-10-25
Enrollment: 457
Sex: ALL
Minimum age (years): 18
Standard ages: ADULT
OLDER_ADULT
Accepts healthy volunteers: no
Allocation: RANDOMIZED
Intervention model: PARALLEL
Intervention model description: While participating in a survey, participants are randomized to see different hypothetical patient clinical vignettes, AI model predictions, and then ask questions about the patient's likely diagnosis and treatment.
Primary purpose: OTHER

Brief summary: Artificial intelligence (AI) shows promising in identifying abnormalities in clinical images. However, systematically biased AI models, where a model makes inaccurate predictions for entire subpopulations, can lead to errors and potential harms. When shown incorrect predictions from an AI model, clinician diagnostic accuracy can be harmed. This study aims to study the effectiveness of providing clinicians with image-based AI model explanations when provided AI model predictions to help clinicians better understand the logic of an AI model's prediction. It will evaluate whether providing clinicians with AI model explanations can improve diagnostic accuracy and help clinicians catch when models are making incorrect decisions. As a test case, the study will focus on the diagnosis of acute respiratory failure because determining the underlying causes of acute respiratory failure is critically important for guiding treatment decisions but can be clinically challenging.

To determine if providing AI explanations can improve clinician diagnostic accuracy and alleviate the potential impact of showing clinicians a systematically biased AI model, a randomized clinical vignette survey study will be conducted. During the survey, study participants will be shown clinical vignettes of patients hospitalized with acute respiratory failure, including the patient's presenting symptoms, physical exam, laboratory results, and chest X-ray. Study participants will then be asked to assess the likelihood that heart failure, pneumonia and/or Chronic Obstructive Pulmonary Disease (COPD) is the underlying diagnosis. During specific vignettes in the survey, participants will also be shown standard or systematically biased AI models that provide an estimate the likelihood that heart failure, pneumonia and/or COPD is the underlying diagnosis. Clinicians will be randomized see AI predictions alone or AI predictions with explanations when shown AI models. This survey design will allow for testing the hypothesis that systematically biased models would harm clinician diagnostic accuracy, but commonly used image-based explanations would help clinicians partially recover their performance.


Eligibility criteria: Inclusion Criteria:

* Physicians, nurse practitioners, and physician assistants that care for patients with acute respiratory failure as part of their clinical practice

Exclusion Criteria:

* Physicians, nurse practitioners, and physician assistants that only provide patient care in outpatient settings

Primary outcomes:
measure: Participant diagnostic accuracy across clinical vignette settings
timeFrame: Day 0
Secondary outcomes:
measure: Treatment Selection Accuracy across clinical vignette settings
timeFrame: Day 0
measure: Diagnosis specific diagnostic accuracy across clinical vignette settings
timeFrame: Day 0
Other outcomes:
Locations:
city: Ann Arbor
country: United States
facility: University of Michigan
state: Michigan
Study officials:
affiliation: University of Michigan
name: Michael Sjoding, MD
role: PRINCIPAL_INVESTIGATOR
References: