Clinical trial NCT03530098: Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment
Official title: Prospective, Multi-Center, Randomized Controlled Trial for Skeletal Age Assessment AI Model
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Bone Age
Keywords: Age Determination by Skeleton
Machine Learning
Deep Learning
Artificial Intelligence
Prospective
Clinical Validation
Lead sponsor: Stanford University
Sponsor class: OTHER
First posted: 2018-05-21
Enrollment: 1903
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: no
Allocation: RANDOMIZED
Intervention model: PARALLEL
Intervention model description: A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the exam using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their skeletal age estimation. In all cases, the decision of the radiologist will be considered final.
Primary purpose: DIAGNOSTIC

Brief summary: The purpose of this study is to understand the effects of using an Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make an estimation of the bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.

Detailed description: The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the radiographic images and health records. As part of this study, the radiologists will estimate skeletal age from radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age.

This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard ("gold standard") composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD). The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups.

The investigators have recruited and analyzed data from a sample size of 1600 exams. Patients getting these exams will not undergo any research procedures that deviate from the current standard practices.

Eligibility criteria: Exams that meet the following inclusion criteria will be included: (1) exams read by radiologists who interpret pediatric skeletal age exams and verbally consent to participate (2) exams that contain a procedure code or study description indicative of a skeletal age exam.

Exams containing more than one radiograph will not be included. Exams for which a trainee provides a preliminary interpretation will be excluded. No further exclusion criteria will be applied on the basis of image quality metrics or manufacturers. No exclusion criteria will be applied on the basis of patient chronological age.

Primary outcomes:
measure: Paired Difference of Skeletal Age Estimate
timeFrame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review
Secondary outcomes:
measure: Time for Diagnosis
timeFrame: Up to approximately 4 minutes
Other outcomes:
Locations:
city: Stanford
country: United States
facility: Stanford University
state: California
city: New Haven
country: United States
facility: Yale New Haven Hospital
state: Connecticut
city: Boston
country: United States
facility: Boston Children's Hospital
state: Massachusetts
city: New York
country: United States
facility: New York University
state: New York
city: Cincinnati
country: United States
facility: Cincinnati Children's Hospital Medical Center
state: Ohio
city: Philadelphia
country: United States
facility: Children's Hospital of Philadelphia
state: Pennsylvania
Study officials:
affiliation: Stanford University
name: Curtis Langlotz, M.D. Ph.D.
role: STUDY_CHAIR
affiliation: Stanford University
name: David Eng, B.S.
role: STUDY_DIRECTOR
affiliation: Stanford University
name: Nishith Khandwala, B.S.
role: STUDY_DIRECTOR
affiliation: Stanford University
name: Safwan Halabi, M.D.
role: PRINCIPAL_INVESTIGATOR
References: