Prospective, Multi-Center, Randomized Controlled Trial for Skeletal Age Assessment AI Model
Phase: NA
Conditions: Bone Age
Sponsor: Stanford University

Prospective, Multi-Center, Randomized Controlled Trial for 
Skeletal Age Assessment AI Model 
Study Protocol and Statistical Analysis Plan 
NCT03530098 
 August 1, 2018 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 1 of 12 
 
Clinical Research Protocol 
SKELETAL AGE ASSESSMENT AI MODEL 
Protocol Number: 
1 
Version Date: 
August 2018 
Investigational Device: 
Skeletal age assessment AI model 
IND Number: 
N/A 
Development Phase: 
N/A 
Sponsor: 
N/A 
Funding Organization: 
N/A 
Principal Investigator: 
Name: Safwan S. Halabi, M.D. 
 
E-mail: safwan.halabi@stanford.edu 
Telephone: (650) 814-1561 
 
Name: Matthew P. Lungren, M.D., M.P.H.  
E-mail: mlungren@stanford.edu 
Telephone: (919) 943-3151 
Medical Monitor: 
N/A 
Coordinating Center: 
Stanford University 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 2 of 12 
 
PROTOCOL SYNOPSIS 
TITLE 
Prospective, Multi-Center, Randomized Controlled Trial for Skeletal 
Age Assessment AI Model 
SPONSOR 
N/A 
FUNDING 
ORGANIZATION 
N/A 
NUMBER OF SITES 
6 
RATIONALE 
To determine whether the AI model improves radiologist 
performance. 
STUDY DESIGN 
This is a prospective, multi-center, randomized controlled trial with 
two arms: Experiment and Control.  Radiologists receive the AI 
model for studies randomized into the Experiment arm before 
dictating a final impression. 
PRIMARY 
OBJECTIVE 
To determine whether radiologists are more accurate at assessing 
skeletal age using the AI model, compared to not using the AI model. 
SECONDARY 
OBJECTIVES 
To determine whether radiologists are more efficient at assessing 
skeletal age using the AI model, compared to not using the AI model. 
NUMBER OF 
SUBJECTS 
1600 interpretations by consented radiologists. 
SUBJECT 
SELECTION 
CRITERIA 
Inclusion Criteria: 
1. Verbal informed consent obtained from interpreting radiologist. 
2. Procedure code or study description indicative of hand radiograph 
for skeletal age assessment. 
Exclusion Criteria: 
1. Trainee provided a preliminary interpretation. 
TEST DEVICE, 
DEPLOYMENT 
The AI model will populate the radiologists’ report templates. 
CONTROL DEVICE, 
DEPLOYMENT 
The AI model will not populate the radiologists’ report templates. 
DURATION OF 
STUDY 
Estimated to be 6 months from the start date. 
CONCOMMITANT 
MEDICATIONS 
N/A 
EFFICACY 
EVALUATIONS 
N/A 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 3 of 12 
 
PRIMARY ENDPOINT 
• 
Mean absolute difference between dictated final impressions and 
the consensus determination of a panel of radiologists. 
SECONDARY 
ENDPOINTS 
• 
Radiologist interpretation time. 
OTHER 
EVALUATIONS 
N/A 
SAFETY 
EVALUATIONS 
Since the device does not pose greater than minimal risk, neither an 
adverse event (AE) or serious adverse experience (SAE) is expected.  
Unanticipated risks will be reported to the IRB, in accordance with 
IRB standards. 
PLANNED INTERIM 
ANALYSES  
N/A 
STATISTICS 
Primary Analysis Plan 
1. Detect differences in the primary endpoint between Experiment 
and Control arms, using a two-sided Student t-test. 
2. Detect differences in the secondary endpoint between Experiment 
and Control arms, using a Wilcoxon rank-sum test. 
Rationale for Number 
of Subjects 
Targeting a power of 0.75, statistical tests achieved a power of 0.79 to 
detect a difference of 0.50 months for 1600 interpretations. 
 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 4 of 12 
 
1 
BACKGROUND 
Radiologists perform a skeletal age assessment to determine the developmental status of a 
pediatric patient.  We trained an AI model on the open-source RSNA dataset to predict the 
skeletal age from a hand radiograph and intend to determine the effect of its clinical use in 
a prospective, multi-center, randomized controlled trial. 
1.1 
Overview of Non-Clinical Studies 
Previous work from Larson, et al. [1] suggests that using an AI model as a diagnostic aid 
could improve the quality of skeletal age assessment. 
[1] Larson, et al. “Performance of a Deep-Learning Neural Network Model in Assessing 
Skeletal Maturity on Pediatric Hand Radiographs” (2017). 
1.2 
Overview of Clinical Studies 
To our knowledge, there do not exist previous clinical studies in the form of a prospective 
clinical trial of any AI model. 
2 
STUDY RATIONALE 
Though the AI model from Larson, et al. [1] performs as well as human radiologists in 
controlled settings, it remains uncertain whether use of an AI model as a diagnostic aid 
can improve the quality of skeletal age assessment in true clinical settings. 
3 
STUDY OBJECTIVES 
3.1 
Primary Objective 
The primary objective is to determine the effect of using an AI model as a diagnostic aid 
for skeletal age assessment on the quality of radiologists’ interpretations, as measured by 
the mean absolute difference between their dictated final impressions and the consensus 
determination of a panel of radiologists. 
3.2 
Secondary Objectives 
The secondary objective is to determine the effect of using an AI model as a diagnostic 
aid for skeletal age assessment on the efficiency of radiologists’ interpretations, as 
measured by their interpretation times. 
4 
STUDY DESIGN 
4.1 
Study Overview 
This is a prospective, multi-center, randomized controlled trial.  1600 interpretations by 
consented radiologists are planned. 
Each of the interpreted studies will be assigned at random to one of two arms: Experiment 
or Control.  The AI model will populate the radiologists’ report templates for studies 
assigned to the Experiment arm.  The AI model will not populate the radiologists’ report 
templates for studies assigned to the Control arm. 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 5 of 12 
 
5 
CRITERIA FOR EVALUATION 
5.1 
Primary Efficacy Endpoint 
The primary endpoint is mean absolute difference between dictated final impressions and 
the consensus determination of a panel of radiologists. 
5.2 
Secondary Efficacy Endpoints 
• 
Radiologist interpretation time. 
5.3 
Safety Evaluations 
• 
Since the device does not pose greater than minimal risk, neither an adverse event 
(AE) or serious adverse experience (SAE) is expected.  Unanticipated risks will be 
reported to the IRB, in accordance with IRB standards. 
6 
SUBJECT SELECTION 
6.1 
Study Population 
Radiologists who interpret hand radiographs for skeletal age assessment will be eligible 
for participation in this study. Studies that meet the inclusion and exclusion criteria will be 
eligible for this study. 
6.2 
Inclusion Criteria 
1. Verbal informed consent obtained from interpreting radiologist. 
2. Procedure code or study description indicative of hand radiograph for skeletal age 
assessment. 
6.3 
Exclusion Criteria 
1. 
Trainee provided a preliminary interpretation. 
7 
CONCURRENT MEDICATIONS 
The study does not involve medications or chronic therapies.  
8 
STUDY TREATMENTS 
8.1 
Method of Assigning Subjects to Treatment Groups 
Studies will be assigned at random to Experiment or Control treatment groups in a 1:1 
ratio using the Python random package. 
8.2 
Blinding 
Due to the objectives of the study, the identity of Experiment and Control treatment 
groups will be known to radiologists. 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 6 of 12 
 
8.3 
Formulation of Test and Control Products 
The study does not involve test or control devices.  The Control arm represents the 
absence of any intervention and is consistent with the current standard of care. 
8.4 
Supply of Study Device at the Site 
The device will be installed at the investigational sites one time prior to randomization by 
the Investigator.  The device will receive studies from the sites, perform randomization, 
and populate the radiologists’ report templates for studies assigned to the Experiment arm 
for the duration of the study. 
9 
STUDY PROCEDURES AND GUIDELINES 
A Schedule of Events representing the required testing procedures to be performed for the 
duration of the study is diagrammed in Appendix 1. 
Prior to conducting any study-related activities, verbal informed consent must be obtained 
from the radiologist.  In addition, written informed consent by the patient must be waived 
by the IRB. 
9.1 
Clinical Assessments 
9.1.1 
Concomitant Medications 
The study does not require documentation of concomitant medication or concurrent 
therapies. 
9.1.2 Demographics  
Demographic information (date of birth, gender, race) will be collected retrospectively. 
9.1.3 Medical History  
Relevant medical history, including history of current disease, other pertinent history, and 
information regarding underlying diseases will be collected retrospectively. 
9.1.4 
Physical Examination 
The study does not require a physical examination. 
9.1.5 
Vital Signs 
The study does not require body temperature, blood pressure, pulse and respirations. 
9.1.6 
Adverse Events 
Since the device does not pose greater than minimal risk, neither an adverse event (AE) or 
serious adverse experience (SAE) is expected.  Unanticipated risks will be reported to the 
IRB, in accordance with IRB standards. 
9.2 
Clinical Laboratory Measurements 
The study does not require clinical laboratory measurements. 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 7 of 12 
 
10 EVALUATIONS BY VISIT 
The device records individual visits. 
1. 
Receive the study as a DICOM file. 
2. 
Record the study as a DICOM file. 
3. 
Randomizes the study. 
4. 
Perform and record the AI model. 
5. 
If randomized into the Experiment arm, populate the radiologists’ report template 
with the AI model. 
11 ADVERSE EXPERIENCE REPORTING AND DOCUMENTATION 
11.1 Adverse Events 
An adverse event (AE) is any untoward medical occurrence in a clinical investigation of a 
patient administered a pharmaceutical product and that does not necessarily have a causal 
relationship with the treatment.  An AE is therefore any unfavorable and unintended sign 
(including an abnormal laboratory finding), symptom or disease temporally associated 
with the administration of an investigational product, whether or not related to that 
investigational product. 
The device is not a pharmaceutical product. 
Since the device does not pose greater than minimal risk, an adverse event (AE) is not 
expected.  Unanticipated risks will be reported to the IRB, in accordance with IRB 
standards. 
11.2 Serious Adverse Experiences (SAE) 
An SAE is defined as any AE occurring at any dose that results in any of the following 
outcomes: 
• 
death 
• 
a life-threatening adverse experience 
• 
inpatient hospitalization or prolongation of existing hospitalization 
• 
a persistent or significant disability/incapacity 
• 
a congenital anomaly/birth defect  
Other important medical events may also be considered an SAE when, based on 
appropriate medical judgment, they jeopardize the subject or require intervention to 
prevent one of the outcomes listed. 
The device is not a pharmaceutical product. 
Since the device does not pose greater than minimal risk, a serious adverse experience 
(SAE) is not expected.  Unanticipated risks will be reported to the IRB, in accordance 
with IRB standards. 
   

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 8 of 12 
 
12 DISCONTINUATION AND REPLACEMENT OF SUBJECTS 
12.1 Early Discontinuation of Study Device  
A radiologist may be discontinued from study treatment at any time if the radiologist or 
the Investigator feels that it is not in the radiologist’s best interest to continue. 
All radiologists are free to withdraw from participation at any time, for any reason, 
specified or unspecified, and without prejudice. 
Reasonable attempts will be made by the Investigator to provide a reason for radiologist 
withdrawals. 
12.3 
Withdrawal of Subjects from the Study 
A radiologist may be withdrawn from the study at any time if the radiologist or the 
Investigator feels that it is not in the radiologist’s best interest to continue. 
All radiologists are free to withdraw from participation at any time, for any reason, 
specified or unspecified, and without prejudice. 
Reasonable attempts will be made by the Investigator to provide a reason for radiologist 
withdrawals.   
12.4 
Replacement of Subjects 
Radiologists who withdraw from the study treatment will not be replaced.   
Radiologists who withdraw from the study will not be replaced. 
 
13 PROTOCOL VIOLATIONS 
A protocol violation occurs when the radiologist or Investigator fails to adhere to 
significant protocol requirements affecting the inclusion, exclusion, radiologist safety and 
primary endpoint criteria.  Protocol violations for this study include, but are not limited to, 
the following: 
• 
Failure to meet inclusion/exclusion criteria 
Failure to comply with Good Clinical Practice (GCP) guidelines will also result in a 
protocol violation. The Investigator will determine if a protocol violation will result in 
withdrawal of a radiologist. 
When a protocol violation occurs, it will be discussed with the Investigator and a Protocol 
Violation Form detailing the violation will be generated. 
14 STATISTICAL METHODS AND CONSIDERATIONS   
14.1 Data Sets Analyzed 
Studies that meet the inclusion and exclusion criteria will be included the analysis. 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 9 of 12 
 
14.2 Demographic and Baseline Characteristics 
The following demographic variables will be summarized: age, sex, skeletal age, and 
clinical histories.   
14.3 Analysis of Primary Endpoint 
Two-sided Student’s t-test will be employed for the primary endpoint. 
14.4 Analysis of Secondary Endpoints 
Wilcoxon rank-sum test will be employed for the secondary endpoint. 
14.5 Interim Analysis 
There is no planned interim analysis. 
14.6 Sample Size and Randomization 
Targeting a power of 0.75, statistical tests achieved a power of 0.79 to detect a difference 
of 0.50 months for 1600 interpretations. 
15 DATA COLLECTION, RETENTION AND MONITORING 
15.1 Data Collection Instruments 
The Investigator will prepare and maintain adequate and accurate documents designed to 
record all observations and other pertinent data for each radiologist. 
The Investigator is responsible for all information collected on radiologists enrolled in this 
study.  All data collected during the course of this study must be reviewed and verified for 
completeness and accuracy by the Investigator. 
15.2 Data Management Procedures 
The data will be entered into a validated database.  The Investigator will be responsible 
for data processing, in accordance with procedural documentation. 
All procedures for the handling and analysis of data will be conducted using good 
computing practices. 
15.3 Data Quality Control and Reporting 
After data have been entered into the study database, a system of computerized data 
validation checks will be implemented and applied to the database on a regular basis.  The 
study database will be updated in accordance with the resolved queries.  All changes to 
the study database will be documented. 
15.4 Archival of Data 
The database is safeguarded against unauthorized access by established security 
procedures; appropriate backup copies of the database and related software files will be 
maintained.  Databases are backed up by the database administrator in conjunction with 
any updates or changes to the database.   

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 10 of 12 
 
At critical junctures of the protocol (e.g., production of interim reports and final reports), 
data for analysis is locked and cleaned per established procedures. 
15.5 Availability and Retention of Investigational Records 
The Investigator must make study data accessible to the IRB/IEC and Regulatory Agency 
(e.g., FDA) inspectors upon request. 
All study documents must be kept secured for a period of two years following conclusion 
of the study. 
15.6 Monitoring 
The study does not involve monitoring by representatives of a Sponsor. 
16 ADMINISTRATIVE, ETHICAL, REGULATORY CONSIDERATIONS 
The study will be conducted according to the Declaration of Helsinki, Protection of 
Human Volunteers (21 CFR 50), Institutional Review Boards (21 CFR 56), and 
Obligations of Clinical Investigators (21 CFR 312). 
To maintain confidentiality, all laboratory specimens, evaluation forms, reports and other 
records will be identified by a coded number and initials only.  All study records will be 
kept in a locked file cabinet and code sheets linking a patient’s name to a patient 
identification number will be stored separately in another locked file cabinet.  Clinical 
information will not be released without written permission of the subject, except as 
necessary for monitoring by the FDA.  The Investigator must also comply with all 
applicable privacy regulations (e.g., Health Insurance Portability and Accountability Act 
of 1996, EU Data Protection Directive 95/46/EC). 
16.1 Protocol Amendments 
Any amendment to the protocol will be written by the investigator.  Protocol amendments 
cannot be implemented without prior written IRB/IEC approval except as necessary to 
eliminate immediate safety hazards to patients.  A protocol amendment intended to 
eliminate an apparent immediate hazard to patients may be implemented immediately, 
provided the IRBs are notified within five working days. 
16.2 Institutional Review Boards and Independent Ethics Committees 
The protocol and consent form will be reviewed and approved by the IRB/IEC of each 
participating center prior to study initiation.  Serious adverse experiences regardless of 
causality will be reported to the IRB/IEC in accordance with the standard operating 
procedures and policies of the IRB/IEC, and the Investigator will keep the IRB/IEC 
informed as to the progress of the study.  The Investigator will obtain assurance of 
IRB/IEC compliance with regulations. 
Any documents that the IRB/IEC may need to fulfill its responsibilities (such as protocol, 
protocol amendments, Investigator’s Brochure, consent forms, information concerning 
patient recruitment, payment or compensation procedures, or other pertinent information) 
will be submitted to the IRB/IEC.  The IRB/IECs written unconditional approval of the 
study protocol and the informed consent form will be in the possession of the Investigator 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018 
Page 11 of 12 
 
before the study is initiated.  This approval must refer to the study by exact protocol title 
and number and should identify the documents reviewed and the date of review. 
Protocol and/or informed consent modifications or changes may not be initiated without 
prior written IRB/IEC approval except when necessary to eliminate immediate hazards to 
the patients or when the change(s) involves only logistical or administrative aspects of the 
study.  Such modifications will be submitted to the IRB/IEC and written verification that 
the modification was submitted and subsequently approved should be obtained.   
The IRB/IEC must be informed of revisions to other documents originally submitted for 
review; serious and/or unexpected adverse experiences occurring during the study in 
accordance with the standard operating procedures and policies of the IRB; new 
information that may affect adversely the safety of the patients of the conduct of the study; 
an annual update and/or request for re-approval; and when the study has been completed. 
16.3 Informed Consent Form  
Written informed consent by the patient must be waived by the IRB. 
A properly executed, verbal, informed consent will be obtained from each radiologist 
prior to entering the radiologist into the trial.  Information should be given in oral form 
and radiologists must be given ample opportunity to inquire about details of the study. 
16.4 Publications  
The publication or presentation of any study results shall comply with all applicable 
privacy laws, including, but not limited to, the Health Insurance Portability and 
Accountability Act of 1996. 
 

Halabi and Lungren 
Confidential 
 
Version #: 1 
Version Date: August 2018  
Page 12 of 12 
 
APPENDIX 1.  SCHEDULE OF EVENTS 
 
REQUIRED 
Install Device 
X 
Informed Consent 
X 
Test Device 
X 
Begin Randomization 
X 
End Randomization 
X 
Collect Data from Device 
X