A Targeted Approach To A Safer Use of Antipsychotics In Youth
Phase: NA
Conditions: Child Behavior Disorders
Sponsor: Kaiser Permanente

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
 
Protocol for Safety Use of Antipsychotics in Youth (SUAY) 
 
Title: Safer Use of Antipsychotics in Youth (SUAY) 
 
NCT: NCT03448575 
 
Dated: 7/18/2018 
 
Last updated: 12/20/2019 
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
 
 
 
Safer Use of Antipsychotics in Youth (SUAY) 
PHASE 3 – PRAGMATIC TRIAL 
 
Contract Number:  HHSN 2712016000002 C 
 
Principal Investigator:  Robert Penfold, PhD 
Contracting Officer’s Representative: Adam Haim, PhD 
 
Version Number:  V5 
20 December 2019 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
1 
 
 
SIGNATURE PAGE 
Signatures constituting approval of this protocol and the attachments, and providing necessary 
assurances that this trial will be conducted according to all stipulations of the protocol, including all 
statements regarding confidentiality, and according to local legal and regulatory requirements and 
applicable U.S. federal regulations and guidelines are maintained electronically due to the multiple 
health systems and IRBs involved in this study Signatures can be found on the study’s private web 
portal. 
 
Signatories for this protocol are:  
Robert Penfold, PhD – Principal Investigator 
Associate Investigator 
Kaiser Permanente Washington Health Research Institute 
 
Arne Beck, PhD – Site Investigator 
Director of Quality Improvement and Strategic Research at the Institute for Health Research 
Kaiser Permanente Colorado  
 
Kelly Kelleher, MD – Site Investigator 
Director 
The Research Institute at Nationwide Children’s Hospital 
 
Gregory Clarke, PhD – Site Investigator 
Distinguished Investigator, Associate Director, Science Programs Kaiser Permanente Northwest  
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
2 
 
TABLE OF CONTENTS 
PAGE 
SIGNATURE PAGE ............................................................................................................................... 1 
TABLE OF CONTENTS .......................................................................................................................... 2 
LIST OF ABBREVIATIONS ..................................................................................................................... 5 
PROTOCOL SUMMARY – SUAY PRAGMATIC EFFECTIVENESS TRIAL ...................................................... 6 
1 
KEY ROLES AND CONTACT INFORMATION .................................................................................. 9 
2 
INTRODUCTION: BACKGROUND INFORMATION AND SCIENTIFIC RATIONALE ............................ 10 
2.1 
Background Information ...................................................................................................... 10 
2.2 
Rationale and Objectives ..................................................................................................... 11 
3 
OUTCOME MEASURES ............................................................................................................. 12 
3.1 
Primary ................................................................................................................................. 12 
3.2 
Secondary ............................................................................................................................. 12 
4 
TRIAL DESIGN ......................................................................................................................... 13 
5 
STUDY ENROLLMENT AND WITHDRAWAL ................................................................................ 14 
5.1 
Prescribing Clinicians: Inclusion and Exclusion Criteria for Provider Subjects ..................... 14 
5.2 
Patients and Families: Inclusion and Exclusion Criteria for Patient Subjects....................... 14 
5.3 
Provider Randomization Assignment Procedures ................................................................ 15 
5.3.1 
Randomization Procedures ..................................................................................... 15 
5.3.2 
Blinding ................................................................................................................... 16 
5.3.3 
Cross-Over Prevention Procedures ......................................................................... 16 
5.4 
Withdrawals ......................................................................................................................... 16 
5.5 
Opting Out of Study Components ........................................................................................ 17 
6 
CONTROL ARM ....................................................................................................................... 18 
7 
INTERVENTION ARM ............................................................................................................... 19 
7.1 
Intervention Arm Medication Alert Intervention Description ............................................. 19 
7.2 
Eligibility Confirmation ......................................................................................................... 19 
7.3 
CAP Case Review and Consultation Intervention Description ............................................. 19 
7.3.1 
Administration of CAP Case Review and Consultation ........................................... 20 
7.3.2 
Procedures for Training Study CAPs and Fidelity Monitoring ................................. 20 
7.4 
Behavioral Health Navigation Description ........................................................................... 20 
7.4.1 
Administration of BH Navigation Intervention ....................................................... 21 
7.4.2 
Procedures for Training of BH Navigators on Intervention .................................... 22 
7.5 
Short-Term Bridging Therapy Description ........................................................................... 22 
7.5.1 
Administration of Bridging Therapy ........................................................................ 22 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
3 
 
7.5.2 
Procedures for Training of Therapists in Telemental Health .................................. 23 
8 
STATISTICAL CONSIDERATIONS................................................................................................ 24 
8.1 
Sample Size Considerations ................................................................................................. 24 
8.2 
Primary Analysis Plan ........................................................................................................... 24 
8.2.1 
Statistical Analysis Plan ........................................................................................... 24 
8.3 
Handling of Missing Data ..................................................................................................... 27 
9 
PRAGMATIC TRIAL OVERSIGHT ................................................................................................ 29 
9.1 
Data Safety Monitoring ........................................................................................................ 29 
9.1.1 
Schedule and Content of DSMB Reports ................................................................ 29 
9.2 
Premature Termination or Suspension of Trial .................................................................... 29 
10 
ASSESSMENT OF SAFETY ......................................................................................................... 30 
10.1 
Specification of Safety Parameters ...................................................................................... 30 
10.1.1 Safe Practice Protocols and Tracking ...................................................................... 30 
10.1.2 Adverse Events (AE) / Serious Adverse Events (SAE) and Reporting ...................... 30 
10.1.3 Unanticipated Problems and Reporting .................................................................. 31 
10.2 
Halting Rules ........................................................................................................................ 31 
11 
CLINICAL SITE MONITORING .................................................................................................... 32 
12 
QUALITY CONTROL AND QUALITY ASSURANCE ........................................................................ 33 
12.1 
Study Procedures ................................................................................................................. 33 
12.2 
Quality Management ........................................................................................................... 33 
12.3 
Protocol Deviations .............................................................................................................. 33 
13 
ETHICS/PROTECTION OF HUMAN SUBJECTS ............................................................................. 34 
13.1 
Ethical Standard ................................................................................................................... 34 
13.2 
Potential Risks and Benefits ................................................................................................. 34 
13.2.1 Potential Risks ......................................................................................................... 34 
13.2.2 Potential Benefits .................................................................................................... 34 
13.3 
Institutional Review Board ................................................................................................... 34 
13.4 
Informed Consent Process ................................................................................................... 35 
13.5 
Exclusion of Women, Minorities, and Children (Special Populations) ................................. 35 
13.6 
Subject Confidentiality ......................................................................................................... 36 
13.7 
Future Use of Identifiable Data ............................................................................................ 36 
14 
DATA HANDLING AND RECORD KEEPING ................................................................................. 37 
14.1 
Data Management Responsibilities ..................................................................................... 37 
14.2 
Data Capture Methods ......................................................................................................... 37 
14.3 
Study Records Retention ...................................................................................................... 37 
15 
PUBLICATION AND NEW SUB-PROJECT POLICIES ...................................................................... 38 
16 
DATA SHARING PLAN .............................................................................................................. 39 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
4 
 
17 
LITERATURE ............................................................................................................................ 40 
PROTOCOL APPENDICES ................................................................................................................... 43 
APPENDIX A: SCHEDULE OF EVENTS .................................................................................................. 44 
APPENDIX B: PROVIDER ADVANCE COMMUNICATIONS AND MEDICATION ALERTS ............................ 46 
APPENDIX C: SUAY CLINICAL GUIDANCE ........................................................................................... 47 
APPENDIX D: BH NAVIGATION .......................................................................................................... 48 
APPENDIX E: TELEMENTAL HEALTH BRIDGING .................................................................................. 49 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
5 
 
LIST OF ABBREVIATIONS 
ADHD 
Attention Deficit Hyperactivity Disorder 
AE 
Adverse Event/Adverse Experience 
APA 
American Psychiatric Association 
BH 
Behavioral Health 
CAP 
Child and Adolescent Psychiatrist 
CD 
Conduct Disorder 
CFR 
Code of Federal Regulations 
DSMB 
Data and Safety Monitoring Board 
EMR 
Electronic Medical Record 
FDA 
Food and Drug Administration 
FTE 
Full Time Equivalent 
FTP 
Secure File Transfer Protocol 
HCSRN 
Health Care Systems Research Network 
HIPAA 
Health Information Portability and Accountability Act 
IRB 
Institutional Review Board 
KP 
Kaiser Permanente 
KPWA 
KP Washington 
KPWHRI 
KP Washington Health Research Institute 
MHRN 
Mental Health Research Network, HCSRN 
NIMH 
National Institute of Mental Health, NIH 
ODD 
Oppositional Defiant Disorder 
OHRP 
Office of Human Research Protections 
PAL 
Partnership Access Line 
QMP 
Quality Management Plan 
SAE 
Serious Adverse Event/Serious Adverse Experience 
SUAY 
Safer Use of Antipsychotics in Youth  
TMH 
Telemental Health 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
6 
 
PROTOCOL SUMMARY – SUAY PRAGMATIC EFFECTIVENESS TRIAL 
Title: 
A Pragmatic Effectiveness Trial: Safer Use of Antipsychotics in Youth (SUAY) 
Précis: 
A pragmatic encouragement trial will test the effectiveness of an intervention 
aiming to improve the targeted and safer use of antipsychotic medications by 
guiding clinician-prescribing behavior of antipsychotics for children aged ≥ 3 
and < 18 years and encouraging psychosocial therapy for eligible youth. The 
intervention includes a medication best practice alert in Epic, consultation with 
a child and adolescent psychiatrist, and extra support for patients and families 
to improve behavioral health service access. The trial will be conducted in 
multiple health systems. Each health system will randomize prescribers to one 
of two study arms, intervention and control. During patient encounters, 
entering an antipsychotic for a potentially eligible patient will cause either the 
control or intervention medication alert to fire in the electronic medical record. 
The control arm medication alert will point prescribing clinicians to relevant 
Choosing Wisely® recommendations. The intervention arm medication alert will 
inform prescribers that: 
1. Antipsychotics are not recommended 1st line treatment for non-
psychotic disorders; 
2. A child and adolescent psychiatrist (CAP) will review antipsychotic 
usage by youth;  
3. Expedited access to bridging therapy, behavior health navigation, 
and/or a proactive consultation with a CAP may be ordered.  
The intervention medication alert will point prescribing clinicians to both 
Choosing Wisely® recommendations and clinical guidelines for SUAY.  
Analytic data will be collected from automated data sources at the health 
systems. The primary outcomes are percent of children taking an antipsychotic 
medication at 6 months of follow up, and total person-months of antipsychotic 
use among participants during the study period – based on medication order 
data.  
Objectives: 
Test the effectiveness of the intervention treatment algorithm vs. usual care 
control in a practical clinical trial involving several hundred subjects in multiple 
health systems.  
Populations: 
Providers authorized to order prescription medications in the health system’s 
electronic medical record.  
Patients aged ≥ 3 and <18 years without diagnosed psychosis, mania, autism 
spectrum disorder, or intellectual disability.  
Number of Sites: 
1. Kaiser Permanente Washington 
2. Kaiser Permanente Colorado 
3. Nationwide Children’s Hospital 
4. Kaiser Permanente Northwest 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
7 
 
Description of 
Intervention: 
The intervention arm medication alert prompts the prescriber to (a) 
keep/remove the antipsychotic order, and/or (b) order behavioral health (BH) 
navigation, and/or (c) order expedited access to psychotherapy (e.g., 
telemental health bridging) for the patient/family, and/or (d) order a provider-
to-provider virtual consult with a child and adolescent psychiatrist (CAP).  
Case review by a child and adolescent psychiatrist will automatically occur for 
all eligible intervention arm patients. A provider-to-provider consult will be 
scheduled if either (a) the prescriber ordered the consult or (b) the CAP needs 
to discuss the case with the prescriber to complete his/her review.  
Following review by the CAP, a BH navigator messages the prescriber and 
reaches out to the eligible intervention arm patient/family to offer extra 
support (unless actively declined by the prescriber or CAP). The navigator’s role 
is to (a) provide extra support to facilitate access and engagement in 
appropriate psychosocial therapies; (b) coordinate short-duration bridging 
therapy sessions for teens/families not engaged in psychotherapy, when 
appropriate; and (c) keep the prescriber informed of any clinically relevant 
updates. 
Trial Duration: 
30 months 
Subject 
Participation 
Duration: 
Up to 30 months for clinicians 
6 months for patients 
Estimated Time to 
Complete 
Enrollment: 
24 months 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
8 
 
Schematic of Study Design: 
 
PHASE 3 PRAGMATIC EFFECTIVENESS TRIAL
Patient Encounter 
Outpatient antipsychotic 
medication entered in 
Epic
Intervention 
medication alert fires; 
patient added 
to study log
Control medication 
alert fires; patient 
added to study log
Standard Care
Case review by child and 
adolescent psychiatrist 
(CAP)
6 months of BH Navigation offered
Analytic data data collected for 6 month follow up period
Provider pre-randomization and advanced study notification 
CONTROL
INTERVENTION
Patient Encounter 
Outpatient antipsychotic 
medication entered in 
Epic
 Eligibility 
confirmed
Excluded
Eligible
Eligibility 
confirmed
Ineligible
CAP recommendations 
shared with prescriber 
and study navigator
Ineligible
Extra support to access and stay 
engaged in BH care
TMH Bridging
Transition to BH Care
declined
accepted
Patient added 
to study log
Patient Encounter 
BHN, bridging, or CAP 
consult ordered directly in 
Epic (i.e., outside the 
medication alert)
Either
AP or CAP 
consult 
ordered?
Eligible
Yes
No

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
9 
 
1 
KEY ROLES AND CONTACT INFORMATION 
Principal Investigator:   
Robert Penfold, PhD, Scientific Investigator II 
Kaiser Permanente Washington Health Research Institute (KPWHRI) 
1730 Minor Avenue, Suite 1600 
Seattle, WA 98101 
Robert.B.Penfold@kp.org 
(206) 287-2232 
Program Official:   
Adam Haim, PhD  
Site Investigators: 
Arne Beck, PhD, Kaiser Permanente Colorado 
Kelly Kelleher, MD, Nationwide Children’s Hospital 
 
Institutions: 
Kaiser Permanente Washington  
Nationwide Children’s Hospital  
Kaiser Permanente Colorado 
Kaiser Permanente Northwest 
Seattle Children’s Hospital 
University of Washington 
Co-Investigators: 
Robert Hilt, MD, Seattle Children’s Hospital 
Kathleen Myers, MD, Seattle Children’s Hospital 
Greg Simon, MD, MPH, KPWHRI 
Rebecca Yates Coley, PhD, KPWHRI 
James Ralston, MD, MPH, KPWHRI 
Arne Beck, PhD, Kaiser Permanente Colorado 
Kelly Kelleher, MD, Nationwide Children’s Hospital 
Gregory Clarke, PhD, Kaiser Permanente Northwest 
Paul Fishman, PhD, University of Washington 
Consultants: 
Katherine (Katie) Odem-Davis, PhD, University of Washington 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
10 
 
2 
INTRODUCTION: BACKGROUND INFORMATION AND SCIENTIFIC RATIONALE  
2.1 
Background Information 
Beginning in the mid 1990’s, rates of prescribing antipsychotic medications to children and adolescents 
grew tremendously (1-6) and remain high in both Medicaid and commercially insured populations (6-9). 
Most antipsychotic medications prescribed to children are “second generation” or “atypical” 
antipsychotic medications. We will refer to these simply as antipsychotics. These medications are argued 
to have milder side effect profiles (e.g., reduced extrapyramidal symptoms) than older “first generation” 
or “typical” antipsychotic medications. However, the side effects of newer APs can still be severe (10, 
11). The side effect profiles of individual medications in the same class differ significantly and their 
impact on the individual patients who take them are also known to differ (12-14).  
The vast majority of antipsychotic use in youth aged 5-17 years is for non-psychotic disorders and is 
particularly prevalent in youth with a diagnosis of attention deficit hyperactivity disorder (ADHD) with 
comorbid conduct disorder (CD) or oppositional defiant disorder (ODD) and/or impulsive aggression (7, 
15, 16). For example, we analyzed a sample of 562,423 youth prescribed psychotropic medications in 
our Mental Health Research Network (MHRN), a consortium of 13 health care delivery systems across 
the U.S., including 11,095 with at least one prescription for an antipsychotic medication, during the year 
2011 alone. In this sample, 66% of boys aged 6-11 years that were prescribed an antipsychotic 
medication had a diagnosis of ADHD and CD or ODD and did not have a psychotic disorder or other FDA-
approved indication. (See Penfold et. al., 2013 for a detailed breakdown of antipsychotic medication use 
in MHRN.) 
In its Choosing Wisely® recommendations, the American Psychiatric Association lists as its 5th 
recommendation: Don’t routinely prescribe an antipsychotic medication to treat behavioral and 
emotional symptoms of childhood mental disorders in the absence of approved or evidence supported 
indications(17). However, not all antipsychotic prescribing for emotional and behavioral disturbances is 
inappropriate. It may be reasonable for clinicians to prescribe antipsychotics when youth are a danger to 
themselves or others and/or not responding to primary treatments in the absence of a psychotic 
disorder. Nevertheless, a new and updated algorithm and workflow is needed to guide clinicians 
regarding when and how to prescribe antipsychotic medications to youth with non-psychotic disorders. 
Importantly, there are a number of reasonable algorithms for antipsychotic prescribing to youth(18-23); 
however, there are two fundamental gaps in our understanding of how to implement such guidelines: 
(a) how to support clinicians most effectively in following the algorithms and (b) how to increase access 
to psychosocial primary treatments that all of the existing guidelines recommend as part of first line 
treatment (in addition to non-antipsychotic medications). A new algorithm will be largely ignored unless 
we address the fundamental issues of implementation(24-31) and long known barriers to changing 
prescribing practices addressed in the academic detailing literature(32-39). 
In phase 1 of SUAY, a sequenced treatment algorithm and clinical workflow were developed. The study 
is inspired by the highly successful Partnership Access Line (PAL) and Second Opinion program operated 
for the Washington State Medicaid program. Implementation of this program in Washington resulted in 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
11 
 
a 50% reduction in antipsychotic medication use among Medicaid insured youth over its first four years. 
Second Opinion reviews have previously generated a 51% decrease in outlier ADHD stimulant 
medication prescribing, coupled with a 10:1 return on investment(40). Similar peer consult programs 
have been demonstrated effective in other settings.(41, 42) In its current form, this program requires an 
outpatient pharmacist to hold a prescription awaiting Medicaid authorization for an antipsychotic 
outside of an established state guideline until there is documentation of a consult between the 
prescribing provider and a child and adolescent psychiatrist (CAP) at Seattle Children’s Hospital. 
Providers can also electively reach out to the team’s consultants via the PAL service line to discuss best 
practice care in a state-wide rapid access program staffed by 1.5 FTE of CAP time. 
Further success and spread into community settings will require understanding and supporting provider, 
patient and caregiver needs at the point of care when therapies are first being considered and 
discussed. The core elements of our proposed intervention are: (a) sequenced ‘treatment alternatives’ 
algorithm based on best evidence that accommodates diversity of practice; (b) effective clinical decision 
support (including human consultation) integrated into the flow of practice; (c) behavioral health 
navigation for psychosocial treatment alternatives that are effective and acceptable to families; and (d) 
expedited access to evidence-based psychosocial therapy. 
 
2.2 
Rationale and Objectives 
Antipsychotics are prescribed to children for a number of behavioral health conditions despite lack of 
FDA approval or evidence of their effectiveness or safety. Such ‘off label’ use of antipsychotics exposes 
children to significant health risks. The SUAY trial aims to minimize use of antipsychotics to treat non-
psychotic behavioral disturbances in children in four real-world health care systems. 
The pragmatic encouragement trial will assess the effectiveness of an intervention aiming to improve 
patient safety by guiding clinician prescribing behavior of antipsychotics for children ≥ 3 and < 18 years 
old. The intervention includes a provider-facing medication alert in Epic and case review by a child and 
adolescent psychiatrist and extra patient-facing support to improve behavioral health service access 
(e.g., navigation, TMH).  
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
12 
 
3 
OUTCOME MEASURES 
Outcome measures will be collected regardless of the extent to which intervention arm clinicians, 
patients, or families accepted or utilized interventional components. That is, we will follow an intent-to-
treat approach and encouragement trial methods. 
3.1 
Primary 
The following primary quantitative outcome measures for the trial will be assessed for the contract 
deliverable: 
• 
Percent of youth using antipsychotics at 6 months, measured by medication orders placed 
within the health system during the 180 day follow up period; 
• 
Total person-months of antipsychotic use by youth, measured by medication orders placed 
within the health system during the 180 day follow up period. 
3.2 
Secondary 
The following secondary quantitative outcome measures for the trial will be assessed: 
• 
Percent of youth using antipsychotics at 6 months, measured by prescription fill data available 
for the 180 day follow up period; 
• 
Total person-months of antipsychotic use by youth, per prescription fill data available for the 
180 day follow up period. 
• 
Emergency department/urgent care visit frequency, both for psychiatric crises and for other 
reasons; 
• 
Percentage of baseline and follow-up safety assessments (i.e., BMI measurement and laboratory 
results);  
• 
Percentage of clinicians that follow CAP advice according to algorithm;  
• 
Modal stage at departure from algorithm (first line, second line, third line treatment prior to 
antipsychotic); 
• 
Percentage of patients with change to psychotropic medication treatment plan following second 
opinion review; 
• 
Percent of patients that agree to behavioral health (BH) navigation; 
• 
Percentage of patients attending two or more system-provided therapy sessions; 
• 
Percentage of patients attending two or more study-provided therapy sessions; 
• 
Of patients in study therapy, percentage who subsequently attend two or more system-
provided therapy sessions. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
13 
 
4 
TRIAL DESIGN 
We will conduct a two-group pragmatic encouragement clinical trial comparing the intervention and 
control at multiple sites. The follow up period for the trial will be 6 months.  
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
14 
 
5 
STUDY ENROLLMENT AND WITHDRAWAL 
5.1 
Prescribing Clinicians: Inclusion and Exclusion Criteria for Provider Subjects 
A site programmer will randomly assign providers to one of two study arms prior to fielding. 
Randomization determines which study medication alert the provider will receive. Intervention arm 
providers have the ability to order extra support from the study: 
• 
A provider-to-provider consult with a child and adolescent psychiatrist (CAP), and/or; 
• 
Behavioral health navigation for the patient/family , and/or; 
• 
Expedited access to psychotherapy (e.g., telemental health) for the patient/family. 
All randomized providers will receive advance notification of the study launch. Providers in the 
intervention arm will be told about the extra support that may be ordered and given a web-link to the 
expert clinical guidelines developed for SUAY. The control arm providers will be informed that a research 
study is being conducted to improve the quality of antipsychotic medication prescribing. 
Inclusions:  
• 
Clinician is credentialed to order prescription medications in the health system’s electronic 
medical record (EMR). 
Exclusions:  
• 
Clinician practices outside of the delivery system (e.g., external or network provider). 
5.2 
Patients and Families: Inclusion and Exclusion Criteria for Patient Subjects 
Potentially eligible patients are logged for study enrollment at the time the medication alert fires. 
Patients are assigned to the study arm of the randomized prescribing provider. Local study staff will 
blindly confirm eligibility via medical record review for all intervention and control arm patients logged 
for eligibility review and enrollment.  
Inclusions: 
• 
Patient is ≥ 3 and < 18 years of age at the time of the encounter at which the study alert fired;  
• 
Study service (BH navigation, bridging therapy, CAP consult) ordered in Epic for the patient;  
o The BPA allows providers to remove the antipsychotic order and still order services. 
• 
Patient is initiating a new episode of outpatient treatment with an antipsychotic medication.  
o New episodes are defined by no record of an antipsychotic medication being ordered 
within the health system as part of an outpatient care plan in the prior 180 days.  
Exclusions:  
• 
Patient has a diagnosed psychotic disorder, mania, autism spectrum disorder, or intellectual 
disability; 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
15 
 
• 
Patient was enrolled in the SUAY pilot study; 
• 
The antipsychotic entered is prochlorperazine (Comazol®) – which is used to treat nausea; 
• 
An outpatient antipsychotic order is entered by a temporary provider in the health 
system. For example, a “doc of the day” signing an order initiated by a pharmacist 
transferring an existing prescription to the health system. This situation may also occur 
when a temporary provider signs a medication order for a prescription initiated in an 
inpatient setting as part of a discharge plan. 
o Note that orders placed by temporary providers do not count towards the 180 
day medication free period for defining a new episode of care. 
• 
The antipsychotic order was placed within an urgent care, emergency department, or inpatient 
setting (to avoid intervening during a crisis).  
o Note that orders placed in these settings do not count towards the 180 day 
medication free period for defining a new episode of care. 
• 
Primary language is not English (e.g., patient requires a translator). 
5.3 
Provider Randomization Assignment Procedures 
5.3.1 
Randomization Procedures  
Randomization will occur before the trial begins. All eligible providers will be block randomized to 
ensure comparable numbers of patients enrolled in each study arm. Providers will be categorized 
according to the expected number of patients for whom they will issue a qualifying prescription during 
the study period. The expected number of qualifying prescriptions will be determined given past 
prescribing behavior and department or specialty. Information on past prescribing behavior may vary 
across study sites based on available provider data but will generally include the absolute number of 
qualifying prescriptions as well as the rate of qualifying prescriptions (per number of visits billed or 
number of unique patients seen, as available). Providers in certain departments may also be more likely 
to write qualifying prescriptions. For example, providers who specialize in mental or behavioral health or 
providers in clinics focusing on higher risk adolescents. The particular departments associated with more 
qualifying prescriptions will be identified separately for each health system based on previous 
prescribing patterns and knowledge of that health system.  
Providers in each health system will divided into as few as two randomization blocks (likely and unlikely 
prescribers) and as many as four randomization blocks (unlikely, likely, high frequency, and very high 
frequency) based on the total number and distribution of expected number of qualifying prescriptions. 
For example, KP Washington providers wrote fewer than 100 qualifying prescriptions in the yearlong 
interval examined and the distribution of qualifying prescriptions per provider had a long right tail 
indicating that a small number of physicians wrote a large number of qualifying prescriptions. In this 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
16 
 
context, we defined four groups for block randomization to protect against a large imbalance in the 
number of patients enrolled across study arms. 
R, an open-source statistical computing software, will be used to generate an even number of control 
and intervention assignments within each provider block. 
 Providers hired after initial randomization will be assigned to the unlikely or likely prescribing 
randomization blocks based on hiring department. Randomization schedules will be generated for 
unlikely and likely prescribing blocks, and randomization arms will be assigned sequentially as new 
providers are hired. Providers previously randomized for the phase 2 pilot will remain in the arm 
assigned for the pilot.  
5.3.2 
Blinding 
Study staff will be blinded to study arm when verifying subject eligibility.  
The details of the study design will not be known by clinicians delivering treatment or writing 
prescriptions or orders (i.e., outcome assessments will be blinded).  
Study child and adolescent psychiatrists, behavioral health navigators, and therapists only interact with 
intervention arm subjects. Group assignment is therefore known to these study staff. 
The PI and Site Clinical Investigators will remain blinded to arm assignment until follow-up is complete. 
Only data coordinating center staff will be un-blinded to arm when preparing harm reports for the 
DSMB). Investigators will be un-blinded for reporting any intervention arm incidental deaths to the 
NIMH DSMB, should such events occur during the course of the trial. 
5.3.3 
Cross-Over Prevention Procedures 
Epic coding will ensure control arm providers cannot access intervention components. Periodic checks 
will be conducted to ensure this remains in place as planned. 
If a control arm provider reaches out for a consultation, the CAP may still discuss treatment options and 
make recommendations as they would as part of their normal role. 
The behavioral health navigator will be responsible for ensuring that the services s/he offers and the 
bridging therapy are only accessible to intervention arm patients and providers.  
Patients who change to the panel of a provider assigned to a different arm during the 6 month follow-up 
period will remain in the arm of their original prescriber for the 6 month follow up period.  
5.4 
Withdrawals 
The algorithm and workflow will be applied to every randomized clinician and eligible patient enrolled at 
each health system.  
Waivers of consent will be obtained from the local Institutional Review Board (IRB) at each site. Neither 
providers nor patient-subjects may withdraw given the pragmatic design and intent-to-treat analysis 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
17 
 
plan. Subjects remain in the study regardless of their acceptability or degree of exposure to 
interventional components, given the encouragement trial design. 
5.5 
Opting Out of Study Components 
Medication alert: Providers may cancel the medication alert by both de-selecting the antipsychotic in 
Epic and not ordering any extra support (i.e., CAP consultation, BH navigation, expedited psychotherapy 
access).  
CAP review: Intervention arm prescribers may not opt out of passive CAP reviews. Control arm 
prescribers are not included in CAP review. 
• 
The CAP is automatically informed of the need for a passive review (i.e., provider did not order a 
consult) after eligibility is confirmed.  
• 
The CAP is automatically informed of the need for an active review (i.e., provider ordered a 
consult), prior to eligibility confirmation to avoid any delay.  
o If the patient is deemed ineligible upon review, other study services will not be offered 
to the patient/family.  
• 
Passive CAP review will still occur if the clinician clicked an exclusion in the BPA (e.g., autism) 
and the patients is subsequently verified to be eligible. That is, study staff (not the prescriber) 
determines study eligibility.  The prescriber maintains his/her clinical autonomy and may opt not 
to follow the CAP’s treatment recommendations. No treatments or medications are withheld at 
any time.   
CAP consultation: The study CAP may still reach out to an intervention arm prescriber to discuss a case 
when a CAP consult was not ordered. The prescriber may decide not to respond to or schedule with the 
CAP. The prescriber maintains his/her clinical autonomy and may opt not to follow the CAP’s treatment 
recommendations. No treatments or medications are withheld at any time.  Secondary outcomes 
include: the percentage of clinicians that follow CAP advice; the percentage of patients with a change to 
psychotropic medication treatment plan following CAP review; and model stage of departure from the 
algorithm (first line, second line, third line treatments prior to antipsychotic). 
Behavioral health navigation: Intervention arm providers may decline behavioral health navigation for 
their patient by responding to a staff message from the navigator. Families/teens contacted by the 
navigator may decline behavioral health navigation at any time.  Secondary outcomes include the 
percentage of patients that agree to behavioral health navigation, and the percent of patient that attend 
2 or more face-to-face therapy sessions. 
Expedited access to bridging therapy: Intervention arm providers may decline bridging therapy for the 
patient/family by responding to a staff message from the navigator. Families/teens offered study 
bridging therapy may decline it at any time.  The percent of patients that attend two of more bridging 
sessions with a study therapist is a secondary outcome of the study. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
18 
 
6 
CONTROL ARM 
The control arm advance communication and medication alert inform the prescriber of Choosing 
Wisely® recommendations regarding antipsychotic medications in children and adolescents. These 
guidelines were developed by the American Psychiatric Association (APA). Available at: 
http://www.choosingwisely.org/clinician-lists/american-psychiatric-association-antipsychotics-in-
children-or-adolescents/ 
The firing of the control arm medication alert will add the patient to the study enrollment log as a 
comparison case. Study staff will confirm eligibility via blinded chart review for all patients logged.  
The follow up period is 6 months. 
Should a control arm provider find SUAY orders in Epic and attempt to order a study service (e.g., 
navigation, expedited access to bridging therapy), an alert will fire to inform him/her that the patient is 
not eligible for the service. 
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
19 
 
7 
INTERVENTION ARM 
7.1 
Intervention Arm Medication Alert Intervention Description 
Randomized providers at each site will be informed in advance of the quality improvement study being 
conducted. The communication will inform intervention arm providers of the nature of the 
interventional components (e.g., medication alert, case review by a child and adolescent psychiatrist, 
behavioral health navigation, bridging therapy). 
The medication alert is triggered by entering an antipsychotic in Epic for a target patient. See Appendix 
B. The patient is added to a study enrollment log at the time of the encounter. The intervention arm 
medication alert links providers to a document containing Choosing Wisely® guidelines, the national 
expert consensus panel clinical guidelines developed for SUAY, and contextual study information.  
Within the alert, the prescriber has the option to: 
• 
keep/remove the medication entered (default = remove) 
• 
keep/remove an order for a provider-to-provider consult with the CAP (default = keep) 
• 
keep/remove an order for BH navigation for the patient (default = keep) 
• 
keep/remove an order for expedited access to bridging therapy for the patient (default = keep) 
The study CAP is automatically notified that a review is needed each time an intervention arm provider 
actively orders a CAP review in the alert and after a patient is verified eligible when a consult is not 
actively ordered.  
7.2 
Eligibility Confirmation 
Study staff (PI or delegate) blinded to arm assignment will conduct a focused medical record review 
within 3 business days of the case being logged to the study when the alert fired. The staff person will 
utilize a checklist of study inclusion and exclusion criteria for this review.  
Patients found not to meet criteria will be flagged as ineligible and the reason(s) noted in the study log. 
Patients found to meet all study criteria will be flagged as eligible. Upon being thus flagged, eligible 
intervention subjects will be ‘released’ to the CAP and navigator. 
7.3 
CAP Case Review and Consultation Intervention Description 
Expert clinical guidance for safer prescribing of antipsychotics to youth was developed in Phase 1 of the 
contract. See Appendix C. This will guide the CAP as s/he reviews the clinical scenario around each 
antipsychotic order (e.g., treatment history, current treatments, diagnoses) to determine the next best 
step in the treatment plan. If information is missing or the prescription is not in line with expert clinical 
guidance, the CAP may attempt to contact the prescriber to discuss the appropriateness of proceeding 
to an antipsychotic for the case and possible revisions to the treatment plan. The CAP will provide the 
prescriber with a written recommendations summary for every review completed. A standardized 
template will be used. The CAP’s recommendations may be noted in the medical record.  

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
20 
 
The prescriber is free to accept or disregard the CAP’s recommendations. There are no negative 
consequences to providers if CAP treatment recommendations are not implemented. That is, the 
prescriber’s clinical autonomy is in no way constrained or diminished.  
7.3.1 
Administration of CAP Case Review and Consultation 
Case review of intervention arm patients will be conducted by a study CAP within 3 business days of 
receiving notification of the case at each site. If a provider-to-provider discussion is deemed necessary 
by the CAP and the prescriber did not order a consult directly, a virtual consult between the CAP and 
prescriber will be arranged within 2 weeks of the visit. 
The CAP’s review outcomes will be sent to the prescriber and may be noted in the patient’s chart. The 
exact procedure will vary by site. The CAPs may also provide guidance to the navigator (e.g., patient is 
not a good candidate for telemental health, etc.) 
Filling of the script is not delayed by the study while the review occurs. However, existing Washington 
State law requires antipsychotic scripts for children on Medicaid to be held at the pharmacy until pre-
authorization is granted by the State. This process is outside the control of the project at KPWA.  
CAP recommendations: A structured template will be used by all study CAPs to convey review outcomes 
and recommendations to prescribers. Feedback will be shared with the prescriber for every case 
reviewed. This includes situations where the CAP agrees with the antipsychotic order and/or if the CAP 
discussed the case with prescriber directly (e.g., by phone, email, or in person).  
7.3.2 
Procedures for Training Study CAPs and Fidelity Monitoring 
Dr. Robert Hilt of Seattle Children’s Hospital created and directs both the Second Opinion and 
Partnership Access Line programs based in Washington State. He will train study CAPs at other clinical 
sites, as needed, on the study protocol, SUAY guidelines, and CAP consultation best practices.  
Site CAPs will meet by regular conference call to discuss fidelity to the SUAY clinical guidelines by 
anonymously discussing case examples.  Recommendations will be periodically quality checked in Epic to 
ensure CAPs are adhering to and completing the structured recommendations template for the study. 
7.4 
Behavioral Health Navigation Description 
Eligibility will be confirmed via medical record review within 3 business days of the patient being added 
to the study enrollment log at each site.  
Initial provider outreach: The behavioral health navigator will contact intervention arm prescribers via 
staff message within 2 business days of the CAP providing his/her summary of review outcomes. When a 
provider actively ordered a study service for their patient, the navigator will send the provider a short 
message. When a study service was not explicitly ordered for an eligible patient, the navigator will 
introduce him/herself to the provider in a staff message, briefly describe the extra support the BH 
navigation can offer the patient, and inform the provider that s/he may respond by staff message to 
decline the extra support on behalf of the patient, if desired. If a provider contacts a navigator directly 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
21 
 
to request a study service, the navigator will request that the provider order the requested service in 
Epic. Ordering the service in Epic will log the patient for study eligibility review. The navigator will 
contact the provider if the patient is subsequently deemed ineligible. 
Existing guidelines all recommend that psychosocial interventions be tried before prescribing 
antipsychotic medications. Unless a provider actively declines the extra support, the navigator will 
contact the teen/family to offer behavioral health navigation.  
Initial patient outreach: Waivers of consent/assent are obtained from the reviewing IRBs. BH navigators 
at each site will administer the equivalent of modified oral consent/assent by phone, as required by the 
local IRB, when calling to offer up to 6 months of navigation services. BH navigation is offered through 
the study at no additional cost.  
Follow up patient outreach: If the teen/family agree to BH navigation, the navigator will initiate contact 
at least monthly during the 6 month follow up period to inquire about the child’s status and re-evaluate 
barriers to access. The navigator will use motivational interviewing techniques (e.g., reflective listening, 
asking open ended questions) with the youth and/or parents to encourage them to engage in 
psychosocial treatments, attempt to motivate them to continue or restart therapy (as needed), and 
identify and help problem-solve ways to minimize barriers to starting or continuing therapy. When 
barriers exist, s/he will offer assistance in identifying behavioral health clinicians that may be a good fit 
for the family/teen, and further assess appropriateness of short-term bridging therapy.  
The navigator will remind teens/families, as appropriate, that laboratory tests are due to be completed 
3 months after starting an antipsychotic.  
The navigator will note any medication side effect information incidentally reported by the teen/family 
in the phone note. If a reported side effect is burdensome, the navigator will encourage the patient or 
parent/guardian to contact their provider about their concern or side effect.  
The navigator will follow the study Safe Practice Protocol if suicidal ideation is incidentally revealed by a 
subject during a BH navigation outreach call.  
Ad hoc patient outreach: The navigator will reach out to assess barriers and attempt to motivate the 
teen/family to continue therapy if a BH appointment is missed and not rescheduled within 2 weeks.  
Final patient outreach call: The navigator will conclude services with a review of the agreed upon plan 
for next steps and encouraging continued contact with the clinical care delivery system. 
Follow up provider outreach: Following each outreach call, the navigator will secure message the 
prescriber with relevant updates, as needed.  
 
7.4.1 
Administration of BH Navigation Intervention 
An Epic reporting workbench will be used to support BH navigation at each site. Navigators will be 
prompted to follow up with subjects at the appropriate time points, log outreach attempts and 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
22 
 
outcomes, note subjects’ acceptance or refusal of navigation and (if offered) short term bridging 
therapy, and track subject adherence to the treatment algorithm.  
7.4.2 
Procedures for Training of BH Navigators on Intervention 
Deborah King, LICSW of Kaiser Permanente Washington will serve as the lead behavioral health 
navigator and train SUAY behavioral health navigators at other clinical sites.  
Navigators will meet regularly by phone to anonymously discuss any difficult cases and ensure fidelity to 
the protocol between the sites.  
7.5 
Short-Term Bridging Therapy Description 
Either the study CAP or the prescribing clinician may decline bridging therapy for the subject based on 
the clinical scenario. The BH navigator will use a decision tree to further assess appropriateness of short-
term bridging therapy. Any reason that bridging is not offered will be noted in the navigator’s reporting 
workbench (e.g., declined by prescriber, CAP advised against, subject already engaged in therapy, etc.).  
The BH navigator will administer modified oral consent/assent for bridging services, as required by the 
local IRB, schedule the initial session, and note if the teen/family requires a loaner device (e.g., tablet) if 
bridging visits are to be delivered via telemental health (TMH) video visits. 
When TMH is to be delivered, a TMH privacy and safety plan with the subject prior to the first TMH 
session. The offer of TMH will be suspended if a privacy or safety concern is identified when the form is 
completed until a clinical decision can be made by the therapist. The plan identifies resources within and 
outside the home to ensure patient privacy during TMH sessions, and plan for any potential crises, 
safety concerns, or inability to contact the teen/family. 
All bridging visits will be entered in Epic as patient visits. There will be no copays or other charges billed 
for these study-related visits. 
7.5.1 
Administration of Bridging Therapy 
Each health system will identify 1 or more licensed and credentialed therapists to provide bridging 
therapy. Study sites will offer clinic-to-home telemental health (TMH) with a study therapist via HIPAA-
compliant video visit technology when appropriate for and agreeable to the participant. Phone-based or 
in-person visits may be offered by sites if technical challenges or patient/family preference prohibit 
video-teleconferencing visits. If a televideo connection cannot be established, the therapist may reach 
out by phone to start (or complete) a TMH session without video. 
Appointments and set up for TMH: Study staff will provide subjects with instructions for joining 
telemental health sessions, as appropriate. Appointment reminders will be sent. Appointment 
reminders for TMH sessions will include a link for the patient to join the next TMH session.  
If a suitable personal device is not available for video-visits, study staff will provide an inexpensive, 
secure loaner device to the subject during their first appointment.  

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
23 
 
Content: The therapeutic content of bridging sessions is solely determined by the therapists delivering 
the care as informed by the consensus panel guidelines developed in Phase 1 and established, 
empirically supported treatments. The study is investigating methods to improve access and use of 
psychotherapy; not evaluating any specific psychotherapeutic interventions. 
Duration and transition: Bridging will conclude after 9 sessions or when the patient can begin long-term, 
in-person therapy, whichever comes first. The extra support provided by the BH navigators includes 
helping the patient/family transition from bridging to ongoing psychosocial therapy when appropriate.  
7.5.2 
Procedures for Training of Therapists in Telemental Health 
Kathleen Myers, MD, MPH of Seattle Children’s Hospital will train study therapists in telemental health 
delivery best practices. She will meet by phone regularly with study therapists to discuss technical or 
other challenges to the telemental health. Protected health information (PHI) will not be shared.  
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
24 
 
8 
STATISTICAL CONSIDERATIONS 
8.1 
Sample Size Considerations 
We anticipate enrolling as many as 800 eligible patients in the trial (up to 400 per arm), with up to 360 
eligible patients in each arm remaining enrolled at 6 months of follow-up (i.e., 10% lost to follow-up).  
 
8.2 
Primary Analysis Plan 
Data will be collected from project tracking systems, the Epic EMR, and other health system data 
systems for analysis. Sites will securely transfer limited datasets to the data coordinating center at 
KPWHRI at defined intervals.  
Data validation and quality assurance checks will be undertaken to ensure data collection during the 
study is sufficiently reliable.  
We will use descriptive statistics to summarize adherence to the study algorithm. Specifically, we will 
summarize the patient characteristics (age, gender, race and ethnicity, diagnoses) of those individuals 
enrolled in the study and we will summarize clinician adherence to the algorithm protocol by provider 
specialty. Modal stage at deviation from the protocol will be evaluated among clinicians deviating from 
the protocol; additionally, modal algorithm stage of drop out among patients in the intervention arm 
who receive antipsychotic treatment during the 6 month window will be summarized.  Our primary 
outcomes are percent of patients taking antipsychotic medication at 6 months post enrollment and total 
person-months of antipsychotic use by youth, based on medication order data. As secondary outcomes, 
we will also compare the percent of patients taking antipsychotic medication at 6 months post 
enrollment and total person-months of antipsychotic use by youth, based on prescription fill data.  
We will also measure: the percentage of patients with a change to psychotropic medication treatment 
plan following second opinion review; percent of patients that agree to BH navigation; percent of 
patients that attend 2 or more bridging therapy sessions; and percent of patients that attend 2 or more 
face-to-face therapy sessions after transitioning from study bridging therapy to standard care.  
In order to capture any negative unintended consequences, we will compare the number of emergency 
department visits, successful and unsuccessful suicide attempts, deaths, and hospital admissions 
between study arms over the 6 months following the intervention.  
8.2.1 
Statistical Analysis Plan 
All primary analyses will follow an intent-to-treat approach, including all individuals assigned to an 
intervention arm regardless of the amount of intervention received. Non-inferiority evaluations will also 
be evaluated in a supplementary “as treated” analysis, comparing outcomes according to treatments 
actually received by patients during their observed period of follow-up. This supplemental comparison is 
useful in a non-inferiority assessment due to the fact that the alternative hypothesis of “no effect” is 
more likely in the presence of protocol violations and therefore the intent to treat approach will be anti-

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
25 
 
conservative if many patients in the intervention arm, for example, revert to control. Also, non-
inferiority of Emergency Department visit rate in the intervention arm compared to usual care control 
will only be assessed on final analysis if superiority is shown for the primary endpoint of no antipsychotic 
use at 6 months.  
We will use Poisson regression to estimate (1) the percent of individuals using antipsychotics at 6 
months post enrollment and (2) the total number of person-months of antipsychotic use by youth in the 
six month follow-up period, per provider medication orders, in each of the study arms. We will use 
generalized estimating equations(43) to estimate the parameters in the Poisson regression model. We 
will use the robust sandwich estimator to ensure correct standard errors for the intervention effect due 
to possible violation of the Poisson variance assumption(44); additionally, this will allow us to account 
for any clustering within clinics(43) and providers. Poisson regression allows comparison of relative 
incident rates among the intervention arms when outcomes are not rare(45). Additionally, Poisson 
regression will allow us to control for pre-specified baseline covariates (age, gender, and race) as well as 
site differences in any antipsychotic use and allow for straightforward estimation of site-specific 
intervention effects in sensitivity analyses. For a secondary analysis, we will repeat this approach 
defining antipsychotic use with prescription fill data rather than orders.  
Data for our primary outcome of interest, any antipsychotic use, will be collected from electronic health 
records on provider orders. Electronic health record data on prescription fills will be used to assess the 
secondary outcome (antipsychotic medication fills). This means that traditional study drop-out due to 
premature participant withdrawal from the study will not lead to missing data in primary analyses. 
However, there will be the potential for differential follow-up rates among individuals due to youth 
disenrolling from their or their parent/guardian’s health plan. Death is also a possibility, but unlikely. We 
will compare those youth for whom we are only able to collect outcome information for a portion of the 
6 month follow-up period to those participants for whom we have complete 6 month follow-up 
information on their demographics, treatment group, and other salient baseline variables. We will 
account for differential follow-up times due to disenrollment in the Poisson regression model using an 
offset(46). Because our power calculations are based on excluding all individuals lost-to-follow-up due 
disenrollment, these estimates are conservative. 
The secondary outcome of emergency department visits will be estimated by Poisson regression in each 
of the arms, again using generalized estimating equations for the reasons listed above for the primary 
outcome. The intervention will be compared for non-inferiority to control using a margin of M=1.5. This 
margin of 1.5 for the rate ratio represents a threshold for the relative increase of 50%, beyond which we 
would consider unacceptable. Sikirica (2014) observed an annual rate of 0.39 per year among 
adolescents with attention deficit/hyperactivity disorder who were taking antipsychotics(47). Using an 
estimated rate of emergency department visits of 0.0325 per month on control(47), the non-inferiority 
margin of 1.5 for the incidence rate ratio would represent a corresponding maximum acceptable rate of 
0.0488 per month in the intervention arm. If the rate of emergency department use is greater than 
expected, say 0.05, then the non-inferiority margin of 1.5 would correspond to a maximum rate of 0.075 
in the intervention arm. The estimated treatment effect (emergency department rate on algorithm 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
26 
 
divided by the rate on control, estimated by Poisson regression) and 95% two-sided confidence interval 
(CI) will be compared to the non-inferiority margin (overall: M = 1.5) and non-inferiority of the algorithm 
approach will be declared if the entire coverage of the 95% CI is less than 1.5. 
In safety review reports, the ratio between arms of emergency department visit rates will be 
accompanied by a confidence interval and by the non-inferiority margin, 1.5, for reference.  Confidence 
intervals will be defined by estimating the rate ratio in 5,000 bootstrap samples. Resampling will be 
done on the cluster (provider) level and stratified by the presence of any event within each cluster. 
Quantile-based confidence intervals will be identified using the appropriate percentile corresponding to 
the Pocock alpha-spending function for interim analyses. Excess harm is observed if the entire coverage 
of the confidence interval for the rate ratio comparing the greater to the lesser rate of emergency 
department and urgent care visits is greater than 1.5. (While the final non-inferiority analysis for this 
safety outcome will use the robust sandwich estimator to ensure correct standard errors, the properties 
of this estimator are based on asymptotic behavior and small sample size performance may be poor.) 
We anticipate initial follow-up of 800 patients, randomized to the two arms, with 720 patients 
remaining enrolled in the study at 6 months of follow-up (i.e., 10% lost to follow-up). Assuming a P<0.05 
alpha level, and based on an anticipated 95% of patients in the control arm using antipsychotics at 6 
months, this sample will ensure over 95% power to detect a 6% reduction in the percent of participants 
with any order for antipsychotic medications at 6 months. Furthermore, we expect to have 80% power 
to detect a 13% reduction in subgroups with exposures occurring with 25% prevalence, and 80% power 
to detect a 17% reduction in subgroups with exposures occurring with 18% prevalence (e.g., 
racial/ethnic minorities). A minimal clinically important difference of 10% was pre-specified to allow for 
evaluation of clinical significance as well as statistical significance as well as statistical significance. We 
have over 90% power to detect a 10% reduction in antipsychotic use at 6 months for patients in the 
intervention arm given a rate of 85%, 90%, or 95% use in the control arm.  
As these power calculations are based on the assumed percent of participants on any antipsychotic in 
the control arm and assumed pattern of intervention effects, we provide examples of detectable 
intervention effects corresponding to alternative scenarios in the table that follows, including varying 
rates of subgroup prevalence to evaluate power regarding intervention effectiveness in less prevalent 
minority groups. In summary, at a range of percentages of participants with any antipsychotic use in the 
control arm, we will have excellent power to detect a variety of clinically meaningful intervention effects 
within the expected range. 
Using an estimated rate of emergency department visits of 0.0325 per month on control(47) and with a 
one-sided 0.025 alpha level with equal allocation between groups, we estimate 71% power(48) with 400 
patients per arm and 67% power with 360 patients per arm (allowing for loss to follow-up) respectively, 
to detect equality of emergency department visit rates (i.e., a true rate ratio of 1) in a non-inferiority 
evaluation with a margin of 1.5. With a greater monthly emergency department rate of 0.05, we would 
have 88% and 84% power with 400 and 360 patients per arm respectively under otherwise equivalent 
conditions. 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
27 
 
Power under varying percentages of participants in each arm taking antipsychotic medication at 6 
months (based on medication orders) and under various subgroup prevalence assumptions. 
 
Subgroup 
prevalence 
n per 
arm* 
Intervention 
Control 
Power 
100% 
360 
89% 
95% 
80% 
100% 
360 
85% 
95% 
>99% 
100% 
360 
80% 
90% 
97% 
100% 
360 
75% 
85% 
92% 
25% 
90 
82% 
95% 
80% 
18% 
65 
78% 
95% 
80% 
10% 
36 
75% 
95% 
67% 
5% 
18 
65% 
95% 
62% 
 
 
8.3 
Handling of Missing Data 
All outcomes for this trial will be collected from electronic health records, which means traditional study 
drop out will not result in missing information. It is possible for youth to disenroll from a health care 
plan in the 6 months following initial randomization. Patients who disenroll from the health system are 
lost to follow up data collection.  
As described above, we will assess the association of baseline characteristics and the probability of 
remaining enrolled in the same health care plan for the full 6 month follow-up. If there are large 
differences between the rate of full follow-up by baseline characteristics we will perform sensitivity 
analyses including these covariates in Poisson regression models to evaluate if missing this missing 
information has led to biased effect estimates. Our primary analyses will use an offset term(46) in the 
Poisson regression models in order to include all individuals regardless of the length of follow-up. We 
have performed conservative power calculations assuming that all individuals without a full 6 months of 
follow-up will be excluded from analyses. If disenrollment rate is high than anticipate or there is missing 
information in important baseline characteristics we will use multiple imputation(49) or weighted(50) 
estimates to ensure that our results are not biased due to missing data. 
One study site (NCH) does not have complete prescription fill data because they serve patients in their 
clinics who are not enrolled in the insurance product for their health plan. Complete capture of 
prescription fill data does not differ with study arm assignment, thus, we do not expect estimates of the 
interventions effect to be biased.  However, missing prescription fill data could lead to underestimates 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
28 
 
of the percentage of youth taking antipsychotics at 6 months and/or the percentage of youth who fill 
prescriptions for lower risk, first line medications. We will conduct quality assurance checks comparing 
the number of medication orders and prescription fills by individual across all sites to determine degree 
to which medication order and prescription fill data differ. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
29 
 
9 
PRAGMATIC TRIAL OVERSIGHT 
9.1 
Data Safety Monitoring  
The NIMH Data and Safety Monitoring Board (DSMB) will provide ongoing oversight for the Phase 3 
pragmatic trial.  
The Board consists of voting members chosen by NIMH for their relevant expertise.  The DMSB operates 
under the rules of an NIMH-approved charter.   
The DSMB will advise NIMH of its findings and reports will be shared with site IRBs, as required. We will 
be responsive to additional specific requests for data reports that may be requested by the DSMB.   
Confidentiality will be maintained during all phases of the DSMB review and deliberations.   
9.1.1 
Schedule and Content of DSMB Reports 
The schedule for delivery of the DSMB reports will be determined by the DSMB; three (3) reports per 
year are expected. DSMB reports will be submitted via email to the COR and DSMB liaison. 
Elements that the DSMB needs for the interim and final reports will be clearly defined in advance on a 
template developed in collaboration with the COR and approved by the DSMB. The report will 
summarize overall trial progress as well as safety data for each site.  
9.2 
Premature Termination or Suspension of Trial 
This trial may be prematurely terminated if, in the opinion of the PI, NIMH, DSMB or IRB there is 
sufficient reasonable cause. Written notification, documenting the reason for study termination, will be 
provided to the PI and/or NIMH by the terminating party. Circumstances that may warrant termination 
include, but are not limited to: 
• 
Determination of unexpected, significant, or unacceptable risk to subjects.  
• 
Insufficient adherence to protocol requirements.  
• 
Failure to obtain sufficient follow-up data to evaluate the effectiveness of the intervention.   
If the project is prematurely terminated or suspended, NIMH will promptly inform the 
investigators/institutions and the regulatory authorities of the termination or suspension and the 
reason(s) for the termination or suspension. The IRB will also be informed promptly and provided the 
reason(s) for the termination or suspension by NIMH or by the investigator/institution, as specified by 
the applicable regulatory requirement(s). 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
30 
 
10 ASSESSMENT OF SAFETY 
10.1 
Specification of Safety Parameters 
It is not feasible to report incidental adverse events (AEs) for this pragmatic trial. Study staff do not have 
contact with control arm patients. Only intervention arm patients have contact with study staff, 
resulting in an inability to compare rates of incidental AEs between arms.   
We will compare the number and rate of emergency room and urgent care visits, inpatient 
hospitalizations (psychiatric and non-psychiatric), suicide attempt (fatal and non-fatal), and deaths from 
all causes to monitor for potential harm to subjects. Comparisons of safety signals between arms will be 
made and reported to the DSMB 3 times per year.  The PI and site investigators will be blinded to the 
results of these assessments. 
10.1.1 
Safe Practice Protocols and Tracking 
BH navigators will have contact with intervention arm patients/families and will follow safe practice 
protocol if a patient or parent/guardian voluntarily discloses thoughts of self-harm. Study navigators do 
not assess suicide risk directly. See Appendix D of this protocol.  
Safe practice protocol use, actions taken, and resolution for each event will be tracked and summarized 
in reports for DSMB.  The events will not be considered adverse events because there is no comparison 
data from the control arm.  
The study therapists are members of the local health system workforce and therefore follow their health 
system’s clinical protocols if thoughts of self-harm are reported by a patient.  
In addition to safe practice protocols, both navigators and therapists are mandatory reporters. Standard 
procedures for reporting to Child Protective Services (CPS) will be followed should a subject voluntarily 
disclose child neglect or abuse. CPS reporting will not be tracked for study purposes. 
10.1.2 
Adverse Events (AE) / Serious Adverse Events (SAE) and Reporting  
As noted above, reporting of incidental adverse events is not feasible for this pragmatic trial.  
The number and rate of events will be compared between the two study arms in DSMB reports. 
• 
Emergency room and urgent care visits (SAE) 
• 
Inpatient hospitalizations (psychiatric and non-psychiatric) (SAE) 
• 
Suicide attempt (fatal and non-fatal) (SAE) 
• 
Death from all causes (SAE) 
 
If study staff have contact with a family and discover the patient died, the death will be reported to 
NIMH within 72 hours of discovery as a serious adverse event. Other SAEs listed above about which 
study staff gain incidental knowledge will be reported in the next DSMB report. 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
31 
 
10.1.3 
Unanticipated Problems and Reporting  
The Office for Human Research Protections (OHRP) considers unanticipated problems involving risks to 
subjects or others to include, in general, any incident, experience, or outcome that meets all three of 
the following criteria: 
• 
Unexpected in terms of nature, severity, or frequency given (a) the research procedures that are 
described in the protocol-related documents, such as the IRB-approved research protocol and 
informed consent document; and (b) the characteristics of the subject population being studied; 
• 
Related or possibly related to participation in the research (“possibly related” means there is a 
reasonable possibility that the incident, experience, or outcome may have been caused by the 
procedures involved in the research); and 
• 
Suggests that the research places subjects or others at a greater risk of harm (including physical, 
psychological, economic, or social harm) than was previously known or recognized. 
Incidents or events that meet the OHRP criteria for unanticipated problems will be reported to the IRB 
and the NIMH DSMB.  Reporting of Unanticipated Problems to NIMH Program and the NIMH DSMB will 
be accomplished by submission of an Unanticipated Problem Report via fax or email to the NIMH.  
Per HHS regulations 46.103(b)(5), the “appropriate time frame for satisfying the requirement for prompt 
reporting will vary depending on the specific nature of the unanticipated problem, the nature of the 
research associated with the problem, and the entity to which reports are to be submitted”.  We will 
apply the following recommended OHRP guidelines to satisfy prompt reporting requirements: 
• 
Unanticipated problems that are serious adverse events should be reported to the IRB within 1 
week of the investigator becoming aware of the event. 
• 
Any other unanticipated problem should be reported to the IRB within 2 weeks of the 
investigator becoming aware of the problem. 
• 
All unanticipated problems should be reported to appropriate institutional officials (as required 
by an institution’s written reporting procedures), the supporting agency head (or designee), and 
OHRP within one month of the IRB’s receipt of the report of the problem from the investigator. 
Once submitted, NIMH will send a confirmation email to the investigator within 2 business days. 
Unanticipated problems will be recorded in the data collection system throughout the trial.  
10.2 
Halting Rules 
Study halting may occur as required by the IRB, NIMH, or the NIMH DSMB.  The PI may temporarily 
suspend enrollment pending review by these authorities if warranted based on significantly more safety 
events occurring in the intervention arm per safety monitoring results.   
The study would not be halted early if the intervention was found to be protective (i.e., significantly 
lower rates of safety events in the intervention arm).  

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
32 
 
11  CLINICAL SITE MONITORING 
Clinical site monitoring is not planned or budgeted for this project; however, the NIMH reserves the 
right to conduct independent audits or clinical monitoring as necessary. 
See the next section for information about quality control and quality assurance, including quality 
management.  

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
33 
 
12 QUALITY CONTROL AND QUALITY ASSURANCE 
12.1 
Study Procedures 
Standard operating procedures for the study will be detailed in the Manual of Procedures at each site. 
An overview of procedures is provided in this protocol.  
Local staff will be trained on study procedures relevant to their role prior to fielding. Staff training logs 
will be maintained by the project manager at each study site. 
12.2 
Quality Management  
A Quality Management Plan (QMP) will be created prior to fielding in order to establish the quality 
management guidelines for tasks related to the study.  At a minimum the quality management plan will 
be reviewed bi-annually by the KP Washington PI and PM.   
This QMP will be a living document that will be updated throughout the life of the project, as necessary, 
in order to document changes in scope or process.  The QMP will include a detailed description of 
quality management activities applicable to the study at each site, including process documents, data 
collection, study record review, technical issues and resolution, essential documents, and staff training 
and qualifications. The QMP will include tools and checklists to be used in the quality management 
process, and describe when and how internal assessments will be summarized and reported to staff.  
12.3 
Protocol Deviations 
Site staff will immediately report protocol deviations to the local PM, who will inform the site 
Investigator or PI.  Any deviations will be considered noncompliance with this protocol or Good Clinical 
Practice.  Protocol deviations will include privacy breaches, and IT-related problems that lead to errors in 
protocol adherence.  
Protocol deviations that may impact patient safety or study data integrity will be immediately reported 
to the KP Washington PI and PM. The DSMB will be informed within 5 business days of KP Washington 
becoming aware of a protocol deviation impacting patient safety or study data integrity. 
Sites will report other protocol deviations as outlined in the QMP, Site IRBs will be notified of deviations 
per local reporting requirements. The NIMH Contracting Officer’s Representative and NIMH DSMB will 
review protocol deviations in the tri-annual data reports. 
Events not considered protocol deviations: Due to the pragmatic and encouragement design aspects of 
the study, the failure of a provider or patient to adhere to the protocol (e.g., bypassing medication alert, 
not responding to request for active CAP consult, not following CAP recommendations, refusal or drop 
out from BH navigation or bridging, not closing a chart note in time for reviews to be conducted), are 
not considered protocol deviations. These uncontrolled and expected events are of scientific interest 
due to the pragmatic nature of the study.  
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
34 
 
13 ETHICS/PROTECTION OF HUMAN SUBJECTS 
13.1 
Ethical Standard 
The investigator will ensure that this research study is conducted in full conformity with the principles 
set forth in The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects 
of Research, as drafted by the US National Commission for the Protection of Human Subjects of 
Biomedical and Behavioral Research (April 18, 1979) and codified in 45 CFR Part 46 and/or the ICH E6.  
13.2 
Potential Risks and Benefits 
None of the individual interventional components of this study are experimental in nature. All 
components have been tested and demonstrated effective elsewhere. No medications or other 
treatments are withheld or delayed. Prescribers are not mandated to accept the CAP’s 
recommendations and remain free to exercise their own clinical judgment. All the participating health 
systems already offer provider-to-provider consultation, care navigation, and psychosocial therapy.  
13.2.1 
Potential Risks 
Control arm subjects receive no benefit from the study.  Potential risks are no different than usual care 
or usual employment. Breach of confidentiality is a possible risk of this research. 
For intervention arm patients, potential risks are no different than those experienced as part of other 
care navigation or psychosocial therapy offered as part of usual care. Some families/teens may not be 
comfortable participating in video-based bridging therapy that may be offered by the study. This may be 
due in some cases to concerns about confidentiality. Information will be provided to ensure 
families/teens understand these risks. They are free to refuse the video visits offered or to request 
phone-only visits (no video-conferencing).  
For intervention arm providers, potential risks are no different than usual employment insofar as 
employers are free to implement quality improvement programs and implement medication alerts.  
13.2.2 
Potential Benefits 
Some providers may find study CAP consultations, BH navigation, and bridging therapy to be valuable 
services. Pediatricians and primary care providers may especially welcome the opportunity to solicit 
input from a study CAP since they infrequently prescribe antipsychotics.  
Youth and families seeing intervention arm providers may have improved access to behavioral health 
services due to BH navigation and may be offered up to 9 bridging sessions. Both bridging therapy 
sessions and navigation services are offered at no additional cost to families.  
13.3 
Institutional Review Board 
The protocol and project materials for providers and patients will be submitted to the IRB at each clinical 
site for local review and approval.  IRB approvals and local waivers of consent and assent must be 
obtained before the research project is initiated at each site.  Amendments require review and approval 
by the IRB before the changes are implemented locally.   

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
35 
 
13.4 
Informed Consent Process 
Waivers of informed consent and assent will be obtained for this study at each clinical site, as granted by 
each local IRB. Written informed consent/assent will not be collected. Providers will be notified of the 
study via a communication sent prior to fielding. Families/teens are free to refuse navigation or bridging 
services at any time. 
The “Common Rule” for protection of research participants (45 CFR 46.116d) lists specific requirements 
for waiver of the usual requirement of individual informed consent to participate in research.  Those 
criteria include: The research involves no more than minimal risk to the subjects – According to 45 CFR 
46.102:  “Minimal risk means that the probability and magnitude of harm or discomfort anticipated in 
the research are not greater in and of themselves than those ordinarily encountered in daily life or 
during the performance of routine physical or psychological examinations or tests.”  
Rationale for waiver of consent: 
• 
We do not believe that any of the components of the intervention expose providers or patients 
to greater than minimal risk. No treatment or intervention will be restricted or withheld; 
providers maintain autonomy with respect to clinical decision making;   
• 
No components of the intervention are experimental in nature. Consultation and case review, 
navigation, psychosocial therapy and medication alerts are all standard health care provisions; 
• 
Engagement in all study activities will be voluntary (e.g., medication alert may be overridden, 
CAP recommendations may be disregarded, navigation and bridging therapy may be declined);  
• 
For control arm providers and patients, the experience is identical to what would have been 
delivered had the study not occurred;  
• 
The research is not practicable without a waiver.  A trial limited to those who would actively 
consent to such a study would not be scientifically valid;   
• 
When appropriate, providers and patients are given pertinent information via staff 
communications (providers) and the BH navigator (patients/families); 
• 
Patients and families are already in the study and randomized prior to being contacted by the 
navigator. Systematically excluding subjects after randomization would produce invalid research 
findings. It would be unethical to expose subjects to even minimal risk when such bias is known 
in advance. 
13.5 
Exclusion of Women, Minorities, and Children (Special Populations) 
Individuals of any gender or racial/ethnic group may participate. Children ≥ 3 and < 18 years of age 
constitute the population of target patient-subjects.  Children diagnosed with psychotic disorders 
including mania, autism spectrum disorders or intellectual disabilities will be excluded.  

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
36 
 
13.6 
Subject Confidentiality 
Appropriate steps will be taken to safeguard participant confidentiality.  This includes the following: 
• 
All project data will be maintained on secure, password protected servers at clinical sites.  Only 
project staff needing this information for their work will have access. Members of the study 
team include Epic programmers from the delivery system;  
• 
Limited data sets from subcontracted clinical sites will be transmitted from to the KPWHRI data 
coordinating center using a HIPAA-compliant secure FTP site;  
• 
Participants will be identified using only subject ID in any study analytic database(s) at each site; 
• 
The protocol, documentation, data and all other identifying information will be held in strict 
confidence.  No information about the project will be released without prior approval of NIMH 
per the terms of the contract.  Authorized representatives of the NIMH may inspect records, as 
needed. 
13.7 
Future Use of Identifiable Data 
Identifiable information will be destroyed within 5 years of the end of the contract, consistent with 
contractual, HIPAA and IRB requirements. 
We have no plans to retain identifiable information beyond this period.  If this plan changes, we will 
obtain appropriate IRB approval. 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
37 
 
14 DATA HANDLING AND RECORD KEEPING 
14.1 
Data Management Responsibilities 
Both institutional and application level access granted by the local information security office and 
simultaneous knowledge of both a valid user name and password are required to access data collected 
in electronic records. Any paper records generated with patient identifying information will be kept in 
locked filing cabinets in secured office buildings. 
Data sent to the KPWHRI data coordinating center will be transferred using a secure file transfer 
protocol (FTP) site.  Data coordinating center files will be kept separately from other study files to 
maintain blinding of study staff working on other aspects of the project. 
No participant data shall be overwritten by project staff.  As necessary, variables may be recoded into 
new variables for analyses, so as to preserve the original record. All changes will be documented. 
Final datasets will be saved electronically, clearly labeled and stored in a secure project folder on the 
KPWHRI server accessible only to project staff.   
14.2 
Data Capture Methods 
The majority of data for this pragmatic study is captured electronically in the site EMR, including the 
study enrollment log. Data entered into the navigator’s personal reporting workbench is also in Epic.  
TMH administrative metrics are captured by study therapists and entered into a de-identified electronic 
database. 
CAP treatment recommendations will be abstracted from Epic by study staff and meta-data entered into 
an electronic tracking system. 
All electronic records will be kept in a 21 CFR Part-11 compliant data capture system, which includes 
password protection. 
14.3 
Study Records Retention 
Identifying information and linking files will be destroyed within 5 years of the conclusion of the contract 
unless consent to retain these files is granted by the IRB. Following this time, no other records will be 
destroyed without the consent of NIMH. It is the responsibility of NIMH to inform the investigator when 
these documents no longer need to be retained. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
38 
 
15 PUBLICATION AND NEW SUB-PROJECT POLICIES 
The Steering Committee has developed and adopted a policy on publications, presentations and 
authorship. The most current version of this policy is available on the SUAY web portal site. 
(https://www.hcsrn.org/share/page/site/suay/dashboard).  Topics covered by the policy include: 
• 
Policy scope;  
• 
SUAY publication committee; 
• 
Use and confidentiality of data from participating sites; 
• 
Decisions regarding authorship; 
• 
Coordination of the writing process; 
• 
Tracking of publication progress; 
• 
Acknowledgement of SUAY support and federal funding; 
• 
Public access to products of NIH-funded research; 
• 
Publicity regarding SUAY publications. 
The Steering Committee has developed and adopted a policy on the proposal and approval of new SUAY 
sub-projects. The most current version of this policy is available on the SUAY web portal site. 
(https://www.hcsrn.org/share/page/site/suay/dashboard).  Topics covered by the policy include: 
• 
Policy scope  
• 
Categories (preparatory or pilot data; affiliated) 
• 
Procedures for proposing and approving preparatory or pilot data activities 
• 
Procedures for proposing and supporting affiliated projects 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
39 
 
16 DATA SHARING PLAN 
Phase 3 of the SUAY contract is scheduled to last 3.5 years (December 25, 2017 – June 24, 2021). The 
data sharing plan is incorporated by reference into the contract governing the project. Any changes to 
the data sharing plan must be pre-approved by the NIH Contracting Officer.  
The project is committed to sharing data created by the proposed research, limited only by the need to 
protect the privacy of potentially identifiable health information. Any datasets for public distribution 
(including to the National Database for Clinical Trials Related to Mental Illness) will be de-identified in 
compliance with HIPAA standard for de-identification.  
As noted in the contracted statement of work, in Phase 3 a de-identified database of the pragmatic 
clinical trial will be delivered to the government, along with a statistical report addressing the research 
questions of the study, and a cost-benefit analysis of the intervention. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
40 
 
17 LITERATURE  
1. 
Olfson M, Marcus SC, Weissman MM, Jensen PS. National trends in the use of psychotropic 
medications by children. J Am Acad Child Adolesc Psychiatry. 2002;41(5):514-21. Epub 
2002/05/17. 
2. 
Zito JM, Safer DJ, dosReis S, Gardner JF, Boles M, Lynch F. Trends in the prescribing of 
psychotropic medications to preschoolers. JAMA. 2000;283(8):1025-30. Epub 2000/03/04. 
3. 
Zito JM, Safer DJ, DosReis S, Gardner JF, Magder L, Soeken K, et al. Psychotropic practice 
patterns for youth: a 10-year perspective. Arch Pediatr Adolesc Med. 2003;157(1):17-25. Epub 
2003/01/09. 
4. 
Zito JM, Safer DJ, Sai D, Gardner JF, Thomas D, Coombes P, et al. Psychotropic Medication 
Patterns Among Youth in Foster Care. Pediatrics. 2008;121(1):e157-e63. 
5. 
Crystal S, Olfson M, Huang C, Pincus H, Gerhard T. Broadened Use Of Atypical Antipsychotics: 
Safety, Effectiveness, And Policy Challenges. Health Affairs. 2009;28(5):w770-w81. 
6. 
Olfson M, Crystal S, Huang C, Gerhard T. Trends in antipsychotic drug use by very young, 
privately insured children. J Am Acad Child Adolesc Psychiatry. 2010;49(1):13-23. Epub 
2010/03/11. 
7. 
Penfold RB, Hunkeler EM, Madden JM, Cummings JR, Owen-Smith AA, Rossom RC, et al. Use of 
antipsychotic medications in pediatric populations: what do the data say? Curr Psychiatry Rep. 
2013;In Press. 
8. 
Olfson M, Blanco C, Liu SM, Wang S, Correll CU. National Trends in the Office-Based Treatment 
of Children, Adolescents, and Adults With Antipsychotics. Arch Gen Psychiatry. 2012:1-10. Epub 
2012/08/08. 
9. 
Merikangas KR, He JP, Rapoport J, Vitiello B, Olfson M. Medication use in US youth with mental 
disorders. JAMA Pediatr. 2013;167(2):141-8. Epub 2013/02/14. 
10. 
Andrade SE, Lo JC, Roblin D, Fouayzi H, Connor DF, Penfold RB, et al. Antipsychotic medication 
use among children and risk of diabetes mellitus. Pediatrics. 2011;128(6):1135-41. Epub 
2011/11/23. 
11. 
Correll CU. Safety and tolerability of antipsychotic treatment in young patients with 
schizophrenia. J Clin Psychiatry. 2011;72(8):e26. Epub 2011/09/09. 
12. 
Maayan L, Correll CU. Weight gain and metabolic risks associated with antipsychotic 
medications in children and adolescents. J Child Adolesc Psychopharmacol. 2011;21(6):517-35. 
Epub 2011/12/15. 
13. 
Cohen D, Bonnot O, Bodeau N, Consoli A, Laurent C. Adverse effects of second-generation 
antipsychotics in children and adolescents: a Bayesian meta-analysis. J Clin Psychopharmacol. 
2012;32(3):309-16. Epub 2012/05/01. 
14. 
Seida JC, Schouten JR, Boylan K, Newton AS, Mousavi SS, Beaith A, et al. Antipsychotics for 
children and young adults: a comparative effectiveness review. Pediatrics. 2012;129(3):e771-84. 
Epub 2012/02/22. 
15. 
Loy JH, Merry SN, Hetrick SE, Stasiak K. Atypical antipsychotics for disruptive behaviour 
disorders in children and youths. Cochrane Database Syst Rev. 2012;9:CD008559. Epub 
2012/09/14. 
16. 
McKinney C, Renk K. Atypical antipsychotic medications in the management of disruptive 
behaviors in children: safety guidelines and recommendations. Clin Psychol Rev. 2011;31(3):465-
71. Epub 2010/12/07. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
41 
 
17. 
ABIM Foundation. American Psychiatric Association: Five Things Physicians and Patients Should 
Question. 2014 [March 22, 2015]; Available from: http://www.choosingwisely.org/doctor-
patient-lists/american-psychiatric-association/. 
18. 
Pappadopulos E, Macintyre Ii JC, Crismon ML, Findling RL, Malone RP, Derivan A, et al. 
Treatment recommendations for the use of antipsychotics for aggressive youth (TRAAY). Part II. 
J Am Acad Child Adolesc Psychiatry. 2003;42(2):145-61. Epub 2003/01/25. 
19. 
AACAP. Practice Parameter on the Use of Psychotropic Medication in Children and Adolescents. 
Journal of the American Academy of Child and Adolescent Psychiatry. 2009;48(9):961-73. 
20. 
Pliszka SR, Crismon ML, Hughes CW, Corners CK, Emslie GJ, Jensen PS, et al. The Texas Children's 
Medication Algorithm Project: revision of the algorithm for pharmacotherapy of attention-
deficit/hyperactivity disorder. J Am Acad Child Adolesc Psychiatry. 2006;45(6):642-57. Epub 
2006/05/25. 
21. 
Steiner H, Remsing L, Work Group on Quality I. Practice parameter for the assessment and 
treatment of children and adolescents with oppositional defiant disorder. J Am Acad Child 
Adolesc Psychiatry. 2007;46(1):126-41. Epub 2007/01/02. 
22. 
Pappadopulos E, Rosato NS, Correll CU, Findling RL, Lucas J, Crystal S, et al. Experts' 
recommendations for treating maladaptive aggression in youth. J Child Adolesc 
Psychopharmacol. 2011;21(6):505-15. Epub 2011/12/27. 
23. 
Findling RL, Drury S, Jensen PS, Rapoport J. Practice Parameter for the Use of Atypical 
Antipsychotic Medications in Children and Adolescents. 2011:27. 
24. 
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and 
efficiency of guideline dissemination and implementation strategies. Health technology 
assessment (Winchester, England). 2004;8(6):iii-iv, 1-72. Epub 2004/02/13. 
25. 
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, 
limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527-30. Epub 1999/02/19. 
26. 
Eccles MP, Grimshaw JM, Shekelle P, Schunemann HJ, Woolf S. Developing clinical practice 
guidelines: target audiences, identifying topics for guidelines, guideline group composition and 
functioning and conflicts of interest. Implement Sci. 2012;7:60. Epub 2012/07/06. 
27. 
Shekelle P, Woolf S, Grimshaw JM, Schunemann HJ, Eccles MP. Developing clinical practice 
guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the 
emerging issues of enhancing guideline implementability and accounting for comorbid 
conditions in guideline development. Implement Sci. 2012;7:62. Epub 2012/07/06. 
28. 
van Engen-Verheul MM, de Keizer NF, van der Veer SN, Kemps H, Reimer W, Jaspers M, et al. 
Evaluating the effect of a web-based quality improvement system with feedback and outreach 
visits on guideline concordance in the field of cardiac rehabilitation: rationale and study 
protocol. Implement Sci. 2014;9(1):780. Epub 2015/01/01. 
29. 
Woolf S, Schunemann HJ, Eccles MP, Grimshaw JM, Shekelle P. Developing clinical practice 
guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and 
presentation and deriving recommendations. Implement Sci. 2012;7:61. Epub 2012/07/06. 
30. 
Goud R, van Engen-Verheul M, de Keizer NF, Bal R, Hasman A, Hellemans IM, et al. The effect of 
computerized decision support on barriers to guideline implementation: a qualitative study in 
outpatient cardiac rehabilitation. Int J Med Inform. 2010;79(6):430-7. Epub 2010/04/10. 
31. 
Grimshaw J, Eccles M, Tetroe J. Implementing clinical guidelines: current evidence and future 
implications. J Contin Educ Health Prof. 2004;24 Suppl 1:S31-7. Epub 2005/02/17. 
32. 
Soumerai SB. Are the peers peers? JAMA. 1995;273(7):523; author reply -4. Epub 1995/02/15. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
42 
 
33. 
Soumerai SB. Principles and uses of academic detailing to improve the management of 
psychiatric disorders. Int J Psychiatry Med. 1998;28(1):81-96. Epub 1998/06/09. 
34. 
Soumerai SB, Avorn J. Physician education and cost containment. JAMA. 1985;253(13):1876-7. 
Epub 1985/04/05. 
35. 
Soumerai SB, Avorn J. Economic and policy analysis of university-based drug "detailing". Med 
Care. 1986;24(4):313-31. Epub 1986/04/01. 
36. 
Soumerai SB, Avorn J. Predictors of physician prescribing change in an educational experiment 
to improve medication use. Med Care. 1987;25(3):210-21. Epub 1987/03/01. 
37. 
Soumerai SB, Avorn J. Changing prescribing practices through individual continuing education. 
JAMA. 1987;257(4):487. Epub 1987/01/23. 
38. 
Soumerai SB, Avorn J. Principles of educational outreach ('academic detailing') to improve 
clinical decision making. JAMA. 1990;263(4):549-56. Epub 1990/01/26. 
39. 
Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: a critical 
analysis of the experimental literature. Milbank Q. 1989;67(2):268-317. Epub 1989/01/01. 
40. 
Hilt RJ, Romaire MA, McDonell MG, Sears JM, Krupski A, Thompson JN, et al. The Partnership 
Access Line: evaluating a child psychiatry consult program in Washington State. JAMA Pediatr. 
2013;167(2):162-8. Epub 2012/12/19. 
41. 
Finnerty MT, Kealey E, Leckman-Westin E, Gupta N, White TM, Engel GM, et al. Long-term 
impact of web-based tools, leadership feedback, and policies on inpatient antipsychotic 
polypharmacy. Psychiatr Serv. 2011;62(10):1124-6. Epub 2011/10/05. 
42. 
Schmid I, Burcu M, Zito JM. Medicaid prior authorization policies for pediatric use of 
antipsychotic medications. JAMA. 2015;313(9):966-8. Epub 2015/03/04. 
43. 
Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 
1986;73(1):13-22. 
44. 
Zou G. A Modified Poisson Regression Approach to Prospective Studies with Binary Data. 
American Journal of Epidemiology. 2004;159(7):702-6. 
45. 
McNutt LA, Wu C, Xue X, Hafner JP. Estimating the relative risk in cohort studies and clinical 
trials of common outcomes. Am J Epidemiol. 2003;157(10):940-3. Epub 2003/05/15. 
46. 
McCullagh P, Nelder JA. Generalized Linear Models. London, UK: Chapman & Hall; 1989. 
47. 
Sikirica V, Pliszka SR, Betts KA, Hodgkins P, Samuelson TM, Xie J, et al. Impact of atypical 
antipsychotic use among adolescents with attention-deficit/hyperactivity disorder. Am J Manag 
Care. 2014;20(9):711-21. Epub 2014/11/05. 
48. 
Stucke K, Kieser M. Sample size calculations for noninferiority trials with Poisson distributed 
count data. Biom J. 2013;55(2):203-16. Epub 2013/02/13. 
49. 
Rubin DB. Multiple imputation for nonresponse in surveys. New York: J Wiley & Sons; 1987. 
50. 
Rotnitzky A, Robins JM. Semiparametric regression estimation in the presence of dependent 
censoring. Biometrika. 1995;82(4):805-20. 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
43 
 
PROTOCOL APPENDICES 
These documents are relevant to, but not considered part of, the protocol.  Each clinical site will modify, 
store, and obtain IRB approval for the documents locally.  
Modifications to the protocol appendices by any clinical site do not constitute a protocol amendment. 
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
44 
 
APPENDIX A: SCHEDULE OF EVENTS 
Event 
Description 
Timing  
Medication Alert and Enrollment Log 
Entry 
The medication alerts fire in Epic during routine clinical encounters at which an 
antipsychotic is entered in Epic for a target patient. The patient is automatically 
added to the study enrollment log when the alert fires. 
Day 0 
Alternatively: Active Order Entered in 
Epic and Enrollment Log Entry 
The intervention arm provider directly orders a study service (i.e., navigation, 
bridging therapy, CAP consult) in Epic outside of the medication alert. The patient 
is automatically added to the study enrollment log when the order is entered. 
Day 0 
Eligibility Validation 
Study eligibility will be validated by chart review for all patients logged.  
Target: Day 0-3 
Child and Adolescent Psychiatrist 
Review and Recommendations 
The study CAP will complete case reviews and send the prescriber 
recommendations within 3 business days of receiving notification of an eligible 
case. Consultation will be scheduled to occur within 10 business day when the 
study CAP requires more information and a consult wasn’t ordered. 
Target: Day 0-6 (review only 
or consult ordered) 
Target: Day 0-13 (consult 
not ordered, discussion 
needed) 
Behavioral Health Navigator Outreach 
to Prescriber 
The BH navigator will outreach the prescriber following CAP review. The navigator 
will also send periodic clinically relevant updates while his/her patient is receiving 
behavioral health navigation support. 
Target: Day 3-180  
(i.e., after the CAP’s review 
is completed) 
Behavioral Health Navigation for 
Patient/Teen 
The navigator will contact families/teens to offer up to 6 months of BH navigation. 
Outreach will occur within 2 business days following CAP review. Teens/families 
accepting this extra support will be contacted at least monthly. Ad hoc contacts 
may be made if psychotherapy appointments are missed and not rescheduled.  
Target: Day 6-180 
(i.e., after BNH outreach to 
provider is complete) 
Bridging Therapy 
Some teens/families will be offered bridging sessions with a study therapist. Up to 
9 sessions may be offered. 
(variable) 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
45 
 
6 Month Follow Up Period 
Data will be collected for the 6 month period after each eligible patient is added 
to the enrollment log. Analytic data will be derived from the EMR and other 
health plan data systems and study tracking systems.  
Day 0-180 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
46 
 
 
APPENDIX B: PROVIDER ADVANCE COMMUNICATIONS AND MEDICATION ALERTS 
 
• 
Advance communication to intervention arm providers 
• 
Advance communication to control arm providers 
• 
Example text of intervention and control arm medication alerts  
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
47 
 
 
APPENDIX C: SUAY CLINICAL GUIDANCE  
 
• 
Expert clinical guidance for the safer prescribing of antipsychotics to youth 
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
48 
 
 
 
APPENDIX D: BH NAVIGATION 
 
• 
Behavioral health navigator script(s) for introductory calls 
• 
Behavioral health navigator script(s) for ongoing and final calls 
• 
Example staff messages to prescriber from behavioral health navigator 
• 
Example staff message to transition patient from bridging to ongoing psychotherapy 
• 
Behavioral health navigation safe practice protocol  
 
 

PHASE 3 TRIAL: Safer Use of Antipsychotics in Youth (SUAY) 
Version 5 
Contract Number:  HHSN 2712016000002 C 
 
 ________________________________________________________________________________________________________  
 
 ____________________________________________________________________________________________ 
49 
 
 
 
APPENDIX E: TELEMENTAL HEALTH BRIDGING 
These items apply only to study therapy sessions delivered via the telemental health modality. They do 
not apply to in-person sessions provided by a study therapist at the clinic. 
• 
Telemental health privacy and safety planning form for video visits 
• 
Patient telemental health introductory and appointment reminder emails 
• 
Reminder telephone script for patient to return loaner device for video visits