Randomized Trial of a Leadership and Organizational Change Strategy to Improve the Implementation and Sustainment of Digital Measurement-based Care in Youth Mental Health Services
Phase: NA
Conditions: Implementation, Behavioral Symptoms
Sponsor: Boise State University

 
1 
 
2 
Study Protocol and Statistical Analysis Plan 
3 
 
4 
Study ID: NCT04096274 
5 
Unique Protocol ID: 8424 
6 
Study Official Title: Randomized Trial of a Leadership and Organizational Change Strategy to 
7 
Improve the Implementation and Sustainment of Digital Measurement-based Care in Youth 
8 
Mental Health Services  
9 
Brief Title: Working to Implement and Sustain Digital Outcome Measures (WISDOM) 
10 
Document Date: December 21, 2023 
11 
 
12 
 
13 
 
14 
 
 
15 

Social and Behavioral Sciences Human Research Protocol  
16 
PRINCIPAL INVESTIGATOR:  
17 
Nathaniel J. Williams, PhD 
18 
1910 University Drive 
19 
Boise ID 83725 
20 
Phone: (208) 426-3145 
21 
Email: natewilliams@boisestate.edu  
22 
 
23 
PROTOCOL TITLE:  
24 
Randomized trial of a leadership and organizational change strategy to improve the implementation and 
25 
sustainment of digital measurement-based care in youth mental health services 
26 
 
27 
SHORT TITLE: 
28 
Working to Implement and Sustain Digital Outcome Measures (WISDOM)  
29 
 
30 
BOISE STATE UNIVERSITY IRB NUMBER:  
31 
041‐SB19‐081 
32 
 
33 
ABSTRACT: 
34 
This study will test the effects of an intervention called Leadership and Organizational Change for 
35 
Implementation (LOCI), relative to implementation as usual (IAU), on clinician fidelity to, and youth 
36 
service outcomes of, a well-established digital measurement-based care intervention in outpatient 
37 
community mental health clinics. The study involves a randomized controlled trial of LOCI in 21 publicly-
38 
funded mental health clinics, incorporating 120 clinicians who deliver outpatient psychotherapy to 
39 
youth and, over two phases, a total of 720 caregivers of youth outpatients who receive services at the 
40 
clinics.  
41 
 
42 
OVERALL OBJECTIVES: 
43 
This study will test the LOCI implementation strategy in two phases to determine whether it improves 
44 
fidelity to a digital measurement-based care intervention called the Outcomes Questionnaire-Analyst 
45 
(OQ-A) as well as service outcomes of youth (i.e., symptoms and functioning during the first 6 months of 
46 
treatment as reported by caregivers). All clinics and clinicians will receive training in the OQ-A system as 
47 
well as  implementation support (technical assistance) from the OQ-A developer (this represents the 
48 
implementation as usual condition). In addition, half the clinics will be randomly assigned to LOCI to 
49 
support implementation and sustainment of the OQ-A system.  
50 
 
51 
Leaders in organizations assigned to the LOCI condition will participate in the LOCI intervention for 12 
52 
months. LOCI provides consultation and training to organizational leaders in how to support the 
53 
implementation of evidence-based practices such as the OQ-A system (see intervention details below). 
54 
To test the effects of LOCI, we will examine outcomes of (a) fidelity to the OQ-A system (measured at 
55 
the youth level), and (b) youth service outcomes (as reported by caregivers) during two phases following 
56 
initial training in the OQ-A system: Phase 1 - initial implementation (months 1-12 post- OQ-A training), 
57 
and Phase 2 - sustainment (months 13-24 post- OQ-A training). Clinician fidelity to the OQ-A system will 
58 
be measured via (a) electronic meta-data from the OQ-A system, which indicates whether or not 
59 
clinicians viewed the feedback reports, and (b) via caregiver reports. Youth service outcomes will be 
60 
measured via caregiver reports of the youth's symptoms and functioning for six months following the 
61 
youth's initiation of treatment. 
62 
 
63 

The project addresses the following specific aims and associated hypotheses:  
64 
 
65 
Aim 1: Test the effects of LOCI on the implementation of a digital measurement-based care intervention 
66 
called the OQ-A (Phase 1).  
67 
Hypotheses: Relative to implementation as usual, (1a) clinicians in LOCI organizations will exhibit 
68 
higher fidelity to OQ-A, and (1b) youths in LOCI organizations will experience superior service 
69 
outcomes (i.e. improvements in symptoms and functioning), for 1-12 months following initial 
70 
OQ-A training.  
71 
 
72 
Aim 2: Test the effects of LOCI on the sustainment of the digital OQ-A intervention (Phase 2). 
73 
Hypotheses: Relative to implementation as usual, (2a) clinicians in LOCI organizations will exhibit 
74 
higher fidelity to OQ-A 13-24 months following initial OQ-A training, and (2b) youths in LOCI 
75 
organizations will experience superior service outcomes (i.e., improvements in symptoms/ 
76 
functioning) 13-24 months following initial OQ-A training.   
77 
 
78 
Aim 3: Test the mechanisms that link LOCI to OQ-A fidelity.  
79 
Hypotheses: (3a) Relative to implementation as usual, LOCI will improve leaders’ 
80 
transformational, transactional, and implementation leadership, organizational implementation 
81 
climate, and clinicians’ intentions to use the OQ-A system; (3b) Improvements in leadership, 
82 
climate, and intentions will mediate LOCI’s effects on OQ-A fidelity. 
83 
 
84 
Primary outcomes variable(s): 
85 
The primary outcomes will be fidelity to the OQ-A system, measured via meta-data from the OQ system, 
86 
and youth clinical outcomes, measured as symptoms and functioning, during the initial six months of 
87 
services for each youth. 
88 
 
89 
Secondary outcome variable(s): 
90 
The secondary outcomes will be LOCI’s effects on first-level leadership behaviors, organizational 
91 
implementation climate, and clinicians’ intentions to use the OQ-A system.   
92 
 
93 
STUDY BACKGROUND 
94 
Psychiatric disorders are the leading cause of mortality and disability among youth in high income 
95 
countries, accounting for 21% of total disease burden, and afflicting 1 in 10 youths in the US with severe 
96 
impairment.1-3 In the US, this extreme disease burden is matched by medical care expenditures, which 
97 
are higher for youth psychiatric disorders than any other childhood illness ($16.8 billion in 2011),4 and 
98 
by investments in clinical research, which have developed over 1,200 effective interventions, or 
99 
evidence-based practices (EBPs), shown to improve youth well-being in randomized trials.5, 6 However, 
100 
despite these significant investments, less than half of youths treated in community settings experience 
101 
symptom improvement,7, 8 a situation largely attributed to the low rates at which community providers 
102 
adopt EBPs and, even when adopted, the low fidelity with which EBPs are implemented and sustained.9-
103 
12  
104 
 
105 
Policymakers and service systems have sought to address this implementation deficit through legislative 
106 
mandates and widespread EBP training programs for clinicians; however, these efforts have failed to 
107 
meaningfully change practice patterns or improve patient outcomes.13-17 Recent reviews indicate that 
108 
many deficits in EBP implementation and sustainment can be traced to a lack of organization-level 
109 
‘social infrastructure,’ that is, social contexts and leadership that do not support and motivate clinicians 
110 
to use EBP. Without this organizational social infrastructure, EBP training and technical efforts fail.12, 
111 

18-20 These observations are consistent with decades of research on organizational climate theory21-24 
112 
and with theories of behavior change,25-27 which we have integrated to generate our primary 
113 
hypothesis: achieving effective implementation and sustainment of EBPs in community settings requires 
114 
mechanisms of a strong organizational implementation climate and high clinician motivation generated 
115 
through effective clinic leadership. With NIH support, we have pilot tested a highly transportable 
116 
implementation strategy called the Leadership and Organizational Change for Implementation (LOCI) 
117 
intervention that targets these mechanisms through (1) short-term, intensive consultation with senior 
118 
leaders (i.e., executives/ administrators) and (2) training and coaching of first-level leaders (i.e., clinical 
119 
supervisors). Preliminary studies indicate LOCI is feasible, acceptable, and improves implementation 
120 
leadership and implementation climate.28, 29 We propose a randomized controlled trial of LOCI in 20 
121 
children’s mental health clinics, incorporating 120 clinicians and a total of 720 youth outpatients during 
122 
two phases of initial implementation (months 1-12) and sustainment (months 13-24), to test LOCI’s 
123 
effects relative to implementation as usual (IAU) on clinician fidelity and youth clinical outcomes of a 
124 
well-established digital measurement-based care (MBC) intervention.30, 31  
125 
 
126 
Digital MBC systems, which integrate with electronic health records to collect treatment outcome data 
127 
from patients and provide clinicians with real-time feedback and recommendations based on ‘big data’ 
128 
actuarial algorithms, are a high-impact digital health technology and EBP32 shown in 29 RCTs to 
129 
generate improvements in clinical outcomes equivalent with the best psychotherapy protocols (i.e., 
130 
d=.3-.5) across patient ages, diagnoses, and treatment modalities.32, 33 Despite the promise of this 
131 
approach, digital MBC systems are rarely used in community settings for youth,34, 35 and when they 
132 
are, fidelity and sustainment are often poor, primarily due to deficits in organizational leadership and 
133 
social context.36-40 This trial tests whether the LOCI strategy can improve the implementation and 
134 
sustainment of digital MBC for youth.  
135 
 
136 
STUDY POPULATION 
137 
 
138 
Target Population: 
139 
We will recruit and collect data from four groups of people for this project: 1) executives and upper-level 
140 
leaders (i.e., CEOs, Executive Directors, program administrators) of outpatient mental health clinics that 
141 
serve youth, 2) clinic first-level leaders (i.e. clinical supervisors) in these clinics, 3) clinicians serving 
142 
children with emotional and behavioral disorders in the clinics, and 4) parents/ caregivers of children 
143 
with emotional and behavioral disorders who receive mental health services from participating clinics.  
144 
 
145 
Accrual:  
146 
Based on our a priori power analyses, we anticipate the study will require the following number of 
147 
participants in each group to generate adequate power to test our hypotheses: 
148 
 
149 
Clinics - N=20 
150 
Executive leaders - N=20 
151 
First-level clinic leaders - N=40 
152 
Clinicians - N=120 
153 
Parents/ caregivers of youth – Total N=720 (recruited in two phases of N=360 per phase) 
154 
 
155 
These figures will serve as recruitment targets. However, we also note that we plan to open enrollment 
156 
in the study to all clinicians at participating clinics and to all leaders in LOCI clinics as a way of improving 
157 
the ecological validity of our results. Thus, our final sample size numbers may exceed those reported 
158 
above. 
159 

 
160 
Eligibility Criteria: 
161 
Inclusion criteria for all groups of participants are intentionally broad to ensure robust and valid results.  
162 
 
163 
Inclusion Criteria for Clinics  
164 
(1) Provide outpatient psychotherapy services to children who have emotional and/ or behavioral 
165 
disorders and their families 
166 
(2) Has at least 3 FTE clinicians on staff  
167 
(3) Not currently implementing a digital measurement-based care system clinic wide 
168 
 
169 
Inclusion Criteria for Executives and Upper Leaders 
170 
(1) Identified as CEO, Executive Director, or high-level administrator at an enrolled clinic 
171 
 
172 
Inclusion Criteria for first-level leaders (i.e., direct supervisors of front-line clinical providers) 
173 
(1) Identified as a clinical supervisor or clinical work-group supervisor/ leader at an enrolled clinic  
174 
 
175 
Inclusion Criteria for Clinicians  
176 
(1) Employed at a participating clinic  
177 
(2) Provides services to youth clients (age 18 or under) 
178 
 
179 
Inclusion Criteria for Parents/ Caregivers of Youth  
180 
(1) Biological parent or custodial parent/ guardian of a youth who: 
181 
      (a) is ages 4- to 17-years-old,  
182 
      (b) has been diagnosed with an emotional or behavioral disorder by clinic staff, 
183 
      (c) has been deemed appropriate for treatment at the clinic by the clinic's staff and has been    
184 
            admitted into services at the clinic. 
185 
 
186 
Subject Recruitment and Screening: 
187 
This project involves multiple levels of recruitment which will be implemented by members of the 
188 
research team including the PI, Co-Is, a Postdoctoral Researcher/Project Coordinator, a full-time 
189 
Research Associate, and trained graduate student research assistants. All members of the research team 
190 
will complete training and certification (CITI) in Human Subjects research prior to recruiting participants 
191 
and will be trained in relevant recruitment protocols by the PI, Co-I Esp, or the Postdoctoral Researcher. 
192 
We note that all recruitment will occur in mental health outpatient clinics located in Idaho, Oregon, and 
193 
Nevada.  
194 
 
195 
Recuitment will proceed in stages beginning with the recruitment of executive directors and CEOs of 
196 
behavioral health clinics that deliver outpatient mental health services to youth. Once these leaders 
197 
have agreed that their clinic will participate in the study, we will recruit first-level leaders (i.e., 
198 
supervisors of clinicians), and clinicians who work in participating clinics. We will obtain a Memorandum 
199 
of Understanding for all clinics that particpiate. After recruiting clinic staff, parents/ caregivers of youth 
200 
who are served by the clinics will be notified about the study via a cover letter and invited to participate 
201 
if they wish. Parents/ Caregivers who have questions about the study will be referred to a member of 
202 
the research team.  The individuals who will compete these various stages of recruitment will be as 
203 
follows: 
204 
 
205 
(1) recruitment of clinic executives - PI (primary), Co-Is (primary), Postdoctoral Researcher/Project 
206 
Coordinator, 
207 

(2) recruitment of first-level clinic leaders and clinicians – PI, Co-I Esp (primary), Postdoc 
208 
Researcher/Project Coordinatory (primary), Research Associate (primary), graduate research 
209 
assistant 
210 
(3) recruitment of caregivers of youth – Postdoc Researcher/Project Coordinator (primary), Research 
211 
Associate (primary), graduate research assistant, PI, Co-I Esp  
212 
 
213 
Details of the recruitment process for each group as follows: 
214 
 
215 
(1) Clinic Executives will be recruited by the PI and Co-Is Esp, Aarons, and Ehrhart via two strategies. 
216 
One strategy involves presenting an overview of the study at standing system meetings (e.g., 
217 
regional behavioral health provider meetings; directors meetings sponsored by Medicaid) and other 
218 
large-scale events and distributing follow up email through system contact lists and targeted emails 
219 
to program/clinic leaders to arrange site visits and consultations. The second recruitment strategy 
220 
involves directly contacting executives of behavioral health clinics in Idaho, Oregon, and Nevada via 
221 
email, phone, and/or postal mail to request a in-person or vitual meeting with clinic leaders to 
222 
present the study and assess their interest in participating. We will directly contact the CEOs/ 
223 
Executive Directors of clinics and the PI, Co-Is Esp, Aarons, and Ehrhart, and/ or the Postdoctoral 
224 
Researcher/Project Coordinator will hold informational meetings with clinic executives to discuss 
225 
the study procedures in detail and request their clinic's participation in the project. Executive 
226 
leaders who agree for their clinic to participate in the study will sign a Memorandum of 
227 
Understanding outlining roles and responsibilities of all parties.  
228 
 
229 
(2) After obtaining permission from clinic executives, we will work to recruit first-level leaders and 
230 
clinicians who work in each clinic. Recruitment of first-level leaders will be completed by the PI, Co-
231 
Is Esp, Aarons, and Ehrhart, as well as the Postdoc Researcher and/ or Research Associate. With the 
232 
executive's support, we will introduce the study to first-level leaders and clinicians (ideally during 
233 
standing staff meetings) and ask them to participate. A member of the research team will make a 
234 
presentation during regularly scheduled staff or team meetings at the participating clinics. Details of 
235 
the project and requirements of participation will be discussed during this presentation. Following 
236 
the presentation, the research team will ask for permission from the Executive and First-level 
237 
leaders to receive a list of eligible clinicians so we can contact them via an automated Qualtrics 
238 
email that includes a link to the survey. The clinicians will also be informed that they will receive an 
239 
email from our team with a link to surveys. Once the research team receives the list of eligible 
240 
clinicians from the agency executive or first-level leader, emails with links to the consent form and 
241 
surveys will be sent via BSU Qualtrics. The email includes the research team’s contact information, 
242 
details about the study, and procedures. Clinicians will have one month to complete the survey if 
243 
interested.  
244 
 
245 
There will be no consequences for clinicians who choose not to participate in the study. Clinicians 
246 
will be assured that choosing to participate or not participate in the research will in no way affect 
247 
their employment. All clinicians will be free to withdraw their participation at any time.   
248 
 
249 
(3) Caregivers will be recruited for the study via a multistep process. 
250 
  
251 
First, during the intake appointment, a clinician or clinic staff member will provide caregivers with a 
252 
cover letter either in person or via electronic health record system. The cover letter will be available 
253 
in English and Spanish (translated professionally).The cover letter will state that the clinic is 
254 
participating in a research study and that the parent is invited to participate if they wish; however, it 
255 

will be stressed that the parent's decision regarding participation in the study will have no effect on 
256 
their ability to receive treatment at the clinic. The letter will provide details regarding the 
257 
assessments caregivers will be asked to complete as part of the study. It will ask parents to indicate 
258 
if they are interested in being contacted by a member of the research team for more information 
259 
about study and to potentially participate. Parents who wish to participate in the study will check 
260 
"YES" on the form and provide their contact information (name, email, and phone number); parents 
261 
who do not wish to participate will check "DECLINE" and/ or leave the form incomplete. Parents will 
262 
return their intake paperwork to a clinic staff member who will remove the cover letter and will fax 
263 
or email the letters to a secure fax line or email address provided by the research team. They will 
264 
then place it in an envelope in a secure area behind a locked door with restricted access (e.g., where 
265 
other medical files are stored). Participating clinics are used to storing sensitive medical files and 
266 
other data and restricting access to these files. After the letter has been received by the research 
267 
team, clinics staff will be asked to shred/discard letters.  If parents have questions about the study, 
268 
they will be directed to the PI or Postdoc/Project Coordinator by providing them with the research 
269 
team's contact information.  
270 
 
271 
Approximately 2-3 times per week, a member of the research team will contact the clinic to see if 
272 
there are any forms to be faxed or mailed. All forms will be saved in a secure, electronic database 
273 
housed on password protected Boise State University computers. This database will be used to track 
274 
contacts with caregivers for the following steps of the recruitment process.  
275 
 
276 
Caregivers who signed the form indicating they are interested in learning more and potentially 
277 
participate in the study, will receive a phone to screen for eligibility and conduct an informed 
278 
consent interview. Please refer to the “Procedures” section of the protocol for details on what 
279 
happens after a participant is recruited and screened.  
280 
 
281 
Early Withdrawal of Subjects: 
282 
At any time during participation, subjects can decide to withdraw from the study. They can do this by  
283 
not completing the surveys or by requesting that the assessments be terminated. 
284 
 
285 
Vulnerable Populations: 
286 
We will not be recruiting minors.  
287 
 
288 
Populations vulnerable to undue influence or coercion: 
289 
Subjects are informed that participation is completely voluntary. There is no penalty for withdrawing or 
290 
terminating their participation at any time during the study. There will be no consequences for clinicians 
291 
who choose not to participate in the study. Clinicians will be assured that choosing to participate or not 
292 
participate in the research will in no way affect their employment. In addition, caregivers will also be 
293 
assured that choosing to participating or not participating in the research will no way affect services at 
294 
their clinic. All subjects will be free to withdraw their participation at any time.   
295 
 
296 
Design:  
297 
This study is a cluster randomized controlled trial. We will randomize 22 clinics (target n = 20) to an 
298 
implementation as usual (IAU) control group or to an experimental group that includes IAU + LOCI. 
299 
Clinics that are randomly assigned to the LOCI condition will begin LOCI activities one month before the 
300 
initial OQ-A training. One month after initiation of LOCI, all clinicians in both LOCI and IAU clinics will 
301 
participate in an initial, in-person, OQ-A training workshop delivered by the system developer. For six 
302 
months following the OQ-A training, all clinics will have access to implementation support provided by 
303 

the OQ-A developer. LOCI activities will continue for a total of 12 months. Recruitment of caregivers into 
304 
the study for Phase 1 will begin immediately after the OQ-A training and will conclude 12 months after 
305 
the training. Measurement of clinician fidelity and service outcomes for these youth will continue for six 
306 
months after completion of LOCI because caregivers recruited in month 12 need to be followed for six 
307 
months after initiation of treatment. Recruitment of caregivers into the study for Phase 2 will begin in 
308 
month 13 after the initial OQ-A training and will continue for 12 months with caregivers recruited in 
309 
month 24 followed for an additional 6 months to ensure adequate follow-up time. Caregivers will be 
310 
asked to complete measures that assess their child/youth's symptoms and functioning as well as the 
311 
clinicians' use of the OQ-A system during a total of 7 telephone calls during a 6 month period (baseline, 
312 
then monthly thereafter for 6 months).  
313 
 
314 
Study Duration:  
315 
The study involves a randomized controlled trial of LOCI in 21 children’s mental health clinics, 
316 
incorporating 120 clinicians and a total of 720 youth outpatients during two phases of initial 
317 
implementation (months 1-12 post- OQ-A training) and sustainment (months 13-24 post- OQ-A training). 
318 
All clinics will receive training in the OQ-A system and implementation support from the OQ-A developer 
319 
for 6 months (this represents implementation as usual). In addition, half the clinics will be randomly 
320 
assigned to LOCI to support implementation and sustainment of the OQ-A system. Leaders in 
321 
organizations assigned to the LOCI condition will participate in the LOCI intervention for 12 months. The 
322 
other half of agencies will be randomly assigned to the web-based leadership condition. Caregivers of 
323 
youth who receive services from clinicians in participating clinics will be surveyed for six months to 
324 
assess service outcomes (measured as changes in the target youth’s symptoms and functioning). 
325 
 
326 
STUDY METHODS 
327 
 
328 
Study Instruments:  
329 
• 
Primary outcomes for Aim 1 are clinician fidelity to MBC and improvement in youth symptoms and 
330 
functioning. Clinician fidelity to MBC will be measured using triangulated data from (a) gold-
331 
standard observational electronic meta-data collected automatically by the OQ-A system, and (b) 
332 
parent/ caregiver reports. For each youth at each session, the OQ-A records whether an outcome 
333 
measure was administered, whether the clinician viewed the feedback report, when the clinician 
334 
viewed the feedback in relation to the session, and whether supplemental measures regarding 
335 
treatment process (e.g., therapeutic alliance, self-efficacy) were administered to aid in diagnosing 
336 
problems. This data will be downloaded from the OQ-A system for each participating youth each 
337 
month and, following validated procedures outlined by Bickman et al.,36, 41 used to generate an 
338 
MBC Fidelity Index that characterizes clinicians’ fidelity to MBC fidelity for each youth’s entire 
339 
course of treatment. In addition, parents will report on MBC fidelity using the MBC Fidelity Scale, a 
340 
3-item measure validated in previous research.90 The 3 items produce a continuous score that 
341 
captures (a) the extent to which the clinician invited the caregiver and/ or youth to complete a 
342 
standardized outcome measure at each session, (b) the extent to which the clinician discussed 
343 
scores on the measure with the caregiver at each session, and (c) the extent to which the clinician 
344 
used outcome measure data to make recommendations about the child’s treatment or changes to 
345 
the treatment plan. Parents will report on clinicians’ MBC fidelity during monthly phone 
346 
assessments described above and these scores will be averaged into a single score characterizing 
347 
the clinician’s total fidelity to MBC for the youth’s entire course of treatment. 
348 
• 
Youth clinical outcomes will be assessed using two caregiver- and youth-reported measures. The 
349 
primary outcome is the 48-item Shortform Assessment for Children (SAC) Total Problems Score. The 
350 
SAC is a well-established, parent-reported, measure of youth internalizing and externalizing 
351 

symptoms used in several previous randomized trials.91-94 The SAC has demonstrated excellent 
352 
score reliability and validity in psychometric evaluations as well as sensitivity to change in clinical 
353 
trials of youth mental health services. The measure provides a total problem score; higher scores 
354 
indicate more severe symptoms. The Columbia Impairment Scale (CIS) is a 13-item measure of the 
355 
youth’s psychosocial functioning and general impairment in the areas of home, school, and 
356 
community settings. The CIS has excellent score reliability, validity, and sensitivity to change as well 
357 
as a clinical cut-score for assessing impairment; it is incorporated into several national 
358 
epidemiological studies of youth psychopathology.98-101 Caregivers will be compensated for 
359 
completing the SAC, SDQ, and CIS at baseline and monthly thereafter (for a total of 6 months) via 
360 
telephone calls.  
361 
• 
We will assess targeted mechanisms of LOCI using the following measures: The Multifactor 
362 
Leadership Questionnaire (MLQ)113 is one of the most widely used measures of transformational 
363 
and transactional leadership in organizations, including mental health agencies, and has excellent 
364 
psychometric properties (alpha = .76 to .90).62-64 Clinicians rate the extent to which their leader 
365 
engages in specific leadership behaviors on a 5-point scale (0=Not at all, 4=Frequently, if not 
366 
always). Transformational leadership is assessed by four subscales: Idealized Influence (8 items, 
367 
α=.87), Inspirational Motivation (4 items, α=.91), Intellectual Stimulation (4 items, α=.90), and 
368 
Individual Consideration (4 items, α=.90). The Implementation Leadership Scale (ILS)70 is a very brief 
369 
12-item measure developed with NIH support by our team to assess leaders’ behaviors that support 
370 
EBP implementation in four areas: Proactive Leadership (α=.95), Knowledgeable Leadership (α=.96), 
371 
Supportive Leadership (α=.95), and Perseverant Leadership (α=.96). The ILS yields subscale scores 
372 
and a total score (α=.98) which have demonstrated excellent reliability and convergent and 
373 
discriminant validity. The Implementation Climate Scale (ICS) was developed by our team with NIH 
374 
support and has demonstrated excellent internal consistency, α = .91 (18 items, 3 items on each 
375 
subscale) and strong validity evidence.73 The ICS assesses clinicians’ shared perceptions of the 
376 
policies, procedures, practices, and behaviors that are rewarded, supported, and expected to 
377 
facilitate effective EBP implementation. Items are scored on a 5-point Likert type scale (0=“Not at 
378 
all” to 4=“To a very great extent”). The six subscales are: Focus on EBP (α=.91), Educational Support 
379 
for EBP (α=.84), Recognition for EBP (α=.88), Rewards for EBP (α=.81), Selection for EBP (α=.89), and 
380 
Selection for Openness (α=.91). Social psychologists have developed measures of intention with 
381 
strong psychometric properties that are meant to be adapted to any behavior of interest; 25, 114 
382 
we have adapted these resulting in a 3-item scale assessing clinician intentions to use MBC. In 
383 
addition, we will modify the psychometrically validated Evidence-Based Treatment Intentions scale 
384 
to assess clinicians’ specific intentions to use MBC in their practice with clients. 
385 
 
386 
Administration of Surveys and/or Process: 
387 
The sources of data for this study will include:  
388 
 
389 
1) Web-based surveys (leaders, clinicians); 
390 
2) Digital meta-data from the OQ-A system regarding the extent to which clinicians used the system 
391 
for each youth; 
392 
3) Web-based or telephone surveys completed by parents/caregivers regarding their youth's service 
393 
outcomes and experiences at baseline and at follow-ups if they choose this mode of administration.  
394 
 
395 
Web-based survey data will be collected on the secure BSU Qualtrics site and managed by the 
396 
professional data manager in BSU’s Office of Information Technology. Clinicians, leaders, and caregivers 
397 
will complete the surveys at a time and location of their choosing.  
398 
 
399 

Telephonic survey data will be collected by trained research assistants supervised by the PI and Co-I and 
400 
entered directly into a secure database housed on professionally-managed and password protected BSU 
401 
computers. Telephone interviews will take place at times and locations that are convenient for 
402 
caregivers.  
403 
 
404 
Members of the research team will conduct telephone, in-person or virtual interviews in an office with 
405 
the door closed. All data maintained by the research project will be coded with a unique identifying 
406 
number for which the key will be separately stored and access limited to investigators and shared only 
407 
to the extent necessary to collect data or for required reporting purposes.  
408 
 
409 
Digital meta-data will be collected automatically by the secure OQ-A software system and securely 
410 
uploaded to password protected BSU servers or computers by the PI or other member of the research 
411 
team. Digital meta-data will also be coded with a unique identifying number to protect participant 
412 
privacy.  
413 
 
414 
Data Management: 
415 
Survey and questionnaire data will be collected from leaders, clinicians, and caregivers of youth via 
416 
either (a) secure web-based surveys, or (b) research administered telephone calls. Web-based survey 
417 
data will be collected on the secure BSU Qualtrics site and managed by the professional data manager in 
418 
BSU’s Office of Information Technology. Telephonic data will be collected by trained research assistants 
419 
supervised by the PI and Postdoc/Project Coordinator and entered directly into a secure database 
420 
housed on professionally-managed secure BSU servers. All data maintained by the research project will 
421 
be coded with a unique identifying number for which the key will be separately stored and access 
422 
limited to investigators and shared only to the extent necessary to collect data or for required reporting 
423 
purposes. Digital meta-data will be collected by the secure OQ-A software system and securely uploaded 
424 
to Boise State servers by the PI or other member of the research team. 
425 
 
426 
STUDY PROCEDURES 
427 
 
428 
Procedures:  
429 
Please refer to the “Subject Recruitment and Screening” section of the protocol for details on how we 
430 
will recruit and screen potential participants. 
431 
 
432 
Details of the procedure process following enrollment for each group as follows:          
433 
 
434 
(1)  Following enrollment in the study and randomization at the clinic level, all participating 
435 
clinicians and first-level leaders at all participating clinics will receive initial training in the OQ-A 
436 
system as well as 6 months of follow-up implementation support from the OQ-A developer. This 
437 
will include an initial in-person training session plus follow-up technical support and 
438 
consultation. We will ask clinicians to attend these trainings and to use the OQ-A system with 
439 
their clients; however, whether or not they use the system is up to them and, in fact, the extent 
440 
to which they use the system (i.e., fidelity) is one of the study outcomes. Details regarding the 
441 
OQ-A system, training, and implementation support is provided below.  
442 
 
443 
(2) Executives, first-level leaders, and clinicians in all conditions will be asked to complete web-
444 
based surveys to assess LOCI's effects on organizational, leadership, and clinician outcomes. 
445 
These surveys will occur a total of 5 times over 18 months (baseline, 4-month, 8-month, 12-
446 
month, 18-month) and are estimated to take 30 to 40 minutes each. The surveys will ask about 
447 

clinicians' perceptions of their work experiences and attitudes toward and use of the OQ-A 
448 
system and other evidence-based practices. Participants will be compensated for their time (see 
449 
Subject Compensation). 
450 
 
451 
(3) Recruitment of caregivers into the study for Phase 1 will begin immediately after the OQ-A 
452 
training and will conclude 12 months after the training. Recruitment of caregivers into the study 
453 
for Phase 2 will begin in month 13 after the initial OQ-A training and will continue for 12 months 
454 
with caregivers recruited in month 24 followed for an additional 6 months to ensure adequate 
455 
follow-up time. Caregivers will be asked to complete measures that assess their child/youth's 
456 
symptoms and functioning as well as the clinicians' use of the OQ-A system on a total of 7 
457 
surveys during a 6 month period (baseline, then monthly thereafter for 6 months). These 
458 
surveys are estimated to take 10-15 minutes each and caregivers will be compensated for their 
459 
time (see Subject Compensation).  
460 
 
461 
All participants receive email and phone call reminders to complete their follow up survey in 
462 
order to boost study engagement and assist with technical support. 
463 
 
464 
Details of the OQ-A intervention: 
465 
 
466 
(1) The Outcomes Questionnaire – Analyst for Youth (OQ-A) is a secure, HIPAA-compliant, cloud-
467 
based digital measurement-based care software application that provides regular (e.g., weekly) 
468 
feedback to clinicians about patients’ progress in treatment. The OQ system has been tested in 
469 
12 clinical trials across 4 countries (US, Netherlands, Norway, Germany) incorporating 7,882 
470 
patients and has been shown to significantly improve psychotherapy outcomes with a mean 
471 
effect size relative to treatment as usual of d = .48.  
472 
 
473 
(2) Each week prior to the session, caregivers (and youths, if applicable) complete brief, 
474 
standardized, psychometrically validated outcome measures and treatment process measures 
475 
(e.g., therapeutic alliance) on an electronic device (e.g., tablet or mobile phone). These digital 
476 
measures are automatically scored by the software application using actuarial ‘big data’ 
477 
algorithms and a feedback report is generated and delivered to the clinician within seconds. 
478 
Feedback reports describe the youth’s progress in treatment benchmarked against data from a 
479 
large population. The software issues empirically-based alerts indicating whether the youth is on 
480 
track toward positive outcomes or at risk for treatment failure or dropout. Data from the 
481 
treatment process measures assesses empirically-validated predictors of treatment outcome 
482 
including therapeutic alliance, motivation, self-efficacy, and social support. The OQ-A is designed 
483 
to detect treatment progress regardless of treatment modality, diagnosis, or discipline of the 
484 
clinician. It is important to note that none of the OQ-A measures completed by caregivers or 
485 
youth will be used in the research; these measures will only be used to provide feedback to 
486 
clinicians via the OQ-A system. 
487 
 
488 
(3) All clinicians in LOCI and implementation as usual will receive initial training from the OQ-A 
489 
developer as well as six months of implementation support covering both clinical and technical 
490 
aspects of OQ-A implementation. Following the six months of implementation support, all clinics 
491 
will continue to have access to the online initial training and booster training and technical 
492 
support from the OQ-A customer care team, which assists with technical troubleshooting and 
493 
minor clinical issues for all licensees. 
494 
 
495 

Details of the LOCI intervention: 
496 
 
497 
First-level leaders and executive leaders in the LOCI condition will participate in LOCI for 12 months. 
498 
There are two key features of LOCI: (1) training and coaching for first-level leaders, and (2) executive-led 
499 
organizational strategies to support the LOCI leaders and their efforts to build an implementation 
500 
climate in their clinics. LOCI has six components which are described in the LOCI manual along with 
501 
slides and handouts. The 6 key components of LOCI are:  
502 
 
503 
(1) Assessment: LOCI is a data-driven process with leader development plans and goals based on 
504 
360º assessment with the leader’s self-ratings, ratings from his/her subordinates (i.e., clinicians), 
505 
and ratings from the leader’s manager/supervisor. The data are synthesized into a detailed 
506 
feedback report and then used in the development of a personal development plan.  
507 
 
508 
(2) Initial Training: The LOCI training begins with a two-day leadership didactic and interactive 
509 
session. This component includes introduction to the full-range leadership model, identifying 
510 
transformational and transactional leadership behaviors, implementation leadership, identifying 
511 
leader behaviors that can be used to build a climate for OQ-A implementation, and group 
512 
activities (e.g., breakout groups, meals) to facilitate social interaction and learning consolidation 
513 
among trainees. The training also addresses implementation climate and the nature of OQ-A so 
514 
that leaders can articulate a rationale for how and why use of the OQ-A system can improve 
515 
client outcomes.  
516 
           
517 
Leadership Development Planning: During the initial training, trainers and coaches work 
518 
individually with each trainee in reviewing their personalized 360º assessment data, identifying 
519 
strengths and areas for development, and setting a timeline for issues to be addressed 
520 
immediately and those to be addressed later in coaching. Leader trainees emerge with data-
521 
based development plans including broad goals and specific action items that will guide 
522 
coaching sessions throughout the remainder of the program.  
523 
 
524 
(3) Coaching: Brief weekly coaching calls are provided for each LOCI participant. Coaching calls 
525 
range from 15-20 minutes in duration and keep participants on track with their goals and 
526 
development plans. The coach will be directly supervised by the developers of LOCI (Co-Is 
527 
Aarons, Ehrhart). The weekly coaching calls focus on tracking the trainees’ progress in their 
528 
development plans, updating the plans based on emergent issues or needs, problem solving, 
529 
providing additional leadership support, and identifying organizational strategy needs. Plan-Do-
530 
Study-Act (PDSA) cycles are utilized to assess incremental progress toward measurable overall 
531 
and time-limited goals. Group conference calls with leader trainees are held monthly to 
532 
facilitate problem solving and networking among LOCI participants and trainees as they discuss 
533 
their progress and solutions to barriers.  
534 
 
535 
(4) Organizational Strategy Development: LOCI facilitators (led by Co-I Esp) meet concurrently with 
536 
executive managers, middle managers, and LOCI trainees (within agency) during monthly 
537 
meetings across the 12 month period to tailor organizational strategies to support the trainee in 
538 
creating an implementation climate that supports use of the OQ-A system. This involves ongoing 
539 
plan-do-study-act cycles using web-survey data to evaluate how strategies are being 
540 
implemented and utilized and if adjustments are needed. As a guiding heuristic, we utilize 
541 
“climate embedding mechanisms” (i.e., what leaders to do communicate that use of the OQ-A is 
542 
expected, supported, and rewarded in the organization) that are tailored for OQ-A 
543 

implementation for each agency and workgroup. Examples of potential strategies include mid-
544 
level managers attending workgroup meetings in support of first-level leaders, providing 
545 
recognition for providers who exemplify excellence in using the OQ-A system, providing 
546 
additional training for the OQ-A or other clinical approaches identified as necessary to get 
547 
clients back on track, altering OQ-A feedback processes, or executives communications to 
548 
clinicians emphasizing the importance of OQ-A in service provision. 
549 
 
550 
(5) Follow-up Sessions: Prior to each booster session, additional 360º assessments are completed. 
551 
LOCI participants attend booster sessions at 4 and 8 months after the initial training. Leadership 
552 
principles, intervention goals, and organizational strategies to support leadership are reinforced 
553 
through group discussion and problem-solving. The booster sessions are considered longer term 
554 
plan-do-study-act cycles in which 360º assessments are utilized to assess progress and guide 
555 
modification of the leaders’ development plans.  
556 
 
557 
(6) Graduation: Graduation is a ritual (i.e., secondary embedding mechanism) deliberately included 
558 
in LOCI to mark completion of the program. Accomplishments of the participants are celebrated, 
559 
challenges are processed, and future plans are shared. 
560 
 
561 
Analysis Plan: 
562 
For all aims, basic data screening procedures will be conducted to screen for errors and explore 
563 
normality, linearity, form, and outliers. Prior to conducting main analyses, we will examine the 
564 
psychometric properties of total scales and subscales of all measures (e.g., coefficient alpha, 
565 
confirmatory factor analyses) to assess the latent constructs captured by the measures and we will 
566 
explore the bivariate associations among all variables. Data will be transformed as appropriate. We will 
567 
confirm randomization validity (e.g., chi-square tests, t-tests, Kruskal-Wallis). Minor differences will be 
568 
statistically controlled during model-building. We will explore for selection bias from attrition. Data 
569 
missing at random will be modeled using full maximum likelihood estimation.  
570 
 
571 
Based on our theoretical model, we have selected generalized linear mixed models (GLMMs) as our 
572 
general analytic framework. GLMMs are ideal for this study because they (1) accommodate the complex 
573 
nested structure of the data (e.g., youth nested in clinicians nested in clinics), (2) permit testing of cross-
574 
level Hypotheses (e.g., effects of organization-level LOCI on clinician- and youth-level outcomes), and (3) 
575 
permit the specification of different functional forms to address different types of outcomes including 
576 
mean differences (e.g., in clinician fidelity) as well as differences in growth rates over time (i.e., youth 
577 
clinical outcomes). Models will include random effects addressing the nesting of repeated observations 
578 
within youth and youth within clinics, as appropriate. All analyses will use an intent-to-treat approach. 
579 
Hypothesis 1a will be tested using a generalized linear mixed effects model with a binomial response 
580 
distribution and logit link function to address the events/trials nature of the MBC fidelity index. To 
581 
facilitate interpretation, we calculated average predicted probabilities for each condition, representing 
582 
the adjusted average fidelity index (or completion rate) in each group. Hypothesis 1b will be tested using 
583 
a linear mixed effects growth model with a condition (LOCI vs. control) by time cross-level interaction. If 
584 
a statistically significant (P<.05) interaction is observed, simple slopes analyses will be conducted to 
585 
examine the rate of change in each condition. Models will be estimated using maximum likelihood, 
586 
which accounts for missing data on the outcome under the missing at random assumption. Prior to 
587 
testing Hypothesis 1b, we will use model comparison tests based on the Bayesian Information Criteria to 
588 
determine the best-fitting functional form for growth and optimal random effects structure.  
589 
 
590 

To address potential imbalance across clusters and to account for variables included in the constrained 
591 
randomization, we will include covariates in all analyses. The familywise error rate for the two primary 
592 
outcomes was controlled using the Benjamini-Hochberg4 procedure.  
593 
 
594 
Controlling for Familywise Error Rate. Across our two primary outcomes (i.e., MBC fidelity index and 
595 
change in SAC total problem score), the Benjamini and Hochberg (1995)98 False Discovery Rate 
596 
procedure will be used to correct for multiple tests, which is preferable to the overly conservative 
597 
Bonferroni correction.  
598 
 
599 
Power Analysis:  
600 
Taking into account 25% attrition of youth and clinicians, our target enrollment is 360 youth nested 
601 
within 20 behavioral health clinics (180 youth in LOCI organizations and 180 youth in IAU organizations) 
602 
for Phase 1. To estimate the minimum detectable effect size give our sample, we assumed: (1) 360 
603 
youth nested within 20 clinics, (2) an alpha level of .05 and a power level of .80, and (3) a level-3 ICC 
604 
ranging from .05 to .10 for youth symptoms/ functioning based on our prior studies. Given these 
605 
assumptions, 360 youth nested within 20 clinics is a large enough sample size to find statistically 
606 
significant intervention effects of d = .44 to .53 (Cohen’s d), classically described as a medium effect.   
607 
 
608 
RISK/BENEFIT ASSESSMENT 
609 
 
610 
Risks: 
611 
There are no known serious health or psychological risks of participating in this study. The digital OQ-A 
612 
measurement-based care system used in this study represents a quality improvement intervention 
613 
designed to provide clinicians with feedback about their patients’ progress in treatment based on 
614 
patients’ self-reported outcomes. The OQ-A has strong empirical support from multiple randomized 
615 
trials. Patients complete an electronic survey on a tablet prior to their session, the OQ-A software 
616 
automatically analyzes this data, and provides clinicians with feedback to inform treatment decisions. 
617 
The LOCI implementation strategy tested in this study represents an effort to strengthen the delivery of 
618 
evidence-based practices for children in publicly-funded mental health clinics, with the direct 
619 
intervention targeted at organizational leaders through training and consultation. The study design does 
620 
not require any changes to treatment or services for any youth or family. All families will receive the 
621 
usual clinical assessments and clinical care provided by clinics.  
622 
 
623 
Potential risks to participants revolve primarily around potential loss of confidentiality. For the mental 
624 
health professional participants (leaders/ clinicians) this relates to confidentiality regarding their 
625 
practice behaviors (i.e., fidelity). Participants will be reassured that their decision to participate (or not) 
626 
in the study will have no impact on their role as providers in the service settings. In addition, participants 
627 
will be informed that their clinic will not have access to research data on clinician fidelity or client 
628 
outcomes thereby eliminating the potential use of this information to evaluate clinicians’ performance.  
629 
 
630 
For youth and caregivers who participate, there is potential risk of loss of confidentiality related to 
631 
sensitive information provided by their caregivers in response to study outcome measures assessing 
632 
child symptoms and functioning. All information obtained from participants will be held confidential 
633 
unless there is a report of serious harm to a child or threat of harm to self or others. This limitation will 
634 
be addressed specifically in the consent form. 
635 
 
636 
Subject Confidentiality: 
637 

When connected only to a numeric identifier, the majority of the research data does not contain 
638 
information that could identify a participant. Therefore, we will use protocols to separate personally 
639 
identifying information from the research data by generating a numeric research identifier in all data 
640 
sets. All digital meta-data, web-based survey data, and telephonic survey data will be labeled using ID 
641 
numbers only and stored on secure, password protected computers and servers that are professionally 
642 
protected and managed by BSU’s Office of Information Technology. Personally identifying information 
643 
(i.e., the key that links names to ID numbers) will be stored in a single database that is located on secure 
644 
servers at Boise State University and separate from the data files. Access to this key will be restricted to 
645 
authorized research personnel only and will include a password that is only known to the PI and 
646 
appropriate research members.  
647 
 
648 
How will confidentiality of data be maintained?  
649 
In addition, we will follow use the following procedures to ensure the confidentiality of electronic data 
650 
including:  
651 
 
652 
(1) all computer screens will have a five-minute time limit that will produce a locked screen if left 
653 
unattended and will require BSU login and password authenticaion to re-access the computer,  
654 
 
655 
(2) The study database will be located on the BSU computer network and will only be available to the 
656 
project team. Boise State University’s Office of Information Technology stores confidential information 
657 
from a multitude of research projects, and thus, maintains stringent security measures,  
658 
 
659 
(3) Research data collected will be coded with a participant’s unique identifier, or number. Personally 
660 
identifying data (i.e., names) will not appear on any research data,  
661 
 
662 
(4) All individuals with access to data will sign confidentiality agreements to never disclose any individual 
663 
information regarding any aspect of the study,  
664 
 
665 
(5) All persons working on this research project will have undergone extensive orientation and training 
666 
on issues regarding the maintenance and protection of confidentiality (sending confidential material to a 
667 
community printer, using names while conducting phone interviews, etc.) as well as standard training in 
668 
research ethics and the protection of human subjects (e.g., CITI).  
669 
 
670 
Subject Compensation: 
671 
1. Executives, first-level leaders, and clinicians will be paid for their time to complete the 5 web-
672 
based surveys. These individuals will receive a $30 for the first (baseline) survey, $30 for the 4-
673 
month survey, $45 for the 8-month survey, $50 for the 12-month survey, and $55 for 18-month 
674 
survey for this study; a total of $210 in gift cards (5 questionnaires = $210). The gift card will be 
675 
delivered either electronically or via mail following completion of each survey.  
676 
2. Caregivers of youth will be paid for their time to complete phone or web-based surveys with 
677 
researchers at baseline and once per month thereafter for up to 6 months. Caregivers will 
678 
receive a $15 gift card upon completion of each survey via either electronic mail or standard 
679 
mail delivery. 
680 
 
681 
Data and Safety Monitoring 
682 
We will convene a 3 member DSMB that will represent experience and expertise in research on 
683 
evidence-based practice implementation in clinical and other service settings for youth as well as 
684 
methodological expertise related to the conduct of randomized trials with vulnerable populations.  
685 

 
686 
The DSMB will convene annually, with ad hoc meetings to be called for by the Chair as needed. The R01 
687 
staff will be responsible for scheduling these meetings and for keeping minutes of their proceedings. 
688 
These minutes will be sent to each member of the DSMB for approval before their distribution to the PI 
689 
and Co-Is. 
690 
 
691 
The PI will be responsible for maintaining communication with the DSMB. Members of the DSMB are 
692 
not involved directly with the trial and do not have conflicts of interest with the study PI or Co-Is.  
693 
 
694 
The DSMB will perform several duties. First, they will review and approve the research protocol and 
695 
plans for data and safety monitoring prior to study commencement. Second, they will evaluate the 
696 
progress of the trial. This will include assessment of participant recruitment and accrual, occurrence of 
697 
adverse events, and study outcomes. This assessment will be performed at meetings every 12 months 
698 
during the trial and more frequently, if decided by the DSMB. The PI will be responsible for responding 
699 
to all recommendations of the DSMB and submitting DSMB reports to the Boise State University IRB and 
700 
NIMH. 
701 
 
702 
Risk / Benefit Assessment:  
703 
Clinic executives, first-level leaders, and clinicians will benefit from the study by receiving access to 
704 
training and digital health technology that they can use to more effectively measure and monitor the 
705 
progress of their clients in treatment. Clinical staff will receive continuing education credits for 
706 
completion of training activities which will be provided free of charge. These resources will improve 
707 
clinicians' and clinics' ability to effectively serve clients and are of value to clinicians because continuing 
708 
education credits can be used to satisfy licensing requirements.   
709 
 
710 
Caregivers of youth who participate in this study may benefit from receiving more effective services and 
711 
from the opportunity to provide feedback to clinicians about their child's progress during treatment.  
712 
 
713 
Leaders and clinics that participate in LOCI will benefit from receiving free leadership training and 
714 
organizational consultation to improve the implementation of evidence-based practices in their 
715 
organizations which may improve the overall leadership ability of participating leaders as well as the 
716 
functioning of the clinic and the overall work environment. 
717 
 
718 
INFORMED CONSENT PROCESS 
719 
 
720 
Informed Consent: 
721 
Please refer to the “Subject Recruitment and Screening” section of the protocol for details on how we 
722 
will recruit and screen potential participants. 
723 
 
724 
This project involves four different groups of participants: (A) clinic executives, first-level clinic leaders, 
725 
clinicians, and (B) parent/ caregivers of youth who are served by participating clinicians. Informed 
726 
consent procedures will be as follows for these groups: 
727 
 
728 
A. Clinic Executives, first-level leaders, and clinicians -  After obtaining permission from clinic executives 
729 
for their clinic to participate in the study (as described above), we will introduce the study to first-level 
730 
leaders and clinicians (ideally during standing staff meetings) and ask them to participate by sending an 
731 
email via BSU Qualtrics that includes a information about the study and a link to the consent form and 
732 
survey at baseline, 4 months, 8 months, 12 months, and 18 months (see recruitment and screening):  
733 

 
734 
1) Each time an executive, first-level leader, or clinician completes a web-based survey for the study, the 
735 
survey will include an introductory page that reminds them of their rights as participants in the study 
736 
and the voluntary nature of participation.  Respondents will indicate they are willing to complete the 
737 
assessments by clicking "AGREE" to continue to the assessments. 
738 
 
739 
There will be no consequences for clinicians who choose not to participate in the study. Clinicians will be 
740 
assured that choosing to participate or not participate in the research will in no way affect their 
741 
employment. All clinicians will be free to withdraw their participation at any time.   
742 
 
743 
B. Caregivers of youth served by participating clinics -  Caregivers will be consented in a multistep and 
744 
ongoing process by a member of the research team (i.e., Postdoc, Research Associate, PI, Co-I Esp, 
745 
graduate research assistant):  
746 
 
747 
 (1) If a caregiver indicates they agree to be contacted by a member of the research team by selecting 
748 
“YES” on the cover letter and sharing their contact information, a member of the research team will 
749 
contact the caregiver via phone. During this call, the research team member will first review the study 
750 
parameters with the caregiver (following a script) and obtain verbal consent that the caregiver wishes to 
751 
participate in the study. The research team member will document this by signing a form indicating the 
752 
caregiver provided verbal consent.  
753 
 
754 
(2) At each subsequent contact with caregivers (total of 7 contacts over 6 months), caregivers will be 
755 
reminded of their rights as research participants and will indicate they are willing to complete the 
756 
assessments. This process will be completed via either (a) an electronic cover page on web-based 
757 
surveys in which caregivers must indicate they consent to participate by clicking "AGREE" to continue to 
758 
the assessments, or (b) a telephone conversation with the researcher in which the caregiver verbally 
759 
indicates he or she is willing to complete the assessments.  
760 
 
761 
Documented consent will be secured prior to the initiation of study participation. Parents/caregivers 
762 
whose preferred language is Spanish will be contacted by a member of the research team who can 
763 
review the study parameters and obtain verbal consent in Spanish. In addition, the web-based 
764 
survey/assessment will be available in English and Spanish.  
765 
 
766 
RESOURCE NECESSARY FOR HUMAN RESEARCH PROTECTION 
767 
 
768 
The team includes academic investigators with experience in research on implementation science, 
769 
leadership and organizational climate, measurement-based care, and youth mental health services. Dr. 
770 
Nathaniel Williams (PI), Associate Professor in the School of Social Work at Boise State University, 
771 
Research and Program Evaluation Coordinator in Boise State University’s Institute for the Study of 
772 
Behavioral Health and Addiction, and a licensed clinical social worker. Dr. Williams is an implementation 
773 
scientist with substantive expertise in issues related to organizational climate and leadership. Dr. Susan 
774 
Esp (Co-I), is an Associate Professor in the School of Social Work at Boise State University. Dr. Esp 
775 
expertise lies in counseling and substance use disorder intervention, behavioral health, and public 
776 
health. Dr. Greg Aarons (Co-I), is a Professor in the Department of Psychiatry at the University of 
777 
California, San Diego (UCSD), a faculty member in the UCSD/SDSU Joint Doctoral Program in Clinical 
778 
Psychology, Director of the Child and Adolescent Services Research Center (CASRC) and Co-Director of 
779 
the Center for Organizational Research on Implementation and Leadership (CORIL). Dr. Aarons extensive 
780 
work focuses on identifying and improving systems, organizational, and individual factors the impact 
781 

successful implementation and sustainment of evidence-based practices and quality of health care and 
782 
public sector settings. Dr. Mark Ehrhart (Co-I), is a Professor in the Industrial/Organizational Psychology 
783 
Program at the University of Central Florida. Dr. Ehrhart’s work focuses on leadership and strategic 
784 
climate, including climate for implementation, and developing measures of implementation climate. Dr. 
785 
Lauren Brookman-Frazee (Co-I), is a Professor in the Department of Psychiatry at the University of 
786 
California, San Diego (UCSD). Dr. Brookman-Frazee research focuses on improving the effectiveness of 
787 
community services for children with mental health and development needed. All investigators and 
788 
project staff supervised by investigators will have documented up-to-date training in human subjects 
789 
research ethics, protections of human subjects, and good clinical practices, as required by law and as 
790 
verified by the BSU IRB.  
791 
 
References 
792 
1. Hedeker D, Gibbons R. Longitudinal Data Analysis Johns Wiley & Sons. Inc, Hoboken, New Jersey. 
793 
2006; 
794 
2. Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis methods. vol 1. 
795 
sage; 2002. 
796 
3. StataCorp L. Stata statistical software: release 17 College Station. TX StataCorp LP. 2021. 
797 
4. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to 
798 
multiple testing. Journal of the Royal statistical society: series B (Methodological). 1995;57(1):289-
799 
300. doi: 10.1111/j.2517-6161.1995.tb02031.x 
800 
5. Spybrook J, Bloom H, Congdon R, et al. Optimal design plus empirical evidence: Documentation for 
801 
the “Optimal Design” software. William T Grant Foundation Retrieved on November. 2011;5:2012.  
802 
6. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. L. Erlbaum Associates; 1988. 
803 
doi: https://doi.org/10.4324/9780203771587 
804