Preventing Substance Use Among Youth: Behavioral and Economic Impact of Enhanced Implementation Strategies for Communities
Phase: NA
Conditions: Substance Abuse
Sponsor: Wayne State University

Study Protocol: MI-LEAP 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Michigan Model for Health: Learning to Enhance and Adapt for Prevention (MI-LEAP) 
 
 
NCT04752189 
10/20/2020 
 

Study Protocol: MI-LEAP 
 
RESEARCH STRATEGY 
SIGNIFICANCE 
Implementation of evidence-based programs and youth substance use. Quality implementation of 
evidence-based programs (EBPs) for youth in community settings is critical for reducing the burden of alcohol, 
tobacco, and other drug (ATOD) use and its consequences. Adolescence marks a time of life with high rates of 
ATOD use.25 Healthy People 2020 identifies ATOD use a leading health indicator associated with morbidity 
and mortality among youth.26 Universal, school-based prevention EBPs reach large broad-based populations 
of adolescents and are essential to reducing ATOD use.2 One such prevention program is the Michigan Model 
for Health (MMH). MMH, the official health curriculum for the State of Michigan, is a theoretically-based, 
comprehensive skills-focused program with a core unit on ATOD use.27,28 In an RCT of MMH, researchers 
found that high school youth receiving MMH were less likely to report ATOD use than youth in the control 
condition.29 In real-world conditions, quality implementation of EBPs is essential to program effectiveness; yet, 
youth rarely receive programs as intended. For example, 76% of 9th grade public school students received the 
MMH unit on ATOD use but 58% of educators did not meet state-designated fidelity requirements.28 Our 
survey of Genesee County high school teachers also suggests limited adherence to the curriculum: 36% use 
less than half and only 27% report using three-quarters or more of the curriculum content.30 The current 
paradigm for intervention development focuses disproportionately on program efficacy (e.g., internal validity) 
and less on how interventions function in real-world contexts (e.g., external validity).8 This paradigm 
contributes to the widely acknowledged research-to-practice gap and limits societal benefits from significant 
intervention research investments.9,10 Implementation science addresses this gap. Research to date, however, 
does not incorporate extensively theoretically-based implementation science frameworks to study factors 
associated with EBP implementation in schools.2 Approaches to enhancing the effective implementation of 
EBPs in schools is also understudied, as most research has focused on clinical interventions.31 The cost of 
implementation fundamentally influences program delivery in schools, who often have carefully allocated 
resources.2,13 Economic evaluation can inform decisions about implementing interventions efficiently and 
effectively but is underutilized. Consequently, theoretically-guided research investigating factors influencing 
implementation, implementation strategies to enhance program delivery, and economic evaluation of these 
strategies is critical to bridging the research-to-practice gap but has not been widely applied in school-based 
ATOD prevention interventions.2 
Theoretical framework for investigating factors associated with EBP implementation in schools. The 
Consolidated Framework for Implementation Research (CFIR) provides a useful guide for the examination of 
factors associated with EBP implementation. Figure 1 depicts this comprehensive framework which includes 
domains across socioecological levels (e.g., intervention,  context, provider characteristics) that can aid in 
understanding EBP delivery in a given setting.11 Researchers have found, for example, that intervention 
characteristics (e.g., program 
packaging) were associated with fidelity 
in a school-based ATOD prevention 
program.32 Researchers also suggest 
that features of the context, are 
associated with fidelity, but this has 
focused on clinical settings.33,34 Provider 
characteristics, such knowledge about 
the EBPs, may also influence fidelity. 
Researchers found that providers with 
negative perceptions of an EBP were 
less likely to deliver it with fidelity.32 This 
framework has been widely applied in 
clinical settings but has rarely been 
applied to implementation in schools. 
Research examining factors across 
multiple domains is essential to 
effectively design strategies to improve implementation.23 Applying well-developed implementation science 
frameworks is a promising approach to understanding ATOD prevention EBPs. 
Figure 1: Overarching conceptual model  

Study Protocol: MI-LEAP 
 
Evidence-based implementation strategies. Evidence-based implementation strategies help facilitate the 
translation of research to practice through mitigating barriers to quality implementation of effective 
interventions.14 Implementation strategies are needed to realize the public health benefits of EBPs.35 
Replicating Effective Programs (REP) is an implementation strategy focused on maximizing flexibility and 
fidelity in EBP delivery.23 Unlike other implementation strategies (e.g., evidence-based quality improvement, 
NIATx),36,37 REP was developed for community-based settings to implement EBPs and has been used with 
prevention-focused interventions. REP, based on the CDC’s research-to-practice framework,38 is guided by 
Social Learning39 and Diffusion of Innovations Theories.40 Standard REP includes three primary components—
program packaging, provider training, and facilitation (Table 3). Standard REP is a low intensity, minimal cost 
intervention that has improved uptake of brief HIV-focused interventions.41 Researchers have also developed 
enhanced REP for more complex clinical behavioral interventions, which includes added customization for 
program packaging and training, and active engagement of internal and external facilitator.42 Information about 
organizational context, providers and the intervention are essential to customizing REP for optimal program-
organization fit. Researchers have found enhanced program fidelity among clinical sites using enhanced 
versus standard REP.43 REP and enhanced REP, however, have not been tested in school settings and very 
little research has been done to investigate the effects of either REP on implementation and behavioral 
outcomes in school settings. Although improved outcomes with enhanced REP are promising, the added costs 
can be substantial.43 Thus, economic evaluation is vital to assess the costs and benefits of different strategies. 
Table 3: Replicating Effective Programs (REP) Components (adapted from Kilbourne et al, 2015) 
Standard REP 
Enhanced REP 
Package 
Intervention manual 
development 
Intervention manuals customized to sites using data from organization needs assessment on 
population and setting resources 
Training 
Standard training 
Customized training based on input from package step above 
Facilitation 
As needed technical 
assistance; Monitor 
intervention uptake via 
observation, qualitative and 
quantitative data  
External facilitation components: Works with IF and providers to review potential barriers and 
set goals; Technical assistance: EF gives specific implementation guidance and facilitates 
information sharing and long-term plans for sustainability  
Internal facilitation components: Regular meetings with EF, providers to set goals, align 
intervention with organization, advocate for implementation to leadership 
Economic evaluation of implementation. The goal of implementation research is to determine the best 
strategies to improve EBP delivery in community settings. Most economic evaluation in implementation, 
however, has focused on the costs of interventions and not the costs of implementation strategies required to 
deploy and sustain interventions in community settings.13 Understanding the resources needed to achieve 
desired behavioral outcomes (e.g., reduced ATOD use) is essential to implementing EBPs.44 Because 
economic factors related to implementation influence the program delivery,12,13,15 economic evaluation can 
inform implementation strategies.45,46 Systematic examination of outcomes related to implementation 
strategies, and the costs of achieving them, is essential for integrating EBPs in real-world settings.13 
Organizations benefit from evidence that supports (or refutes) investment in specific implementation strategies 
as an efficient use of resources and can help prioritize implementation efforts. The Cost of Implementing New 
Strategies (COINS) provides a useful framework for evaluating implementation costs.44 COINS aids in 
identifying a range of costs across multiple phases of implementation (e.g., pre-implementation, 
implementation, and sustainability) and is congruent with REP phases (see Table 7).23,44 Many types of 
economic evaluation exist, but one well suited to implementation science is cost-effectiveness. Cost-
effectiveness assesses whether incremental benefits of one strategy versus another are sufficient to justify 
additional costs and has been used to support selective EBPs for youth in clinical settings47,38 Cost-
effectiveness data can help inform decisions about resource allocation for program selection and 
implementation.45 Economic analysis of implementation strategies, however, is understudied.  
 
Significance of the Expected Research Contribution 
I expect to make the following unique and significant contributions to the field of implementation science. First, 
through identifying barriers and facilitators to implementation, I will contribute to the application of 
implementation science theory across non-clinical settings (e.g., school) with a focus on prevention. Second, 
my theoretically guided research will inform implementation strategy customization for school-based ATOD 
prevention EBP, with attention to the complexity of an educational context and competing demands of schools’ 
primary goals and objectives (e.g., student learning & retention). My research will help expand the application 
of evidence-based implementation strategies (e.g., REP) in non-clinical settings to enhance EBP delivery in 
schools to prevent youth ATOD use. Finally, I will contribute to advancing economic evaluation of 

Study Protocol: MI-LEAP 
 
implementation through identifying the cost-efficient strategies to enhance program delivery. This is especially 
important in schools that reach a broad range of youth, where resources are often limited. The economic 
evaluation will aid in identifying feasible, sustainable implementation strategies to enhance quality EBP 
implementation. 
Consideration of sex as a biological variable 
I will consider sex as a biological variable in terms of participant outcomes for substance use. Researchers 
have found differences in substance use between adolescent males and females, with males having generally 
higher rates of ATOD use.25 Researchers have also found that adolescents may respond differently to clinical 
and school-based ATOD treatment and prevention programs.29,49 Consequently, it is vital to account for these 
potential differences when investigating the implementation of youth ATOD prevention programs. 
INNOVATION 
The proposed research has several innovations: 1) Application of implementation science frameworks in 
non-clinical community (i.e., school) settings. Although implementation research in non-clinical settings is 
growing, most research has focused on psychotherapy treatment. Also, the application of well-developed 
clinically-focused implementation science theories and frameworks for universal prevention in schools is 
limited. The proposed research applies implementation science theories to universal ATOD prevention in 
schools. 2) Testing implementation strategies with a scaled-up intervention. Researchers generally follow 
a linear paradigm of intervention development from efficacy trials to dissemination. Bridging the research-to-
practice gap will require a shift to include implementation research with EBPs in various stages of 
development. Researchers have not systematically applied implementation science frameworks to school-
based ATOD prevention programs even though some programs have already been adopted and widely 
disseminated. 3) Economic evaluation of implementation and implementation strategies: Researchers to 
date have focused primarily on quantifying intervention costs; less has focused on implementation strategy 
costs.12,13 Yet, implementation strategies range from less cost-intensive (e.g., standard REP) to more cost-
intensive (e.g., plan-do-study-act cycles).50,51 This research is innovative because it will focus on economic 
evaluation of implementation strategies in schools, an underdeveloped area of implementation science.   
APPROACH 
The Aims are reorganized based on the response to the summary statement, December 2018. 
AIM 1: Conduct secondary data analysis comparing the effectiveness and cost-effectiveness of two 
implementation strategies.   
Mid-year progress Aim 1a: I am currently working on options to address missing data for Aim 1a.  
In Aim 1a, I will compare using Replicating Effective Programs (REP) and Enhanced REP to deliver the Life 
Goals Collaborative Care: LGCC intervention on program fidelity and participant alcohol, tobacco, and other 
drug (ATOD) use. LGCC is an EBP developed for psychiatric disorders, which also addresses substance 
use.60 Similar to MMH, LGCC emphasizes skill building for behavior change to reduce negative outcomes.61 
LGCC has undergone customization guided by feedback on factors related to implementation in initial clinical 
trials and additional adaptations to include two conditions of enhanced REP. I will compare these conditions on 
fidelity and ATOD use. This will provide critical experience investigating implementation strategies to prepare 
me for applying REP in a novel setting- schools (Aim 3). In Aim 1b, Guided by the Costs of Implementing New 
Strategies (COINS) framework, I will estimate the costs and cost-effectiveness of REP and Enhanced REP on 
LGCC intervention fidelity and participant ATOD use. Sample. Eighty community clinics received standard 
REP at the start of the study. After 6 months, the non-responsive sites (n=60) were randomized to receive 
enhanced REP with both external facilitation: EF and internal facilitation: IF (n=30, 600 participants) or EF only 
(n=30, 600 participants) to facilitate LGCC implementation. Procedures. The study incorporated a sequential, 
multiple assignment, randomized trial design that included implementation and effectiveness components.43 
Data were collected at 6 and 12 months following randomization.  
Mid-year progress Aim 1b: I have a manuscript for the ADEPT cost-effectiveness analysis under 
review. 
Aim 1a. Measures. ATOD use. Using items from the National Institute on Drug Abuse’s Modified Alcohol, 
Smoking and Substance Involvement Screening Test63 and the Alcohol Use Disorders Identification Test64 we 
assessed past month marijuana, illegal drug, and tobacco use and binge drinking. We will create a summed 

Study Protocol: MI-LEAP 
 
composite scale using these items. Implementation outcome. Our implementation outcome will be adherence, 
a dimension of fidelity based on Proctor et al.,12 defined as the extent to which the program occurred as 
intended (see Table 4).32 Researchers debate the best approach to measuring fidelity dimensions, but we 
adopted an approach that was minimally invasive and suitable to a real-world context.42 We measured 
adherence by creating a total score of the number of group sessions and care management contacts using a 
combination of provider logs and confirmatory data from patient surveys.42 Data Analysis. We will compare 
fidelity (i.e., adherence) by REP condition (EF and IF vs. EF only). We will also use growth curve modeling to 
investigate individual change over time in ATOD use and between-individual change due to group membership 
(e.g., EF and IF vs. EF only), accounting for nesting effects. We will control for baseline and demographic 
characteristics and will examine potential differences in ATOD use by sex. Standard longitudinal methods such 
as growth curve modeling can be used to investigate the main effect aims in factorial experimental designs.65 
These analyses will provide an opportunity to work with data from an ongoing implementation study with ATOD 
outcomes. Expected outcome: We expect that patients receiving LGCC in clinics with enhanced REP using 
EF and IF will report less ATOD use than those in clinics with enhanced REP and EF only. 
Aim 1b: Measures. Implementation costs: Implementation costs will be guided by the COINS approach (see 
Table 7). Implementation costs include indirect costs such as time spent training and supervising, training 
compensation (e.g., pay during non-work hours), time costs of assisting with intervention delivery, and 
nonlabor costs such as costs of the curriculum (manual and materials), training time, technical assistance, 
facilitator time, and travel costs.13,72  Data analytic approach. We will estimate the costs of each REP 
condition using data in Table 7, with the base case analysis being the enhanced REP+EF (external facilitation) 
only. The primary outcome will be change in reported ATOD use during the intervention period. We will use net 
costs (net increase in costs from the enhanced REP condition with external: EF and internal: IF facilitation 
compared to EF only) and net effectiveness (net decrease in substance use from the enhanced REP condition 
with EF and IF compared to EF only) to calculate the incremental cost-effectiveness ratio for behavioral and 
implementation outcomes: 
𝐶𝐶𝐶𝐶𝐶𝐶𝑡𝑡𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅:𝐼𝐼𝐼𝐼+𝐸𝐸𝐸𝐸 −𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅:𝐸𝐸𝐸𝐸
𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅:𝐼𝐼𝐼𝐼+𝐸𝐸𝐸𝐸−𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅:𝐸𝐸𝐸𝐸. We will conduct uncertainty analyses to evaluate 
variability in results from potential bias in parameter estimates (e.g., duration of consultations for facilitation) 
and use these results to construct confidence intervals around cost and cost-effectiveness estimates as a 
measure of uncertainty around results.73 We will also conduct sensitivity analyses to characterize how 
extensive an implementation strategy must be to justify a change in approach from a behavioral (e.g., ATOD 
use) and economic perspective. We will vary certain assumptions to meet a range of possible scenarios (e.g., 
extent of facilitation).45 These analyses will be essential in informing economic evaluation for the pilot study. 
Expected outcome: We expect patients in the enhanced REP with IF and EF will report a greater reduction in 
ATOD use relative to dollar spent compared to those in REP with IF.  
Mid-Year Progress Aim 2: Since the end of Year 1, I have had two papers related to this research 
published, one in Prevention Science, and one in the Journal of School Health. I have also submitted a 
qualitative study related to the project that is currently under review. 
AIM 2: Evaluate factors associated with the fidelity of a school-based ATOD prevention program (the Michigan 
Model for Health: MMH) to tailor an implementation strategy.  
The objective of Aim 2 is to investigate factors associated with implementation guided by CFIR to better 
understand implementation in school settings (Figure 1). This will inform Replicating Effective Programs (REP) 
design for MMH delivery. This research is expected to provide information regarding barriers and facilitators to 
MMH implementation and foundational knowledge to effectively design implementation strategies.  
Overview. I will use data from an ongoing study (Health Education Experts Study) of high school health 
teachers and administration to assess factors related to the implementation of an EBP.30 This includes 
quantitative and qualitative data addressing factors across multiple domains guided by CFIR. Sample and 
setting. Participants are Michigan high school health class teachers, high school administrators, and other key 
personnel in MMH implementation (e.g., Regional Health Coordinators). We currently have survey (n=20) and 
interview (n=4) data from participants in Genesee County collected in collaboration with the GISD Regional 
Health Coordinator. We will recruit additional participants in collaboration with MI DHHS through their network 
of Regional Health Coordinators. MI DHHS has successfully recruited health teachers and administrators in 
previous health curricula studies and they are deeply committed to supporting MMH delivery.28 We expect to 
recruit 200 more teachers for the survey based on response rates from previous MI DHHS surveys. With this 

Study Protocol: MI-LEAP 
 
sample size (n=220), we would have 80% power to detect a 50% change in odds of our outcome (e.g., fidelity). 
We are also recruiting 10-15 more teachers for interviews. Procedures. We will use a mixed methods 
approach to develop a comprehensive understanding of ATOD prevention programs in schools. Mixed 
methods combine statistical results with in-depth personal experiences and aids in prioritizing implementation 
challenges and developing solutions.55 The survey is administered online through the U-M Qualtrics survey 
system. Participants received $10 remuneration for completing the survey. Health teachers willing to 
participate in the interviews will be contacted by project staff via email and postal mail. Those agreeing to 
participate will be consented using an online consent form and interviewed either in-person or virtually (e.g., 
Skype). These interviews will be audio recorded for in-depth analysis. Measures. See Table 5 for a summary 
of quantitative measures.56 We developed measures for domains listed based on the Consolidated Framework 
for Implementation Research (CFIR),57, and Replicating Effective Programs (REP) 23 unless otherwise 
indicated. CFIR is a comprehensive framework that aids in identifying the most relevant, context-specific 
factors related  
to implementation.33 Factors included in this study are informed by 
previous school-based research and input from community partners 
at GISD and MI DHHS; this includes constructs related to the 
intervention (e.g., packaging/manual), context (e.g., implementation 
climate) and providers (e.g., knowledge and beliefs).32 The 
implementation outcome is dose delivered: a fidelity dimension.12 We 
assess dose delivered by asking about the proportion of the MMH 
curriculum teachers reported delivering to students, from 0: none to 
4: 75% or more. The measure is consistent with dose delivered in a 
previous study of MMH.28 The semi-structured interviews focus on 
the intervention, context, and provider characteristics, including 
barriers and facilitators to MMH delivery. Data Analysis: Quantitative 
data analysis: We will use multinomial logistic regression models to 
estimate the probability of an outcome relative to a baseline category 
for a given set of predictors.58 We will account for nesting by school 
as needed; our community partners indicate, however, that many 
high schools in Michigan are staffed by one certified health teacher. 
Guided by Hosmer and Lemeshow’s approach,59 we will choose 
variables for inclusion based on statistical and substantive 
contributions to the overall model.59 In the interest of parsimony and 
reducing model over fitting, we will use both theoretical 
considerations and the Wald test to eliminate variables that do not 
contribute to the model. Qualitative Data analysis: We will analyze 
the structured interviews using N-vivo. First, we will mark separate segments responses that represent distinct 
constructs. Next, we will review data chunks to establish agreement and then assign chunks into meaningful 
categories guided by CFIR. Next, we will combine categorized data chunks. Finally, we will review data to 
identify common themes. Results from qualitative data analysis will inform model building and provide context 
for interpreting and applying quantitative results. Expected outcomes: We expect to identify specific factors 
across CFIR domains that influence dose delivered.33 For example, we may find that teachers report MMH is 
not fully meeting students’ needs around substance use and that this is associated with less dose delivered. 
These results will be critical to informing REP for the pilot study (Aim 3). 
Mid-Year Progress for Aim 3: We have completed meetings with an advisory committee and developed 
a plan for tailoring the MMH curriculum. Related to this, the research team created a website for the 
tailored curriculum to be amenable to online instruction partly in response to COVID-19. We are 
designing a tailored training to complement the curriculum. I am working with Amy Kilbourne to design 
an Implementation Facilitation training for health coordinators. We have also begun designing an 
implementation guide for deploying Enhanced REP for MMH. This process of designing Enhanced REP 
has been slowed due to the COVID-19 pandemic. The adapted curriculum, teacher and facilitation 
training, and pilot study are planned for early 2021, consistent with the original proposed timeline, 
dependent on schools and the pandemic status. 
Table 5: Study implementation measures 
Domain/construct 
Framework 
Intervention 
Adaptability 
CFIR11 
Packaging 
CFIR11; REP23 
Training 
REP23 
Technical assistance 
CFIR11; REP23 
Context 
Student needs 
CFIR11 
External policies  
CFIR11 
Networks and communication 
CFIR11 
Structural characteristics 
CFIR11 
Implementation climate 
CFIR11 
Provider characteristics 
Knowledge 
CFIR11,  IMB39 
Beliefs 
CFIR11,  IMB39 
Self-efficacy 
CFIR11,  SCT39 
Outcome expectations 
CFIR11,  SCT39 
Behavioral capability 
CFIR11,  SCT39 
Implementation Outcomes 
Appropriateness (fit) (Aims 2-3) 
IOF12,53,54 
Acceptability (Aims 2-3) 
IOF12,53,54 
Feasibility (Aims 2-3) 
IOF12,53,54 
Fidelity: adherence (Aims 2-3) 
IOF12,42 
Fidelity: dose delivered (Aim 1) 
IOF12,28 
CFIR: consolidated framework for implementation 
research, REP: replicating effective programs, ICS: 
implementation climate survey, IMB: information-
motivation-behavioral skills model, SCT: social 
cognitive theory, IOF: implementation outcomes 
framework 

Study Protocol: MI-LEAP 
 
Aim 3: MMH REP pilot: building on the skills developed in Aim 1 and Training Goals 1–3, I will conduct a 
prospective implementation pilot of the MMH in the GISD. REP is well suited for implementing MMH because it 
includes a curriculum manual and teacher training processes. The overall purpose of this pilot is to assess the 
appropriateness, feasibility, and acceptability of using REP for MMH, a universal ATOD prevention program.62 I 
will create a customized package for delivering MMH using standard and enhanced REP. Informed by data 
from Aim 1, I will investigate the effectiveness of standard and enhanced REP on implementation and 
behavioral outcomes. Sample and setting. We will study high schools from Genesee County, MI. Like other 
communities facing declining populations and economic challenges, the county has had higher unemployment 
levels compared to state and national averages for over a decade,66 and youth report higher rates of ATOD 
use compared to state averages (Table 6). This pilot study will include 8 schools from the Genesee 
Intermediate School District (GISD), as informed by other implementation pilot studies62 and study resource 
limitations. GISD includes 21 school districts and 25 public high schools. In collaboration with GISD partners, 
we will identify high schools using MMH that fail to meet MI DHHS standards for implementation (75% or less 
of the curriculum) to contact regarding participation. We will select schools using data from our MMH 
implementation study that have comparable student demographics (e.g., % free/reduced lunch). Procedures. 
We will recruit schools in collaboration with GISD leadership. Identified schools will be randomized to either 
standard REP or enhanced REP (see Table 3). The standard REP condition will receive a customized MMH 
manual (adapted from the current manual) based on results from Aim 2 to meet the needs of the context and 
providers; this may include creating on online manual. Standard REP will also include customized training 
based on Aim 2 (e.g., adapting content to new substances or substance forms, updated teacher resources) 
and brief, as needed technical assistance. Enhanced REP will also include internal and external facilitators 
with specific expertise in health education and implementation with weekly teacher contact to provide specific 
guidance for overcoming MMH delivery barriers. Enhanced REP will also include additional training and 
packaging customization (e.g., supplemental ATOD resources). The study team will meet with teachers and 
parents at participating schools to share study information. Teachers will complete a survey on MMH 
implementation and using Enhanced REP to deliver MMH post-intervention. We will also conduct qualitative 
interviews of teachers in participating schools to obtain in-depth information about using MMH delivery with 
REP. Students receiving the MMH at participating schools will be eligible for student-level surveys. Students 
will be recruited in participating schools in conjunction with GISD partners. Following parental consent and 
participant assent, students will complete a self-administered questionnaire through a secure, online survey 
about ATOD and related behaviors. As MMH is integrated as part of the school curriculum, students will 
complete the initial survey during their 9th or 10th grade health class prior to MMH delivery and 3 months post-
delivery. Students who do not consent will receive an alternate activity. I expect 300 students to participate 
based on a 60% response rate.67 Students will receive $10 for the surveys. 
Table 6: Demographics and youth substance use 
 
Michigan 
Genesee 
County 
Flint 
Median household income 
$49,087 
$41,879 
$24,679 
Per capita income 
$26,143 
$22,536 
$14,527 
% living in poverty - children >18yo 
23.7 
32.1 
62.4 
% racial/ethnic minority 
20 
25 
40 
Adolescent substance use % (past 30 
days)a 
 
 
Marijuana 
19.3 
25.6 
n/a 

Study Protocol: MI-LEAP 
 
Aim 3a Measures. Fidelity: adherence 
(MMH): Adherence, the extent to which 
an EBP is delivered as intended, will be 
assessed by teacher self-report. The 
measure includes specific procedures 
used (e.g., active learning) and engagement in specific MMH activities (e.g., four corners discussion) as 
detailed in the curriculum.12,68 We will create teacher checklists of MMH lesson components and create a 
composite adherence score based on the checklist. Acceptability, appropriateness, and feasibility (REP): We 
will use Weiner et al.’s53,56 measures to assess acceptability, appropriateness, and feasibility. Each construct 
has 4 items (e.g., REP is appealing, REP seems suitable), from 1: Strongly disagree to 5: Strongly agree. 
ATOD and associated outcomes (behavioral outcomes): We will use items from CDC’s Youth Risk Behavior 
Survey,69 that includes measures for substance use including tobacco, marijuana, alcohol, and other drugs, 
and associated consequences (e.g., injury, DUI). We will also assess MMH-specific constructs using an 
adapted survey of MMH from O’Neill et al.; this includes drug refusal skills (e.g., identifying proper responses 
to peers when offered drugs) and self-management skills (e.g., identifying appropriate ways to express anger 
or frustration).27 Implementation factors and outcomes (qualitative data): We will use the CFIR interview 
guide57 to guide qualitative investigation of using REP to deliver MMH; this will focus on factors identified in 
Table 5, and adapted based on results from Aim 1. In addition, we will ask for in-depth feedback on 
acceptability, feasibility, and appropriateness from teachers regarding the delivery of MMH using REP. The 
questions are guided by the constructs listed in Table 4. Data analytic approach. Implementation outcomes: 
We will assess adherence using the composite adherence score. We will evaluate appropriateness, 
acceptability and feasibility using data from the teacher survey and interviews. Qualitative data will be analyzed 
as described in Aim 1 and we will triangulate the findings using descriptive data from the Weiner et al. 
measures.53 Behavioral intervention effectiveness: Given the sample size associated with this pilot study, we 
will focus on estimating intervention effects, variance of the treatment effect, and directions of associations with 
outcomes of interest as opposed to statistical significance.70 We will use generalized linear models to 
investigate the effects of implementation condition on ATOD use and related consequences at 3 months post-
intervention, controlling for demographics and baseline functioning, and accounting for clustering by school. 
We will test the overall main effect of implementation condition on outcomes and investigate mean values of 
outcomes (e.g., substance use) by condition.71 We will also investigate ATOD use by sex to determine if the 
intervention condition had differential effects. Finally, we will use variance estimates of outcomes to inform 
sample size calculation for a larger, multi-site effectiveness study that will be the focus of a future R01 
proposal. Expected outcome: We expect that enhanced REP will increase adherence to the MMH curriculum 
compared to standard REP and teachers will report REP is feasible, acceptable, and appropriate to guide 
MMH delivery. We also expect that students in schools with enhanced REP will report less ATOD use than 
those with standard REP. 
Aim 3b: Procedures. Guided by Costs of Implementing New Strategies (COINS), we will monitor the activities 
listed in Table 7 to estimate implementation strategy costs. The study team will track time for labor costs-most 
of the resource/costs. This will include tracking time in study team activities as well as time logs for community 
partners to track REP activities. The study team will also track and compile all non-labor costs. Measures. 
Costs: Implementation cost measures are guided by COINS and include indirect and non-labor costs.72 We will 
measure implementation costs using available cost data from the school district, summary data from previous 
empirical studies on implementation costs, and routinely available cost data using market prices.45 
Alcohol (any) 
25.9 
27.5 
n/a  
Alcohol (binge) 
12.5 
16.1 
n/a  
Cigarettes 
10 
10.4 
n/a  
Prescription drug misuse 
15.8 
6.9 
n/a  
aamong 11th graders: Michigan Youth Risk Behavior Survey 2015  
Table 7: Intervention costs informed by Costs of Implementing New Strategies (COINS) 
Phases 
Description 
Activities 
Resource/cost type 
Pre-
implementation  
Identify barriers 
Stakeholder meetings 
RF, PM/ labor costs 
Survey  
Secondary data, PM, RF/labor costs 
Interviews 
Secondary data, PM, RF/labor costs 
Draft package 
Identify adaptations 
RF, CP/ labor costs 
Identify core elements 
RF, CP/ labor costs 
Program adaptations: online 
P/ labor costs 
domain costs/nonlabor 
Program adaptations: manual 
PM, RA/ labor costs 
Pilot test package 
refinement 
RF, RA, PM/ labor costs 
Orientation 
identify schools 
RF, PM, CP/ labor costs 
Logistics of delivery 
RF, PM/ labor costs 
Refine training 
RF, CP/ labor costs 

Study Protocol: MI-LEAP 
 
Data Analytic 
Approach. I will report 
summary statistics for 
implementation costs 
including mean costs 
with 95% confidence 
intervals. All costs will 
be adjusted to current 
year US dollars. I will 
estimate costs of REP 
and E-REP using the 
cost data, with the 
comparator strategy 
being REP. The 
outcome will be 
change in reported 
ATOD use. I will use net costs (net increase in costs from E- REP compared to REP) and net effectiveness 
(net decrease in ATOD use for E-REP versus REP) to calculate the incremental cost-effectiveness ratio for 
student outcomes: 
𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅 −𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝑅𝑅𝑅𝑅𝑅𝑅
𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑢𝑢𝑢𝑢𝑢𝑢 𝐸𝐸−𝑅𝑅𝑅𝑅𝑅𝑅−𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑢𝑢𝑢𝑢𝑢𝑢 𝑅𝑅𝑅𝑅𝑅𝑅.  I will conduct a one-way sensitivity analysis on all 
input parameters to provide estimates of the costs and incremental cost-effectiveness to decision-makers. The 
analysis will include 2- and 3-way sensitivity analyses on the parameters most sensitive in influencing the cost-
effectiveness ratio.6 Expected outcome: I expect that students in the enhanced REP condition will report 
greater reduction in ATOD use relative to dollar spent on the implementation strategy compared to those in 
standard REP. 
Missing data. We will conduct tests of selective attrition as needed to explore if those lost to follow up and with 
missing data on outcomes differ from those retained. In addition, we will use approaches such as multiple 
imputation and full-information maximum likelihood as needed to address missing values on model predictors 
to avoid biases due to complete case analysis.74 Full-information maximum likelihood and multiple imputation 
are both preferred over deletion approaches or nonstochastic imputation methods (e.g., mean imputation).74,75 
Potential Problems and Alternative Strategies: Enrollent biases: Schools, teachers, and administrators that 
support the MMH standardized curriculum may be more likely to enroll in the study, which may create 
enrollment bias; I will work with school district partners guided by my mentorship team and training in 
community-engaged research to reduce risk of enrollment bias. Through effective research-community 
collaborations, we facilitate enrollment of schools best suited to the study. Self-report biases: Participants, 
students, and staff may be influenced by social desirability biases. To minimize this, study staff and 
participants will be clearly informed of confidentiality and anonymity procedures, sensitive questions will be 
asked via a self-administered survey, and interviews will take place in private areas where others cannot 
overhear. School/district confidentiality: We will work with our partners at the GISD and local school districts to 
address potential concerns to school/district level confidentiality. All formal reports will only include aggregated 
information and be reviewed by district partners prior to dissemination.  
Identify EF IF 
CP/ labor costs 
Training 
Staff training: intervention 
RF, CP/ labor costs 
travel costs, RF/nonlabor 
training compensation/nonlabor 
space/nonlabor 
Implementation 
hosting/nonlabor 
Disseminate package 
PM/labor costs 
Technical assistance 
Internal facilitation 
CP/ labor costs 
External facilitation 
PM/labor costs 
Evaluation 
Interviews 
RF, RA/labor costs 
remuneration/nonlabor 
Survey  
RF, RA/labor costs 
remuneration/nonlabor 
Refinement 
Analyze data 
Secondary data, PM, RF/labor costs 
training, package refinement 
PM, RF, CP/labor costs 
Sustainability 
Organizational 
changes 
Recommendations for REP 
PM, RF, CP/labor costs 
Reproduce package 
CP/ labor costs 
Continue to refine 
CP, RF/labor costs 
PM: project manager; RA: research assistant; RF: research faculty, CP: community partner, P: programmer 
Table adapted from Saldana et al44