Improving Mental Health Outcomes: Building an Adaptive Implementation
Phase: NA
Conditions: Bipolar Disorder, Depression, Mood Disorder
Sponsor: University of Michigan

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
I. Project Goal 
The overarching goal of this study is to develop an adaptive implementation strategy involving 
previously established implementation strategies: Replicating Effective Programs (REP) and 
External/Internal Facilitation (EF/IF) to further disseminate an evidence-based clinical program 
for mood disorders (Life Goals- LG) in community-based routine care settings..  This study is 
designed to implement promising ways to improve health care operations through Facilitation. 
 
II. Project Aims 
It often takes years if not decades to translate evidence-based practices (EBPs) into community-
based settings. This research-to-practice gap is especially pronounced for psychosocial EBPs for 
mental disorders. Despite the availability of psychosocial EBPs, they are not getting into the hands 
of frontline providers in community-based practices and ultimately, improving care for persons 
with mental disorders. Implementation strategies, which are highly specified, operationalized 
approaches, involve educating and supporting frontline providers and clinical managers in 
promoting the use of EBPs. Notably, Replicating Effective Programs (REP) is a previously 
established implementation strategy that includes standardized EBP toolkits, training and limited 
technical assistance focused on supporting frontline providers in implementing EBPs in their care 
settings. REP is a low-intensity strategy with minimal costs from the site’s and provider’s 
perspective. However, more enhanced implementation strategies might be needed especially 
when sites and providers face competing demands or when the EBPs require additional support 
from higher-level organizational leadership. Hence, we enhanced REP to include Facilitation, a 
previously established implementation strategy that provides more personalized coaching to 
frontline providers and clinical managers in promoting the use of clinical EBPs in routine care that 
addresses  site-level organizational barriers to EBP adoption. Specifically, there are two 
Facilitation roles: External and Internal Facilitators. External Facilitators (EFs) resided outside the 
clinic and provided technical expertise to providers in adapting EBPs. Internal Facilitators (IFs) 
resided within each site with designated time to support EBP implementation through day-to-day 
provider engagement. REP in combination with EF and IF (REP+EF/IF) versus standard REP 
alone when applied to implement LG improved patient outcomes, notably mental health quality of 
life. However, IF requires additional time from sites, and not all sites may need IF. Therefore, an 
adaptive implementation strategy approach is necessary, whereby the REP implementation may 
need to be augmented if sites are not responding (i.e., not adopting EBP). In contrast to simply 
measuring correlates of implementation non-response, adaptive implementation strategies are 
augmented, or stepped, in direct response to limited uptake of EBPs among specific sites based 
on circumstances that may not be observable at baseline.  
 
The overarching goal of this study is to build the most cost-effective adaptive implementation 
strategy involving REP and Facilitation  to enhance the update of an EBP for mood disorder and 
subsequent changes in patient outcomes using clinical assessments available in routine care. 
The EBP, Life Goals (LG), is an evidence-based psychosocial treatment shown to improve 
outcomes among patients with mood disorders and will be implemented as part of routine clinical 
care at each organization. In this study, sites initially receiving REP that have not fully 
implemented LG (e.g., <10 patients receive the LG EBP) will be randomized to receive additional 
External Facilitation (REP+EF) or External plus Internal Facilitation (REP+EF/IF). At 12 months, 
sites that are non-responsive to External Facilitation (REP+EF) will be re-randomized to receive 
REP+EF or REP+EF/IF and then followed for another 12 months to determine whether longer-
term exposure to Facilitation is needed. Because sites and not patients will be randomized, the 
EBP (LG) is already an established evidence-based clinical practice that will be implemented as 
part of routine clinical care, and because all sites will receive some additional state-of-the-art 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
implementation support through Facilitation, this project involves health care operations 
improvement within select healthcare systems. 
 
  
Primary Study Aim. To determine whether sites that do not exhibit response to REP alone after 
6 months (e.g., <10 patients receiving LG), the effect of adding an External and Internal Facilitator 
(REP+EF/IF) versus adding an External Facilitator alone (REP+EF) on use of the EBP (LG) and 
changes in routinely collected patient clinical assessments (health-related related quality of life 
and mood  symptoms). 
 
Secondary Aim 2. To estimate the costs of REP+EF/IF compared to REP+EF at the sites. 
 
Secondary Aim 3. To describe the implementation of EF and EF/IF, including interaction between 
the two roles and the specific strategies EFs and IFs use to facilitate LG uptake across different 
sites. 
 
Results from the project will be used to inform activities designed to implement promising ways 
to improve clinical care and healthcare operations. 
 
III. Background and Significance 
 
Persons with mental disorders are not getting adequate care, leading to poor outcomes: 
In particular, mood disorders (depression, bipolar disorders) are common, undertreated, 
represent the top ten causes of disability according to the World Health Organization, and are 
associated with significant functional impairment, high medical costs, and preventable mortality. 
For many patients, pharmacotherapy is not enough to improve outcomes, and psychosocial 
treatments in addition to pharmacotherapy are recommended. 
 
Evidence-based implementation strategies are needed to promote uptake of psychosocial 
EBPs & improve mental health outcomes: Despite the availability of evidence-based 
psychosocial treatments, they rarely leave the academic shelf and get translated to community-
based practices. As a result, quality of care and outcomes for persons with mental disorders 
remain suboptimal. This research-to-practice gap can lead to millions of dollars of funded 
research being wasted when the evidence-based practices themselves never reach the 
populations in need. New health care initiatives including medical home models, bundled 
payments, and health care exchanges that are designed to improve efficiency and value will 
substantially impact publically-funded community-based sites that serve a disproportionate 
number of persons with mental disorders. For EBPs to make a difference under these emerging 
models of care, they need to be implemented with fidelity, yet be flexible in responding to 
organizational changes so that they are also used effectively to improve patient outcomes. Hence, 
improving the uptake of EBPs will require evidence-based implementation strategies that promote 
the rapid deployment of EBPs. 
 
Replicating Effective Programs improves uptake of psychosocial programs: There have 
been few rigorous trials of implementation strategies in community-based practices. Among the 
frameworks that guide implementation efforts, few have been operationalized for use as strategies 
to improve EBP uptake and patient outcomes. Study investigators have used Replicating Effective 
Programs (REP) to promote the uptake of psychosocial EBPs. REP is based on the Centers for 
Disease Control and Prevention’s (CDC’s) Research to Practice Framework project, and includes 
1) translation of the EBP’s scientific protocol into non-technical, user-friendly language 
(“packaging”), 2) formal training in implementing the package, and 3) supporting the 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
implementation through six months of brief, structured technical assistance. The concepts 
underlying REP include diffusion of innovations and social learning theory. In a national study of 
implementation of HIV prevention EBPs, AIDS service organizations receiving the REP package 
dissemination, training, and brief technical assistance were more likely to adopt HIV EBPs than 
sites receiving the package and training alone in 6 months (54% vs. 31%). 
 
Facilitation is needed to address organizational barriers: REP uses key tactical strategies 
that can promote effective EBP adoption in community-based practices. However, many sites 
face multiple organizational barriers including staff turnover, lack of leadership support, competing 
priorities, or lack of guidance regarding supervision that are beyond the scope of technical 
assistance. Addressing these barriers may require additional strategic thinking and multilevel 
organizational alliances that are tailored to the site’s unique circumstances, especially with the 
advent of new healthcare initiatives. Moreover, while support in EBP adoption from leadership is 
important, involvement and buy-in from frontline clinicians may lead to a greater likelihood of EBP 
uptake and sustainability. Hence, REP needed to be augmented to address these organizational 
barriers to adoption, to ultimately show value of the EBP through the triple aim (improving patient 
outcomes, experience, lowering costs). 
 
Consequently, study investigators enhanced REP to include Internal and External Facilitation, 
based on the Promoting Action on Research Implementation in Health Services (PARiHS) 
Framework. Facilitation is defined as the process of interactive problem solving and support that 
occurs in the context of a recognized need for improvement and a supportive interpersonal 
relationship. Study investigators (R01 MH79994, MH QUERI) operationalized Facilitation into two 
roles: External (EF) and Internal Facilitators (IF). EFs are off-site and provide technical assistance 
on EBP implementation (akin to REP technical assistance). IFs reside within each site to support 
the use of the EBP through strategic thinking and relationship-building, notably by incorporating 
provider input to support the adoption of the EBP, and aligning the EBP with organizational 
priorities via a direct reporting line with leadership. The EF/IF strategy proposed in this study 
extends the work of other programs such as the National Demonstration Project and the 
Evidence-Based Quality Improvement Model, whereby the EF and IF roles used in this study 
emphasize inter-organizational relationships across mental health and general medical care.  
 
Adaptive implementation strategies can determine how best to augment REP using 
Facilitation in routine clinical care: An adaptive implementation strategic approach is needed 
to more efficiently improve EBP uptake and patient mental health outcomes using Facilitation, 
whereby REP is augmented given early signs of site non-response. While REP might be sufficient 
for some sites in adopting EBPs, our preliminary studies and prior research suggest that the 
majority needed additional assistance (see below). Moreover, sites initially not responding to REP 
(i.e., limited adoption of EBPs) were unlikely to do so in the future. Adaptive implementation 
strategies are analogous to the concept of stepped care strategies, which have been applied for 
the treatment of mood disorders in a sequential fashion. In such strategies, care is augmented if 
the patient does not initially respond to treatment. In contrast to simply measuring correlates of 
implementation non-response across sites, adaptive implementation strategies allow for 
immediate augmentation at individual sites. For this proposed study, more intensive 
implementation strategies (e.g., EF/IF) are offered sequentially to sites that demonstrate limited 
EBP uptake under REP. In a preliminary study by the investigators (Figure 1, see below), REP in 
combination with EF and IF (REP+EF/IF) versus standard REP alone when applied to implement 
LG improved patient outcomes, notably mental health quality of life (R01 MH 79994).  
 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
EBP for community-based practices (Life Goals) improves outcomes in mood disorders: 
The EBP to be implemented using REP and Facilitation in this proposed study is Life Goals (LG). 
LG is an evidence-based psychosocial treatment that has been shown to improve medical and 
psychiatric outcomes in patients with mood disorders (including bipolar disorder or depression) 
from primary care and community mental health practices. LG is based on social cognitive theory 
and delivered in four two-hour weekly group sessions and six brief tailored contacts that 
encourage active discussions focused on individuals’ personal goals that are aligned with healthy 
behavior change and mood symptom management strategies. As LG is an EBP, it will be 
implemented in this project as part of routine clinical care by existing providers using 
established implementation strategies (REP + EF/IF). 
 
In five randomized controlled trials across mental health and primary care settings, LG improved 
outcomes, including mental and physical health-related quality of life, and reduced impaired 
functioning and mood symptoms. LG has also been shown to be equally effective across patients 
with co-occurring substance use and general medical comorbidities. LG was originally developed 
for bipolar disorder, and in partnership with community-based providers (R34 MH74509), 
expanded to include patients with any mood disorder (unipolar depression or bipolar disorder) as 
both are common, contribute to substantial functional limitation, and are considered the most 
expensive mental disorders in primary and mental health care settings. As with many 
psychosocial EBPs, LG has not been widely implemented across different community-based 
practices. LG has already been manualized for REP based on feedback from community partners. 
It is an appropriate choice in EBP to conduct an adaptive implementation trial of Facilitation, given 
that group sessions and follow up provider contacts require the support of frontline providers and 
leaders to implement. 
 
IV. Preliminary Work: 
REP and REP+EF applied to LG implementation for mood disorders. Study investigators 
defined and assessed variation in LG uptake and outcomes in patients diagnosed with unipolar 
depression or bipolar disorder using REP across primary care practices from three states. A 
provider (MSW) from each site was trained in LG based on REP in July 2010 and began 
implementing LG in August 2010. Each provider was asked to provide LG to 20 patients during a 
six-month period. Non-response was defined as <10 out of 20 patients receiving at least one LG 
session within the three-month period after REP training. Overall, three of the four sites (75%) 
were non-responsive to REP. Sites unable to implement LG faced organizational barriers that 
were observed during REP, notably lack of awareness of LG among the other providers and 
competing demands of the LG provider. EF technical assistance was then offered to non-
responsive sites, and included strategies to facilitate patient recruitment and tips on marketing 
the value of LG. Subsequently, two additional sites were responsive (50%) when EF was added. 
Pre-Conditions 
(completed February 2012) 
 
Identify need & EBP (LG) 
 
Identify sites 
 
 
Develop EBP Package 
REP: Pre-Implementation 
 
  Disseminate EBP     
  (LG) package 
 
Train providers in EBP  
 
Monitor LG uptake  
  
 
External Facilitation 
     Provider contact 
     Technical assistance 
     Monitor EBP milestones 
 
Internal Facilitation 
       Initiation 
       Relationship-building  
       Benchmarking, Rapport 
      Cultural adaptation 
      Ongoing marketing 
Sustainability 
  Evaluation 
 
Uptake and Fidelity 
 
Patient outcomes 
 
Costs 
 
Community feedback 
 
 Refinement and national 
roll-out of LG, REP+EF/IF 
Figure 1: Replicating Effective Programs (REP) & Internal/External Facilitation (EF/IF) 
  

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
While preliminary, these findings suggest that most sites will be initially non-responsive after REP, 
and some but not all improved LG uptake through EF. 
 
REP+EF/IF versus REP alone improved patient outcomes & LG fidelity, with additional IF 
costs: The Recovery-Oriented Collaborative Care Study (ROCC, R01 MH79994) involved sites 
from Michigan and Colorado randomized to REP+EF/IF vs. REP alone to implement LG for bipolar 
disorder. Primary outcomes included LG implementation and patient-level mental health-related 
quality of life (SF-12). REP consisted of dissemination of the LG package and training for site 
providers. REP+EF/IF added external facilitation (6 months of technical assistance by study staff) 
as well as internal facilitation, which consisted of consultation with clinic leadership over a 6-month 
period to address organizational barriers to implementing LG. Each site identified a behavioral 
health provider to offer LG to patients, and sites randomized to REP+EF/IF also identified an IF, 
who was typically an MSW-level clinical manager supervising the LG provider and with direct line 
of authority to the clinic director. Site providers completed organizational assessments and study 
staff conducted patient baseline, 6-, and 12-month computerized assessments that included 
quality of life (SF-12, symptoms), and functioning (WHO-DAS). The LG fidelity assessment was 
used to assess degree to which core components of LG were implemented using provider logs 
and chart review.  
 
Five sites were randomized to receive REP+EF/IF and four randomized to REP alone. A total of 
384 patients were enrolled (mean age=42, 66% female, 31% African-American). Patients from 
REP+EF/IF sites compared to those from standard REP sites had improved 12-month mental 
health-related quality of life scores (37.2 vs. 35.6, beta=1.22, p=.04) and decreased PHQ-9 scores 
(10.2 vs. 13.4, beta=-2.3, p=.04). Mean number of LG sessions was also greater for REP+EF/IF 
versus REP only patients. REP+EF/IF and REP sites did not differ on organizational 
characteristics. 
 
Fidelity/cost measures for REP, EF, and IF were also operationalized based on careful 
observation, review of study staff and provider logs, and clarification of EF/IF roles via expert 
panel input. These tools were used in the R01 to monitor potential contamination across EF/IF 
roles as well as competence. IF costs were estimated using monthly provider logs. Total costs for 
IF for each site was $5,500 over the 12-month period, and involved specific activities listed in 
Table 1. Nonetheless, the value of IF in addition to EF has not been assessed in a fully powered 
clustered randomized controlled trial.  
 
V. Methodology 
Overview: This is an adaptive implementation study involving cluster randomization at the site 
level of previously established implementation strategies designed to implement promising ways 
to improve health care operations. Providers will implement an EBP (Life Goals) as part of routine 
clinical care and no patients will be randomized. Outcomes data include site-level use of the EBP 
as well as assessments already available as part of routine clinical care (e.g., symptoms, quality 
of life).  
 
A total of 80 community-based clinics (sites) from Michigan (MI), Colorado (CO), or Arkansas 
(AR) that provide care for persons with mood disorders (depression or bipolar disorders) for a 
total of 1,600 patients with mood disorders (N=20 per site). The primary aim of this study will 
determine whether non-responsive sites (i.e., limited LG uptake) randomized to receive REP and 
additional Internal and External Facilitation (REP+EF/IF) implementation strategies to help adopt 
an EBP (Life Goals-LG) versus sites randomized to receive REP and additional External 
Facilitation only (REP+EF).  
 
A. Site study population: Site inclusion criteria include the following:  

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
1. Community-based mental health or primary care clinic located in MI, AR, or CO with at 
least 100 unique patients diagnosed with or treated for mood disorders in a given year (to 
ensure adequate patient N). 
2. Availability of a bachelor’s or master’s level health care provider with a mental health 
background and experience with implementing group sessions (core modality of LG) who 
can be trained to provide LG to up to 20 adult patients with mood disorders in the clinic in 
a one-year period. 
3. Availability of an employee at the site with direct reporting authority to the leadership of 
the site and network who could serve as potential internal facilitator. 
 
B. Site selection, representation: A total of 80 community-based mental health and primary 
care clinics (e.g., Federally-Qualified Health Centers) out of a total of 128 eligible sites from three 
states (Michigan, Colorado, Arkansas) will be randomly selected to participate in the study. Since 
the EBP to be implemented (Life Goals) has been shown to be effective in improving outcomes 
across different settings (primary care, community mental health), a diverse array of sites was 
recruited to maximize the potential generalizability of this proposed adaptive implementation 
study. These sites are also community-based safety net clinics serving a disproportionate number 
of low income and minority patients. 
 
For the purposes of this study, a site is a stand-alone clinic providing outpatient health services. 
The sites’ parent practice organizations were approached at the following state-level primary care 
and mental health association meetings: the Michigan Association of Community Mental Health 
Boards (November 2011), the Michigan Primary Care Association (February 2012), Community 
Health Centers of Arkansas, and Mental Health Council of Arkansas (January 2012), and the 
Colorado Regional primary care and mental health boards (January 2012). At these meetings 
study investigators provided an overview of Life Goals and the proposed study. A total of 128 out 
of 192 mental health or primary care sites representing several practice networks agreed to 
participate, including 40 from MI, 62 from CO, and 26 from AR. Primary reasons that sites gave 
for not wanting to participate included not having an individual that could serve the function of 
Internal Facilitator, no on-site mental health provider to implement LG, or too busy to participate. 
 
C. Study Design flow.  Primary care or mental health outpatient clinics agreeing to participate 
will be randomly selected (N=80 total) and initially offered REP for six months in order to 
implement LG. Based on previous research and preliminary data, it is expected that after six 
months of REP at least two-thirds of sites will be non-responsive to REP. The primary focus of 
the randomized comparisons in this study are sites that are initially non-responsive to REP (k=40), 
defined based on our preliminary studies as <10 patients receiving LG within six months of 
initiating REP and that patients received <75% group sessions. (Although not part of the primary 
randomized comparisons, sites that are responsive at 6 Months will continue to be followed and 
outcomes will be assessed). The non-responsive sites will be randomized 1:1 to receive additional 
External Facilitation (REP+EF) or External plus Internal Facilitation (REP+EF/IF). After another 
six months (at Month 12), sites that are still non-responsive (based on the same criteria) will be 
randomized 1:1 to either continue REP+EF or augmentation with IF (REP+EF/IF). All sites are 
followed for a total of 24 months. Patient-level outcomes will be assessed by study staff using 
assessments considered part of routine clinical care (e.g., symptoms and quality of life) starting 
at the initiation of REP and then at 6, 12, 18, and 24 months later among a pre-selected cohort of 
n=20 patients per site who are identified by site providers for LG (total patients participating in the 
projectl N=k*n=800). Patients receiving Life Goals as part of their routine clinical care may opt in 
our out in participating in the outcomes assessment component.  As with our prior implementation 
work (R01 MH79994), independent evaluators (study staff who are not aware of the assignment 
to REP+EF or REP+EF/IF) will conduct study outcomes assessments to ensure consistency of 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
data collected across sites and to prohibit provider/site or investigator assessment bias. Pre-
identification of LG recipients also maximizes consistency of the patient population across sites 
(see patient criteria below).  
   
D. Site Organizational Randomization: The unit of randomization is the site and will be stratified 
using a minimization allocation method. This procedure will ensure that groups are balanced for 
site variables that may correlate highly with outcomes. The Month 6 randomization for REP non-
responsive sites, will be stratified by state, type of practice (primary care or mental health site), 
and site-aggregated mean mental health related quality of life measured at Month 6. The Month 
12 randomization for non-responders to REP+EF will be stratified by site-aggregated quality of 
life measured at Month 12. Based on interviews with site leaders and the literature, participating 
sites differ more by state and practice type than by type of network, and state and practice are 
correlated with mental health related quality of life and number of LG encounters. 
 
E. Implementation Components (Table 1) 
REP: All sites will receive REP (Table 1) in the first three months and will be monitored monthly 
up to the sixth month in the use of LG. REP includes dissemination of the LG package, training, 
and LG uptake monitoring for up to six months. REP includes an off-site trainer, which is neither 
an External nor Internal Facilitator. The following REP components were previously 
operationalized by study investigators (R01 MH79994): 
 
Step 1: Dissemination of LG package, in-service, and patient selection: Each site will designate 
at least one provider with a mental health background (master’s in social work, nursing degree, 
or bachelor’s with two years’ experience in mental health psychosocial treatment) to implement 
LG (“LG provider”). Regional two-hour in-services will be provided by study investigators who will 
give an overview of LG including the evidence, and details on how to implement LG in their setting. 
During the in-service, the LG package will be disseminated and the REP trainer will coach the LG 
providers in pre-identifying a cohort of patients.  
 
Table 1: Summary of REP, REP+EF, and REP+EF/IF Implementation and Fidelity 
Measures 
Implementation Component 
REP 
REP+EF 
REP+EF/IF 
REP (REP) 
All sites 
Randomize to 
Non-Responding 
Sites 
Randomize to 
Non-Responding 
Sites 
Step 1: Disseminate package: LG in-service and identify 
appropriate patients using medical record review  
√ 
√ 
√ 
Step 2: Train site providers in LG 
√ 
√ 
√ 
Step 3: Monitor LG uptake via monthly reporting sheets 
√ 
√ 
√ 
REP+External Facilitator (REP+EF) 
 
 
 
Step 1: Provider Contact: EF sets measurable goals in LG 
uptake 
 
√ 
√ 
Step 2: Technical assistance: EF makes structured calls to 
site’s LG providers, giving specific guidance on implementing 
LG components 
 
√ 
√ 
REP+External and Internal Facilitator (REP+EF/IF) 
 
 
 
Step 1: Initiation: EF identifies IF within the site; IF meets 
with EF, LG provider, and leadership, establish measurable 
goals in LG uptake 
 
 
√ 
Step 2: Relationship-building: IF identifies site priorities per 
leadership input, identifies other LG program champions 
 
√ 
√ 
Step 3: Benchmarking and Ongoing Rapport: EF 
continues coaching IF, IF measures LG progress, IF develops 
rapport with leadership 
 
 
√ 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
Step 4: Cultural adaptation: IF uses local site culture 
knowledge to facilitate LG, addressing potential barriers, 
aligning LG goals with site 
 
 
√ 
Step 5: Ongoing marketing and sustainability: IF, EF and 
LG provider summarize progress with leadership, develop 
business/training plans 
 
 
√ 
 
REP LG Package: The LG REP package includes all of the components needed to implement 
LG, including the LG provider manual, group session scripts and focus points covered in each 
session in a semi-directed fashion, the registry template for monitoring mood and LG progress, 
scripts for follow-up calls, patient workbooks, and an implementation manual describing logistics 
(e.g., identifying group session rooms, identifying patients for LG, medical record templates for 
LG sessions, billing codes). 
 
Identifying appropriate LG target population: In REP, providers identify the appropriate target 
population for the EBP. We chose to focus on mood disorders (depression and bipolar disorder) 
because they are common, considered to be high priority populations for intervention based on 
input from community partners, and LG was shown to improve outcomes in this group. The REP 
trainer will work with LG providers to identify up to 40 patients who are appropriate for the LG 
program and who have appointments within a short time period of each other (within the first three 
months after the in-service) so that LG groups can be scheduled by the LG provider with minimal 
delay. Although LG providers will be expected to offer LG to 20 patients, identifying 40 ahead of 
time allows an ample number of participants to complete LG in the event that some patients either 
refuse or are ultimately not eligible due to illness or cognitive impairment (see criteria below). In 
prior LG implementation studies (R01 MH79994), medical record reviews were successfully used 
to pre-identify appropriate participants in both primary care and community mental health sites, 
and of those approached, ~10% were ultimately ineligible and ~25% refused to participate in LG.  
Patients who receive Life Goals at the clinic will be asked if they would be willing to participate in 
periodic outcomes assessments involving routine clinical assessments to measure the effect of 
Life Goals on long-term outcomes. 
 
E.1. Patients considered appropriate for LG (i.e., eligible for the study) include: 
1. Adults 21 years or older with a diagnosis and current documentation of antidepressant or 
mood stabilizer for a mood disorder (depression, bipolar disorder) based on medical 
record review; 
2. Not currently enrolled in intensive mental health treatment (e.g., assertive community 
treatment, residential treatment) based on medical record review and confirmation by 
treating clinician; 
3. No terminal illness or cognitive impairment that precludes participation in outpatient 
treatment based on medical record review and confirmation by treating clinician. 
 
 
Table 2: Core Components of Life Goals Program 
Component 
Description 
Group Sessions 
Four sessions lasting 60-80 minutes focused on active discussions around personal goals, 
psychiatric symptoms, stigma, and health behaviors 
Session 1: Personal goals 
Personal goals and self-management; Understanding stigma; Symptoms & wellness 
Session 2: Depressive symptoms (sx) 
Overview, triggers to depressive episodes; Action plan for depression, self-assessment 
Session 3: Anxiety/manic sx 
Overview, triggers to episodes; Action plan: anxiety/mania, self-assessment 
Session 4: Wellness plan 
Building behavior change goals; Relapse prevention and monitoring, medications 
Individualized sessions 
Provider makes 6 weekly individual contacts(15-20 min), encouraging ongoing healthy 
behavior change tied to sx coping strategies, identifying strategies to overcome barriers to 
behavior change, and encouraging ongoing sx and behavior monitoring 
 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
Life Goals (LG) description and operationalization: The evidence for Life Goals was described 
earlier. There are three stages to the Life Goals program: initial contact, group sessions, and 
follow-up contacts/referral (Table 2). First, upon creating a list of patients appropriate for LG, 
providers will review medical records and confirm eligibility (see above). The LG provider will 
make an initial call to the patient, and introduce the LG program and schedule group sessions. 
After the LG provider identifies 6-10 patients, he or she will initiative the first group session, and 
will hold cohorts of group sessions until enrolling 20 participants. Participants can make up 
sessions over the phone or in person if they are unable to make group sessions. In the sessions, 
the LG provider will encourage active discussions that progressively have participants identify 
triggers to the symptom or episode, develop an action plan for identifying warning signs of 
symptoms, and an activity plan for adopting a specific health behavior to mitigate symptoms and 
promote wellness. Participants will be given a workbook with exercises on behavior change goals, 
symptom assessments, and coping strategies. The LG provider will then contact the patient six 
times after the end of the group sessions on a weekly basis to review symptoms, behavior change, 
and any concerns the patient might have. LG providers will also track patients’ mood symptoms 
and health behaviors using a registry. Once participants complete the LG program, a one-page 
summary of their current mood symptoms and health behavior changes will be routed to their 
primary care or mental health provider for treatment planning. As in previous studies, LG providers 
will be trained to inform providers immediately if patients have elevated symptoms or suicidal 
ideation. This is incorporated into the clinic’s routine clinical care.  Study staff (outside outcomes 
assessors) will complete outcomes assessments on up to 20 patients who are receiving Life 
Goals at the clinic in order to measure the program’s reach over time. 
 
Step 2: REP LG training and provider competency: LG providers will undergo a one-day training 
program developed by study investigators that has been provided to over 100 clinicians nationally. 
The LG trainer will first provide an orientation to the evidence behind LG and core elements, then 
a step-by-step walk-through of LG components. The trainer will then demonstrate each LG group 
session and follow-up contact procedures, as well as review options for adapting LG to sites 
without compromising core components (Table 2). After each demonstration, LG providers will 
break into groups and practice each component. The trainer will then go over record keeping in 
the LG registry log and how to market groups to patients and other providers.  
 
Step 3: REP LG tracking - monitor uptake of LG: The final phase of REP consists of a standard 
monitoring form sent by the trainer in which each LG provider reports on the number of patients 
approached and number receiving each LG session. Monitoring forms will be used to assess non-
response across sites and corroborated based on patient self-report of LG use. 
 
REP+EF: External Facilitators (EFs) are part of the study team and reside outside the clinic and 
provide technical expertise to providers in adapting EBPs to address organizational and financial 
barriers. There will be one centralized external facilitator on the study team who has previously 
provided technical expertise in implementing LG. The EF’s core functions include monthly 
technical assistance calls and dissemination of additional materials on LG based on the site 
needs. The EF will initially contact the LG provider and set measurable objectives in implementing 
LG (e.g., number of patients completing at least one group session), and review progress based 
on these measures via monthly calls for six months. Based on the previous implementation trial 
(R01 MH 79994), the monthly calls will last approximately one hour and be used to discuss 
barriers to LG implementation and specific guidance on implementing LG components.  
 
REP+EF/IF: In contrast to EFs, Internal Facilitators (IFs) (Table 1) reside within each site and 
have an internal working knowledge of the site. IFs are responsible for enhancing the uptake of 
LG, notably by incorporating input to support the adoption of the EBP from frontline providers and 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
aligning the EBP with organizational priorities via a direct reporting line with leadership. The IF 
will be identified at the time of randomization to EF/IF and will be a clinician with direct reporting 
authority to the clinical director (or be the clinic director himself or herself). The IF will work to 
build relationships with site personnel and leaders during the six-month period. Using their 
knowledge of the local site’s culture and needs, the IF will identify the site’s priorities, align the 
goals of implementing LG with these priorities, and identify other provider champions to assist in 
implementing LG. The EF will continue to coach the IF on how to monitor uptake to LG 
components, and continue to contact the IF and LG provider via monthly one-hour conference 
calls to review progress based on the established site goals as well as mitigate barriers to 
implementation. 
 
Ensuring Fidelity to REP, EF, IF: Fidelity monitoring will be used to assess whether each site is 
receiving the core components of each implementation strategy (REP, EF, and IF), and to ensure 
that there is no contamination. Data from LG provider logs and REP technical assistance and 
EF/IF activities completed by study staff will be used to ascertain fidelity within each six-month 
period of implementation exposure using established checklists. All sites will get the same LG 
REP package, and LG training will be conducted by the same study investigators who will hold 
regional trainings in each state. As with ROCC (R01 MH79994), EFs will be trained by study 
investigators based on a two-day training program developed for national roll-out of both the REP 
and EF/IF programs. To avoid potential contamination across EFs and IFs, a separate training for 
IFs will be held in each state once randomization occurs in months 6 and 12. REP study staff, 
EFs, or IFs with suboptimal fidelity (described below) at 3 months into each study wave will 
undergo additional training by study investigators. Fidelity to REP is defined based on number of 
sites receiving an LG package, number of providers completing the one-day LG training program, 
and number of completed monthly LG logs. Fidelity to EF will be defined based on whether the 
EF developed specific measurable goals to LG uptake and number of completed out of the six 
monthly technical assistance calls to the LG providers at each site. Fidelity to IF is defined as 
establishment of measurable goals to LG uptake, number of meetings with leadership by IF and 
EF, documentation of LG barriers and strategies to mitigate barriers, and development of a 
strategic plan to implement LG. Sites not providing reporting sheets will be visited to assess 
implementation and further training will be offered.  
 
Mitigating contamination between implementation components: The possibility that EFs might 
inadvertently provide IF components is minimized given that EFs are study staff external to the 
site, while IFs are site employees. Nonetheless, there is a chance that EFs in the EF only group 
might inadvertently provide internal facilitation (e.g., contact providers or leaders about LG 
implementation, which is a core component of IF). We will monitor for potential contamination by 
assessing whether the EF makes any contacts to site personnel other than LG provider at EF 
only sites and provide additional training to EFs if necessary. 
 
F. Data Collection and Measures  
Data for primary and secondary aims will be collected by study staff from patient surveys, 
organizational surveys, and provider logs. The assessment package previously developed and 
implemented by study investigators (R34 MH74509, R01 MH79994) was informed by the RE-AIM 
framework for evaluating implementation of EBPs. Key measures include patient-level outcomes, 
LG uptake and fidelity, organizational factors, and REP, EF, and IF activities (Table 3). Data 
sources include patient-level (surveys), site-level (provider surveys), and project staff logs. 
 
Patient data: Lists of patients and contact information that were identified by the LG providers 
during the first 3 months of REP (pre-randomization) will be sent to study outcomes assessors 
who will conduct phone surveys at baseline (after the REP provider training is complete), and 6, 
12, 18, and 24 months later. Patient baseline assessments will include assessments already used 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
in routine clinical care and will be conducted by project staff by phone after the LG provider 
identifies eligible patients but before the first LG sessions are implemented. Once project staff 
receive patient lists, they will contact patients, describe the purpose of the study, and ask if the 
patients would like to participate.  Patients will be compensated $10 for completing each 
assessment. Patient-level data will be coded with a unique identifier. There will be a crosswalk 
file that it password protected that is only available to be access by study staff that will link unique 
study ID with a link to the person’s identity. 
 
Table 3: Primary and Secondary Assessments – Baseline and Follow-ups 
Aims 
Measure 
Source 
1a. Primary outcomes 
Health-related quality of life-SF-12 mental health score 
Patient survey 
1b. Secondary outcomes 
Functional impairment-WHO-DAS 
Patient survey 
 
Psychiatric symptoms-PHQ-9, GAD-7, ISS 
Patient survey 
 
LG fidelity and uptake (# sessions, contacts) 
LG provider logs, patient surveys 
Secondary aims 
LG implementation costs (patient, site perspective) 
Patient survey, chart review, logs 
 
REP costs 
Study staff logs 
 
Facilitation costs 
Staff, LG provider, Facilitator logs 
Covariates 
Patient demographic, clinical factors, behaviors use 
Patient survey, chart review, logs 
 
Organizational factors (OTM, ORCA, EBPAS) 
Site leader, provider surveys 
 
Patient outcomes will be ascertained from a brief, previously established survey used in routine 
care settings that includes health-related quality of life (the primary outcome, SF-12), functional 
impairment (World Health Organization Disability Assessment Scale- WHO-DAS), and mental 
health symptoms (secondary outcomes), including the PHQ-9, GAD-7 for anxiety symptoms, and 
the Internal State Scale for manic symptoms. Additional questions on LG use will also be included 
to provide confirmatory information on LG fidelity and uptake (see description of LG fidelity 
measures below). 
 
Patient demographic, clinical, and use data will be ascertained from the survey including 
race/ethnicity, age, marital status, education, employment status, social support, incarceration 
history, homelessness, adherence, and substance use. Utilization, including inpatient, outpatient, 
as well as data on medical and psychiatric comorbidities will be ascertained from the patient 
assessment. 
 
 
Provider surveys: Provider surveys will be used to ascertain information on organizational 
factors.  These organizational surveys will be used to inform activities designed to implement 
promising ways to improve clinical care and health care operations.  The site clinical leader or 
administrator as well as the LG provider and the IF from sites randomized the REP+EF/IF will 
complete surveys on organizational factors that might impact implementation and outcomes, and 
inform the identification of more or less responsive sites for future adaptive implementation work. 
First, site leaders will be contacted prior to the initiation of the REP in-service to complete an 
organizational survey previously developed by study investigators (with a >90% completion rate) 
at the beginning of the study. This survey is used to determine organizational characteristics that 
might potentially explain variation in EBP uptake, including resources, staff turnover, and degree 
of integrated primary and mental health care. 
 
Additional organizational factors based on surveys completed by the site clinical leaders, LG 
provider and IFs will be assessed prior to the initiation of REP, then again at 12 and 24 months 
later to assess changes in organizational features that are potentially impacted by REP and 
Facilitation.  
 
LG fidelity: Because this study is designed to assess real-world implementation of routine clinical 
care, minimally invasive measures developed by study investigators will be used to assess use 
of LG services. LG providers will complete a computerized log of LG encounters, which will 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
ascertain the number of completed group sessions and provider follow-up contacts received by 
each patient. LG fidelity will also be assessed using a previously established measure that 
combines information from the LG provider logs with confirmatory information from patient 
surveys. The fidelity monitor calculates a total score based on number of group sessions (0-4) 
and follow-up contacts (0-6) completed by each patient. Average patient completion of group 
sessions of >=75% was associated with improved mental health-related quality of life. 
 
Table 4: Recording Time Spent on REP, Facilitation, and LG 
REP (all sites) 
LG provider and site representative time spent at in-service, training 
LG provider time spent with LG trainer on phone follow-ups, documentation 
Project staff (investigators, personnel): Packaging LG, training, site contacts, note-taking 
EF (REP+EF sites) 
LG provider time spent with External Facilitators (phone contacts, email follow-ups)  
LG provider actions taken to facilitate LG, documentation 
Project staff (EF): Time spent with provider contacts, follow-up technical assistance, documentation of goals with 
each site 
IF (REP+EF/IF sites) 
Time IF spent with EF, LG providers in meetings and establishing goals, site priorities, develop rapport 
Time IF spent working out LG barriers, strategizing and aligning LG goals with site goals 
Time IF spends with  EF and LG provider in summarizing implementation progress with leadership, business plan 
development 
LG Implementation (all sites) 
Group sessions: Number of outpatient clinic visits w/ LG patients actually completed 
Follow-up contacts: Time spent on each contact 
Other Face-Time: Any time spent seeing LG patients face-to-face or on phone outside of sessions/contacts 
Phone Time: Any time spent on the phone only with LG patients. Include time spent leaving messages, playing 
“phone tag”, etc.  
Care Coordination: Any and all time spent on behalf of a LG patient when patient is not present (e.g., phone, 
consultation) 
Charting Time: Any time spent charting for a LG patient (e.g., registry, progress notes, updating treatment plans, 
etc.) 
 
Implementation Strategy (REP, EF/IF) Use and Costs (Table 4): Personnel time is likely to 
account for the vast majority of costs associated with REP and Facilitation. All activities involving 
REP, EF, and IF will be documented using previously established log forms (e.g., R01 MH 79994) 
and all costs will be multiplied by personnel wage rates including fringe. We will estimate time 
costs incurred by both site employees as well as project staff (REP trainer, Facilitators). We chose 
to focus on a cost analysis from the perspective of the sites because the added cost of paying for 
and providing Internal Facilitation will be the major concern of these stakeholders, who are the 
target audience for potentially spreading value-based implementation frameworks  
 
REP Costs from the site’s perspective: We will estimate the costs of the REP implementation from 
the site’s perspective by collecting information on the total time LG providers spent on 1) in-
services, 2) training, and 3) monitoring by surveying the providers involved in each step based on 
the LG registry log. 
REP and External Facilitation costs: Total costs of REP and EF implementation will be estimated 
based on study staff logs by summing up the total time spent by the study staff in LG package 
dissemination, in-services, training, and follow-up site calls and documentation.  
Internal Facilitation Costs: IF costs are primarily borne by the site and are detailed in Table 4. 
Each IF will complete a log of activities using similar procedures in ROCC. The total time spent in 
IF activities will be multiplied by personnel wage rates. 
LG Costs from the site’s perspective will also be estimated for each site. Study staff will ascertain 
information from the LG provider registry on number and time for each group session, follow-up 
contact, as well as for additional provider and patient encounters and clinical documentation 

Full NIH Title: Improving Mental Health Outcomes: Building an Adaptive Implementation Strategy 
Protocol # HUM00065539 
Protocol version: v3 7.8.14 
PI: Amy Kilbourne, PhD, MPH 
 
(Table 4). 
 
 VI. Analyses and Evaluation 
Intent-to-treat. Every effort will be made to collect patient level outcomes across all 80 sites and 
patients (N=1600) to conduct the primary and exploratory analyses (see below). The 40 sites 
randomized at month 6 (total 800 patients) will be included in the intent-to-treat randomized 
sample. The Primary Aim analysis will compare strategies in non-responding sites (k=40) 
beginning with REP+EF/IF versus strategies beginning with REP+EF on longitudinal change in 
patient-level SF-12 mental health scores (MCS- primary outcome). This analysis is a comparison 
of cells A+B+C vs. D+E (Figure 2). For this analysis, the longitudinal outcome is SF-12 mental 
health scores measured at Months 6 (pre-randomization), 12, 18, and 24. The primary contrast is 
the between groups difference in change from Month 6 to Month 18. The follow-up contrast at 
Month 24 will also be examined in this and all subsequent analyses. Linear mixed models (LMM), 
also known as random effects models, fit with SAS PROC MIXED will be used to analyze the 
longitudinal data. LMMs use all available measurements, allowing subjects to have an unequal 
number of longitudinal observations and producing unbiased parameter estimates as long as 
unobserved values are missing at random. The analysis will fit a 3-level (repeated measures for 
each person clustered within site) LMM with fixed effects for the intercept, time, group, and a 
group-by-time interaction term, where group is an indicator of REP+EF/IF vs. REP+EF. The LMM 
will also include random effects for site and time, an unstructured within-person correlation 
structure for the residual errors, and it will adjust for the following pre-randomization measures 
collected through Month 6: state and type of practice (primary care- or mental health site). Model 
diagnostics will be used to determine the suitability of more parsimonious (e.g., autoregressive) 
correlation structures, and nonlinear (e.g., piecewise linear or quadratic) effects for time. LMMs 
similar to the above for the primary outcome will be conducted for the secondary outcomes: 
REP+EF/IF vs. REP+EF on change in number of LG encounters, functional impairment, and 
psychiatric symptoms.  
 
The goal of the Secondary Aim 1 analysis is to determine whether continuing REP+EF versus 
augmenting with REP+EF/IF leads to changes in outcomes, among sites who are non-responsive 
at Month 12. This analysis is a comparison of cells B vs. C (Figure 2). Both primary and secondary 
longitudinal outcomes above will be examined using an LMM similar to that described above, 
except (a) including only the subset of sites that do not respond at Month 12 to REP+EF, (b) using 
monthly longitudinal outcomes from Month 12 to Month 24, and (c) the LMM will use dummy-
indicators for time (i.e., time-saturated model since there are only 3 measurement times for each 
longitudinal outcome).  
 
The Secondary Aim 2 analysis will involve an analysis of implementation costs. The 
implementation (REP, EF, IF) costs will be summed for each site as a measure of total costs and 
changes in costs will also be assessed over time (6, 12, 18, 24 months). An exploratory analysis 
of the incremental effectiveness of IF by comparing costs from sites given REP+EF/IF versus 
those with REP+EF will be conducted from the site’s perspective. In regression models of costs, 
Generalized Linear Models with log link functions will be used to correct for heteroscedasticity 
and reduce the impact of outliers. Effectiveness will be measured by changes in health utilities 
using the SF-6 (a subset of SF-12 questions) as well as the Euro-QOL 5D. Cost-effectiveness will 
be calculated as the ratio of incremental average site-level cost and incremental average 
effectiveness (change in patient-level utilities) of IF. A standard nonparametric bootstrapping 
approach will be used to calculate confidence intervals and cost-effectiveness acceptability 
curves.