Evaluation of Implementing FLOW: Transitioning Stabilized Mental Health Patients to Management in Primary Care (PEC 19-302)
Phase: NA
Conditions: Mental Health
Sponsor: VA Office of Research and Development

 
 
 
Study Protocol and Statistical Analysis Plan 
 
Evaluation of Implementing FLOW in VISN 19:  
Transitioning Stabilized Mental Health Patients to Primary Care 
 
PEX 19-004 
8/15/2023 
 
 
 

Specific Aims 
Adequate access to mental health treatment is one of the highest priority problems facing the Veterans 
Health Administration (VHA) and VISN 19. Chronic staffing shortages exist for both psychologists and 
psychiatrists.1,2 Our clinical partner, VISN 19, notes that access to mental health clinics impact their 
performance on SAIL access and quality metrics. Poor access to care can contribute to suicide and disability, 
and long wait times contribute to patient dissatisfaction and disengagement from care.3,4,5 
Common mental health conditions are expected to remit in many cases6 or to be substantially improved 
with treatment7 such that patients no longer need intensive specialty mental health services. International 
clinical guidelines encourage the use of stepped care, in which patients should be treated at the least intensive 
level of care that is appropriate to their condition8,9 and VA has embraced stepped care.10 As part of this, VA 
has rolled out primary care mental health integration (PCMHI) programs at every VA.11 Primary care, in 
partnership with PCHMI, provides access to maintenance treatment (e.g., ongoing psychotropic prescriptions). 
VA mental health manuals encourage the transition from specialty mental health to primary care for patients 
whose mental health has stabilized and can be addressed in the primary care.11,12 Unfortunately, VA manuals 
do not provide clear guidance on which patients are appropriate for transition back to primary care or how the 
transition should take place.11 In the absence of guidance, few patients are appropriately transitioned and 
specialty mental health panels remain full, contributing to access problems.  
To address these issues, our team partnered with VISNs 16 & 17 to pilot a demonstration project called 
FLOW. This program consists of an algorithm to identify patients who are potentially appropriate for 
transition, a user-friendly online report to communicate this information to providers, materials to 
explain this process to patients and providers, and an electronic medical record (EMR) note template 
to document care transitions. In cases where providers believe continued specialty mental health treatment 
is warranted or patients object to the transition, patients remain in specialty mental health. This approach has 
high acceptability to remitted patients because of perception of less stigma and reduction in appointment 
burden when care is managed in primary care.13 The FLOW program fits with VA national priorities to ‘focus 
resources more efficiently’ and ‘improve timeliness of services.’ To implement FLOW, external facilitators work 
with site-based internal facilitators and providers in specialty mental health and primary care using evidence-
based implementation facilitation strategies and feeding back data to individual sites to guide implementation. 
FLOW was first implemented at 4 sites in VISNs 16 and 17 and is now being implemented at all VISN 17 
sites. Data from the first site indicates that 424 unique patients transitioned from specialty mental health to 
primary care and of those, only nine (2.1%) returned to MH after that transition, suggesting that the majority of 
transfers were successful. Implementation success differed dramatically across the original pilot sites but data 
collected did not offer an explanation for this or allow examination of implementation factors associated with 
success. To maximize timely access for new mental health patients, VISN 19 is embracing the FLOW program. 
Given that VACO is moving towards implementing this program nationwide (personal communication 
from Dr. Marsden McQuire in OMHSP and Dr. Timothy Dresselhaus in Office of Primary Care), it is 
essential to examine the impact of this program on VA patients and clinics and understand the factors that 
facilitate implementation success. We are partnering with VISN 19 to evaluate this program using a cluster-
randomized stepped wedge design with 9 sites randomly allocated to receive the intervention in one of 
3 steps. This provides an ideal opportunity to conduct rigorous evaluation of FLOW’s clinical impact and the 
implementation factors affecting success at each site.  
Specific aims for this proposal, co-developed with VISN 19, are: 
1. To evaluate the impact of FLOW on patient care, relative to standard care, using the evaluation framework 
RE-AIM, including: 
a. 
Reach of the program: % of clinic patients transitioned to PC  
b. 
Effectiveness: % of patients with successful transition to primary care and impact on clinic access and 
provider time for incoming SMH patients 
c. 
Adoption: % of providers in the selected clinics transitioning patients to primary care 
d. 
Implementation: use of all FLOW components, based upon the implementation checklist (see 
Appendix) 
e. 
Maintenance: sustainment of FLOW after withdrawal of external facilitation 
2. To evaluate the relationships between successful FLOW implementation and implementation factors as 
aligned with the Consolidated Framework for Implementation Research model (CFIR)14, including 
intervention characteristics, inner and outer context (e.g., staffing, resources, readiness to change), 
characteristics of individuals (e.g., leadership, facilitators and providers), and the process of implementation 
(e.g., the process of implementation facilitation). 
 

A. Background and Significance  
Access is a priority problem. Improving access and timeliness of services is one of the five priorities in 
the 2018 VA Strategic Plan. Our clinical partner, VISN 19, notes that access to mental health impacts their 
performance on SAIL access and quality metrics. Unfortunately, chronic staffing shortages exist in specialty 
mental health (SMH)1,2 and inefficiencies and poor coordination of care combine to reduce mental health 
access and impact wait times and frequency of visits. Poor access to care contributes to suicide and disability, 
and long wait times lead to patient dissatisfaction and disengagement from care.3,4,5  
To make the most efficient use of existing SMH providers, patients who are improved or remitted should 
be transitioned from intensive SMH treatment to a less intensive setting, primary care (PC). For our purposes, 
transition to PC is defined as an intentional, planful discharge from SMH and transfer of any prescription and 
monitoring responsibilities to the PCP, when the patient has sufficiently recovered such that they no longer 
need intensive SMH care. This will increase availability in specialty providers’ panels and provide faster access 
for new patients referred to SMH. This approach also has high acceptability to remitted patients because of 
perception of less stigma and reduction in appointment burden in PC.13 This creates little burden on PC 
physicians to renew medications of patients already in their panel. VISN 19 notes that their SMH providers face 
barriers to transitioning patients, including lack of guidance on whom to transition, a method to identify 
candidates, a lack of processes and incentives to support this practice, and lack of coordination across clinics. 
Clinical Guidelines Encourage Recovery and Transition out of Mental Health. Many mental health 
conditions are expected to remit.6 Others may be dramatically improved with evidence-based prescribing and 
psychotherapy,7 such that patients no longer need intensive SMH services. International clinical guidelines and 
VA practice guidelines encourage stepped care, in which patients are treated at the least intensive level of care  
appropriate.8,9,13 VA has embraced stepped care10 and rolled out primary care mental health integration 
(PCMHI) programs at every VA.11 Primary care, in partnership with PCHMI, provides access to maintenance 
treatment (e.g., ongoing psychotropic prescriptions). VA mental health manuals encourage the transition from 
SMH back to PC for patients who have stabilized and can be addressed in the primary care setting.11,12 Having 
a mechanism for transition back to PC is consistent with the VA and SAMHSA-supported Recovery 
Model,15,16,17,18 which facilitates recovery rather than continued dependence upon treatment. In contrast to this 
guidance, however, lack of a way to ‘graduate’ successfully sends patients the message that continued 
dependence upon SMH is expected. 
 
 
 
Recently, VA’s Office of Mental Health and Suicide Prevention (OMHSP) has embraced the Mental 
Health Continuum of Care Model (CoC; Figure 1).19 It incorporates stepped care, the recovery model, and 
measurement-based care to determine when to step up or step down in care intensity. According to OMHSP’s 
Continuum of Care workgroup report, “Within stepped care, the basic tenet is to start with the least resource-
intensive, yet likely to be effective, treatments, only 'stepping up' to more intensive/specialist services as 
clinically indicated and ‘stepping down’ once maximum clinical benefit has been obtained from the higher level 
of care.” Further, it indicates that “discussions regarding stepping down should occur when Veterans have met 
their treatment goals or are stabilized as demonstrated by validated assessment of their symptoms and 
functioning.”19 The processes for stepping up care are already well defined in clinical practice guidelines and 
policy manuals, but there is very little to determine when or how to step down. The FLOW program will assist 
VA providers and clinics in making this clinically important decision and implementing processes to facilitate it.  
There is no current solution for the problem of care transitions. In the absence of processes to 
facilitate discharge, patients who recover and no longer need SMH are left with 3 options: 1) stay in SMH and 
continue attending multiple unneeded yearly visits; 2) disengage from SMH by no-showing and discontinuing 
medications without supervision; or 3) ask their SMH provider to transition them to PC. Given the current poor 
coordination, lack of agreements between SMH and PC, and lack of experience with these transitions, option 3 
is unlikely to be easy or seamless. VA Central Office indicated that electronic medical record (EMR) templates 
for transition to PC exist in only a few facilities nationwide and are rarely used.19 
Inpatient
Residential
Specialty Mental Health
General Mental Health
Primary Care Mental Health Integration 
(PCMHI) and PACT
Self-Directed Care
Flow 
Program 
Impacts 
Figure 1. FLOW fits 
with OMHSP’s Mental 
Health Continuum of 
Care Model 

The FLOW program assists facilities with transitions. The FLOW program was created to address 
this need. FLOW consists of EMR-based criteria to identify patients who may be eligible for transition to PC 
and a user-friendly online report to communicate this information to clinicians. The criteria were developed by 
the FLOW team in consultation with stakeholders and a Veteran community advisory board. Current criteria 
are that patients must be taking three or fewer psychotropic medications, not be taking antipsychotics, lithium, 
or a mood stabilizer when bipolar disorder is present, had no new psychotropic medications in the past 6 
months, no psychiatric inpatient or psychiatric ER visits in the past 12 months, and have no suicide risk flag. 
Recent refinements of these criteria added that the patient must not be taking Suboxone and must not be high 
risk for suicide based upon REACHVet score. Because these criteria rely upon information available in 
administrative databases, they may miss subjective signs that the patient is not ready for transition such as 
milder suicidal ideation or near-hospitalization episodes. As such, provider clinical judgment based upon the 
individual patient’s current functioning, strengths, and resources, is the final determination of eligibility. The 
process of transition starts first with identification of potential eligibility via the online FLOW report. SMH 
providers review the list and determine which patients they believe are clinically appropriate, and discuss this 
with the patient at their next appointment to determine if the patient feels ready to transition. If yes, the SMH 
provider contacts the PCP to facilitate transition. If a patient is reluctant to transition, the SMH provider may 
make a plan for what level of functioning would indicate recovery sufficient to transition. FLOW materials 
discuss how measurement-based care can be used to guide decisions about transition and education 
materials for providers in SMH and PC about how to have therapeutic conversations with patients about 
transition. It also contains education materials for patients about the transition, how to address service-
connection concerns, what to do if they have a resurgence in symptoms, and an EMR template to document 
transitions and facilitate communication to PCPs. 
Implementation of FLOW begins with weekly meetings between the external facilitator, Dr. Smith, and the 
identified local champion at a site. Local champions are selected by sites based upon a list of criteria including 
interpersonal characteristics (e.g., strong communicator) and role and tenure at an organization.  The FLOW 
external facilitator meets with the local champion weekly to feed back data on transitions, provider adoption, 
transition success, process data, and to problem solve barriers that arise. VISN 19 believes that having an 
official channel with processes, coordination, and support for these transitions will enhance the Veteran’s 
experience with transition to PC and encourage a focus on recovery, all while increasing access for future 
patient referrals to SMH (personal communication, Dr. Herbert Nagamoto, 10/5/18). 
FLOW is effective in pilot programs. We piloted FLOW in VISN 16 and 17. The process began with 
meetings with leadership and provider stakeholders. Formative qualitative interviews were conducted with 
providers in SMH, PCMHI, and PC before implementation. Qualitative data indicated that providers from all 
settings noted several strengths, including an increase in access to care for newly referred Veterans, less 
appointment burden on recovered patients, and signifying to recovered patients that they were better. 
Providers were concerned that PC could not offer the same level of care as SMH and that some Veterans may 
be reluctant to transition due to concerns about symptom reoccurrence, loss of service connection, or alliance 
with their SMH provider. To address these concerns, provider and patient brochures were altered where 
necessary to allay concerns. After multiple meetings with VISN and local facility operations partners, providers 
in the various clinics, and stakeholders, FLOW was implemented.  
Evaluation Framework. Guiding the evaluation of FLOW was the Reach, Effectiveness, Adoption, 
Implementation, Maintenance framework (RE-AIM).20 RE-AIM provides a way to assess outcomes in five 
facets: reach, or % of the population affected by a program; effectiveness, or the impact of the program upon 
the patients and clinics; adoption, or the proportion of providers who use the program; implementation, whether 
the program was implemented as intended; and maintenance of the program over time after implementation.  
FLOW increases access. During implementation, 1,566 patients in participating clinics were identified as 
potentially meeting criteria for transition and 411 unique recovered MH patients transitioned from MH to PC. Of 
those, only nine (2%) returned to MH for care after the transition. The patients transitioned represented 16% of 
all unique patients (n=2,504) treated in the clinics implementing FLOW, suggesting that FLOW had a 
significant impact on workflow and access. After our team left the site, they continued using FLOW to transition 
a further 205 patients and independently implemented the program at a nearby CBOC. Initial data suggests 
that FLOW increased access and timeliness for new patients to obtain a SMH appointment (Figure 2). This 
increase was not due to an increase in mental health staffing, as MH FTEE per 1000 patients decreased 
during the initial implementation of FLOW in 2017. At 12-months post-implementation, qualitative data 
indicated that providers felt FLOW increased access for patients, increased communication among providers, 
and for SMH providers, decreased their own stress related to workload, although they reported that sometimes 

the transition process was not as smooth as they would have liked. Qualitative patient interviews suggested 
that many patients felt ready for transition, although some were surprised by being asked to transition and 
recommended that patients be prepared earlier, which is incorporated in FLOW. 
 
Figure 2. % patients who received a SMH appointment within 7 days of the preferred date before and after 
implementation of FLOW at site 1, compared with MH FTEE. 
 
  
 
Other sites. FLOW is being piloted at 3 other sites in VISNs 16 and 17. Initial data suggest substantial 
variability in uptake of FLOW. Sites 2 and 3 have transitioned 7 and 8% of unique MH patients. At site 4, 
implementation is paused due to inability to enact project tasks. Substantial variation also exists in provider 
adoption (100% and 92% at sites 1 and 2, 33% at site 3). Sites differ significantly in readiness to implement 
and more work is needed to understand key barriers and facilitators to refine implementation methods. 
Need for further evaluation.  FLOW has recently been disseminated to other V17 sites through a 
passive implementation strategy--providing sites with FLOW materials and a monthly consultation call. Of the 5 
sites receiving the passive implementation strategy, 3 sites have transitioned 0 or 1 patient in the year since 
receiving FLOW materials, 1 site has had minimal success, whereas 1 site has achieved rates roughly similar 
to that in the official pilot. Given the wide variation in program success and evidence that passive 
implementation is generally insufficient, it is clear that successful implementation depends heavily on context, 
local implementation efforts, and internal facilitators. As VACO is moving towards implementing this program 
nationwide, it is essential to examine the impact of this program on patients and clinics and understand what 
facilitates implementation success.  Further evaluation in a new VISN-wide setting will a) indicate effectiveness 
in a larger scale evaluation effort and b) allow us to understand which implementation factors are influential 
and provide insights that can shape the implementation strategies used in a later national rollout.  
With approximately 1.8 million Veterans receiving specialty mental health care nationally (Clinic Practice 
Management Dashboard [CPMD], data extracted 4/18/19), a national rollout of FLOW has the potential to 
dramatically increase access. If sites achieve the same transition rate as in the pilot (7%), an estimated 
126,000 to 288,888 patients would transition. Given that the average SMH patient has 7.5 SMH visits per year 
(including assessments, medication management, therapy, groups, and social work visits (CPMD, data 
extracted 4/18/19), this translates into nearly 1 to 1.4 million freed up appointment slots in SMH per year. 
 
B. Research Overlap  
We searched the suggested websites using the search terms ‘mental health’ and ‘access,’ ‘transition,’ 
‘discharge,’ ‘stepped care’ and ‘coordination of care’ and found no similar studies. Existing studies of care 
transitions focus on transition into MH treatment, or from inpatient treatment to outpatient SMH. VACO 
indicated that no similar program exists and reported that only a few sites nationwide had EMR notes for 
formalizing the transition from SMH to PC but that these were rarely used, had little impact on transitions, and 
that no incentives or guidance exists to facilitate use. Clinical Practice Guidelines discuss the importance of 
using stepped care to intensify treatment but do not discuss when or how to transition to less intensive care.  
 
C. Methods (Evaluation Plan)  
Partnership process. Partners in VISN 19 originally approached the FLOW team after hearing about the 
success of FLOW in VISN 17. Initial champions of FLOW in VISN 19 were Dr. Edward McPhee, Deputy ACOS 
in Eastern Colorado, and Dr. Lisa Smith, Mental Health Director at Rocky Mountain Regional VA. With their 
enthusiasm about FLOW, they connected our team to the VISN 19 Deputy Chief Medical Officer, Dr. Herbert 
Nagamoto, and Dr. Lisa Noe, who oversees mental health programs throughout VISN 19. Through a series of 
calls and emails facilitated by Dr. Nagamoto, we discussed the mental health priorities of VISN 19 and their 
current status on SAIL measures and VISN leadership performance metrics. Dr. Nagamoto assisted us with 
obtaining support of VISN leadership, Dr. Ralph Gigliotti. VISN 19 leadership noted that many of their large 
facilities struggle with access, with many patients waiting until 14 days or more after their desired date to obtain 
an appointment. At their request, we provided them with data on how FLOW impacted these metrics when 
88%
88%
91%
93%
96%
86
91
96
5/2016-10/2016
11/2016-5/2017
5/2017-10/2017
11/2017-5/2017
5/2017-9/2018
FLOW
implemented 
May 1, 2017
FLOW maintence 
phase began May 
1, 2018
MH FTEE per 10,000 patients

implemented at Site 1 and provided data on current access metrics in their VISN for comparison. Although all 
sites expressed interest in participating, 7 sites were able to promise 4 hours per week of protected time for a 
provider to be the internal facilitator at their sites, with two more tentatively committing. Our partners noted that 
they were willing to commit to this because they were certain that the financial and time investment “would be 
returned many times over in freed up FTEE to see new patients in MH” (personal communication, Dr. Herbert 
Nagamoto, 10/5/18). Project aims were developed in collaboration with VISN 19 to align with their priorities, 
and methods were tailored to fit their preferences, such as the use of a stepped wedge design. 
At the request of VISN 19 we will also provide a less intensive implementation approach to the remaining 
VISN 19 sites, not included in the 7-9 sites in this project. The other sites will receive a toolkit for implementing 
FLOW and optional monthly calls for support. This may provide supplemental data regarding whether the 
intensive implementation approach at the primary 7-9 sites is necessary, or how many sites receiving the 
scaled down approach are able to implement FLOW. Because the EMR template will be implemented at all 
VISN 19 facilities, we will obtain data on the number of patients transitioned to primary care throughout VISN 
19, regardless of their participation in this program evaluation process. Exploratory, descriptive analyses will 
compare implementation of FLOW at the 7-9 primary sites to the less intensive implementation sites. 
Study Design Overview. A cluster-randomized stepped wedge design21,22 will examine implementation 
of FLOW in VISN 19. We chose a stepped wedge (see Table 1) because VISN 19 requested that all 
participating sites receive the FLOW program, and this design allows balancing of practical considerations with 
statistical considerations and randomization. This design is also appropriate because we expect the intra-class 
correlations in sites to be large. We will use the new CONSORT extension for cluster-randomized stepped-
wedge trials to report methodological considerations and outcomes.23 Sites will be matched on resources and 
staffing ratios and simultaneously randomized to the time in which they begin the intervention (i.e., step 1, 2, or 
3) on the basis of matching. Implementation of the program will be examined in three phases. First, data will be 
collected about the frequency of patient transition during the baseline phase. Next, FLOW will be implemented 
for 1 year with external and internal facilitation, and program evaluation will be guided by RE-AIM (see Table 
1). Finally, there will be a minimum of a 1-year maintenance phase during which only data collection will occur. 
Sites. Seven to nine VAMCs and very large CBOCs (>10,000 unique patients) from VISN 19 will 
participate, depending upon final leadership commitment at that time. VISN 19 is the geographically largest in 
the nation.  
Randomization. Sites will be randomized based upon balancing relevant characteristics across cohorts. 
We propose to match upon: facility complexity, staffing ratios in SMH and PC, and PCMHI penetration rate. We 
will create a composite variable based upon each site’s performance compared to the average participating 
site. A z-score will be created for each selected variable and averaged to create a final composite variable. 
Based upon FY19 data, it appears that the three larger VAMCs would be in the higher resource group whereas 
the smallest VAMC and the 3 very large CBOCs would be in the lower resource group. These calculations will 
be repeated with the newest data when the study commences and the final number of sites is determined. We 
will ensure that each cohort in the stepped wedge has 1-2 higher resource sites and 1-2 lower resource sites. If 
9 sites participate, we will use a tertile split for matching.  A random number generator will be used to assign 
sites to the step in which they receive the intervention. Sites will be randomized to receive FLOW 
implementation immediately in sequence 1 or to wait until steps 2 or 3 (see Table 1 for timeline) but will not be 
informed of their randomization until it is time for the site to begin implementation. 
 
Table 1. Project timeline.  
 
Year 1 
Year 2 
Year 3 
Months 
Q1 
Q2 
Q3 
Q4 
Q1 
Q2 
Q3 
Q4 
Q1 
Q2 
Q3 
Q4 
Startup 
X 
 
 
 
 
 
 
 
 
 
 
 
Cohort 1 (2 sites) 
 
Implementation 
Maintenance 
 
Cohort 2 (3 sites) 
 
 
Implementation 
Maintenance 
 
Cohort 3 (2 sites) 
 
 
 
Implementation 
Maintenance 
 
Assessment timepoints 
 
 
 
 
 
 
 
 
 
 
 
 
Data analysis, papers 
 
 
 
 
 
 
 
 
 
X 
X 
X 
Report to VISN leaders 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
 
Pre-implementation data-collection. Before implementation (i.e., Q1, Q2, or Q3), baseline rates of 
patient transition from SMH to PC will be examined. Some sites currently have an EMR note template for 
discharge or transition out of specialty mental health; others document this as needed in medication 
management session EMR notes. We will use VA administrative databases to identify the pre-FLOW discharge 
rate. We will also collect assessments of implementation factors tapping the five CFIR domains: intervention 

characteristics, outer setting, inner setting, characteristics of individuals, and process.  
Implementation of FLOW. Implementation of FLOW will occur by implementation facilitation, an 
evidence-based strategy to change clinical practice, guided by the VA-based implementation facilitation 
manual.25 Facilitators partner with key stakeholders to plan change strategies, adapt them to the local context, 
and address barriers to implementation. Implementation facilitation is a flexible strategy that has been used 
effectively to create change in VA mental health services.26,27 Our multifaceted implementation strategy will use 
both external facilitation provided by Dr. Smith, who has received training in facilitation from Dr. Kirchner, and 
internal facilitation provided by a local champion at each site.  
As in the pilot, each site will receive FLOW materials and internal and external facilitation, in partnership 
with a local champion. The process starts with meetings with clinic leadership and identification of a local 
champion, typically a SMH provider. Sites will be given a list of ideal Internal Facilitator (IF) characteristics 
(e.g., strong communicator, opinion leader, prior QI or PE experience, well-established in their current position) 
and will nominate a provider who fits these. Next, the FLOW implementation team meets with providers from 
the affected clinics: SMH, PCMHI, and PC, to explain the rationale and procedures and answer any questions 
or concerns. With the input of providers and the local champion, local processes are identified to support 
communication between different clinics about patients transitioning. Sites are given access to the online 
FLOW report identifying patients who meet criteria for transition, provider and patient materials to explain the 
process of transition, and an EMR template with a progress note title to document the transition. The project 
team initially holds weekly meetings with the local champion to troubleshoot implementation, and 
implementation is flexibly differentiated to meet each site’s identified barriers (see Table 4 for information about 
strategies employed by facilitators). Data will fed back to the internal facilitator on a monthly basis, including 
information about both the process of implementation and implementation success. Process data will include 
barriers and facilitators, types of actions or interventions the IF enacted to overcome barriers or leverage 
facilitators, assistance from the FLOW team, and local climate and context information arising that affects 
implementation. Implementation success metrics include the number of patients eligible and transitioned, 
which providers are engaging in FLOW and which providers have yet to transition any patients, etc. The 
internal facilitator is responsible for local implementation activities guided by regular calls with the FLOW team. 
Although the implementation strategies used are more resource intensive than passive implementation, 
they are nevertheless more resource efficient than other mental health program implementation efforts in VA 
such as the evidence-based psychotherapy rollouts28 and thus likely to be scalable for future national 
implementation. The primary ‘cost’ to each facility implementing FLOW is 4 hours per week of one provider’s 
time in the early stages of implementation, with smaller time investment as time goes on. Based upon pilot 
data, this provides a return on investment of freeing up many hours of specialty mental health provider time to 
see new mental health patients and substantial improvements in SAIL access metrics. 
 
Measures/Data Sources 
Outcomes. Program effectiveness outcomes were developed with VISN 19 leaders and aligned with the RE-
AIM framework. VISN 19’s priorities are to 1) increase access to timely mental health services for new patients, 
and 2) ensure that patients who are transitioned back to primary care have a clinically appropriate transition. 
Considering these priorities, we developed the outcomes discussed in detail in Table 2.  
Assessment of implementation factors/determinants and their relationship to FLOW implementation 
success is guided by the consolidated framework for implementation research (CFIR),14 which is composed of 
five major domains: intervention characteristics, outer setting, inner setting, characteristics of the individuals 
involved, and process. Table 3 shows the proposed implementation factors as aligned with CFIR constructs. 
 
Table 2. RE-AIM outcomes measured during implementation and maintenance phases (Specific Aim 1). 
RE-AIM Domain 
Project Outcome 
Measured by: 
Level of 
Analysis 
Timepoint 
Reach 
Patient transition to PC, before and after FLOW 
implementation 
Administrative data 
for EMR template 
Patient 
Baseline; 
quarterly  
Effectiveness 
Successful patient transition to PC, defined as 
PCP renews psychotropic prescriptions and 
patient does not return to SMH within 1 year 
VA administrative 
data for FLOW 
EMR template use 
Patient 
Baseline; 
quarterly in impl. 
phase 
Patient satisfaction with FLOW 
Qual. interviews 
Patient 
Post-transition 
Provider satisfaction with FLOW 
Qual. interviews 
Provider 
Post-
implementation  
Impact on access, measured as (1) gain in 
provider hours available and (2) gain in % of 
CDW Completed 
Appointments 
Provider 
Baseline; 
quarterly 

patients who receive appointments within 7 
days of desired date  
Cube 
Adoption 
Providers who use FLOW for >2 patients 
Providers’ average % of patients engaged 
If turnover exists, new provider use of FLOW  
VA administrative 
data for FLOW 
EMR template use 
Provider 
Quarterly in impl. 
phase 
Implementation Fidelity in implementation of FLOW 
components 
Implementation 
checklist 
Facility 
Monthly 
Maintenance 
Sustainment, including all RE-AI metrics 
Continuing facilitation activities taken over by 
the local facility 
All metrics above 
Assessment of IF 
behavior 
Patient & 
provider 
Quarterly during 
maintenance 
phase 
 
Table 3. Assessment of implementation factors guided by CFIR (Specific Aim 2). 
CFIR Domain 
Construct measured 
Measured by 
Timepoint(s) 
Intervention 
characteristics 
Attitudes about FLOW 
Qualitative interviews 
Baseline; 
Implementation 
Outer setting 
Facility resources 
Facility complexity 
Baseline 
Inner setting 
Readiness to change 
Organizational Readiness for Implementing Care  
Baseline 
Facility staffing 
Staff to patient ratio in SMH; PCMHI penetration rate; PCP mean 
panel size; Turnover in staffing 
All phases 
Facility organization 
Degree with which PCMHI is integrated in PC; Existence of 
active Behavioral Health Interdisciplinary Program teams 
All phases 
Facility climate 
VA Clinical Practice Organizational Survey 
Provider qualitative interviews 
Baseline;  
Implem. phase 
Inter-service clarity of roles Existence of MOU between SMH and PC outlining roles and how 
patient care responsibilities are divided  
All phases 
Characteristics 
of the 
individuals 
Provider support for change Provider qualitative interviews 
Implem. phase 
Leadership support 
Provider survey and qualitative interview data regarding 
leadership support 
Baseline; 
Implem. phase 
Process 
Facilitator behaviors 
% of FLOW meetings attended; IF’s documentation of their 
activities; Actual protected time; Provider interview questions re: 
IF behaviors; Qualitative interviews of Ifs 
Implementation 
phase 
 
Implementation phase data collection. The majority of data during the implementation phase will be 
collected via VA administrative databases. Use of the FLOW EMR template, extracted from administrative 
databases, will provide information on reach (% of patients transitioned pre vs. post FLOW implementation), 
effectiveness (% of patients who receive follow up MH care in PC, based upon filled prescriptions, and % of 
MH patients who return to SMH, based upon consult and appointment data), and adoption (% of providers 
using the template at least three times, average rates of transition by provider). We will also track demographic 
characteristics of transitioned patients to examine whether transition rates differ by gender and ethic/racial 
background. CDW clinical/administrative data will also be used to collect information on the impact of FLOW 
on access by tracking the percentage of all SMH clinic patients who receive an appointment within 0 to 7 days 
of the preferred date. A second measure of effectiveness will be the expected gain in provider time for new 
patient appointments as a consequence of transitioned patients. This will be calculated for each provider as # 
of patients discharged multiplied by the average yearly minutes spent in appointments with a patient. We will 
also examine impact on provider time available for new patients by tracking the number of completed new 
patient appointments by provider from pre-post implementation of FLOW.  
Brief qualitative interviews will assess the satisfaction of Veterans and providers (see interview guides in 
Appendix). Veterans will be recruited via ‘opt out’ letter sent to those who transitioned, stratified by site. Final n 
will be determined by saturation, but we propose 45 Veterans across the seven sites. Veteran qualitative 
interviews assess satisfaction, perception of appropriateness of the transition and the transition process, and 
satisfaction with care post-transition. Veterans will complete the interview via phone and receive compensation 
of $30. Provider qualitative interviews will assess perceptions of clinical appropriateness, effectiveness, 
implementation barriers and facilitators such as leadership support and local champion actions that facilitated 
implementation (see Appendix for draft interview guide).  Providers will be recruited by email and will not be 
compensated. We will interview at least 3 providers at each site (n = 27) to enable understanding of 
differences across sites. All interviews will be audio recorded, transcribed by a professional transcription 
service (see revised budget), and coded by two experienced qualitative coders using rapid qualitative 
analysis.29   
 To assess implementation, the FLOW implementation team, in collaboration with the local champion, will 
continuously update, at least once per month, each site’s FLOW checklist to monitor which aspects of FLOW 

are being implemented with fidelity. Local champions will also be asked to document their time spent in various 
activities (for example, logistical support vs. influencing providers) in order to understand the key ingredients of 
facilitation in the local context.  We will also examine the number of FLOW meetings the local champion 
attended (% of scheduled meetings) and their actual protected time, to understand whether the champion’s 
availability and time influence implementation.  
Maintenance phase data collection. After a 1-year implementation phase, the FLOW team will phase 
out external facilitation and the facility will be encouraged to continue using FLOW locally. Maintenance phase 
data will continue to be collected on all reach, effectiveness, adoption, and implementation outcomes.  
Power analysis. Power calculations were conducted for the reach and effectiveness outcomes using 
the Steppedwedge command in Stata Version 14.30 Although between 7 and 9 sites will participate, power 
calculations were estimated conservatively, assuming only 6 facilities (clusters) with 2 randomized into each of 
the 3 steps of the predominately open cohort (i.e., different patients at different timepoints) complete design. 
Calculations assume a two-tailed alpha of 0.05. VISN 19 leadership reports that  few patients are currently 
being transitioned from MH to PC. Transition rates are <1% of patients in facilities who do not receive the 
FLOW intervention and between 1-16% of patients in facilities who do receive the FLOW intervention. Given 
an average of 1000 patients per quarter per facility based upon the Clinic Practice Manager data for VISN 19, 
a sample size of 6 clusters (24,000 patients in total) achieves 80% power to detect a difference in transition 
rates from 1% at baseline to 1.65% upon receiving FLOW. Power is >99% to detect a two-fold increase in 
patients who transition from SMH to PC.  Reach, effectiveness, and most maintenance outcomes are binary 
variables at the patient level (e.g., whether one was transitioned to PC). These outcomes represent an open 
cohort trial, as different patients will be included at each timepoint within a given site. Adoption outcomes are 
both binary (e.g., whether a provider engaged in the FLOW process for at least 3 patients) and continuous 
(e.g., % of patients engaged in FLOW for each provider) at the provider level. These outcomes represent a 
closed cohort trial, as the same providers will be included at each timepoint. Adequate effectiveness will be 
declared if at least 90% of transitioned patients successfully transition without return to SMH in the 1 year 
follow-up period.  Given a very conservative estimation that 4% of SMH patients transition from SMH to PC, an 
average of 40 patients per quarter per facility (4% of 1000 = 40), a sample size of 6 clusters (960 transitions in 
total) will allow for 80% power to detect a difference in the percentage of patients who successfully transitioned 
to primary care from 76.50% to 90% upon receiving FLOW.  
Downstream effectiveness, or impact upon access, will be measured as increase in appointments 
within 7 days of requested date from pre-implementation to post-implementation. Data extracted from the Clinic 
Practice Manager suggests a fairly high baseline rate of clinic patients receive appointments within 7 days of 
desired date in VISN 19 (~75 to 80%). Given an average of 1000 patients per quarter per facility, a sample size 
of 6 clusters (24,000 patients in total) achieves 80% power to detect a difference in the percentage of patients 
receiving appointments within 7 days of desired date from 80% at baseline to 83% upon receiving FLOW.  
Power calculations are consistent with ICC values of .10, .15, .20, and .25 and would be similar to what is 
described above for the maintenance phase. 
Quantitative data analysis. We will begin by describing the number of participants, providers, and 
sites (clusters) who were randomly assigned to each sequence and the numbers included at each quarter.  
Intervention conditions will initially be compared using generalized linear mixed models (HGLMs) for binary 
outcomes and hierarchical linear mixed models for continuous outcomes. Each model will include intervention 
(yes/no) and quarter as fixed predictors and models for reach, effectiveness (except for gain in provider hours), 
and maintenance will also include the value of the outcome during the baseline quarter as a fixed predictor.  
Site will be included in all models as a random effect. As we expect the intervention effect may be delayed and 
reach its peak at the end of the implementation period or that the effect may plateau or even decay during the 
maintenance period, we will also conduct a set of models where the treatment by time interaction is included 
as a fixed effect. Furthermore, given the sequences have different lengths of maintenance we will be able to 
examine whether maintenance varies with time since removal of external facilitator. 
Qualitative analysis. We will examine associations between the proposed implementation factors 
(e.g., staffing levels, leadership support) and outcomes. To do so, we will use a mixed-methods comparative 
case study approach,31 a method that has been used to examine determinants of successful implementation of 
evidence-based practices in integrated healthcare systems.32,33 In this method, patterns of similarities and 
differences in relationships between implementation factors and outcomes are examined qualitatively and 
quantitatively across sites. This method is particularly suited for understanding why program implementation 
fails and succeeds at sites and is an appropriate evaluation design choice when it is not feasible to randomize 
sites or to have a large enough number of participating sites to determine statistical associations.34,35 We will 

follow the steps of QCA, including: 1) operationalize the metric of success (example: successful 
implementation of FLOW leading to high reach), 2) identify which sites are classified as high implementation 
success vs. low implementation success, 3) Identify key conditions theoretically affecting implementation 
success (e.g., our implementation determinants), 4) create a matrix of scores signifying whether each site was 
high or low on each determinant, 5) analyze which implementation determinants in the matrix appear to be 
associated with high implementation success, 6) use freely available software (e.g., fsQCA 2.0) to analyze all 
possible configurations of selected implementation determinants, and 7) examine the results and resolve 
contradictions (e.g., if a contradictory result is potentially explained by a third, unmeasured variable or by the 
combination of variables) and re-run analyses if necessary. QCA can appropriately handle mixed methods data 
because the potential implementation determinants included in the analyses can be quantitative data such as 
staffing ratios, or can be qualitative data, such as presence or absence of a theme arising from qualitative 
interviews at each site. Outcomes produced from QCA will be information regarding whether a particular 
implementation determinant or combination of implementation determinants can be used to successfully 
distinguish between sites with high implementation success vs. low implementation success. This method has 
successfully been used for small n implementation studies such as the currently proposed study.32,33 

References 
 
1. Office of Inspector General. (2015). OIG Determination of Veterans Health Administration’s 
Occupational Staffing Shortages. Washington, DC: OIG. 
2. Office of Inspector General. (2017). OIG Determination of Veterans Health Administration’s 
Occupational Staffing Shortages. Washington, DC: OIG.  
3. Tondo, L., Albert, M.J., & Baldessarini, R.J. (2006). Suicide rates in relation to health care access in the 
United States: An ecological study. Journal of Clinical Psychiatry, 67(4), 517-23. 
4. Adolph, M., Wu, J., Feldman, S.R., & Balkrishnan, R. (2012). Who will take care of us? Exploring 
differences in respondents’ satisfaction with primary care vs. specialty care physicians. Health 
Outcomes Research, 3 (1), e3-e10. 
5. Galluci, G., Swartz, W., & Hackerman, F. (2005). Impact of the wait for an initial appointment on the 
rate of kept appointments at a mental health center. Psychiatric Services, 56(3), 344-346. 
https://doi.org/10.1176/appi.ps.56.3.344 
6. Whiteford, H. A., Harris, M.G., McKeon, G., Baxter, A. (2013). Estimating remission from untreated 
major depression: A systematic review and meta-analysis. Psychological Medicine 43(8), 1569-85.  
7. Veterans Health Administration (VHA). (2017). VA/DoD Clinical Practice Guidelines for the 
Management of Posttraumatic Stress Disorder and Acute Stress Disorder. Washington, DC: Author.  
8. Bower P, Gilbody S. (2005). Stepped care in psychological therapies: access, effectiveness and 
efficiency. Narrative literature review. British Journal of Psychiatry, 186, 11–17. doi: 
10.1192/bjp.186.1.11. 
9. National Collaborating Centre for Mental Health. (2009). Depression: the treatment and management of 
depression in adults: NICE clinical guideline. London: National Institute for Health and Clinical 
Excellence. 
10. Rosenberger, P.H. & Philip, Errol & Lee, A & Kerns, Robert. (2011). The VHA's National Pain 
Management Strategy: Implementing the stepped care model. Federal Practitioner, 28, 39-42. 
11. Dundon, M., Dollar, K., Schohn, M., Lantinga, L.J. (2011). Primary Care-Mental Health Integration Co-
Located Collaborative Care: An Operations Manual. Syracuse, NY: Center for Integrated Healthcare.  
12. VHA. (2008). VHA Handbook 1160.01: Uniform Mental Health Services in VA Medical Centers. 
Washington, DC: VHA.  
13. World Health Organization (WHO). (2007). Integrating mental health services into primary health care. 
Geneva: World Health Organization.  
14. Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A., & Lowery, J.C. (2009). 
Fostering implementation of health services research findings into practice: a consolidated framework 
for advancing implementation science. Implementation Science, 4, 50.   
15. Repper, J. & Perkins, R. (2006). Social Inclusion and Recovery: A Model for Mental Health Practice. 
London: Bailliere Tindall.  
16. Sheedy, C.K. & Whitter, M. (2009). Guiding Principles and Elements of Recovery-Oriented Systems of 
Care: What Do We Know from the Research? HHS Publication No. (SMA) 09-4439. Rockville, MD: 
Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration. 
17. SAMHSA. (2012). SAMHSA’s Working Definition of Recovery: 10 Guiding Principles of Recovery. 
Rockville, MD: Substance Abuse and Mental Health Services Administration. 
18. VHA (2011). VHA Handbook 1163.03. Psychosocial rehabilitation and recovery centers (PRRC). 
Washington, DC: VHA.  
19. Office of Mental Health and Suicide Prevention (OMHSP; 2017). Report from the Office of Mental 
Health and Suicide Prevention (OMHSP) Continuum of Care Workgroup. Washington, DC: OMHSP.  
20. Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999). Evaluating the public health impact of health 
promotion interventions: the RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. 
21. The Gambia Hepatitis Study Group, The Gambia Hepatitis Intervention Study, Cancer Research, 
1987;47(21):5782–87. 
22. Brown, C.A. & Lilford, R.J. (2006). The stepped wedge trial design: a systematic review. BMC medical 
research methodology, 6: 54. 
23. Hemming, K., Taljaard, M., McKenzie, J.E., Hooper, R., Copas, A., Thompson, J.A.,…Grimshaw, J.M. 
(2018). Reporting of stepped wedge cluster randomized trials: Extension of the CONSORT 2010 
statement with explanation and elaboration. BMJ, 363: k1614. 
24. Yano, E., Fleming, B., Canelo, I. et al. (2008). National Survey Results for the Primary Care Director 

Module of the VHA Clinical Practice Organizational Survey. Sepulveda, CA: VA HSR&D Center for the 
Study of Healthcare Provider Behavior. 
25. Ritchie MJ, Dollar KM, Miller CJ, Oliver KA, Smith JL, Lindsay JA, Kirchner JE. (2017). Using 
Implementation Facilitation to Improve Care in the Veterans Health Administration (Version 2). 
Veterans Health Administration, Quality Enhancement Research Initiative (QUERI) for Team-Based 
Behavioral Health, 2017. Available at: 
https://www.queri.research.va.gov/tools/implementation/Facilitation-Manual.pdf 
26. Pomerantz, A.S., Kearney, L.K., Wray, L.O., Post, E.P., & McCarthy, J.F. (2014). Mental health 
services in the medical home in the Department of Veterans Affairs: factors for successful integration. 
Psychol Services, 11(3), 243-53. 
27. Ritchie, M.J., Parker, L.E., Edlund, C.N., & Kirchner, J.E. (2017). Using implementation facilitation to 
foster clinical practice quality and adherence to evidence in challenged settings: A qualitative study. 
BMC Health Services Research, 17, 294. 10.1186/s12913-017-2217-0 
28. Karlin, B. E., Brown, G. K., Trockel, M., Cunning, D., Zeiss, A. M., & Taylor, C. B. (2012). National 
dissemination of cognitive behavioral therapy for depression in the department of veterans affairs 
health care system: Therapist and patient-level outcomes. Journal of Consulting and Clinical 
Psychology, 80(5), 707-718.  
29. Beebe, J. (1995). Basic concepts and techniques of rapid appraisal. Human Organization, 54 (1), 42-
51. 
30. Hemming, K., & Girling, A. (2014). A menu-driven facility for power and detectable-difference 
calculations in stepped-wedge cluster-randomized trials. The Stata Journal, 14 (2), 363-380. 
31. Goodrick, D. (2014). Comparative Case Studies: Methodological Briefs - Impact Evaluation No. 9. New 
York, NY: Unicef 
32. Cragun, D., Pal, T., Vadaparampil, S.T., Baldwin, J., Hampel, D., & DeBate, R.D. (2016). Qualitative 
comparative analysis: A hybrid method for identifying factors associated with program effectiveness. 
Journal of Mixed Methods Research, 10(3), 251-272. 
33. Misra-Hebert, A., Perzynski, A., Rothberg, M.B., Fox, J., Mercer, M.B., Liu, X., Hu, B., Aron, D.C., & 
Stange, K.C. (2018). Implementing team-based primary care models: A mixed methods comparative 
case study in a large, integrated health care system. 
34. Chan, C.M.L. & Pan, S.L. (2008). User engagement in e-government systems implementation: A 
comparative case study of two Singaporean e-government initiatives. The Journal of Strategic 
Information Systems, 17 (2), 124-139. 
35. Pascual, C., Escarti, A., Llopis, R., Gutierrez, M., Marin, D., & Wright, P.M. (2011). Implementation 
fidelity of a program designed to promote personal and social responsibility through physical education: 
A comparative case study. Research Quarterly for Exercise and Sport, 82(3), 499-511. 
36. Dogherty, E. J., Harrison, M. B., Baker, C., & Graham, I. D. (2012). Following a natural experiment of 
guideline adaptation and early implementation: a mixed-methods study of facilitation. Implementation 
Science, 7(1), 9. 
37. Kotecha, J., Han, H., Green, M., Russell, G., Martin, M. I., & Birtwhistle, R. (2015). The role of the 
practice facilitators in Ontario primary healthcare quality improvement. BMC Family Practice, 16(1), 93.