FINAL REPORT - Amended

Social Innovation Fund

Healthy Futures / IMPACT Expansion

Subgrantee Evaluation

June 2013 – June 2017
 

SIF IMPACT Expansion ii Final Report: October 2018 

ACKNOWLEDGEMENTS

This publication represents a collaborative effort by the John A. Hartford Foundation and the University of Washington 

AIMS Center. 

This work was funded by the John A. Harford Foundation and the Corporation for National and Community Service 

Social Innovation Fund. 

Principal Authors: 

Diane Powers, MBA, MA 

Joan Russo, PhD 

Rob Arao, MS 

Mindy Vredevoogd, MS 

Jürgen Unützer, MD, MA, MPH 

Recommended citation: 

Powers D, Russo J, Arao R, Vredevoogd MA, , Unützer J. (2017) FINAL REPORT: Social Innovation Fund: Healthy 

Futures/IMPACT Expansion, Subgrantee Evaluation, June 2013 – June 2017. Report presented to the Corporation for 

National and Community Service (CNCS). 

For more information please visit: 

http://aims.uw.edu/

http://uwaims.org/sif

http://www.jhartfound.org/grants-strategy/current-strategies/models-of-care/social-innovation-fund/

The contents of this document are solely the responsibility of the University of Washington AIMS Center and The John 

A. Hartford Foundation and do not necessarily represent the official views of the Corporation for National and 

Community Service (CNCS). 

This document may be reproduced in whole or part without restriction as long as the University of Washington AIMS 

Center and The John A. Hartford Foundation are credited for the work. Upon request, the contents of this document 

will be made available in alternate formats to serve accessibility needs of persons with disabilities. 
 

SIF IMPACT Expansion iii Final Report: December 2017

TABLE OF CONTENTS

EXECUTIVE SUMMARY ........................................................................................................................................ 5

INTRODUCTION .................................................................................................................................................. 7

METHOD ........................................................................................................................................................... 14

ANALYSIS ........................................................................................................................................................... 21

FINDINGS .......................................................................................................................................................... 22

DISCUSSION ...................................................................................................................................................... 38

RECOMMENDATIONS ....................................................................................................................................... 41

REFERENCES.................................................................................................................................................................. 43

APPENDIX.......................................................................................................................................................... 46
 

SIF IMPACT Expansion 4 Final Report: October 2018 

TABLES

Table 1. IMPACT Logic Model 

Table 2. Patient Enrollment 

Table 3. Demographic Profile of Enrolled Patients 

Table 4. Demographic Characteristics of Providers 

Table 5. Training and Employment Characteristics of Providers 

Table 6. Patient Survey Data Collection 

Table 7. Clinical Survey Data Collection 

Table 8. Goals and Expectations: Processes of Care 

Table 9. Processes of Care: Program Exposure and Quality of Program Delivery 

Table 10. Patient Satisfaction 

Table 11. Provider Satisfaction with Patient Outcomes, Support from Clinic 

Table 12. Provider Satisfaction with Quality of Work Life, Level of Burnout 

Table 13. Baseline Depression Symptom Severity 

Table 14. Depression Outcomes: Decrease in Depression Symptoms 

Table 15. Depression Outcomes: Response and Remission Rates by Baseline PHQ-9 Category 

Table 16. Predicting Depression Response by Clinic, Age, Race, Gender, and Treatment Variables 

Table 17. Predicting Depression Remission by Clinic, Age, Race, Gender, and Treatment Variables 

Table 18. Effects of Collaborative Care on Social and Occupational Functioning 

Table 19. Effects of Collaborative Care on Medical Care Utilization 

Table 20. Effects of Collaborative Care on Depression Care Utilization

FIGURES 

Figure 1. Collaborative Care 

Figure 2. Implementation Support Phases 

Figure 3. SIF Implementation Timeline 

Figure 4. Patient Survey Recruitment Report 

Figure 5a/b. Impact of Patient Survey Incentive 

Figure 6. Power, Sample Size and Detectable Difference in PHQ-9 scores initial to 6 month 

Figure 7. Key Components of Collaborative Care 

Figure 8. Stages of Implementation Completion 

Figure 9. Duration of Program Start-up and Time to Competency 
 

SIF IMPACT Expansion 5 Final Report: October 2018 

EXECUTIVE SUMMARY 

Grantee Name

John A. Hartford Foundation (JAHF) 

Subgrantee Name(s)

Cohort I (Awarded May 2013) 

Mat-Su Health Services, Wasilla, AK 

Partnership Health Center, Missoula, MT 

Community Health Center of Central Wyoming (CHCCW), Casper, WY 

Peninsula Community Health Services, Port Orchard, WA 

Valley View Health Center, Chehalis, WA 

Cohort 2 (Awarded May 2014) 

Kodiak Area Native Assoc (KANA), Kodiak, AK 

Bighorn Valley Health Center (BVHC), Hardin, MT 

Southwest Montana Community Health Center (SWMCHC), Butte, MT 

Evaluation Contractor 

University of Washington AIMS (Advancing Integrated Mental Health Solutions) Center 

Program Name 

Healthy Futures/IMPACT Expansion 

Program Synopsis 

Through the Social Innovation Fund (SIF) initiative, the John A. Hartford Foundation (JAHF) is promoting 

expansion of the IMPACT model of depression treatment to improve care in medically underserved rural communities 

in the Washington, Wyoming, Alaska, Montana and Idaho (WWAMI) region. IMPACT is the name of the largest 

research trial demonstrating effectiveness of the Collaborative Care model and these two terms are used 

interchangeably. Collaborative Care is an approach to treating depression in primary care settings that is based on the 

Wagner Chronic Care model. Collaborative Care supports primary care providers (PCP) treating depression with a 

clinic-based care manager and a designated psychiatric consultant who use a measurement-based, treat-to-target 

approach for the population of patients needing care. The long-term effects for participating subgrantee clinics 

implementing IMPACT will be increased access to effective, evidence-based depression treatment and improved social 

and economic functioning among participating patients. The AIMS (Advancing Integrated Mental Health Solutions) 

Center at the University of Washington (http://aims.uw.edu/) provided the training and support to participating clinics 

to help them plan, launch and sustain IMPACT, in addition to evaluating effectiveness of the program. 

Prior Research 

More than 80 randomized controlled research trials have established a robust evidence base for Collaborative 

Care.1-5 The IMPACT study was one of these trials.6 Patients receiving Collaborative Care were more than twice as 

likely as those in usual care to experience a substantial improvement in their depression over 12 months. They also 

had less physical pain,7 better social and physical functioning,8 and better overall quality of life9 than patients in care 

as usual. The IMPACT trial, which included more than 1,800 patients from 18 primary care clinics in five states, 

generated over 50 peer-reviewed research publications. The target population of the IMPACT trial was older adults 

while this implementation initiative served adults of all ages and a small number of teens.

Level of Evidence 

Collaborative Care has an extensive existing evidence base1-9 with diverse populations10-12 and settings.13

Therefore, the SIF expansion of IMPACT to low-income rural communities does not focus on establishing the efficacy 
 

SIF IMPACT Expansion 6 Final Report: October 2018 

of IMPACT compared to care as usual. The evaluation seeks to identify whether implementation in non-profit rural 

community health clinics participating in SIF will result in depression improvements that are similar to benchmarks 

established in published studies.14-15 It also examines variations in the quality and effectiveness of clinic-level 

implementation and explores effects on use of health care services and social and economic functioning among 

participating patients. This evidence is important since it addresses key factors in scaling the program to new 

populations, especially low-income and underserved communities, and adds to the substantial body of research on 

Collaborative Care. This SIF evaluation targets a preliminary level of evidence. 

Implementation and Evaluation Design 

The John A. Hartford Foundation contracted with the AIMS Center to provide training and support to 

participating subgrantee clinics to help them plan, launch and sustain Collaborative Care. Implementation support was 

provided through a variety of means, including onsite coaching, in-person training, webinars, conference calls, and a 

website designed specifically for this initiative (http://uwaims.org/sif). The AIMS Center also conducted an evaluation 

of the IMPACT implementation in subgrantee organizations, including clinical processes of care, patient outcomes and 

economic effects of the program. The AIMS Center’s training and implementation support team was a different group 

of faculty and staff than those who led evaluation of the program. Training was led by Dr. Anna Ratzliff, MD, PhD, in 

her role as the AIMS Center’s Associate Director for Education. Implementation support and technical assistance was 

led by Diane Powers, MA, MBA in her role as the AIMS Center’s Associate Director for Translation & Implementation. 

Evaluation was led by Dr. Jürgen Unützer, MD, MPH, MA, an internationally recognized mental health services 

researcher Director of the AIMS Center, in partnership with Melinda Vredevoogd, MS who was the AIMS Center’s 

Assistant Director for Research and Evaluation. These three units functioned independently but cooperatively to 

achieve the initiative’s goals. (Note: Listed roles reflect the positions these people occupied during the SIF expansion of 

IMPACT not their current roles.) 

Evaluation Questions 

This evaluation seeks to answer five confirmatory questions on program impact and one exploratory 

question. In addition, the evaluation examines implementation process questions. The primary outcome is the effect 

of IMPACT implementation on patient-level depression response and remission among enrolled patients, how this 

compares with published research trials and real-world implementations, and predictors of depression improvement. 

Secondary outcomes include patient-level effects on social functioning, occupational functioning, use of depression 

care, and satisfaction with depression care. Exploratory outcomes include effects on patient’s healthcare utilization. 

Clinic-level implementation outcomes include how well clinics implement practice change, the effect of this on 

evidence-based processes of care, and provider satisfaction with Collaborative Care. 

Measures and Instruments 

The AIMS Center used a variety of measures to evaluate program effectiveness. These included data captured 

from a web-based patient registry used by providers as part of routine clinical care delivery, quantitative and 

qualitative surveys with clinic leadership, a quantitative survey of clinicians, quantitative surveys with a subset of 

patients receiving depression treatment at participating clinics, and data from clinic administrative data systems. The 

Subgrantee Evaluation Plan (SEP) was revised in January 2015 to include the Stages of Implementation Completion 

(SIC) measure to describe variation and key components of the implementation process amongst the subgrantee 

clinics, and, in turn, how these variations might affect evidence-based processes of care16-17 and/or clinical outcomes 

(depression response and remission).18 In February 2016, a revised Subgrantee Evaluation Plan was submitted to add 

a final quantitative survey and phone interview for clinic leadership. The survey was designed to capture feedback 

from care teams regarding their experience with AIMS Center implementation coaching and support so as to identify 

and characterize critical components related to clinic-level implementation. 
 

SIF IMPACT Expansion 7 Final Report: October 2018 

Analysis Approach 

The Care Management Tracking System19 patient registry served as the primary data source. This was 

supplemented by patient, provider, and organizational leadership surveys as well as clinic administrative data. 

Observational analyses examined clinic-level implementation and adherence to evidence-based processes of care as 

well as patient-level intervention outcomes on depression symptom severity. We compared the number of patients 

enrolled across participating clinics, the proportion of patients receiving proactive follow-up, the proportion of 

patients whose care was discussed with the psychiatric consultant, improvement in patients’ depression symptoms, 

changes in social and occupational functioning, medical care and depression care utilization, as well as patient and 

provider satisfaction. A logistic regression model that adjusted for clinic site and all patient demographic variables 

examined variation in the odds of depression response (>50% reduction in depressive symptoms) and depression 

remission (virtual absence of symptoms) among participating subgrantee clinics. Comparisons are made across 

participating sites and are compared with existing benchmarks in the published literature from depression care 

programs implemented in similar populations13 and practice settings.12, 14

Key Findings 









Rural clinics serving low-income communities can implement Collaborative Care, increasing access in to 

depression treatment in underserved areas. 

These clinics can achieve clinic-level processes of care and patient-level clinical outcomes comparable to research 

trials and large implementations. 

Large variation exists among clinics in processes of care and clinical outcomes despite consistency in training and 

implementation support. 

Primary care providers and patients are satisfied with Collaborative Care. 

INTRODUCTION

Implementation Study Context 

Program Synopsis and Delivery Timeline 

Through this Social Innovation Fund (SIF) initiative, The John 

A. Hartford Foundation promoted dissemination of Collaborative 

Care (IMPACT) to improve care in non-profit community health clinics 

in the Washington, Wyoming, Alaska, Montana and Idaho (WWAMI) 

region serving low-income, uninsured and underinsured patients. 

Much of this region is medically underserved and/or a healthcare 

provider shortage area.20-21 Collaborative Care is a team-based model 

of treatment in which the primary care provider (PCP) is supported by a care manager and a psychiatric consultant to 

treat patients with depression in the primary care clinic using existing evidence-based treatments and established 

principles of chronic illness care.6Figure 1 shows a graphical representation of Collaborative Care. The AIMS Center 

provided training, implementation support, and evaluation for this initiative. A detailed timeline of implementation 

and evaluation activities is provided in Appendix A. 

“Every provider, whether it be the 

primary care provider, behavioral 

health provider, or care manager is an 

important piece to the puzzle.” 

– Care Manager at Clinic H
 

SIF IMPACT Expansion 8 Final Report: October 2018 

Figure 1. Collaborative Care Team 

Program Beneficiaries 

The goal of the IMPACT Expansion initiative was 

to disseminate evidence-based Collaborative Care for 

depression to patients of community primary care clinics 

in the WWAMI region, a largely rural and underserved 

area that comprises 27% of the land-mass of the United 

States but contains only 3.3% of the population. While 

the overall poverty rate for the WWAMI region was 12% 

in 2011 and 13% in 2014,22 in each of these states the 

proportion of residents living in poverty is significantly 

higher in rural counties. Prior to 2014, Medicaid 

participation ranged from 13% of the state’s population 

in Montana to 18% in Washington and the prevalence of 

uninsured residents was similar across all five WWAMI 

states at about 16% of the population.23 In all of these 

states the proportion of residents living in poverty, the 

proportion of older adults, and the proportion of ethnic 

minorities is greatest in non-metropolitan areas.24 With 

only a few exceptions representing the largest 

metropolitan areas, the vast majority of the WWAMI 

region is identified by HRSA as medically underserved 

and/or a health professional shortage area and 

community primary care clinics in this region primarily 

serve low-income, uninsured, underinsured and Medicaid patients.25

The target population for this initiative was adults (age>=18) living in rural WWAMI communities. The eight 

subgrantees had to meet the following requirements for funding to meet the initiative’s target population: 

a. Non-profit community primary care organization 

b. Located in a rural WWAMI county or serving a significant proportion of rural dwelling patients 

c. Service area designated by the federal government as medically underserved and/or a health professional 

shortage area 

d. Serve at least 1,500 unique patients each year 

e. Patient population at least 50% uninsured or Medicaid or other proxy for low socio-economic status 

Subgrantees either replicated or expanded Collaborative Care. Expansion subgrantees were those that already 

had a collaborative care program in place for a subset of their patients prior to the Social Innovative Fund (SIF) award 

wanted to expand these services to the remainder of their patient population. Replication subgrantees did not have an 

existing collaborative care program at the time of their SIF award. The two Expansion subgrantees (Clinics D and E) 

implemented collaborative care about five years earlier as part of the Washington State Mental Health Integration 

Program (MHIP)14 but were only able to offer Collaborative Care to a small proportion of their patients due to 

eligibility requirements for that program which was offered by one managed Medicaid plan but no others. MHIP clinics 

also implemented Collaborative Care with minimal training and implementation assistance from the AIMS Center. The 

managed Medicaid plan sponsoring MHIP received implementation support and guidance from the AIMS Center and 

deployed their own practice coaches to the clinics. The AIMS Center offered one-time training, periodic webinars, and 

a psychiatric consultant support call. Clinics in the other sites were all Replication subgrantees. 
 

SIF IMPACT Expansion 9 Final Report: October 2018 

Subgrantees (Cohort 1): received awards in June 2013 











Community Health Center of Central Wyoming (CHCCW) in Casper, Wyoming 

Mat-Su Health Services in Wasilla, Alaska 

Partnership Health Center in Missoula, Montana 

Peninsula Community Health Services in Kitsap County, Washington 

Valley View Health Center in Lewis County, Washington 

Subgrantees (Cohort 2): received awards in May 2014 







Kodiak Area Native Association (KANA), Kodiak, AK 

Bighorn Valley Health Center (BVHC), Hardin, MT 

Southwest Montana Community Health Center (SWMCHC), Butte, MT 

Program Components / Activities 

The AIMS Center provided training and implementation coaching/support 

to participating subgrantees. IMPACT adapts principles of effective chronic illness 

care26 for treatment of common mental health conditions in primary care. These 

principles include measured-based care and treatment to target.27 Systematic 

measurement of clinical outcomes using brief, patient rating scales such as the nine 

item Patient Health Questionnaire (PHQ-9) for depression28 helps clinicians track whether patients are improving as 

expected or if treatment needs to be adjusted. Research has shown that certain scores on the PHQ-9 are strongly 

correlated with a subsequent major depression diagnosis, with PHQ-9 ≥10 indicating a moderate or greater level of 

depression. Adjusting the treatment plan based on symptom measures is one of the most important components of 

Collaborative Care which requires a change in treatment every 10-12 weeks if the patient has not had at least a 50% 

improvement in symptoms as measured with a validated instrument like the PHQ-9. Such systematic treatment to 

target can overcome the clinical inertia often responsible for ineffective treatment of depression in primary care.29

Figure 2. Implementation Support Phases 

Figure 2 shows the implementation process the AIMS Center 

used to work with subgrantees, both individually and collectively, to 

provide training and implementation support. Support activities 

included implementation coaching, webinars, clinic-level trainings 

delivered onsite (at each participating clinic), group in-person training 

(in Seattle), webinars, and a variety of implementation support 

conference calls. Each subgrantee clinic spent 2-3 months completing a 

guided pre-launch implementation planning phase. Then, subgrantees 

gathered together in-person in Seattle for a 2-day pre-launch training 

meeting. This training occurred in September 2013 for Cohort 1. 

Cohort 2 met in September 2014 for their 2-day pre-launch training. 

Cohort 1 attended the second day of the 2014 training for post-launch booster training. Within 2 weeks of the 2-day 

pre-launch training, each subgrantee launched their IMPACT expansion or replication program. Post-launch support 

included intensive skills-based training (certification in Problem-Solving Treatment for care managers and 

psychotherapists offering psychotherapy as part of treatment), webinars and case calls on various clinical topics, as 

“IMPACT has grown my skills at using 

brief therapy, solution-focused and 

problem-solving skills to help clients 

learn how to problem solve and to 

become more independent in their 

healing, taking more control of their 

outcomes, and enabling us to provide 

services to more people.” 

– Psychiatric Consultant at Clinic B

“The effects have been 

much more widespread 

than simply improving care 

of our behavioral health 

patients. I feel like it’s 

benefitted the entire 

organization, made us 

more effective in teams, 

more focused on 

measurement, more 

interested in improving 

quality.” 

– Primary Care Provider at Clinic F
 

SIF IMPACT Expansion 10 Final Report: October 2018 

well as group and individual implementation support conference calls. The timeline presented in Appendix A shows 

pre-launch and post-launch training and support activities was well as implementation coaching activities. Cohort 1 

completed the post-launch implementation support phase at the end of June 2016 and all but one of these subgrantee 

clinics moved into sustainment. Cohort 2 clinics completed the post-launch implementation support phase at the end 

of June 2017. See Figure 3 for a graphical representation of these implementation and sustainment phases. 

Figure 3. Social Innovation Fund Implementation Timeline 

The original award from The John A. Hartford Foundation to the AIMS Center for training, implementation 

support and evaluation was awarded through 12/31/2015, based on the original award timeframe from CNCS. In 

September 2014, the award from CNCS to The John A. Hartford Foundation was extended through August 2017 to 

accommodate three years of funding for the second cohort of clinics. In June 2015, The John A. Hartford Foundation 

approved an augmentation/extension of the AIMS Center award to extend the timeframe for training, implementation 

assistance and evaluation activities through June 2017 with the final Subgrantee Evaluation Report due December 31, 

2017 after completing data analysis. The extended timeline is shown in Appendix A. This extension allowed postlaunch training, implementation support and evaluation data collection to extend through June 2016 for Cohort 1 and 

June 2017 for Cohort 2. 

 

Program Outputs 

Table 1 lists the actions (Outputs) subgrantees completed as part of IMPACT implementation. In brief, these 

included: 





o

o

o

o

o

o

Pre-launch development of a detailed implementation plan outlining personnel needs and organizational and 

workflow changes required to align with Collaborative Care. 

Post-launch IMPACT patient enrollment and treatment, including: 

1) assessing patients for depression, 

2) patient engagement and education, 

3) evidence-based treatments such as medications and/or brief, structured counseling using 

established evidence-based techniques such as Behavioral Activation and Problem-Solving 

Treatment in Primary Care, 

4) proactive follow-up focusing on treatment adherence, treatment effectiveness, and treatment 

side effects, 

5) team communication about processes of care and treatment outcomes, and 

6) regular, systematic review of the entire population of patients engaged in care with a psychiatric 

specialist and adjustment of treatment plans for patients not improving after 10-12 weeks with the 

current treatment. 
 

SIF IMPACT Expansion 11 Final Report: October 2018 

Program Outcomes and Impacts 

As outlined in Table 1, the key anticipated primary outcome for patients treated with IMPACT in participating 

subgrantee clinics is decreased depression symptoms characterized as response and remission. Response is a >50% 

reduction in symptoms while remission is a near elimination of symptoms.18 A single item from the PHQ-9, shown to 

predict suicide attempts30 was used to assess suicidal ideation. Process of care outcomes related to 1) psychiatric 

consultation and 2) receiving at least one contact within the first four weeks of treatment were measured using the 

record of contacts documented in the CC registry. We used CMTS patient registry data to measure this for all patients 

enrolled in Collaborative Care. Registry data was also used to analyze secondary outcomes including: 1) initial clinical 

assessment, 2) patient follow-up, 3) patients engaged in treatment for at least 12 months, and 4) patients with at a 

least one psychiatric consultation. Registry data was supplemented by additional data sources including: 1) patient 

surveys with a subsample of treated patients at each clinic, 2) administrative data from each of the clinics, 3) surveys 

and interviews with leadership from each subgrantee clinic, 4) surveys with clinical providers (primary care providers, 

care managers, psychotherapists, psychiatric consultants) at each clinic, 5) completion of the Stages of 

Implementation Completion (SIC)

31-33measure for each subgrantee clinic, and 6) end of study survey and interview 

with clinic leadership. These supplemental data sources served as the basis for four secondary outcomes: 1) changes 

in social functioning, 2) changes in occupational functioning, 3) use of depression care, and 4) satisfaction with 

Collaborative Care. Combining clinical registry data with quantitative surveys and semi-structured qualitative 

interviews will help us understand and interpret observed variations in implementations and outcomes. This approach 

has been used successfully in the original evaluation of the IMPACT program and several subsequent real-world 

implementations.6,13,15 The SIC measure describes variation in implementation process and whether/how these may 

affect processes of care and/or clinical outcomes.

In addition to measuring key outcomes for treated patients, we explored outcome trends in health care 

utilization specifically examining whether treated patients used health care services differently before and after 

receiving Collaborative Care. In the original IMPACT study, patients receiving Collaborative Care were more than twice 

as likely as those in usual care to experience a substantial improvement in their depression over 12 months.6 They also 

had less physical pain,7 better social and physical functioning,8 and better overall quality of life9 than patients in care 

as usual. IMPACT was strongly endorsed by patients and primary care providers.34 The data source for examining 

health care utilization and economic impact is baseline and six-month surveys collected with a subsample of patients 

enrolled in Collaborative Care at participating subgrantee clinics (n=351 paired surveys). The patient survey included 

questions on healthcare utilization and economic outcomes. 

The implementation and effectiveness of Collaborative Care in rural, low-income, underserved areas is an 

important question. Rural communities face the “twin burden” of higher need and lower resources.35 As a result, 

residents of rural and frontier areas experience a variety of disparities as compared with urban and suburban 

counterparts. Rural residents are more likely to be poor, in part because they have less access to education, jobs, and 

community services and facilities like broadband internet.36 People who grow up in rural areas are more likely to 

experience Adverse Childhood Experiences (ACEs) like abuse, neglect, and family challenges like food insecurity and 

unstable housing.37 People living in rural areas have less access to health and mental health providers and are less 

likely to receive evidence-based treatments.38 They are more likely to have chronic health conditions, demonstrate 

high-risk health behaviors, and live shorter lives.39 It is perhaps no 

coincidence that population density per square mile correlates with 

the highest rates of completed suicide and, according to the 

American Foundation for Suicide Prevention, Wyoming, Montana 

and Alaska have the highest suicides rates in the United States. 

While the annual age-adjusted suicide rate was 13.26 per 100,000 

individuals for the overall US, the rates in these three states were 

more than twice as high – ranging from 26.83 to 28.24 per 

100,000.40

“I like the availability of meaningful 

treatment interventions and 

recommendations on the part of 

behavioral health staff, as well as the 

methodical identification of patient 

who may benefit, many of whom were 

previously overlooked.” 

– Primary Care Provider at Clinic F
 

SIF IMPACT Expansion 12 Final Report: October 2018 

Table 1. IMPACT Implementation Initiative Care Logic Model 

Inputs Activities Outputs Outcomes Impacts











CNCS

John A. Hartford 

Foundation

University of 

Washington AIMS 

Center

Care Management 

Registry

Subgrantee

community health 

clinics; serving 

low-income 

patients in rural 

communities that 

are medically 

underserved











Pre-launch Kickoff Webinar: 

Subgrantees participate in kickoff webinar. 

Pre-launch Technical Assistance:

Subgrantees attend Introduction 

to IMPACT and Creating the 

Collaborative Care Team 

webinars; individual onsite prelaunch TA visit; ad hoc technical 

assistance during planning. 

Pre-launch Training:

Subgrantees attend 2-day 

IMPACT training; PCPs and 

psychiatric consultants unable to 

participate in training attend 

webinars geared to their role. 

Post-launch Training: Care 

managers providing 

psychotherapy complete 

training and certification in PSTPC; Bi-monthly topic webinars 

and case discussions;

subgrantees participate in an inperson training convening one 

year following implementation. 

Post-launch Technical 

Assistance: Monthly 

implementation conference call 

with each subgrantee site; 

Quarterly conference call with 

psychiatric consultants at all 

subgrantee sites. 













Subgrantees assemble primary 

care based teams, including 

primary care providers, care 

manager(s) and psychiatric 

consultant(s). 

Subgrantees adapt clinical 

workflows to incorporate key 

components of collaborative 

care model. 

Care manager(s) work closely 

with the primary care provider 

to deliver and coordinate 

patient care. 

Care manager(s) delivers 

IMPACT care to patient, 

including: 1. screening for 

depression; 2. patient 

education; 3. pro-active followup; 4. brief counseling using 

evidence-based techniques; 5. 

facilitation of communication 

between PCP and psychiatric 

consultant; and 6. referrals to 

and coordination with outside 

agencies. 

Care manager and psychiatric 

consultant engage in regular 

review of patients whose 

depression symptoms are not 

improving, facilitated by care 

management registry. 

Psychiatric consultant provides 

treatment recommendations to 

the primary care team, focused 

on development and 

modification of treatment plans. 

Primary Outcome for 

participating patients:

 Decreased 

depression 

Secondary Outcomes 

for Participating 

Patients:









Improved social 

functioning 

Improved 

occupational 

functioning 

Increased use of 

depression care 

Increased 

satisfaction with 

depression care 

Additional 

Exploratory Outcome: 

 Identify trends in 

health care 

utilization, health 

care costs, and 

other economic 

outcomes for 

patients. 





Increased access 

to effective 

evidence- based 

depression 

treatment for lowincome patients in 

rural areas that 

are medically 

underserved. 

Improved 

economic wellbeing among 

participating 

patients and their 

families. 

 

This evaluation seeks to answer five confirmatory questions on program impact and one exploratory question. 

In addition, the evaluation examines implementation process questions. The primary outcome is the effect of IMPACT 

implementation on patient-level depression response and remission among enrolled patients, how this compares with 

published research trials and real-world implementations, and predictors of depression improvement. Secondary 

outcomes include patient-level effects on social functioning, occupational functioning, use of depression care, and 

satisfaction with depression care. Exploratory outcomes include effects on patient’s healthcare utilization. Clinic-level 

implementation outcomes include how well clinics implement practice change, the effect of this on evidence-based 

processes of care, and provider satisfaction with Collaborative Care. 
 

SIF IMPACT Expansion 13 Final Report: October 2018 

Impact Evaluation 

Study Design Overview 

Observational analyses examined clinic-level implementation and adherence to evidence-based processes of 

care as well as patient-level intervention outcomes on depression symptom severity. We compared the number of 

patients enrolled across participating clinics, the proportion of patients receiving proactive follow-up, the proportion 

of patients whose care was discussed with the psychiatric consultant, improvement in patients’ depression symptoms, 

changes in social and occupational functioning, medical care and depression care utilization, as well as patient and 

provider satisfaction. A logistic regression model adjusting for clinic site and patient demographic variables examined 

variation in the odds of depression response and depression remission among participating subgrantee clinics. 

Comparisons are made across participating sites and are compared with existing benchmarks in the published 

literature from depression care programs implemented in similar populations13 and practice settings.12, 14 

 

Description of Design 

The evaluation of clinical, social and economic effects of the program on patients is an observational design 

using clinical registry data supplemented by self-reported survey data collected from a subsample of patients. The 

primary outcome, depression severity, was assessed at treatment initiation (baseline) and compared to the last 

observed measurement. Secondary outcomes that focused on clinical processes of care, shown in previous research 

to predict better and faster patient improvement, were also assessed with registry data. Supplementary patient 

survey data was used to measure four other secondary outcomes: 1) social functioning, 2) occupational functioning, 3) 

healthcare and depression care utilization, and 4) satisfaction with depression care. 

Provider satisfaction and organizational factors that may influence implementation were measured with 

quantitative and qualitative data collected from clinic leadership (pre-launch and 18 months post-launch) and 

clinicians (18 months post-launch). . 

The Stages of Implementation Completion (SIC) measure was completed for each subgrantee clinic. This 

measure of variation in implementation process has been shown in previous studies to effectively predict 

organizations that will launch an innovation, maintain competence in that innovation, and sustain it over time.31-33 

This measure allows characterization of variation in implementation process and describe fidelity to key components 

of IMPACT amongst the subgrantee clinics and whether/how these affect processes of care and/or clinical outcomes. 

Important Note: Although the SIF-CC initiative was originally intended for adult patients only, some clinics 

requested permission to treat adolescents. In recognition of the fact that many of these rural primary care clinics have 

no other treatment option for adolescents and Collaborative Care has been demonstrated to be effective for 

adolescents,41 they were granted permission to provide care to them with SIF-CC resources. However, for purposes of 

evaluation, all patients under the age of 18 were excluded from analysis to allow for comparison to the Collaborative 

Care published literature which does not intermix adult and adolescent patient populations. Overall, 3% of enrolled 

patients were under the age of 18 with a range of 0% to 4% at all clinics, except Clinic G where 10% of enrolled 

patients were under the age of 18. 

Previous Research/Literature 

Program outcomes are compared to national benchmarks established by the original IMPACT trial6, other 

RCTs of Collaborative Care,1-6 replications of the IMPACT trial in diverse populations10-12 and settings13-14, as well as 

more recent implementation studies of Collaborative Care in non-experimental, real-world health care settings.15 

Rationale for Design 

More than 80 randomized controlled trials (RCTs) have established the effectiveness of Collaborative Care for 

depression.1-6 Given the strong existing evidence and the SIF program’s purpose in disseminating this model to lowincome, underserved populations in under-resourced rural community health care clinics, we felt it is not necessary, 

and perhaps not even ethically justifiable, to repeat an RCT trial involving a usual care control group, or to use an 
 

SIF IMPACT Expansion 14 Final Report: October 2018 

interrupted time-series approach to create a lag in time for conducting repeated measurements before allowing 

underserved patients to obtain needed depression care. Instead, we chose an observational design. 

Targeted Level of Evidence 

Since Collaborative Care has a strong existing evidence base, the SIF expansion to low-income, rural 

communities does not focus on providing further causal evidence for the overall approach of Collaborative Care. 

Rather, this evaluation examines whether similar changes in patient-level outcomes to the original studies are seen in 

the target population of low-SES patients served in rural community health centers, examines variations in 

implementation and outcomes across participating rural clinics, and explores effects of Collaborative Care on health 

care utilization and other economic outcomes among rural, low-income patients. The targeted level of evidence is 

preliminary. 

METHOD

Descriptive Participation and Program Delivery

 Number of Program Units/Outputs 

The AIMS Center worked with subgrantees both 

individually and collectively to provide training and 

implementation support. Support activities included 

coaching, webinars, trainings, and conference calls delivered 

onsite (at each participating clinic), in-person (in Seattle), or 

via the internet (webinars, website) or telephone (conference 

calls). Program units/outputs included training and 

implementation activities targeted to subgrantee 

organizations, and clinical delivery activities targeted to 

enrolled patients. 

Table 2. Cohorts 1 & 2

Patient Enrollment Target

Total 

Enrolled

Variance

Clinic A 450 728 +62%

Clinic B 375 405 +8%

Clinic C 600 1,542 +257%

Clinic D 350 890 +254%

Clinic E 450 633 +40%

Clinic F 300 207 -31%

Clinic G 375 262 -30%

Clinic H 350 538 +54%

TOTAL 3,250 5,187 +63%

Table 2. Adult (age 18+) Patient Enrollment

Date Program Delivery Began/Ended 

 The five Cohort 1 subgrantees were awarded funding in May 2013, began pre-launch training in July 2013 and 

launched IMPACT clinical services in September/October 2013. The three Cohort 2 subgrantees were awarded funding 

in May 2014, began pre-launch planning and training in July 2014, and launched IMPACT clinical services in October 

2014. Details regarding program delivery are provided in Appendix A. 

Number of Program Patients / Participation Rate 

Table 2 shows the number of patients enrolled in IMPACT 

depression treatment at each of the participating clinics. All but two 

sites exceeded targeted enrollment, some of them significantly 

exceeding anticipated program reach. Together, the eight subgrantee 

clinics enrolled over 5,000 patients, exceeding targeted reach by 63%. 

Clinics varied substantially in the unique number of patients 

treated annually prior to SIF with an average of 11,537 (range 1,356 – 22,934; data unavailable for Clinic G). 

Over the 2.75 years clinics participated in SIF they enrolled 1,886 patients per year, representing 16% of the 

total annual patient population. According to the Centers for Disease Control and Prevention, during the period 

2013 – 2016 “8.1% of American adults aged 20 and over had depression in a given 2 week period” while “15.8% 

of adults from families living below the federal poverty level had depression.”42 The SIF-CC clinics engaged 

patients in treatment at the level of anticipated prevalence. This is no small feat in rural, under-resourced 

“I think they did such a good job, 

they got me right into IMPACT. It was 

amazing because I didn’t realize how 

sick I was.” – Patient at Clinic A
 

SIF IMPACT Expansion 15 Final Report: October 2018 

community clinics and achieves the goal of increasing access to depression treatment in these high need – low 

resource communities. 

Table 3. Demographic Profile of Enrolled Patients 

Clinic A Clinic B Clinic C Clinic D Clinic E Clinic F Clinic G Clinic H All

n=728 n=405 n=1524 n=890 n=633 n=207 n=262 n=538 n=5187

Gender #

%

#

%

#

%

#

%

#

%

#

%

#

%

#

%

#

%

Women 510 

(70.1)

273 

(67.4)

973 

(63.8)

525 

(59.0)

426 

(67.3)

139 

(67.1)

174 

(66.4)

385 

(71.6)

3405 

(65.6)

Age

18-24 119 

(16.3)

63 

(15.6)

215 

(14.2)

134 

(15.1)

73 

(11.5)

22 

(10.7)

52 

(19.8)

97 

(18.3)

775 

(15.0)

25-45 387 

(53.2)

172 

(42.6)

729 

(48.1)

493 

(55.4)

305 

(48.2)

99 

(48.1)

122 

(46.6)

240 

(45.2)

2,547 

(49.3)

46-64 187 

(25.7)

140 

(34.7)

495 

(32.7)

261 

(29.3)

220 

(34.8)

69 

(33.5)

72 

(27.5)

152 

(28.6)

1,596 

(30.9)

> 65 35 

(4.8)

29 

(7.2)

69 

(4.6)

2 

(0.2)

35 

(5.5)

16 

(7.8)

16 

(6.1)

42 

(7.9)

244 

(4.7)

Unknown - - 8 (0.5) - - - - - 8 (0.5)

Race / Ethnicity

American Indian 

or Alaska Native

7 

(1.0)

5 

(1.2)

51 

(3.3)

9 

(1.0)

15 

(2.4)

106 

(51.2)

188 

(71.8)

14 

(2.6)

395 

(7.6)

Asian 4 (0.5) 3 (0.7) 3 (0.2) 17 (1.9) 4 (0.6) 0 (0.0) 5 (1.9) 0 (0.0) 36 (0.7)

African 

American

15 

(2.1)

7 

(1.7)

6 

(0.4)

32 

(3.6)

4 

(0.6)

- 2 

(0.8)

4 

(0.7)

70 

(1.3)

Pacific Islander 0 (0.0) 0 (0.0) 6 (0.4) 14 (1.6) 6 (0.9) 2 (1.0) 1 (0.4) 1 (0.2) 30 (0.6)

White 657 

(90.2)

363 

(89.6)

1334 

(87.5)

773 

(86.9)

489 

(77.3)

88 

(42.5)

51 

(19.5)

514 

(95.5)

4269 (

82.3)

Mixed 18 

2.5)

1 

(0.2)

11 

(0.7)

24 

(2.7)

13 

(2.1)

3 

(1.4)

10 

(3.8)

2 

(0.4)

82 

(1.6)

Hispanic - - - - 77 (12.2) - 5 (1.9) - 82 (1.6)

Other / 

Unknown

27 

(3.7)

26 

(6.4)

113 

(7.4)

21 

(2.4)

25 

(3.9)

8 

(3.9)

- 3 

(0.6)

223 

(4.3)

Demographic Characteristics of Adult Patients Enrolled in Collaborative Care

Table 3 shows demographic characteristics for adult (age 18+) patients. Despite using three different data 

sources (CMTS clinical registry, clinic administrative data, patient survey) to insure demographic data was as 

complete as possible, this is a real-world implementation and it was not possible to collect demographic data on 

every patient. As expected, two thirds of treated patients were women and one third men. This is consistent with 

both research studies and implementation projects focused on depression treatment and there was no significant 

variation among the participating subgrantee clinics. Patient mean age was 39.8 (SD 14.5) and only 5% of patients 

were age 65 or older. 

Overall, 82% of patients reported their race/ethnicity as White. According to Census data,43 the combined 

population of the four participating states is 75% White. The most likely reason this initiative treated a higher 

proportion of White patients as compared with statewide statistics is that racial groups are not evenly distributed 

throughout these states. The two states that treated a smaller proportion of White patients as compared with their 

statewide prevalence are Alaska (67% of statewide population; 54% treated in SIF initiative) and Montana (89% of 

statewide population, 75% treated in SIF initiative). These are the two states participating in the SIF initiative with the 
 

SIF IMPACT Expansion 16 Final Report: October 2018 

largest proportion of American Indian/Alaska Native patients and each state had only one clinic treating a large 

percentage of indigenous patients as compared with statewide averages. 

In fact, the second largest ethnic minority group treated in this initiative was American Indian / Alaska Native 

(8%). This is particularly notable because of the dearth of published data regarding effective depression treatment 

interventions for this underserved patient population and the small sample sizes associated with published data. 

Demographic and Other Characteristics of Participating Providers 

 Table 4 shows demographic characteristics for treatment providers (care managers, primary care 

providers, psychiatric consultants, psychotherapists) who completed the survey 18 months following launch of 

Collaborative Care. Totals in the bottom row of the table use the entire sample as the denominator for that 

column whereas the rows represent each clinic and use the total number of responding providers from that 

clinic as the denominator. 

Table 4. Demographic Characteristics of Providers

Organization # of 

Providers 

in Survey

Percent

Provider 

Mean Age 

(SD)

Female

Providers

Provider Racial Ethnicity (Percent)*

White Asian/ 

Pacific 

Islander

Hispanic Native 

American / 

Alaska Native

African 

American

Clinic A 11 11% 40.6 (9.4) 60% 91% 0% 0% 0% 9%

Clinic B 7 7% 45.4 (9.8) 86% 71% 0% 29% 0% 0%

Clinic C 32 31% 36.3 (9.5) 56% 84% 9% 0% 3% 0%

Clinic D 19 18% 42.2 (10.8) 60% 53% 21% 16% 0% 0%

Clinic E 11 111% 48.2 (12.6) 80% 91% 9% 0% 0% 0%

Clinic F 5 5% 45.2 (9.9) 60% 40% 20% 0% 40% 0%

Clinic G 8 8% 43.3 (9.4) 86% 88% 0% 0% 13% 0%

Clinic H 10 10% 46.6 (12.3) 90% 90% 0% 0% 0% 10%

Total 103 100% 41.7 (10.9) 68% 78% 9% 5% 4% 2%

* 3 providers did not provide Race data

There are significant differences in the mean ages of the providers between the organizations (p < .03). 

Clinic C had the youngest age on average (36 years) in comparison to Clinic E, which had on average the oldest 

age (48 years). Two thirds of the providers are women. Although the differences in percentages of woman as 

providers in the organizations was not significantly different (p = .35), the organizations had a significant range 

from a high of 90% (Clinic H) to a low of 56% (Clinic C). The racial ethnicity of the providers differed significantly 

among the organizations (p < .004): Seventy-eight percent of the providers reported their race/ethnicity as 

white, 9% reported Asian / Pacific Islander, 5% Hispanic, 4% Native American / Alaska Native, and 2% African 

American. 

Table 5 reveals that the majority of providers had worked in the clinic for at least a year and differences 

among the clinics were not statistically significant. In terms of months working at the clinic, the entire provider 

sample worked 3 years (36 months) on average, with provider groups varying from less than 2 years at Clinic A 

to over 4 years for Clinic G and Clinic D. The percentage of providers employed in various roles relevant to 

Collaborative Care was similar across the organizations (p = .85). In total, PCPs comprised about 60% of the 

survey sample, while about 20% are Care Managers and the other 20% are split between Psychiatric 

Consultants, Behavioral Health Providers and other. Of the 65 Primary Care Providers, 45% (n=30) were 

physicians and the remaining 55% were mid-level providers, including physician assistants (n=12), nurse 

practitioners (n=18), medical residents (n=6). 
 

SIF IMPACT Expansion 17 Final Report: October 2018 

Table 5. Training / Employment Characteristics of Providers

Organization Worked in 

the Clinic 

> 1 year

Total Months 

Working at Clinic

Mean (SD)

Professional 

Practice < 5 

years*

Provider Role Percentage (N)

Care 

Manager

PCP Psychiatric 

Consultant

Beh Health 

Provider

Other

Clinic A 82% 19.5 (10.9) 54% 27% 55% 9% 0% 9%

Clinic B 100% 34.0 (25.6) 29% 14% 43% 14% 29% 0%

Clinic C 67% 25.6 (32.0) 57% 13% 63% 6% 16% 3%

Clinic D 88% 53.6 (43.9) 44% 16% 79% 5% 0% 0%

Clinic E 100% 36.0 (30.8) 73% 27% 73% 0% 0% 0%

Clinic F 60% 30.4 (24.2) 20% 20% 80% 0% 0% 0%

Clinic G 67% 49.0 (71.6) 57% 29% 57% 0% 14% 0%

Clinic H 67% 44.0 (43.3) 50% 20% 50% 10% 10% 10%

Total 78.5% 35.7 (37.6) 52% 19% 

(19)

64%

(65)

6%

(6)

9%

(9)

3%

(3)

*Not including residency and training 

Implementation Data Collection and Measurement 

Measures Used for Each Dimension 

Outcome measures for each dimension are described in Appendix B.

Description of Data Collection Methods / Amount of Data Collected 

Data was collected from a variety of sources, including clinic leadership, clinicians, administrative data 

systems and patients. Additional detail regarding measures, sources, timing, and data collection methods is provided 

in Appendix B and Appendix C.





i

Care Management Tracking Registry (CMTS) – Primary Data Source

The primary purpose of the CMTS clinical registry is to assist clinicians with delivery of high quality evidencebased Collaborative Care. All patients enrolled in Collaborative Care 

are entered into the registry. In addition, the CMTS patient registry 

data served as the primary data source for the primary outcomes, 

depression response and depression remission. The primary 

outcome was calculated using data from the care management 

registry at baseline and last recorded measurement. The CMTS collects data on all patients enrolled in care. CMTS 

tracks clinical outcomes (depression symptoms) and processes of care (e.g. number and type of follow-up 

contacts, proportion reviewed with psychiatric consultant). Process of care data was used internally at each 

subgrantee clinic and during monthly implementation support calls between each clinic and the AIMS Center to 

help them identify and resolve implementation challenges. 

Leadership Surveys – Secondary Data Source 

Organizational leaders provided quantitative and qualitative information about the organization and their efforts 

to implement Collaborative Care immediately prior to program launch, 18 months post-implementation, and 

mmediately prior to completing implementation support with the AIMS Center. Organizational leaders varied 

from site to site but could include the CEO, CFO, Medical Director, Behavioral Health Director and similar 

members of the leadership team at subgrantee clinics. 

“I really like the registry and being 

able to track the people with 

depression.” – Care Manager at Clinic H
 

SIF IMPACT Expansion 18 Final Report: October 2018 

 Patient Surveys – Secondary Data Source 

Supplemental surveys were collected with a subsample of patients, representing 441 patients across the 8 

subgrantee clinics. The goal was to collect at least 325 paired surveys (baseline and follow-up surveys), reflecting 

a max attrition rate of 30%. The survey was collected shortly after patients enrolled in treatment and again 6 

months after program enrollment. The general eligibility requirements for survey participation were: age >18, 

English-speaking, have a telephone for contact, and agreement to participate in surveys. After completing the 

second survey, each survey participant received a $40 gift card incentive for participating in the two surveys as a 

method of insuring a max attrition rate of 30%. The gift cards were from local gasoline merchants in each 

respective clinic neighborhood. 

The collection of baseline patient surveys was completed in May 2015 for both cohorts. A total of 1,138 patients 

were approached to participate in the evaluation surveys. This number is lower than the total enrolled in IMPACT 

since it does not include ineligible or deferred patients (ineligibility includes: enrolled before survey start date, 

enrolled after survey enrollment ended, age ≤18yrs, non-English speaking, and no telephone). Of the 837 (74%) 

patients that agreed to participate in the survey, we completed 441 (53% of agreed) baseline surveys (see Table 6 

below). 

Table 6. Patient Survey Data Collection 

The six month follow-up survey collection began in July 2014 for Cohort 1 and April 2015 for Cohort 2. Collection 

of follow-up surveys concluded for both cohorts in February 2016. A total of 351 follow-up surveys were 

collected, representing 80% of completed baseline surveys, exceeding our goal of ≥70% retention. 

On February 2, 2015, the incentive component of the patient survey was launched. This provided a $40 gas card 

to survey participants following completion of the 6 month follow-up survey as an incentive to complete both 

surveys. Figure 4 illustrates the follow-up survey completion trends before and after the incentive. It shows a 

sharp steepening of the curve following implementation of the incentives. The trend flattens out in September 

2015 as the number of open follow-up surveys dropped. 

Figure 4. Follow-up Survey Completion Trends before and after incentive (dashed red line represents launch of incentive, 

February 2015)

CHCCW Mat-Su Partnership Peninsula

Valley 

View Bighorn KANA SWMCHC

Total patients approached 234 8 6 355 224 128 4 2 4 6 2 3

Refused at clinic 6 0 2 4 140 2 4 1 9 1 4 2 0 0 301 26%

Agreed to participate 174 6 2 215 200 109 2 8 2 6 2 3 837 74%

Completed Baseline Surveys 9 7 3 3 9 4 102 7 0 1 6 1 2 1 7 441 39%

Completed Follow-up Survey 7 4 2 6 7 6 7 7 5 9 1 2 1 1 1 6 351 31%

All Clinics

1138

0

10

20

30

40

50

60

70

80

90

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Year 2015

CHCCW

Mat-SU

Partnership

Peninsula

Valley View
SIF IMPACT Expansion 

19 Final Report: October 2018 

Figure 5a. Patient Survey Completion Timeframes

27%

19% 18%

36% 36%

13% 14%

37%

0%

10%

20%

30%

40%

50%

Within 7

days

8-14 days 15-28 days Over 28

days

Proportion Completed Survey Before Incentive

Proportion Completed After Incentive

18

10

24

16

10

38

33

45 46

35

20

33

12

7 

3 2 

0

10

20

30

40

50

Figure 5b. Patient Survey Completion by Month

Figures 5a and 5b provide additional details regarding impact of the patient survey incentive. Figure 5a

shows the proportion of patients who completed the follow-up survey before and after the incentive (as of 

December 1, 2015), aggregated by time to completion of the survey. Most notably, before the incentive 27% of 

patients completed the follow-up survey within 7 days and this jumped to 36% after the incentive was 

introduced. Figure 5b details the frequency of patients completing the follow-up survey starting in September 

2014 through December 2015. The monthly completes range from 2 to 46 completed surveys. The dotted red 

line shows when the incentive was introduced. The average before the incentive was 15.6 completes per month. 

After the incentive, the average completes rose to 36 per month. By September 2015, the number of completes 

dropped due to fewer open surveys. 

Clinician Surveys

 Clinicians (primary care providers, care managers, psychiatric consultants) at each of the subgrantee clinics 

participated in an online survey regarding their experience with 

clinical practice change 18 months following program launch. 

Collection of the clinician survey is complete for both Cohort 1 

and Cohort 2, with a total of 103 collected of the 124 requested 

(See Table 7). We achieved an average response rate of 83% 

across the 8 clinics, with rates ranging from 67% to 100%. 

 Table 7. Clinician Survey Collection

“I appreciate how the IMPACT model 

is thorough and produces improved 

outcomes.”

– Psychiatric Consultant at Clinic D

Clinic Approached Completed %

CHCCW 11 11 100%

Mat-Su 8 7 88%

Partnership 39 32 82%

Peninsula 21 19 90%

Valley View 12 11 92%

Bighorn 7 5 71%

KANA 11 8 73%

SWMCHC 15 10 67%

Totals 124 103 83%
 

SIF IMPACT Expansion 20 Final Report: October 2018 

 Administrative Data 

All subgrantee clinics provided administrative data for patients enrolled in IMPACT treatment. This data 

supplemented demographic information captured in the CMTS registry. 

 Stages of Implementation Completion (SIC) 

The Stages of Implementation Completion (SIC) was adapted for Collaborative Care as part of the SIF initiative. 

This measure allows us to describe variation in the implementation process among subgrantee clinics and 

whether/how these affect processes of care and/or clinical outcomes. The SIC data was collected from the AIMS 

Center and entered into a web portal located at the Oregon Social Learning Center. See Appendix E for the 

Collaborative Care SIC. 

Impact Evaluation Questions

Implementation and Impact 

This evaluation seeks to answer five confirmatory questions on program impact and one exploratory 

question. In addition, the evaluation examines implementation process questions.

Confirmatory (Impact) 

The primary question is: 

1. Do patients in participating clinics treated with IMPACT achieve depression remission and response 

at levels comparable with published research trials and real-world implementations and which 

factors predict likelihood of remission and response? 

In addition, the evaluation will seek to answer: Do patients in participating clinics treated using IMPACT 

experience the following at 6-months post-baseline: 

2. Improved social functioning? 

3. Improved occupational functioning? 

4. Increased use of depression care? 

5. Increased satisfaction with depression care? 

Exploratory (Impact) 

As an exploratory question, the evaluation explores: 

6. How will IMPACT treatment in rural clinics affect participating patients’ health care utilization? 

Implementation 

The final research questions deal with program implementation and variation across sites: 

7. Do participating clinics implement IMPACT Collaborative Care? 

8. How does implementation vary across subgrantee clinics and which factors account for variation in 

implementation across clinics? 

The implementation and effectiveness of Collaborative Care in rural, low-income, underserved areas is an 

important question. For this project, we expected to replicate results of prior studies and implementation initiatives 

targeted to patients treated in community health clinics. The primary goals are to increase access and quality of 

depression care delivered to patients at these rural clinics. 
 

SIF IMPACT Expansion 21 Final Report: October 2018 

Counterfactual Condition 

Baseline equivalence analysis 

We used an observational design to compare program 

outcomes to benchmarks published by the original IMPACT trial,6

other RCTs of Collaborative Care,1-5 replications of the IMPACT trial in 

diverse populations10-12 and settings,13-14 as well as more recent 

implementation studies of Collaborative Care in non-experimental, 

real-world health care settings.15 Because we did not use an RCT 

design and all patients received Collaborative Care care, there was no control group. Instead, patients and clinics 

served as their own controls. This natural experiment design explored the effect of Collaborative Care care 

experienced across the participating subgrantees and allowed for an initial evaluation on whether similar benchmark 

results can be achieved in community health clinics in the WWAMI region that serve low-income patients in 

medically underserved rural communities. 

“Some people feel that the care 

manager calls are too frequent and 

get annoyed.” 

– Care Manager at Clinic H

ANALYSIS 

Implementation 

Types of Analysis 

We evaluated the following clinical process of care measures captured in the CMTS registry for all enrolled 

IMPACT patients at the time of report: 

 # of patients enrolled in treatment (completed initial assessment/PHQ-9) 

 % of patients receiving follow- up within < 4 weeks and # of follow-ups over the course of treatment 

 % of patients staying in contact for at least 12 weks 

 % of patients engaged >40 weeks 

 % of patients having at least one psychiatric consultant review during treatment 

 Demographic characteristics of enrolled patients 

Frequency distributions for these outcomes and processes of care were calculated and examined across the 

program as a whole and for individual participating subgrantee clinics. 

Analysis Procedures/steps 

The AIMS Center extracted registry data from the SQL data tables contained within the CMTS registry. This 

data was imported into SAS for data cleaning and analysis. Where necessary, calculated variables were created, and 

analytic outputs were checked for accuracy. 

Impact Evaluation 

Types of Analysis 

We examined unadjusted baseline depression severity as measured by the PHQ-9 across the eight 

participating clinics and compared this to mean PHQ-9 score at last measurement. Frequencies for the PHQ-9 

were calculated two ways, one continuous and one categorical. This measure has a continuous range of 0-27 

but can also be categorized as mild (<10), moderate (10-14), moderately severe (15-19), and severe (20-27).28

Suicidal ideation, a single item on the PHQ-9 with a response scale of 0-3 was dichotomized as “yes” (any score 
 

SIF IMPACT Expansion 22 Final Report: October 2018 

above zero) and “no” (a score of zero). For the logistic regression model we calculated maximum PHQ-9 score 

during treatment as a proxy for “baseline” severity; the initial PHQ-9 was not used since it is part of the 

calculation of depression response. If the patient had only one depression treatment contact, the baseline 

value was carried forward using an “intent to treat” approach. Depression response is defined as a 50% 

reduction in symptoms and depression remission is defined as a final PHQ-9 score less than 5. 

Analysis Procedures / Steps 

Patient survey data was collected electronically in a REDCap system. REDCap, a web-based application for 

Electronic Data Capture (EDC) developed by a consortium of institutional partners, is currently free to all researchers 

in the five-state WWAMI region. At the conclusion of patient survey data collection, survey data was exported from 

REDCap to Excel and uploaded into SPSS v.23 (Statistical Package for the Social Sciences) for analysis. Data was 

prepared for analysis by running range checks and other data cleaning steps. Qualitative interviews were transcribed 

immediately after data collection and analyzed for themes relevant to primary and secondary outcomes. Data was 

analyzed with SPSS and Microsoft Excel.® 

To examine potential sources of variation depression response and remission we examined CMTS clinical 

registry data frequencies for outliers and none was found. We used paired-t tests to determine if statistically 

significant changes existed in depression severity over time in the entire patient sample. We created a logistic 

regression model to examine variation in the odds of our outcomes: response and remission. Our model adjusted for 

clinic site, using the largest clinic as the reference group. We added all patient demographic variables (i.e., age, 

gender, and race/ethnicity) to the model. To represent patient’s depression severity level we tested the maximum 

PHQ-9 score during treatment in the model as a proxy for “baseline” severity. The initial PHQ-9 was not used since it 

is part of the calculation of depression response. Lastly, we fit all the process of care variables and retained only 

those in the model that explained a statistically significant amount of variance. These process of care variables 

included time in treatment from first to last contact, prior depression treatment episode at the clinic, number of 

follow-ups during treatment (exposure) and whether a follow-up occurred within 31 days of enrollment. We present 

the adjusted odds ratios (ORs) and their 95% confidence intervals as well as the unadjusted percentage of 

depression response and remission for the categorical strata within the model. The R2 maximum was calculated for 

the model. 

Missing Data Analysis 

The patient registry was used as the data source for the main outcome (impact evaluation) and, because it 

captures clinical care as it is delivered, there was no missing data in the comparison of depression symptoms at 

baseline and final measurement. Only surveys for which both a baseline and follow-up were collected were included 

in the subsample analysis of secondary outcomes. 

FINDINGS

The AIMS Center has 13 years’ experience 

supporting 1,000’s of organizations implementing 

Collaborative Care programs. This experience has yielded 

many lessons about effective implementation of a complex 

healthcare innovation in a wide range of clinical settings and 

patient populations. 

Successful implementation of Collaborative Care 

balances increased access for the entire population of 

patients needing mental health treatment and a proactive, 

stepped care approach to treatment intensification if patient 

Figure 7. Principles of Collaborative Care
 

SIF IMPACT Expansion 23 Final Report: October 2018 

symptoms are not substantially reduced within 10-12 weeks. Figure 7 shows the 4 key principles of Collaborative 

Care along with an environmental intervention (accountable care) that facilitates faster and more effective adoption 

of these principles. Evaluation of a pay-for-performance intervention with community health clinics (mostly FQHCs) 

implementing Collaborative Care demonstrated that putting as little as 25% of reimbursement at risk based on 

quality metrics dramatically changes clinician behavior and cuts the time from treatment initiation to response in 

half.44 Clinics that implement Collaborative Care successfully meet minimum standards for the four key principles. 

1. Patient-Centered Collaboration

The minimum expectation is that the primary care provider, care manager (+/- behavioral health provider), 

and psychiatric consultant collaborate on development and implementation of the treatment plan. The primary care 

provider remains the locus of care, rather than referring patients to behavioral health specialists who deliver parallel 

treatment. Figure 1 (page 6) shows the Collaborative Care team and how an effective team functions. Treatment 

plans are individualized for each patient and may include medication, psychotherapy, or both. 

2. Population Based Care 

The minimum expectation is that all patients being treated for depression are tracked in a registry that 

allows all of the members of the treatment team to quickly and easily see which patients are not improving and 

need a change in treatment and which patients are sufficiently improved to be moved out of active care 

management. The latter action is critically important in expanding 

access to behavioral health services in primary care. Each clinic had a 

minimum caseload target based on care manager FTE and care 

manager model (all-in-one or shared; see page 35 for more details). 

The registry also identifies patients who have stopped contact with 

the Collaborative Care team. Isolation and avoidance of interpersonal 

contact are symptoms of depression that can impede a patient’s effective engagement with the treatment team. 

The treatment team is expected to proactively reach out to all patients with active symptoms rather than simply 

responding to the patients who show up at the clinic. 

3. Treatment to Target 

The minimum expectation is that the treatment team regularly measures depression symptom severity with 

a quantitative tool like the PHQ-9. Most important, these measurements are used to drive the course of treatment. 

If the patient’s symptoms have not reduced by at least 50% from baseline within 10-12 weeks the treatment plan is 

changed and the clock resets for another 10-12 weeks. The individual items on the PHQ-9 are used to inform 

changes in treatment so that interventions best suited to addressing the unresponsive treatments are used. The 

Collaborative Care team persistently adjusts the treatment plan. Prior research has shown that each change in the 

treatment plan results in achievement of response for 20% of patients who are unresponsive to treatment up to that 

point. 

4. Evidence-based Care 

The minimum expectation is that patients are offered treatments that are evidence-based and appropriate 

for the primary care setting. All care managers were trained in behavioral activation and licensed care managers 

were also trained and required to demonstrate minimum proficiency in Problem-Solving Treatment (PST), a brief, 

structured psychotherapy delivered in six to eight 30 minute sessions in which the patient is taught a technique to 

use on their own. PST has a very strong evidence base for treating depression.45

Fidelity to program design 

Every organization, however highly functioning and however well prepared, experiences implementation 

challenges following program launch. The key to implementation success is the ability to honestly and accurately 

“I feel like there is less chance for 

these patients to fall between the 

cracks.” 

– Primary Care Provider at Clinic A
 

SIF IMPACT Expansion 24 Final Report: October 2018 

identify these challenges, brainstorm potential solutions and implement successive adaptations designed to address 

these challenges until the implementation is achieving its goals. In short, it’s the ability of the organization to be 

resilient that is most likely to determine whether the implementation will succeed. 

All five of the clinics in Cohort 1 launched Collaborative Care in late September or early October 2013. The 

three Cohort 2 clinics launched IMPACT in October of 2014. See Appendix A for a graphical representation of the 

program timeline. Fidelity to program design is best examined by evaluating processes of care and how well these 

clinical processes match expectations based on the implementation experience of the AIMS Center. Goals and 

expectations for processes of care based on implementation experience with other community primary care clinics 

serving low-income, underserved patients14 are provided in 

Table 8. 

As part of post-launch support and coaching (Appendix 

A), the AIMS Center reviewed process of care reports from the 

CMTS registry as part of a monthly implementation coaching 

call with each subgrantee clinic. These real-time reports 

included metrics for all patients currently “active” in the CMTS 

registry and allowed clinics to see their performance in 

comparison with the other clinics in the initiative (Appendix F).

included the following metrics (among others): number of 

active patients, average baseline and most recent PHQ-9 score, 

proportion of patients engaged in follow-up, length of time in 

treatment, mean number of follow-up contacts, proportion of 

follow-up contacts conducted in-person and by phone, 

proportion of patients reviewed with the psychiatric 

consultant, proportion of patients experiencing a five point or 

greater reduction in symptoms as measured by the PHQ-9, and proportion of patients with 50% or greater 

improvement on the PHQ-9 after at least ten weeks in treatment. 

The AIMS Center used these program monitoring reports to help clinics identify implementation challenges 

and brainstorm solutions. For the first year of clinical implementation, the AIMS Center led the monthly calls using 

these reports and modeled how to read and interpret them. Thereafter, clinics were expected to review and 

interpret the reports in advance of the call and to lead the call. The goal of this approach is to teach clinics selfsufficiency with program monitoring and to prepare them for sustainment. 

Collaborative Care includes the ability to provide brief, evidence-based psychotherapy in the primary care 

clinic when that is a part of the patient’s treatment plan (typically 30-50% of patients have psychotherapy as part of 

the treatment plan). During pre-launch training in Seattle, care managers and other clinicians who offer 

psychotherapy as part of treatment participated in training in an evidence-based psychotherapy called Problem 

Solving Treatment (PST) in Primary Care. The training consisted of didactic learning, case-based learning and audiorecording sessions with actual patients. Trainees who completed all three steps and demonstrated minimum 

proficiency in the technique received certification as a PST practitioner. Every participating clinic had at least one 

staff member trained in PST. 

Table 8. Processes of Care: 

Expectations / Goals

Mean number of contacts 5 - 8

In-clinic contacts typically 65-80%

Phone contacts typically 20-35%

% of patients completing >1 follow-up 

<4 weeks of treatment start >50%

% of patients engaged in follow up ≥80%

% of patients engaged in treatment ≥ 

12 weeks ≥40%

% of patients engaged in treatment > 

40 weeks <20%

% of patients with at least 1 

psychiatric consultation ≥75%
SIF IMPACT Expansion 25 Final Report: October 2018 

Program Exposure / Dosage 

Table 9 shows program exposure/dosage data for the first 

episode of care for each patient. Some patients had only one episode 

of care during the evaluation timeframe while other patients had up 

to 6 separate episodes of care. Prior analysis of Collaborative Care 

implementation data focuses on the first episode of care because 

that represents the majority of patients. 

Overall, clinics had 4.5 contacts per patient (range 8.0 to 1.5). 

Across an entire patient population in a community health center 

setting, the average range is 5-8 contacts. There is not a goal per se 

but the range helps to identify when a clinic may be struggling with 

engagement (below 5 average contacts) and when a clinic may be keeping patients in treatment longer than is 

typically warranted in a primary care setting (more than 8 average contacts). Essentially, there is a “sweet spot” that 

balances access to services and intensity/duration of services for the large proportion of primary care patients who 

need mental health care. Collaborative Care is not simply co-locating 

specialty care, which is often open-ended and provides high intensity 

care to a small number of patients, into a primary care clinic. It is a 

fundamental shift in the philosophy and approach to mental health 

services that considers the entire population needing care and 

attempts to meet the needs of a larger proportion of that population 

than would be possible with a more traditional approach. Primary 

care patients, especially those who might not otherwise engage in 

care, are more likely to accept mental health services that take their cues from the primary care culture; that is, they 

are timely/immediate, focused on urgent symptom remission, and time-limited. 

“I love that we are regularly following 

up with depressed patients. I have 

heard from several of the patients on 

my caseload that it helps them ‘stay 

on their toes’ when they know they 

have someone regularly checking in 

with them.” 

– Care Manager at Clinic H

“I like that the IMPACT model treats 

the whole patient in a manner that is 

less stressful to the patient. The 

collaboration between the PCPs and 

care managers ensures holistic care.” 

– Care Manager at Clinic A

Table 9. Processes of Care: Program Exposure and Quality of Program Delivery

Total number of contacts

Clinic A Clinic B Clinic C Clinic D Clinic E Clinic F Clinic G Clinic H All Clinics

 4,107 1,453 5,928 3,791 5,061 928 1,453 2,162 24,883

Mean (SD) Contacts per Patient 5.5 (5.3) 1.5 (2.3) 3.7 (4.9) 4.1 (4.5) 8 (7.1) 4.4 (4.5) 4.8 (5.4) 3.9 (5.3) 4.5 (5.4)

In Clinic % 56% 84% 89% 99% 84% 95% 86% 91% 86%

Phone % 44% 16% 11% 1% 16% 5% 14% 9% 15%

>1 follow-up w/in < 4 weeks 69% 9% 55% 57% 82% 56% 58% 54% 55%

% completing follow-up (2+ contacts) 83% 32% 59% 73% 95% 68% 70% 61% 68%

% engaged in treatment ≥ 12 weeks 38% 40% 27% 34% 42% 36% 30% 36% 35%

% engaged in treatment > 40 weeks 7% 9% 4% 5% 11% 5% 4% 5% 6%

% at least 1 psych consult 91% 71% 81% 88% 94% 87% 76% 98% 86%

The proportion of in-clinic vs. telephone contacts is observed as a potential indicator of using a more 

traditional co-located psychotherapy approach, which often shows a high proportion of in-person contacts (over 

85%) and a low proportion of phone contacts (under 15%). As shown in Table 8 the typical range for phone contacts 

is 20-35%. Seven of the eight clinics were under this average and one clinic was above this average. As part of 

monthly implementation support Clinic A leadership were asked about the very high proportion of phone contacts. 

They responded that a significant subset of their patients are long-haul truckers who travel through Casper, 

Wyoming on their way across the country. The clinic used phone contacts to keep these patients engaged in 

treatment until the next time they came through on their way to another destination. Variation in this metric is 

normal and there is not a target goal per se. What is shown in the target metrics is the typical range for community 

primary care clinics. The fact that the two clinics with the lowest proportion of phone contacts (Clinic D and Clinic F) 
 

SIF IMPACT Expansion 26 Final Report: October 2018 

also have the highest proportion of patients receiving follow-up shows that this metric can serve as a general 

indicator of program fidelity but does not predict quality of program delivery. 

Quality of Program Delivery 

 Table 9 also shows metrics that indicate quality of program 

delivery, including the proportion of patients receiving follow-up 

within 31 days of treatment initiation, the proportion of patients 

remaining in treatment for at least 12 weeks, the proportion of 

patients in treatment for more than 40 weeks and the proportion of 

enrolled patients being reviewed by the psychiatric consultant. One 

of the key differences between Collaborative Care and usual care is proactive engagement with patients and 

ongoing measurement of symptoms to inform proactive treatment changes. Proactive engagement is reflected in 

the percentage of patients receiving follow-up. The goal is for 80% of patients enrolled in treatment to receive at 

least 2 follow-up contacts. Only two of the clinics met or exceeded this goal – Clinic A and Clinic E. Clinic E was also 

the clinic with the largest proportion of patients engaged in treatment for more than 40 weeks, which makes it 

easier to hit this target but for the wrong reason. The clinic that missed the mark by the greatest margin, at about 

half the rate of the next best clinic, was Clinic B at only 32%. 

 Based on the published literature,16-17 two key components of Collaborative Care predict better patient 

outcomes: Two processes of care emerged from this analysis: 1) the proportion of patients receiving at least one 

follow-up within the first four weeks following enrollment in treatment, and 2) the proportion of patients discussed 

with the psychiatric consultant at least once. 

 All of the participating subgrantees except Clinic B exceeded the goal for engaging patients during the first 

month of treatment and Clinic E far surpassed the goal at 82%. Clinic B achieved this this key component of effective 

patient engagement with only 9% of their enrolled patients. Similarly, all of the clinics except Clinic B exceeded the 

target goal for proportion of patients discussed at least once with the psychiatric consultant, though they missed the 

mark by a narrower margin. 

The percentage of patients actively engaged in treatment for at 

least 12 weeks is one measure of the quality of program delivery 

because it indicates how well the clinic engages and retains patients in 

treatment. While it is possible for some depressed patients to complete 

active treatment within 12 weeks, this is not common because the 

symptoms of depression often require at least this amount of time to respond to treatment, regardless of the 

treatment plan. Plus, clinics are expected to monitor patients for at least 1 month (and often 2-4 months) following 

successful completion of active treatment to ensure that improvement is not temporary. Low-income patients have 

a large number of psychosocial factors affecting their ability and willingness to remain engaged in active treatment, 

especially once their symptoms begin to significantly improve. The goal (≥40%) is based on experience with 

community primary care settings serving low-income patients similar to those enrolled in this initiative. Clinic E 

(42%), and Clinic B (40%) are the only clinics that met this goal. Most of the other clinics were close, with the 

exception of Clinic C at 27%. 

 While it is important to retain patients in treatment long enough to implement at least one treatment plan 

and measure its success in reducing patient symptoms, it is equally important to move patients through care and 

close active care management when symptoms are adequately improved and/or the care team has determined the 

patient’s needs cannot be met in primary care. This increases access. In short, there is a “sweet spot” when it comes 

to primary care behavioral health. This is a significant culture shift for many behavioral health providers who are 

accustomed to open-ended treatment that may keep a patient in active treatment for years. Long-term treatment 

creates access barriers for the typically large volume of primary care patients who need behavioral health services. 

To assist clinics in evaluating how well they are moving patients through treatment, the AIMS Center reported to 

clinics the proportion of actively enrolled patients engaged in treatment for more than 24 weeks. Based on the AIMS 

“I think they did a really good job 

because there was follow-up and I 

was active in my treatment plan.” 

– Patient at Clinic C

“I feel like they’re involving me in 

the treatment instead of just 

treating me.” 

– Patient at Clinic C
 

SIF IMPACT Expansion 27 Final Report: October 2018 

Center’s experience with diverse community primary care clinics, the goal for the metric is less than 20% of patients. 

All of the clinics achieved this goal. 

Stages of Implementation Completion (SIC) 

 Although many evidence-based practices 

(EBPs) have been developed, large knowledge gaps 

remain regarding how to routinely move EBPs into 

usual care. The lack of understanding of what it takes 

to install EBPs has costly public health consequences 

including a lack of availability of the most beneficial 

services, wasted efforts and resources on failed 

implementation attempts, and the potential for 

engendering reluctance to try implementing new 

EBPs after failed attempts. 

 As shown in Figure 8, the Stages of 

Implementation Completion (SIC) is an 8-stage tool of 

implementation process and milestones, with stages 

spanning three implementation phases (preimplementation, implementation, sustainability). 

One of the most attractive features of this measure is that it is agnostic to the specific intervention being 

implemented. The measure was originally developed to measure implementation of youth social service 

interventions in the foster care system but has since been adapted for implementation of many other social service 

and healthcare EBPs. Comparison of these multiple versions of the SIC has yielded the Universal SIC, which is a 

measure of items that are common across all implementation initiatives regardless of the topic. 

 Items delineate the date that a site completes implementation activities, yielding an assessment of duration

(time to complete a stage), proportion (of stage activities completed), and a general measure of how far a site moved 

in the implementation process (stage score).31-33 The SIC categorizes missing data into four categories: 1) truly not 

completed, 2) not necessary because clinic is an expansion site (e.g. Clinic E and Clinic D), 3) not applicable, and 4) 

completed but date of completion unknown. 

 The SIC was adapted through 

an iterative process for Collaborative 

Care and piloted with the SIF clinics. 

Through the use of retrospective and 

prospective data collection, this pilot 

of the Collaborative Care SIC will 

inform the need for potential 

modifications to the measure. This is 

an important step in the validation 

process. Second, the use of 

retrospective and prospective data 

collection may detect meaningful 

patterns in implementation behavior 

of SIF subgrantee sites. Identification 

of such patterns has real world 

implications for informing the level of 

training and implementation coaching 

that is recommended for new sites 

adopting Collaborative Care. 

Figure 8. Stages of Implementation Completion
 

SIF IMPACT Expansion 28 Final Report: October 2018 

“They should make the psychiatric 

consultant the primary person in 

charge of the program and allow 

them to choose the program staff to 

run the program because this is their 

specialty.” 

– Care Manager at Clinic B

“This emphasized ongoing 

difficulties related to 

integrated care at our 

agency. I believe most of 

the difficulty is our agency 

and the ongoing struggles 

with leadership and 

direction.”

– Care Manager at Clinic B

The eight SIF clinics were compared to the Universal SIC and preliminary results are shown in Figure 9. The 

graph at right shows the average duration for program start-up and achievement 

of competency as measured across implementations with the universal SIC (red 

bars) as compared with Cohort 1 SIF clinics (light teal) and Cohort 2 (dark teal). 

Although Cohort 1 is significantly longer in duration that both the Universal SIC 

and Cohort 2, the results for Cohort 2 are very similar to the Universal SIC. This 

indicates that the AIMS Center learned from their implementation experience with 

the Cohort 1 sites and used these insights to make adjustments to the training and 

coaching they provided to Cohort 2, reducing both program start-up time and time 

to competency. 

Despite the small sample (8 clinics), the SIC disentangled implementation 

of a complex intervention and successfully predicted discontinuation of the Clinic B 

clinic. This outcome was predicted by the fact that they did not select an 

appropriate implementation leader, declined to participate in recommended remedial onsite coaching and 

implementation assistance, never achieved the recommended minimum caseload of patients, and participated 

minimally in long-term sustainability planning.

Responsiveness 

All of the subgrantees met minimum requirements for 

participation and engagement with the exception of Clinic B. Clinics 

were required to designate an IMPACT implementation leader who 

was responsible for insuring compliance with participation 

requirements, monitoring metrics and making adjustments as needed 

to achieve improved access and quality. All clinics except Clinic B 

appointed a person with the authority to make decisions and garner 

necessary resources to achieve implementation goals. Clinic B 

appointed an administrative assistant with no clinical training or 

experience and no authority within the clinic. Key leadership, including the Medical Director of the primary care 

clinic and the Behavioral Health Director, were openly hostile to the idea of Collaborative Care and obstructed 

implementation despite the efforts of the care manager and psychiatric consultant who supported implementation. 

They dropped out of sustainability planning earlier than any of the other subgrantees and openly admitted they had 

no intention of continuing Collaborative Care after the end of the grant. 

Program Differentiation 

Many of the metrics shown in Tables 10-20 are based on the existing Collaborative Care literature and what 

is known from both research1-6 and real-world implementations13-15 regarding the critical components that are 

essential for producing positive outcomes.16-17 

Satisfaction - Patient 

Patient satisfaction was measured on a 5 point Likert scale 

where 1 = very satisfied and 5 = very dissatisfied. Table 10 shows 

results from the patient survey regarding satisfaction with care. The 

only two clinics for which the difference was significant (Clinic B and 

Clinic E) both showed reductions in satisfaction. The decrease in 

satisfaction at Clinic B is not surprising given their poor 

implementation of the treatment model, including proactive 

“I needed them to listen and they did.”

- Patient at Clinic E

“I just want to thank you and everyone 

who takes the time to care about me.”

- Patient at Clinic H
SIF IMPACT Expansion 29 Final Report: October 2018 

engagement within the first month of treatment with only 9% of patients. The decrease in satisfaction at Clinic E is 

unexplained by any of the other data, including processes of care, and is not supported by clinical outcomes 

reported below. 

Table 10. Patient Satisfaction with Depression Treatment 

Clinic A Clinic B Clinic C Clinic D Clinic E Clinic F Clinic G Clinic H All Clinics

n= 73 n= 28 n= 76 n= 76 n= 58 n= 13 n= 11 n= 15 n= 350

Mean (SD) baseline 1.81 (0.70) 1.54 (0.56) 1.80 (0.63) 1.93 (0.71) 1.81 (0.75) 1.84 (3.5) 2.35 (3.5) 1.68 (4.2) 1.83 (3.6)

Mean (SD) six month follow-up 1.88 (0.90) 1.87 (0.82) 1.87 (0.82) 1.86 (0.60) 2.06 (0.85) 1.71 (3.1) 1.67 (3.7) 1.62 (3.7) 1.87 (3.6)

Difference -4.4% -21.4%* -3.3% 4.1% -13.9%** 7.1% 28.9% 3.6% 2.2%

*p <0.05

**p <0.001

Satisfaction - Providers 

Provider satisfaction was measured with four questions that were part of a survey 18 months after the clinic 

launched Collaborative Care. All of the clinic providers were approached for participation, including care managers, 

primary care providers, psychiatric consultants, and behavioral health providers. See Tables 4 and 5 for 

demographic, training and employment characteristics of providers who responded to the survey. 

Table 11. Provider Satisfaction with Patient Outcomes, Support from Clinic 

Organization Do you believe that the IMPACT 

program improved clinical outcomes 

for your patients?

Did you receive adequate support from the 

clinic to provide depression care to your 

IMPACT patients?

None Moderate 

Improvement

Definite 

Improvement

Not enough 

support

Adequate 

support

More than 

adequate support

Clinic A 0% 0% 100% 9% 18% 73%

Clinic B 0% 29% 71% 29% 71% 0%

Clinic C 0% 38% 62% 7% 59% 34%

Clinic D 0% 12% 88% 6% 56% 38%

Clinic E 0% 11% 89% 0% 67% 33%

Clinic F 0% 40% 60% 0% 60% 40%

Clinic G 14% 29% 57% 14% 71% 14%

Clinic H 10% 50% 40% 11% 79% 11%

Total 2% 27% 71% 9% 58% 33%

Across the eight participating clinics, nearly three quarters of providers believe the IMPACT program 

resulted in “definite improvement” for their patients. Only two clinics (Clinic B and Clinic H) had providers who 

responded there was no improvement in clinical 

outcomes for their patients. Not surprisingly, Clinic B is 

the only clinic where none of the providers surveyed said 

they received “more than adequate support” to provide 

depression treatment to their patients and more than 

twice as many providers as any other clinic responded they received “not enough support.”

“The PCPs at this clinic have not embraced the model 

and after two years (and education) they still have not 

grasped simple concepts.”

- Primary Care Provider at Clinic B
 

SIF IMPACT Expansion 30 Final Report: October 2018 

Table 12. Provider Satisfaction with Quality of Work Life, Level of Burnout 

Organization Has IMPACT improved the quality of 

your work life?*

How would you rate your level of burnout?

None Moderate 

Improvement

Definite 

Improvement

None Some A Lot

Clinic A 18% 0% 82% 27% 55% 18%

Clinic B 43% 57% 0% 0% 57% 43%

Clinic C 11% 36% 53% 11% 43% 46%

Clinic D 19% 44% 37% 18% 41% 41%

Clinic E 0% 56% 44% 11% 67% 22%

Clinic F 0% 40% 60% 40% 60% 0%

Clinic G 14% 14% 72% 14% 71% 14%

Clinic H 20% 70% 10% 0% 60% 40%

Total 15% 39% 46% 14% 52% 34%

*Clinics differ significantly (p = .02) 

At seven of the eight clinics, at least 80% of providers reported that IMPACT improved the quality of their 

work like at least moderately. The only clinic at which none of the providers reported “definite improvement” in 

work life was Clinic B. These differences are statistically significant. Providers at Clinic F reported the lowest levels of 

burnout while providers at Clinic B reported the highest levels and providers at Clinic H were second highest.

“In our rural setting, the ability to 

provide better mental health care 

has made a big impact for my 

patients and has definitely improved 

my job satisfaction.”

– Primary Care Provider at Clinic E
 

SIF IMPACT Expansion 31 Final Report: October 2018 

Impact Evaluation 

Baseline Depression Severity 

Table 13 shows baseline depression severity by clinic. Baseline PHQ-9 is the measure of symptoms at 

treatment initiation. Maximum PHQ-9 shows the highest PHQ-9 score during the first (if more than one) episode of 

care. Baseline suicidal ideation is based on a single item from the PHQ-9. 

Table 13. Baseline Depression Symptom Severity

Clinic A Clinic B Clinic C Clinic D Clinic E Clinic F Clinic G Clinic H All

n=728 n=405 n=1524 n=890 n=633 n=207 n=262 n=538 n=5187

Baseline PHQ-9 #

%

#

%

#

%

#

%

#

%

#

%

#

%

#

%

#

%

Mild < 10 62 

(8.5)

53 

(13.1)

203 

(13.3)

64 

(7.2)

85 

(13.4)

44 

(21.3)

62 

(23.7)

33 

(6.1)

606 

(11.7)

Moderate 10-14 226 

(31.0)

105 

(25.9)

386 

(25.3)

242 

(27.2)

144 

(22.7)

76 

(36.7)

94 

(35.9)

138 

(25.7)

1411 

(27.2)

Mod / Severe 15-19 238 

(32.7)

122 

(30.1)

449 

(29.5)

295 

(33.1)

194 

(30.6)

49 

(23.7)

72 

(27.5)

186 

(34.6)

1605 

(30.9)

Severe 20-27 202 

(27.7)

125 

(30.9)

486 

(31.9)

289 

(32.5)

210 

(33.2)

38 

(18.4)

34 

(13.0)

181 

(33.6)

1565 

(30.2)

Mean 

(Std Dev)

16.1 

(5.2)

16.0 

(5.6)

16.1 

(6.1)

16.4 

(5.3)

16.5 

(5.9)

13.9 

(5.8)

13.1 

(5.8)

17.1 

(5.2)

16.1 

(5.7)

Maximum PHQ-9

Mild / Moderate 0-14 229 

(31.5)

127 

(31.4)

490 

(32.2)

243 

(27.3)

183 

(28.9)

100 

(48.3)

139 

(53.1)

126 

(23.4)

1637 

(31.6)

Mod / Severe 15-19 249 

(34.2)

126 

(31.1)

460 

(30.2)

292 

(32.8)

191 

(30.2)

55 

(26.6)

75 

(28.6)

177 

(32.9)

1625 

(31.3)

Severe 20-27 250 

(34.3)

152 

(37.5)

574 

(37.7)

355 

(39.9)

259 

(40.9)

52 

(25.1)

48 

(18.3)

235 

(43.7)

1925 

(37.1)

Baseline suicidal 

ideation

Yes (Q9>0) 226 

(31.0)

144 

(35.6)

458 

(30.1)

359 

(40.3)

247 

(39.0)

79 

(38.2)

59 

(22.5)

90 

(16.7)

1662 

(32.0)
 

SIF IMPACT Expansion 32 Final Report: October 2018 

Primary Outcomes – Decrease in Depression Symptoms 

Table 14. Depression Outcomes: Initial to Last Measurement 

Mean 95% CI Standard 

Deviation

Depression severity (range 0-27)

PHQ-9 at baseline 16.1 15.9, 16.2 5.7

PHQ-9 at last measurement 10.9 10.8, 11.1 7.2

Change from baseline to last 5.1a,b 4.9, 5.3 6.7

Suicidal ideation (range 0-3)

Baseline 0.59 0.57, 0.62 0.90

Last measurement 0.35 0.33, 0.37 0.73

Change from baseline to last 0.25a,b 0.22, 0.27 0.77

a

 P-value<0.0001 b Paired t-test 

The unadjusted change in depression severity and suicidal ideation from baseline to last measurement is 

shown in Table 14. Unadjusted baseline depression severity as measured by the PHQ-9 (range 0-27) averaged 16.1, 

with a standard deviation (SD) of 5.7 (95% confidence interval 13.1, 17.1) across the eight participating clinics. Mean 

PHQ-9 score at last measurement was 10.9 (SD 7.2; 95% CI 10.8, 11.1). The mean change in PHQ-9 score from first to 

last measurement was 5.1 points (SD 6.7; 95% CI 4.9, 5.3), a drop in depression severity which is both statistically 

and clinically significant.46 Suicidal ideation, a single item on the PHQ-9, was reported (at any severity level) by 32% 

of patients at baseline and by 20% of patients at the last follow-up with the mean severity of suicidal ideation 

decreasing from 0.60 to 0.35 on a scale from 0-3. The mean change from first to last measurement was 0.25, which 

is both statistically and clinically significant (p<0.0001). 

Table 15. Depression Response and Remission Rates by Baseline PHQ-9 Category 

Overall Response Remission

Baseline PHQ-9 (range 0-27) N (%) N (%) N (%)

Mild (< 10) 606 (11.7) 538 (88.8) 307 (50.7)

Moderate (10-14) 1411 (27.2) 723 (51.2) 398 (28.2)

Moderately Severe (15-19) 1605 (30.9) 658 (41.0) 330 (20.6)

Severe (20-27) 1565 (30.2) 531 (33.9) 203 (13.0)

Table 15 shows PHQ-9 scores at baseline categorized into four groups: mild, moderate, moderately severe, 

and severe with the corresponding proportion of patients achieving depression response or remission. Both 

outcomes show a linear relationship between PHQ-9 score and likelihood of response or remission with the 

likelihood of each outcome decreasing as the baseline PHQ-9 score increases. 
 

SIF IMPACT Expansion 33 Final Report: October 2018 

“It’s really hard to keep in touch with 

patients when it’s fishing season and 

they are at fish camp. A lot of the 

camps are too remote for cell phone 

service and they’ve got to fish when 

they can so their family has food to 

eat in the winter. So, quite a few of 

my patients fell off the radar until 

fishing season was over.”

– Care Manager at Clinic G

Primary Outcomes –Depression Response and Remission 

Odds ratios, their 95% CIs, Ns, and unadjusted percentages for the logistic regression model are shown 

for depression response (Table 16) and remission (Table 17). The models accounted for 26% and 21% of the 

variance in response and remission, respectively. Clinic explained a significant amount of variance, with 

depression response ranging from 42% to 66% across the eight participating clinics and depression remission 

ranging from 19% to 34%. There were no differences in the main outcomes between racial/ethnic groups or 

genders. Age was significantly related to depression remission with younger patients (< 66 years) experiencing 

a significantly lower rate of remission than patients over sixty-five (27% - 38%). 

 Maximum PHQ-9 score explained a significant amount of variance in outcomes, with positive outcomes 

related to lower maximum depression severity. Rates of response and remission for patients with a baseline PHQ-9 

score below 15 were 68% and 51%, respectively. Patients with severe 

symptoms at baseline (score 20 - 27) had the lowest rates of response 

(33%) and remission (13%). Receiving a follow-up contact within 30 

days from enrollment was the only process variable not significant in 

the model and, consequently, it was dropped. Each month in 

treatment increased a patient’s odds of having a depression response 

7% and of remission 3%. Patients with prior depression treatment 

episodes in the same clinic were slightly less likely to have a response 

or remission. Number of follow-up contacts during the treatment 

episode was strongly linearly associated with better response and 

remission rates. Patients with four or more visits were about 4 times 

more likely to improve as compared to those with only one visit. 
 

SIF IMPACT Expansion 34 Final Report: October 2018 

Table 16. Predicting Depression Response by Clinic, Age, Race, Gender, and Treatment Variables 

Depression Response (2450/5187)a

N % Odds Ratio LowerCL UpperCL p-value

Organization

Clinic C (Referent) 638 41.86 - - - -

Clinic A 377 51.79 1.45 1.16 1.80 0.001

Clinic B 200 49.38 1.59 1.20 2.12 0.001

Clinic D 435 48.88 1.84 1.49 2.29 < 0.001

Clinic E 308 48.66 1.19 0.95 1.49 0.140

Clinic F 137 66.18 3.34 2.12 5.24 < 0.001

Clinic G 138 52.67 1.06 0.70 1.60 0.798

Clinic H 217 40.33 1.05 0.82 1.34 0.699

Age

Age 18-24 (referent) 345 44.52 - - - -

age 25-45 1185 46.53 0.98 0.80 1.21 0.867

age 46-64 762 47.74 0.90 0.72 1.13 0.366

age >=65 148 60.66 1.47 1.01 2.14 0.043

Race

White (referent) 2004 46.94 - - - -

AI/NA 207 52.41 1.14 0.80 1.63 0.475

Other 239 45.70 0.93 0.73 1.17 0.532

Gender

Male (referent) 749 45.92 - - - -

Female 1661 48.78 1.05 0.91 1.22 0.495

Severity of max PHQ-9 

None/Moderate (0-14) (referent) 1111 67.87 - - - -

Moderately severe (PHQ 15-19) 708 43.57 0.24 0.20 0.29 < 0.001

Severe (PHQ 20-27) 631 32.78 0.12 0.10 0.15 < 0.001

Treatment exposure (months)

Continuous - - 1.07 1.04 1.11 < 0.001

Total episodes

1 episode (referent) 2209 47.80 - - - -

Two or more episodes 241 42.58 0.69 0.56 0.86 < 0.001

Total follow-up (FU)

1 FU (referent) 277 34.32 - - - -

2-3 FU 601 52.35 2.52 2.04 3.12 < 0.001

4-7 FU 714 61.34 3.61 2.87 4.53 < 0.001

8+ FU 740 63.96 3.84 2.85 5.16 < 0.001

a

R2max=0.2596
 

SIF IMPACT Expansion 35 Final Report: October 2018 

Table 17. Predicting Depression Remission by Clinic, Age, Race, Gender, and Treatment Variables 

Depression Remission (1238/5187)a

N % Odds Ratio LowerCL UpperCL p-value

Organization

Clinic C (referent) 294 19.29 - - - -

Clinic A 231 31.73 1.97 1.56 2.48 < 0.001

Clinic B 123 30.37 2.25 1.68 3.01 < 0.001

Clinic D 201 22.58 1.47 1.16 1.86 0.001

Clinic E 139 21.96 1.01 0.78 1.31 0.952

Clinic F 72 34.78 2.35 1.57 3.50 < 0.001

Clinic G 70 26.72 1.23 0.80 1.91 0.345

Clinic H 108 20.07 1.19 0.90 1.58 0.218

Age

Age 18-24 (referent) 159 20.52 - - - -

age 25-45 580 22.77 1.14 0.91 1.44 0.252

age 46-64 405 25.38 1.24 0.98 1.58 0.078

age >=65 89 36.48 1.80 1.25 2.60 0.002

Race

White (referent) 1011 23.68 - - - -

AI/NA 95 24.05 0.83 0.57 1.21 0.331

Other 132 25.24 1.22 0.95 1.56 0.124

Gender

Male (referent) 359 22.01 - - - -

Female 860 25.26 1.16 0.99 1.36 0.065

Severity of max PHQ-9 

None/Moderate (0-14) (referent) 632 38.61 - - - -

Moderately severe (PHQ 15-19) 357 21.97 0.34 0.28 0.41 < 0.001

Severe (PHQ 20-27) 249 12.94 0.16 0.13 0.19 < 0.001

Treatment exposure (months)

Continuous - - 1.03 1.00 1.06 0.051

Total episodes

1 episode (referent) 1123 24.30 - - - -

Two or more episodes 115 20.32 0.79 0.62 1.01 0.056

Total follow-up (FU)

1 FU (referent) 112 13.88 - - - -

2-3 FU 287 25.00 2.29 1.77 2.97 < 0.001

4-7 FU 418 35.91 3.96 3.04 5.15 < 0.001

8+ FU 381 32.93 3.84 2.77 5.32 < 0.001

a

R2max= 0.2087

 
SIF IMPACT Expansion 36 Final Report: October 2018 

Secondary Outcome – Effects on Social and Occupational Functioning 

“I’ve experienced professional 

satisfaction and gratitude when 

young patients (20s and 30s) get to 

redirect their life trajectory away 

from seeking disability to finding 

employment, pursuing education and 

improving their level of self-care and 

the quality of their relationships.”

– Care Manager at Clinic E

Effects of depression treatment on interference with 

social and occupational functioning was measured in the subset 

of patients who completed the patient survey at baseline and 6 

months following treatment initiation, whether or not the 

patient was still engaged in depression treatment at the clinic. 

Occupational interference was used to measure economic wellbeing (see Impacts, Table 1). Table 18 shows the effects of 

IMPACT depression treatment on functional interference. 

Reductions in interference indicate an improvement in 

functioning. Both social and occupational functioning improved a 

highly statistically significant amount. 

Table 18. Effects of Collaborative Care on Social and Occupational Functioning 

*p< .001

Secondary Outcome – Effects on Medical Care Utilization 

Effects of depression treatment on utilization of medical care 

services, including primary care, medical specialists, emergency room, and 

inpatient medical care was measured in the subset of patients who 

completed the patient survey at baseline and 6 months following treatment 

initiation, whether or not the patient was still engaged in depression 

treatment at the clinic. Table 19 shows the effects of IMPACT depression 

treatment on medical care utilization. The only category showing a 

substantial difference is primary care medical visits, which dropped by 

almost half from baseline to the 6 month follow-up.

Social Functioning Interference n= 341

Mean (SD) baseline interference 4.66 (3.68)

Mean (SD) six month follow-up interference 3.69 (3.55)

Difference

Occupational Functioning Interference n= 337

Mean (SD) baseline interference 4.85 (3.46)

Mean (SD) six month follow-up interference 3.92 (3.37)

Difference 19.2%*

20.8%*

Total

“I believe IMPACT has greatly 

decreased patient visits to the ER, 

which positively benefits our 

patients’ and our community 

hospital’s financial stability and the 

hospital’s ability to help patients in 

truly emergency situations.”

– Care Manager at Clinic E
SIF IMPACT Expansion 37 Final Report: October 2018 

Table 19. Effects of Collaborative Care on Medical Care Utilization 

Utilization of Medical Care at Baseline

Mean (SD) primary care medical visits in past 6 mos

Mean (SD) medical specialist visits in past 6 mos

Mean (SD) emergency room visitsin past 6 mos

Mean (SD) inpatient medical nights in past 6 mos

Utilization of Medical Care at 6 months

Mean (SD) primary care medical visits in past 6 mos

Mean (SD) medical specialist visits in past 6 mos

Mean (SD) emergency room visits in past 6 mos

Mean (SD) inpatient medical nights in past 6 mos

All Clinics

4.1 (5.4)

0.7 (1.3)

1.2 (3.8)

8.0 (9.9)

4.0 (5.1)

0.6 (1.4)

1.2 (3.2)

4.2 (4.3)

Secondary Outcome – Effects on Depression Care Utilization 

Effects of depression treatment on utilization of depression care 

services, including primary care, mental health specialists, and inpatient 

psychiatric care was measured in the subset of patients who completed the 

patient survey at baseline and 6 months following treatment initiation, 

whether or not the patient was still engaged in depression treatment at the 

clinic. Table 20 shows the rates of prior depression treatment among this 

group of patients. As expected, over three quarters of patients reported prior 

treatment for depression including two thirds within the past year. Table 20

also shows effects of IMPACT depression treatment on depression care 

utilization. The only category showing a substantial difference is primary care 

depression visits, which more than doubled from baseline to the 6 month 

follow-up.

Table 20. Effects of Collaborative Care on Depression Care Utilization 

“She makes me feel like I have 

a true friend. I can really open 

up to her. She’s a gift. I’ve 

needed her for many, many 

years.”

– Patient at Clinic E

“This program helped me get 

the help I needed. It was a 

good program I really 

appreciated it. I had been 

taking medication but now I 

don’t need it.”

– Patient at Clinic A

Utilization of Depression Treatment at Baseline

Any prior depression treatment

Depression care within past year

Mean (SD) primary care depression visits in past 6 mos

Mean (SD) mental health specialist visits in past 6 mos

Mean (SD) inpatient psychiatric nights w/in past 6 mos

Utilization of Depression Treatment at 6 month Follow-up

Mean (SD) primary care depression visits in past 6 mos

Mean (SD) mental health specialist visits in past 6 mos

Mean (SD) inpatient psychiatric nights w/in past 6 mos

67.3%

All Clinics

76.7%

6.0 (4.7)

3.6 (6.4)

5.1 (6.5)

5.0 (1.6)

2.0 (4.6)

2.5 (3.2)
 

SIF IMPACT Expansion 38 Final Report: October 2018 

DISCUSSION

This SIF initiative demonstrates that rural primary care clinics serving low-income patients can achieve 

depression remission and response rates comparable to those published in research trials1-6 and large-scale 

implementations.13-16 About 24% of SIF-CC patients achieved remission, which is within the range found by 

other large implementations, which have documented remission rates between 24% and 52% among 

participating clinics, indicating substantial clinic variation. Variation in clinic-level processes of care and patientlevel clinical outcomes is well documented in other Collaborative Care implementations.13,15,16 Collaborative 

Care is multi-component, team-based treatment model requiring two new roles and substantial, sustained 

changes to clinic workflows and processes.47 A growing body of implementation literature indicates it is more 

difficult to unlearn old habits than learn new ones,48 which is problematic when quality improvement requires 

replacing old roles, workflows, and processes with new ones. 

As seen in other depression studies,49 patients with more severe baseline were less likely to experience 

remission. Remission of symptoms (PHQ-9 score less than 5) is a difficult target, especially with a low-income 

population experiencing the compound stressors associated with economic disadvantage.50 This is particularly 

challenging in rural areas because there may not be options for referral to specialty care. These findings also 

suggest that dose and duration of Collaborative Care matters. The mean duration of treatment was 3.5 months 

with longer duration of treatment associated with better depression outcomes. Similarly, patients with more 

CM contacts had better depression outcomes, suggesting future Collaborative Care implementations in rural 

clinics should focus on maintaining patients in treatment longer. Older patients were more likely to experience 

depression remission, a finding that contradicts prior randomized controlled trials6 but mirrors real-world 

Collaborative Care implementation.15 

Primary care providers, in short supply in rural clinics, were positive about their experiences with 

Collaborative Care which replicates findings from other implementations.13,34 This is important because 

provider burnout is associated with poorer health outcomes for patients51 and lower retention of providers.52 

Innovations to address the accelerating gap between the need for mental health services and the 

available workforce, especially in rural areas where the gap is greatest, are needed. Understanding 

implementation methods that effectively scale evidence-based intervention that requiring systems and 

providers to change roles and workflows are also necessary.53-54 

Lessons Learned 

Implementation 

The AIMS Center has over 13 years’ experience assisting 1,000’s of organizations implement Collaborative 

Care. Their SIF experience mirrors lessons learned with other implementations. These include: 





Collaborative Care is a complex multi-component clinical practice 

change that affects the entire workflow of the clinic and all clinic 

staff. As such, every organization experiences unexpected 

implementation challenges no matter how well they planned and 

prepared for implementation. 

Clinics and providers rarely understand the true scale of disruption to 

usual practice that will be required to effectively implement 

Collaborative Care when they embark on implementation, no matter how much they are told that this is a 

large-scale systemic change to clinic operations. Experience is believing.

“The pressure to make changes 

quickly was challenging. When 

change happens fast it isn’t well 

thought-out and the clinic can 

quickly lose buy-in”

– Behavioral Health Provider at Clinic C
 

SIF IMPACT Expansion 39 Final Report: October 2018 















Some of the same principles that drive better clinical outcomes for patients in IMPACT also drive better 

clinical outcomes at the organizational level. Regularly reviewing data about both processes of care and 

clinical outcomes and using this information to make adjustments to program implementation mirrors the 

treat-to-target principles clinics are taught to employ at the individual patient level. 

Clinics with an existing co-located behavioral health program who intend to keep that program while 

simultaneously implementing IMPACT as a complementary service often experience more implementation 

challenges than clinics without an existing behavioral health service or clinics remaking all of their behavioral 

health services to align with IMPACT. This is referred to as “de-implementation” in the Implementation 

Science literature and is a well-recognized impediment to adoption and sustainment.55-56

Patients in rural areas can be more challenging to engage and retain in depression treatment. Reasons for 

this include travel distances, lack of reliable/available telephone service, higher prevalence of stigma 

associated with mental health care, and more prevalent cultural expectations of independence and selfreliance. 

If an organization is struggling with implementation challenges it’s 

important to recognize those as early as possible and intensify 

coaching and assistance. Proactive identification and intervention 

increase the likelihood that the clinic will be able to resolve the 

challenges before they derail implementation. 

Genuine buy-in and enthusiasm for practice change is necessary 

among key stakeholders, including clinic leadership, medical 

providers, support staff and behavioral health staff. 

Implementation is a process that evolves over time. Patience with 

this process is often one of the biggest challenges for clinics because 

they expect everything to change on the day they launch the 

program and they expect everything to be in place immediately 

when that is not realistic with such a complex clinical practice 

change. 

Long-term sustainability planning requires significant individual assistance and a sustained effort because 

the strengths and challenges faced by clinics in this topic area are highly individual and ongoing. Intensive 

onsite assistance after laying a foundation is particularly effective in helping organizations accomplish 

sustainability. 

“I called a meeting with the 

CEO and the CFO and I said, 

“We have a decision to make. 

Either we fall in with this or we 

just let it go. We can’t be half 

in and half out. It has to be the 

whole clinic bought in and 

meeting the requirements.” I 

was extremely irritated...this is 

where they said “No, no, 

whatever you need.” And so 

from then on I just had to keep 

reminding them - You said 

whatever I need.”

– Implementation Leader at Clinic A

Specific lessons learned for the Social Innovation Fund IMPACT initiative: 

 All of the subgrantee sites participated in planning, training, and post-launch coaching activities conducted 

by the AIMS Center, though for some participation has been more fully engaged and for others it has been 

more cursory. The clinics with more cursory participation in training and implementation support activities 

had poorer clinical outcomes. 
 

SIF IMPACT Expansion 40 Final Report: October 2018 





















The Cohort 1 clinic that struggled most with early implementation (reported in a prior interim evaluation 

report) received intensive onsite coaching and training and used this assistance to turn things around and 

they are now one of the highest performing clinics. 

All but two clinics exceeded their enrollment goals. The two sites serve a largely American Indian / Alaska 

Native patient population. Their challenges identifying and engaging patients in depression treatment mirror 

the AIMS Center’s prior experience with Alaska Native health clinics as part of an earlier implementation 

initiative. 

Consistent with our prior experience with low-income patients served at community primary care clinics, 

the eight subgrantee clinics are seeing more than half (61%) of their IMPACT patients with moderately 

severe or severe depression symptoms at baseline and a mean PHQ-9 of 16.1 (moderately severe). Through 

the SIF IMPACT initiative, the mean PHQ-9 dropped 5.1 points to 10.9 (p value <0.0001) at last 

measurement. This change is both statistically and clinically significant. 

Clinic explained a significant amount of variance in depression response and remission, a finding consistent 

with the published literature. 

Maximum PHQ-9 score explained a significant amount of variance in outcomes, with positive outcomes 

related to lower maximum depression severity. 

Each month in treatment increased a patient’s odds of having a depression response 7% and of remission 

3% and the number of follow-up contacts during the treatment episode was strongly linearly associated with 

better response and remission rates. 

Clinics located in rural and frontier areas face significant workforce challenges for all types of providers, 

including primary care and behavioral health. To accommodate this, we offered clinics the option of two 

different models for the care manager role: 1) all-in-one, which is a licensed behavioral health provider 

capable of the full range of care manager responsibilities including treatment planning and delivering 

psychotherapy, and 2) shared care manager, in which all of the tasks that don’t require a license are 

performed by a medical assistant, community health worker, or similar paraprofessional so licensed 

behavioral health providers focus their limited time on the tasks that require a license. 

Similarly, the AIMS Center helped the participating clinics be creative about how to fill the psychiatric 

consultant role. In most Collaborative Care implementations the consultant role is performed by a 

psychiatrist. However, psychiatrists are exceedingly rare in rural areas. Several clinics utilized the services of 

a psychiatric nurse practitioner as their psychiatric consultant and several clinics had psychiatric consultants 

that supported them remotely via the CMTS registry and telephone. 

The culture of the rural west was another challenge the clinics grappled with. Stoic self-reliance is a strong 

cultural value and can be an impediment to seeking or accepting help. 

Rural areas are defined by their low population density. Isolation can trigger or exacerbate depression and it 

can be difficult to address this effectively with patients. Some clinics created groups for patients, in part as a 

way to provide them with an opportunity for social interaction. 
 

SIF IMPACT Expansion 41 Final Report: October 2018 

Evaluation 

Lessons learned about evaluation of this SIF initiative: 















Clinic leadership at all of the subgrantee sites participated in baseline data collection about their clinic, 

patient population and other organizational factors that may influence program implementation. 

Cohort 1 and cohort 2 sites completed follow-up (18 month) data collection about organizational factors 

that may influence program implementation. 

Baseline patient survey data collection was completed after the evaluation team, in consultation with CNCS 

and JAHF, decided to make the patient registry the data source for the primary outcome. 

Implementing a patient incentive ($40 gas card) positively affected completion rates for the follow-up 

survey. 

We can accomplish an 80% retention rate for the patient survey, collecting a total of 354 paired surveys 

(baseline and follow-up surveys), beating our goal of a 30% max attrition rate. 

However, recruiting rural, low-income patients to participate in a baseline and 6 month survey was difficult. 

When it became clear we could not recruit a sufficiently representative sample of patients we requested 

and received permission to revise the SEP to make the CMTS registry the primary data source. 

We can achieve an average response rate higher than 80% for clinician surveys. 

RECOMMENDATIONS

Program Recommendations 





Recognize that implementation in rural and frontier communities requires additional time and support than 

typically necessary for urban and suburban clinics. However, rural communities also have strengths that can 

facilitate effective implementation if these are recognized and leveraged. Consider using the “Rural 

Implementation Formula”55 which recommends: 

1. “Leverage rural communities’ strengths by investing in relationships as an implementation asset. 

2. Fund and build creative individual and organizational capacity in rural communities. 

3. Partner with rural communities to translate research-based programs and practices to the rural 

context. 

4. Support extended implementation timelines.”

Further develop and test the efficacy and effectiveness of the Organizational Relapse Prevention process. 

Identify opportunities to test this process with other evidence-based clinical innovations, either 

Collaborative Care or other interventions, to increase understanding of the predictors of sustainment. 
 

SIF IMPACT Expansion 42 Final Report: October 2018 

Evaluation Recommendations 







Further analyze data from the Collaborative Care version of the Stages of Implementation Completion (SIC) 

and identify a future opportunity in which to use this tool prospectively as a real-time feedback mechanism 

for organizations implementing Collaborative Care and evaluate effects on implementation success. 

Analyze clinical outcomes and processes of care for American Indian / Alaska Native patients to better 

understand effective delivery of depression treatment to this marginalized and underserved population 

which has the highest completed suicide rate of any ethnic group. 

Analyze clinical outcomes and processes of care for clinics that continue to use the CMTS registry after 

conclusion of funding to observe sustainment. 
 

SIF IMPACT Expansion 43 Final Report: October 2018 

REFERENCES 

1. Thota AB, Sipe TA, Byard GJ, et al. Collaborative care to improve the management of depressive disorders: 

A community guide systematic review and meta-analysis. Am J Prev Med. 2012;42(5):525-538. 

doi:10.1016/j.amepre.2012.01.019 

2. Archer J, Bower P, Gilbody S, et al. Collaborative care for depression and anxiety problems. Cochrane 

Database Syst Rev. 2012;10:CD006525. doi:10.1002/14651858.CD006525.pub2 

3. Bower P, Gilbody S, Richards D, Fletcher J, Sutton A. Collaborative care for depression in primary care. 

Making sense of a complex intervention: Systematic review and meta-regression. Br J Psychiatry. 

2006;189:484-493. doi:10.1192/bjp.bp.106.023655 

4. Coventry PA, Hudson JL, Kontopantelis E, et al. Characteristics of effective collaborative care for treatment 

of depression: A systematic review and meta-regression of 74 randomised controlled trials. PLoS One. 

2014;9(9):e108114. doi:10.1371/journal.pone.0108114 

5. Gilbody, S., et al., Collaborative care for depression: a cumulative meta-analysis and review of longer-term 

outcomes. Arch Intern Med, 2006. 166(21): p. 2314 - 21. 

6. Unutzer J, Katon W, Callahan CM, et al. Collaborative care management of late-life depression in the 

primary care setting: A randomized controlled trial. JAMA. 2002;288(22):2836-2845 

7. Lin EH, Katon W, Von Korff M, Tang L, Williams JW, Jr., Kroenke K, et al. Effect of improving depression care 

on pain and functional outcomes among older adults with arthritis: a randomized controlled trial. JAMA

2003;290(18):2428-9.

8. Callahan CM, Kroenke K, Counsell SR, Hendrie HC, Perkins AJ, Katon W, et al. Treatment of depression 

improves physical functioning in older adults. J Am Geriatr Soc 2005;53(3):367-73.

9. Hunkeler EM, Katon W, Tang L, Williams JW, Jr., Kroenke K, Lin EH, et al. Long term outcomes from the 

IMPACT randomised trial for depressed elderly patients in primary care. BMJ 2006;332(7536):259-63.

10. Areán PA, Ayalon L, Hunkeler E, et al. Improving depression care for older, minority patients in primary 

care. Med Care. 2005;43(4):381-390 

11. Ratzliff AD, Ni K, Chan YF, Park M, Unutzer J. A collaborative care approach to depression treatment for 

Asian Americans. Psychiatr Serv. 2013;64(5):487-490. doi:10.1176/appi.ps.001742012 

12. Huang H, Chan YF, Katon W, et al. Variations in depression care and outcomes among high-risk mothers 

from different racial/ethnic groups. Fam Pract 2012;29(4):394-400. doi:10.1093/fampra/cmr108 

13. Bauer AM, Azzone V, Goldman HH, et al. Implementation of collaborative depression management at 

community-based primary care clinics: An evaluation. Psychiatr Serv. 2011;62(9):1047-1053. 

doi:10.1176/appi.ps.62.9.1047 

14. Joesch JM, Short R, Mancuso D, Krupski T, et.al. Evaluation of the general assistance managed care pilot in 

King and Pierce counties for the period January 2008 through September 2009. King County website: 

https://www.kingcounty.gov/healthservices/MentalHealth/Board/~/media/health/mentalHealth/Board/20

11/Handouts/DL_GA_U_report_2_16_11c_BRIEF_REPORT_FINAL.ashx. Accessed: September 24, 2018 

15. Rossom RC, Solberg LI, Magnan S, et al. Impact of a national collaborative care initiative for patients with 

depression and diabetes or cardiovascular disease. Gen Hosp Psychiatry. 2017;44:77-85. 

doi:10.1016/j.genhosppsych.2016.05.006 

16. Bao Y, Druss B, Jung H-Y, Chan Y-F, Unützer J. Unpacking collaborative care for depression: Examining two 

essential tasks for implementation. Psychiatr Serv. 2016;67(4):418-424. doi:10.1176/appi.ps.201400577 

17. Bower P, Gilbody S, Richards D, Fletcher J, Sutton A. Collaborative care for depression in primary care. 

Making sense of a complex intervention: Systematic review and meta-regression. Br J Psychiatry. 

2006;189:484-493. doi:10.1192/bjp.bp.106.023655 
SIF IMPACT Expansion 44 Final Report: October 2018

18. Rush AJ, Kraemer HC, Sackeim HA, et al. Report by the ACNP Task Force on Response and Remission in 

Major Depressive Disorder. Neuropsychopharmacology. 2006;31(9):1841-1853. 

doi:10.1038/sj.npp.1301131

19. Unutzer J, Choi Y, Cook IA, Oishi S. A web-based data management system to improve care for depression 

in a multicenter clinical trial. Psychiatr Serv 2002;53(6):671-3, 8.

20. Health Resources and Services Administration. Defining the Rural Population. Available from: 

http://www.hrsa.gov/ruralhealth/policy/definition_of_rural.html.

21. Health Resources and Services Administration. Health Professional Shortage Areas (HPSAs). Available from: 

http://bhpr.hrsa.gov/shortage/hpsas/.

22. Kaiser Family Foundation. Available from: www.statehealthfacts.org.

23. United States Census. Income, Poverty and Health Insurance Coverage in the U.S.: 2014. Census.gov. 

Retrieved 18 October 2017.

24. Albrecht, D.E., The changing west - a regional overview, in Population Brief: Trends in the Western U.S.

2008, Utah State University: Logan, Utah.

25. Rural Policy Institute, Alaska, Idaho, Montana, Washington, Wyoming, in RUPRI State Demographic and 

Economic Profile Series. 2008-2009, Rural Policy Institute: Columbia, MO.

26. Wagner, E.H., B.T. Austin, and M. Von Korff, Organizing care for patients with chronic illness. Milbank Q, 

1996. 74(4): p. 511-44.

27. Trivedi, M.H., Treating depression to full remission. J Clin Psychiatry, 2009. 70(1): p. e01.

28. Kroenke, K., R.L. Spitzer, and J.B. Williams, The PHQ-9: validity of a brief depression severity measure. J Gen 

Intern Med, 2001. 16(9): p. 606-13.

29. Henke, R.M., et al., Clinical inertia in depression treatment. Med Care, 2009. 47(9): p. 959-67.

30. Simon GE, Rutter CM, Peterson D, et al. Does response on the PHQ-9 Depression Questionnaire predict 

subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64(12):1195-1202. 

doi:10.1176/appi.ps.201200587

31. Chamberlain, P., Brown, C.H., Saldana, L., Observational measure of implementation progress in 

community based settings; the Stages of Implementation Completion (SIC). Implement Sci, 2011 Oct. 6; 

6:116.

32. Saldana, L., The stages of implementation completion for evidence-based practice: protocol for a mixed 

methods study. Implement Sci, 2014. 9(1): p. 43.

33. Saldana, L., et al., Predicting program start-up using the stages of implementation measure. Adm Policy 

Ment Health, 2012. 39(6): p. 419-25.

34. Levine, S., et al., Physicians' satisfaction with a collaborative disease management program for late-life 

deprdimu, T.F., Martinex, A.P., Minyard, K.ession in primary care. Gen Hosp Psychiatry, 2005. 27(6): p. 383-

91.

35. Smith, T.T., Adimu T.F., Martinez, A.P., Minyard, K. Selecting, adapting and implementing evidence-based 

interventions in rural settings: an analysis of 70 community examples. J Health Care for the Poor and 

Underserved, 2016. 27(4), 181-193.

36. United States Department of Agriculture, Rural America at a Glance: 2016 Edition. Washington, DC, 2016. 

Retrieved from https://www.ers.usda.gov

37. United States Department of Health and Human Services, Health Resources and Services Administration, 

Maternal and Child Health Bureau. The health and well-being of children in rural areas: A portrain of the 

nation. Rockville, MD, 2011-2012. Retrieved from https://mchb.hrsa.gov.

38. Dotson J.A.W, Roll, J.M., Packer, R.R., et.al. Urban and rural utilization of evidence-based practices for 

substance use and mental health disorders. J Rural Health, 2014. 30:292-299.

39. Eberhardt, M.S., Pamuk, E.R. The importance of place and residence: examining health in rural and nonrural areas, Amer J Public Health, 2004. 94(10), 1682-1686.
 

SIF IMPACT Expansion 45 Final Report: October 2018 

40. Centers for Disease Control and Prevention, Data and Statistics Fatal Injury Report for 2015. Atlanta, GA, 

2015. Retrieved from https://webappa.cdc.gov/sasweb/ndpc/mortrate.html. 

41. Richardson, L., Ludman, E., McCauley, E., et al., Collaborative care for adolescents with depression in 

primary care: A randomized trial. JAMA, 2014;312(8):809-816. 

42. Brody, J., Pratt, L., Hughes, J., Prevalence of Depression Among Adults Aged 20 and Over: United States, 

2013-2016. US Department of Health and Human Services, Centers for Disease Control and Prevention, 

National Center for Health Statistics. Data Brief No. 303, February 2018. 

https://www.cdc.gov/nchs/data/databriefs/db303.pdf

43. CensusViewer, Data and Statistics Pages (by state), 2010 Census Data. Retrieved from 

http://censusviewer.com/free-maps-and-data-links. 

44. Unutzer, J., et al., Quality improvement with pay-for-performance incentives in integrated behavioral 

health care. Am J Public Health, 2012. 102(6): p. e41-5.

45. Malouff, J. M., Thorsteinsson, E. B., Schutte, N. S. The efficacy of problem solving therapy in reducing 

mental and physical health problems: a meta-analysis. Clin Psychology Rev, 2007; 27: 46-57. 

46. Lowe B, Unutzer J, Callahan CM, Perkins AJ, Kroenke K. Monitoring depression treatment outcomes with 

the Patient Health Questionnaire-9. Med Care. 2004;42(12):1194-1201 

47. Gunn J, Diggens J, Hegarty K, Blashki G. A systematic review of complex system interventions designed to 

increase recovery from depression in primary care. BMC Health Serv Res. 2006;6:88. doi:10.1186/1472-

6963-6-88

48. Niven DJ, Mrklas KJ, Holodinsky JK, et al. Towards understanding the de-adoption of low-value clinical 

practices: A scoping review. BMC Med. 2015;13. doi:10.1186/s12916-015-0488-z 

49. Souery D, Oswald P, Massat I, et al. Clinical factors associated with treatment resistance in major 

depressive disorder: Results from a European multicenter study. J Clin Psychiatry. 2007;68(7):1062-1070 

50. Falconnier, L. Socioeconomic status in the treatment of depression. Am J Orthopsychiatry, Apr 2009; 79(2): 

148-58. 

51. Bodenheimer T, Ghorob A, Willard-Grace R, Grumbach K. The 10 building blocks of high-performing primary 

care. Ann Fam Med. 2014;12(2):166-171. doi:10.1370/afm.1616 

52. Linzer M, Poplau S, Grossman E, et al. A cluster randomized trial of interventions to improve work 

conditions and clinician burnout in primary care: Results from the Healthy Work Place (HWP) study. J Gen 

Intern Med. 2015;30(8):1105-1111. doi:10.1007/s11606-015-3235 

53. van Bodegom-Vos, L., Davidoff, F., Marang-van de Mheen, P.J., Implementation and de-implementation: 

two sides of the same coin?. BMJ Qual Saf, 2017 Jun; 26(6):495-501. 

54. Montini, T., Graham, I.D., Entrenched practices and other biases: unpacking the historical, economic, 

professional, and social resistance to de-implementation. Implement Sci, 2015 Feb 13; 10:24. 

55. Louison, L., Fleming, O. Context matters: recommendations for funders and program developers supporting 

implementation in rural communities, National Implementation Research Network. Brief 4, Retrieved from 

http://nirn.fgp.unc.edu. 
 

SIF IMPACT Expansion 46 Final Report: October 2018 

APPENDIX 

A. Implementation and Evaluation Timeline 

B. Outcome measures, sources, timing, fidelity 

C. Evaluation Data Collection Procedures 

 
APPENDIX A

Implementation and Evaluation Timeline 
SIF 2014‐2017 Original Timeframe Au

Yr 2012 Year 2013 Year 2014 Year 2015 Year 2016 

ACTIVITIES

Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 

IMPLEMENTATION J A S OND J FMAM J J A S OND J FMAM J J A S OND J FMAM J J A S OND J FMAM J J A CNCS Award to JAHF XCohort 1 Subgrantee site selection XXXXXCohort 1 Subgrantee pre‐launch planning and training XXXCohort 1 Subgrantee launch IMPACT X XCohort 1 Subgrantee post‐launch training XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXCohort 1 Subgrantee post‐launch coaching and support XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXCohort 1 Subgrantee $ sustainability support XXXXXXXXXXXX XXXXXXCohort 2 Subgrantee site selection XXXXXXXCohort 2 Subgrantee pre‐launch planning and training XXXXCohort 2 Subgrantee launch IMPACT XCohort 2 Subgrantee post‐launch training XXXXXXXXXXXXXXX XXXXXXXXCohort 2 Subgrantee post‐launch coaching and support XXXXXXXXXXXXXXX XXXXXXXXCohort 2 Subgrantee $ sustainability support XX XXXXXXXXEVALUATION DEVELOPMENTDevelopment of measures, evaluation materials, procedures XXXXXXXXXXXXXXXCohort 1 subgrantees trained in patient approach procedures X X XXXCohort 2 subgrantees trained in patient approach procedures XXX X EVALUATION: DATA COLLECTION ‐ COHORT 1Patient recruitment and consenting XXXXXXXXXXXXXXXXSubgrantees use Registry: patient care/implementation monitoring XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXPatient Baseline Survey XXXXXXXXXXXXXXX6‐month Patient Follow‐up Survey XXXXXXXXXXXXXXXXXX XXXLeadership Baseline Survey/Interviews X XLeadership Follow‐up Survey/Interviews (collect at 18mo) XLeadership End of Study survey and Phone Interview X XProvider Survey/Interviews (collect at 18 mo) XSubgrantees provide information from administrative data systems X XStages of Implementation (SIC) analysis XXXXXXXXEVALUATION: DATA COLLECTION ‐ COHORT 2Patient recruitment and consenting XXXXXXXSubgrantees use Registry: patient care/implementation monitoring XXXXXXXXXXXXXXX XXXXXXXXPatient Baseline Survey XXXXXXXX6‐month Patient Follow‐up Survey XXXXXXXXX XXXLeadership Baseline Survey/Interviews X XLeadership Follow‐up Survey/Interviews (collect at 18mo) XLeadership End of Study survey and Phone Interview Provider Survey/Interviews (collect at 18mo) XSubgrantees provide information from administrative data systems Stages of Implementation (SIC) analysis XXXXXXXXREPORTING AND ANALYSISMonthly review of Implementation Process Reports for TA XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXMonthly review of Evaluation Reports XXXXXXXXXXXXXXXXXXXXXXXXXXX XXXAnnual Progress Report to CNCS X XXFinal Report to CNCS Annual Progress Report to JAHF XXX XFinal Report to JAHF Data analysis XXXXX
APPENDIX B

Outcome measures, sources, timing, fidelity 
Subgrantee Evaluation Plan Final Report 

December 2017 

1 

Outcome Source Timing Implementation Fidelity

Measures for impact evaluation

 Depression (Patient Health 

Questionnaire-9)

 CMTS Registry (all 

IMPACT patients)

 Patient survey

 Baseline survey

 6 month follow-up survey

 Intake assessment 

(registry)

 Follow-up contacts 

(registry)

 CMTS baseline and follow-up initiated Sep/Oct 

2013 

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow up began July/Aug 2014 

 PHQ-9 is measured at baseline and most 

subsequent follow-up contacts

 Social functioning (Sheehan 

Disability Scale)

 Patient survey  Baseline survey

 6 month follow-up survey

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow up began July/Aug 2014 

 Occupational functioning 

(WHO Work Performance

 Questionnaire, Work 

Productivity and

 Activity Impairment 

Questionnaire, income )

 Patient survey  Baseline survey

 6 month follow-up survey

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow up began July/Aug 2014

 Patient satisfaction  Patient survey  Baseline survey

 6 month follow-up survey

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow up began July/Aug 2014

 Patient use of depression 

care

 CMTS Registry (all 

IMPACT patients)

 Patient survey

 End of study 

interview

 Baseline survey

 6 month follow-up survey

 Follow-up contacts 

(registry)

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow began July/Aug 2014

 CMTS baseline and follow-up initiated Sep/Oct 

2013 

 Health service utilization 

(patient survey questions 

adapted from Cornell 

Service Index and Medical 

Expenditure Panel Survey 

Data questionnaire)

 Clinic Administrative 

data (all IMPACT 

Patients)

 Patient survey

 Baseline survey

 6 month follow-up survey

 Six months before and 24 

months following 

treatment initiation

 Patient survey baseline started Jan/Feb 2014

 Patient survey follow up began July/Aug 2014

 Health Utilization data collection began in the 

spring of 2015 

Measures for implementation evaluation

 Initial assessment 

completion

 Follow-up assessment 

completion

 Patients staying in contact 

for at least 12 weeks

 Patients had at least one 

psychiatric consultant

review during treatment

 Depression improvement

 CMTS Registry (all 

IMPACT patients)

 Intake assessment 

(registry)

 Follow-up contacts 

(registry)

 CMTS baseline and follow-up initiated Sep/Oct 

2013 

 Review of current CMTS data done at monthly 

Implementation calls with Subgrantees

Measures for leader and provider interview

 Leadership survey

 Clinician survey

 Quantitative survey 

and Qualitative 

Interview

 Quantitative survey

 End of study 

qualitative interview

 Baseline and 18 months 

after program 

implementation

 18 months after program 

implementation

 Baseline leadership survey and interviews 

completed Sept 2013 (Cohort 1) Sept 2014 

(Cohort 2). Follow-up leadership interviews 

completed for Cohort 1 February 2015.

 Follow up leadership survey and interviews done 

at 18 months post-launch

 Clinician surveys done at 18 months post-launch

 End of study qualitative interview done at 32 

months post-launch
APPENDIX C

Evaluation Data Collection Procedures 
Subgrantee Evaluation Plan Final Report 

December 2017 

1 

Leadership Surveys 

Subgrantees identified members of the leadership team involved in the implementation planning to participate in 

data collection. One or more members of the leadership team completed an online survey with quantitative 

information about the organization (e.g. number of patient visits in the prior year, payer mix). Following completion 

of that online survey, the entire leadership team (as identified by the subgrantee) participated in a qualitative 

interview with 2 members of the evaluation team at the AIMS Center. The AIMS Center used an interview guide 

outline to help structure the conversation and insure that all of the key topics of interest were covered. This 

interview was conducted by telephone and was audio-recorded and subsequently transcribed. The same method 

was used at 18 months following clinical program launch. 

Care Management Tracking Registry 

Clinicians at each subgrantee clinic enter patient information as part of ongoing clinical care. 

Patient Surveys

Trained interviewers at the AIMS Center complete a ~20 minute telephone interview with each patient agreeing to 

complete the survey (both baseline and 6 month follow-up). Interviewers make 10 call attempts during eligibility 

period. During the first week, there are 3 calls made distributed over day of the week (including weekends) and 

times of day (including evenings) to increase the likelihood of completing the baseline survey within the 8 week time 

frame for baseline data collection. A staff member evaluates the incomplete surveys each day to prioritize them for 

contact and to insure that each one has been given the appropriate amount and type of contact attempts. Each 

survey is audio-recorded and a randomly selected 10% of all surveys are reviewed by a data collection supervisor to 

insure appropriate data collection methods were used and to provide feedback to interviewers when necessary. 

To ensure a high response and retention rate, the patient survey collection procedures were expanded to include 

the following components: 













The AIMS Center developed and launched an online version of the baseline and follow-up supplemental 

patient surveys to increase participation rates, especially among patients who have telephone service with 

limited minutes and those who will feel more comfortable completing the survey in a more anonymous 

manner. 

The AIMS Center sends a reminder postcard to all patients prior the follow-up survey. The postcard will also 

provide the online survey link for the participant to complete the survey online if they prefer. 

The AIMS Center Implemented texting as a strategy to communicate with patients, including sending the link 

to the online survey, which has produced immediate results for both telephone and online surveys and 

responses from patients otherwise out of contact. 

AIMS Center staff provided ongoing training and support to clinics regarding effective recruitment of patients 

for the supplemental patient survey, including a live answer line for patients to call and complete the survey 

immediately and setting up a computer in some clinics so patients can complete the survey online at the clinic. 

The addition of a patient incentive for completion of both the baseline and follow-up survey. Upon 

completion of the follow-up survey, participants will receive a $40 gas card from a gas merchant located near 

their respective clinic. 

For patients who complete a baseline supplemental survey, and are not reachable at the time of the 6 month 

follow-up (after using all means to locate), we will use the PHQ-9 score from their final clinical follow-up 

assessment as entered in the registry. This will allow us to include information from those patients lost at 

follow-up in the final analysis of the main impact question (depression severity). 

Clinician Surveys 

All clinic staff providing IMPACT care to patients (care managers, consulting psychiatrists, and primary care 

providers) will complete an online survey and a qualitative interview with AIMS Center staff. The AIMS Center will 
Subgrantee Evaluation Plan Final Report 

December 2017 

2 

use an interview guide outline to help structure the conversation and insure that all of the key topics of interest are 

covered. Interview will be conducted by telephone, audio-recorded and subsequently transcribed. 

Administrative/Claims Data 

Subgrantees provided data from their administrative data systems regarding demographics, These data elements 

are already collected by the clinics for the Uniform Data System (UDS) reporting to HRSA so can be collected 

accurately and consistently across clinics. This data was used to enrich and validate the patient registry data and 

serve as a source for capturing missing registry data. The AIMS Center provided a detailed description of the desired 

data elements, data format, etc. and worked with staff at the subgrantee clinics to transmit the data to the AIMS 

Center in a HIPPA secure manner. 

End of Study Survey and Qualitative Interview

To provide additional data on utilization trends, we collected an end-of-study survey and phone interview with clinic 

leadership and care teams. The online survey captured feedback from the care teams on their experience with AIMS 

center technical assistance and support to identify and characterize critical components that lead to improved 

implementation. The qualitative phone interview captured the care team’s experience with IMPACT and the 

complex practice change. It also captured information on health care utilization and how the IMPACT program has 

changed how patients use clinic services. 

Stages of Implementation Completion (SIC) and Fidelity Score 

The Stages of Implementation Completion (SIC) measure allowed us to describe variation in implementation 

process and fidelity to key components of IMPACT amongst the subgrantee clinics and whether/how these affected

processes of care and/or clinical outcomes. The measure was completed by AIMS Center evaluation staff and 

entered into the Oregon Social Learning Center web portal for scoring and interpretation..