Implementing Health Plan-Level Care Management for Solo & Small Practices
Phase: NA
Conditions: Bipolar Disorder, Depression
Sponsor: University of Michigan

 
Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
NCT02041962 
Protocol # HUM00073753  
Protocol version: v5  
Protocol date: 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
I. Objective 
The vast majority of primary and behavioral health care services are provided by solo or small practices, yet to 
date collaborative chronic care models (CCM) have not been implemented in these settings. Mood disorders 
are common and represent the most expensive mental disorders for commercial health plans and employers.  
The goal of this study is to implement and test a CCM for mood disorders for solo and smaller practices in 
partnership with Aetna Health Plan, with an eye towards making the business case and demonstrating value 
for CCMs for these settings that can reach patients wherever they receive treatment.  
 
II. Specific Aims/Hypotheses 
A 2010 HHS report highlighted the prevalence, morbidity, and cost associated with clusters of co-occurring 
chronic conditions, both physical and mental.  The report also underscored the lack of sustainable treatment 
strategies for these afflicted individuals, and the difficulties in customizing patient-centered interventions. 
 
Collaborative chronic care models (CCMs) are effective in treating chronic medical and mental illnesses at little 
to no net healthcare cost. To date CCMs have primarily been implemented at the facility level and primarily 
developed for and adopted by larger healthcare organizations.  However, we have determined that the vast 
majority of primary care and behavioral health practices providing commercially insured care are far too small 
to implement such models. Health plan-level CCMs can address this unmet need. 
 
Chronic mood disorders (e.g., bipolar disorders, depression) are common and are associated with extensive 
functional impairment, medical comorbidity, and personal and societal costs. While unipolar depression is more 
common, bipolar disorder is more costly on a per patient basis due to its chronic and severe nature. Moreover, 
bipolar disorder is the most expensive mental disorder for U.S. commercial health plans and employers.  While 
evidence-based care parameters have been well established for mood disorders, quality of care and health 
outcomes in general mental health practice are suboptimal. The majority of these patients suffer from clusters 
of comorbid conditions, both physical and mental.  Thus mood disorders represent optimal tracer conditions 
with which to improve management strategies for individuals with multiple chronic conditions. 
 
Accordingly, we have partnered with Aetna Inc. to develop and implement a CCM designed to improve 
outcomes for persons with mood disorders for solo or small practices, with an eye towards developing a 
business case for a generalizable plan-level CCM for chronic disorders. We will conduct an RCT of a health 
plan-level CCM vs. education control. The population of interest will be Aetna beneficiaries across the country 
hospitalized for depression or bipolar disorder treated in solo or small primary care or behavioral health 
practices.  Patients will be randomized to one year of outpatient treatment augmented by the CCM or 
education control, for a total of 344 participantsCCM care management will be fully remote from practice 
venues and patients, implemented by existing providers (the Aetna care management center). A business case 
will be developed using the Replicating Effective Programs (REP) strategy that identifies generalizable 
facilitators for CCM spread and value added of CCMs to be vetted to key industry and policy stakeholders. 
 
Our Specific Aims are to: 
1. Determine whether individuals with mood disorders treated with a health plan-level CCM demonstrate 
improved health outcomes and quality of care at 12 months compared to those who are mailed educational 
material alone (Educational Control-EC).  
 
Our primary hypothesis is that compared to EC, the mood CCM will result in a) reduced mood symptoms in 
12 months, or b) improved health-related quality of life. 
 
Our secondary hypotheses is that patients receiving CCM vs. EC will have improved guideline-concordant care 
(e.g., mood disorders treatment, cardiometabolic monitoring), reduced hospitalizations, and improved work 
productivity within 12 months. 
 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
2. To support subsequent CCM dissemination by identifying key organizational, provider, and patient 
characteristics associated with CCM engagement and outcomes.   
 
3. To conduct a cost-effectiveness analysis of CCM versus EC on changes in patient utilities over 24 months 
and apply findings to develop a business case for CCMs in smaller practices. 
 
This proposed R18 addresses AHRQ’s research demonstration and dissemination priorities, particularly 
around prevention and care management, with an emphasis on vulnerable groups including more rural 
populations (the majority of Aetna enrollees are seen in smaller practices). It also confronts unanswered 
organizational and financial questions raised by a recent AHRQ report (“A National Agenda for Research on 
Collaborative Care”). As one of the first cross-diagnosis CCM-based implementation studies focused on small 
practices, this groundbreaking study is informed by an extensive partnership with a network-model health plan 
(Aetna). The study also applies a novel implementation framework (REP) in which the CCM is delivered by 
existing providers. In addition to the specific importance of this practice-research partnership to further 
implement CCMs among small practices at the health plan level within a natural experiment setting, this study 
will contribute to the evolution of the business case for CCMs in general, enhance the utility of plan-level panel 
management focused on vulnerable populations across different treatment settings, and inform remote 
technologies that will facilitate emerging healthcare initiatives (e.g., accountable care organizations, health IT). 
 
III. Background and Significance 
 
A.  Significance.  
1. Majority of care for mental disorders is provided by private solo or small practices.  
The 2011 HHS report on multiple chronic conditions (1) highlighted the prevalence, morbidity, and cost 
associated with clusters of co-occurring chronic conditions, and lack of current treatment strategies. A recently 
completed AHRQ report (2) and meta-analysis by our group (3) concluded that collaborative chronic care 
models (CCMs) are effective in treating chronic medical and mental illnesses at little to no net healthcare cost 
(4-6). CCMs consist of: (a) patient self-management skill enhancement, (b) expert decision support to 
providers via evidence-based practice guidelines, and (c) enhanced access and continuity via care managers 
working in collaboration with physicians to support information flow between patients and providers (7, 8).  
 
Our recent meta-analysis of CCMs also found that they have primarily been implemented at the facility level in 
publically funded settings (2, 3, 9, 10).  Still, between 50-70% of patients with mental health conditions are 
managed by commercial health plans (11). Moreover, in a recent analysis we have also determined that 85% 
of those with mood disorders receive care from solo or small practices (12)  that are less likely to be able to 
implement care management processes at the practice level (13).  
 
CCMs will become increasingly important as healthcare delivery systems evolve into accountable care 
organizations (14, 15), thereby taking on broader responsibility for care coordination and quality while bearing 
financial risk for complex, chronic conditions. CCMs can provide either the foundation of, or an annex to, 
emerging medical home methodologies (16-18). However, dissemination of CCMs is limited to larger practice 
venues with sufficient infrastructure (19, 20). To date CCMs for chronic disorders have been implemented as 
clinic-based interventions (21) or within integrated healthcare systems, but not in smaller practices (22). 
 
2. Mood disorders are common exemplars for complex patients at high risk in need of CCMs.  
Mood disorders (depression and bipolar disorder) are chronic illnesses associated with substantial social 
dysfunction (23) and rank in the top ten causes of disability worldwide (24). National studies of the U.S. 
population estimate the lifetime prevalence for bipolar spectrum disorders as 6.4% (25, 26) and 16.6% for 
major depressive disorder (27). Medical comorbidity is the rule for mood disorders, in which the majority of 
patients suffer from at least one co-occurring chronic medical illness (28, 29). Persons with mood disorders die 
younger, mainly from cardiovascular disease (30-32). Quality of care is suboptimal for both chronic medical 
(33) and mental (34-41) disorders, underscoring the need for coordinated, comprehensive care. Not 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
surprisingly, the recently released IOM comparative effectiveness priority list includes among its top-quartile 
priorities research on comprehensive care coordination for chronic diseases (42). While unipolar depression is 
more common, patients with bipolar disorder incur the most health care costs of any mental illness (43). Up to 
70% of direct treatment costs for mood disorders are generated outside the mental health sector, notably in 
primary care (44-46). In response to extremely high costs and high disease burden associated with mood 
disorders, Aetna has partnered with us to develop and implement care management methods for this group. 
 
3. Health plan-level cross-diagnosis CCMs can address gaps in quality & outcomes for mood disorders 
To date CCMs have not been implemented across mood disorder diagnoses (depression or bipolar disorder), 
nor have they simultaneously addressed medical and psychiatric outcomes in this group. Given these clear 
public health deficits, there is a need for adaptation and greater reach of care management strategies beyond 
single diagnoses. The current state-of-the-science indicates that research efforts should focus more on cross-
diagnoses models such as the one proposed in this study as well as issues of spread and sustainability. CCMs 
have been found to be effective in reducing symptom burden and improving health-related quality of life for 
depression (3, 47-49) and bipolar disorder (7, 8, 50-53) in separate studies and are now recommended in 
practice guidelines (54, 55). Our preliminary study of a cross-diagnosis CCM for mood disorders also 
demonstrated improved outcomes (Section C). Moreover, long-term CCM treatment for mood disorders can 
be cost-neutral (51), and effective among patients with co-occurring substance use disorders (56).  
 
Thus adapting CCMs for this population will require population or health plan level intervention. Moving to plan 
level care management requires primary reliance on remote rather than clinic-based methods, such as 
telephone contacts with patients and providers. We have identified 14 reports of 10 trials of telephone care 
management for depression but none for bipolar disorder (2). These studies recommend a combination of 
rapport-building telephone care manager-patient contacts with tailored health IT modalities (e.g., web-based 
self-management, EMRs) (57-69). Several initiatives by our study team also hold promise in a health plan-level 
implementation of the CCM (Section C), notably a telehealth program for bipolar disorder and a  web-based, 
tailored medical care management program for mood disorders (40, (70). 
 
B.  Innovation 
1. To date this is the first study to implement CCMs in solo or small practices, where many patients 
with mood disorders are seen: CCMs have mostly been implemented at the facility level, and primarily 
developed for and adopted by larger healthcare organizations. However, 50-70% of Americans with mood 
disorders are managed by commercial insurance plans such as Aetna. Aetna is one of the largest health plans 
in the U.S., and among its enrollees >90% were seen in solo or small practices (12). This study also 
complements ongoing efforts in the public sector to implement CCMs through medical home models (10).  
 
2. Translating CCMs to health plan level models is essential for maximal generalizability: Plan-level 
CCMs have rarely been tested in rigorous RCTs (31). A focused implementation of a cross-diagnosis CCM at 
the national level has implications for the tailoring of evidence-based programs to smaller and rural settings, 
personalized health care, and implementation of health IT. As technologies around large databases become 
more sophisticated and complete, implementation programs that successfully apply these rich resources to 
helping vulnerable populations will serve as important milestones in the nation’s transition to a more public 
health model of care. By focusing on measurement dimensions of the Berwick Triple Aim (care, health, and 
cost), study findings are potentially generalizable across complex patient populations (71).  
 
3. Implement CCM using an established strategy to address organization, financing (Figure 1): We will 
apply a state of the art framework to implement the CCM across Aetna nationally. Developed by study 
investigators, the Replicating Effective Programs (REP) implementation framework has been used to facilitate 
the uptake of mood disorder CCMs in prior studies (R01 MH 79994) (72, 73). A variety of implementation 
frameworks and supporting theories exist (74-78), with relative advantage conferred on those that are (a) 
theory-based, (b) highly specified operationally, and (c) widely used in relevant contexts. REP is based on the 
Centers for Disease Control and Prevention’s Research Practice Framework (76, 77, 79). Based on Social 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
Pre-Conditions 
(completed) 
 
Identify need & evidence-
based practice (CCM) 
 
Identify underserved setting 
(solo/small practices) 
 
Develop EBP Package 
Pre-Implementation 
 
Disseminate CCM 
package 
 
Train providers in EBP  
 
Assess organizational 
context using OTM 
 
Monitor EBP uptake 
Implementation 
Technical assistance 
Organizational incentives 
      Financial incentives 
 
Outcomes monitoring 
      Provider assessments 
             Organizational change 
     assessment using OTM 
     Triple Aim 
Sustainability 
Evaluation 
      Patient outcomes (Aim 1) 
       Mechanisms (Aim 2) 
       Cost-effectiveness (Aim 3) 
 
Develop business plan (Aim 3) 
 
Stakeholder feedback 
 
Refinement, national roll-out 
Figure 1: Enhanced Replicating Effective Programs (REP) to Facilitate Implementation of CCMs 
Learning Theory (80) and Rogers’ diffusion model (81), REP is highly specified operationally into three 
components that are directly applied to the CCM: Packaging (i.e., translation of CCM materials into user-
friendly language), Training, and ongoing facilitated Technical Assistance to address barriers to uptake.  
 
Enhanced REP: In a recent implementation of REP (R01 MH79994), community-based administrators and 
providers made specific recommendations to the Technical Assistance component that would help develop a 
business case for the CCM based on organizational as well as financial incentives (73). This Enhanced REP 
framework includes ongoing assessment of organizational barriers or facilitators to implementation (11, 82-87) 
based on the Organizational Transformation Model- OTM (88) that assesses both context as well as 
organizational change factors that might influence implementation. The OTM assessment (72) measures 
organizational processes found to be associated with improved quality (89), including the practice’s readiness 
to implement new programs (e.g., CCM), leadership commitment, degree to which frontline providers are 
involved in the operationalization of a new program, and alignment of new program goals with existing priorities 
and day-to-day processes. Enhanced REP emphasizes dissemination of data to health plan leaders and other 
stakeholders, and aligning CCM components with reimbursement strategies that may inform the 
implementation of emerging initiatives including accountable care organizations. 
 
4. This proposal represents a true private-public partnership in light of other AHRQ initiatives (90). Community-
based providers involved in our previous studies (R01 MH74509, MH79994) have strongly emphasized the 
need for CCMs across mood disorders from smaller practices (73). Aetna will provide all care management 
infrastructure, data, and staff while federal research funds are limited to support of training and evaluation. All 
products developed from this project will be made available in the public domain based on AHRQ guidelines. 
Finally, this R18 also addresses unanswered questions raised by a recent AHRQ report on setting a National 
Agenda for Research in Collaborative Care (2).  
 
IV. Preliminary Studies: 
 
Our preliminary studies, including a national study of care management in Aetna and implementation of a 
cross-diagnosis CCM across the National Network of Depression Centers demonstrate our collective expertise 
in development and implementation of mood disorder CCMs to improve outcomes across a wide variety of 
practices, and support the readiness of the mood CCM for a plan-level RCT. 
 
C.1. Mood disorders in Aetna: Identifying and addressing the need in solo/small practices. Aetna Inc. is 
the fifth largest healthcare insurer in the country, providing benefits through employers in all 50 states, with 
products and services targeted specifically to small, mid-sized and large multi-site national employers. Aetna 
Behavioral Health serves approximately 12 million covered lives, with 244,971 providers across 157,523 
practices filing claims in the past year. Thus while this specific RCT is sited in a single health plan, Aetna has 
the size and diversity sufficient to represent a wide range of commercially funded care for mood disorders in 
the US.  Aetna has had a longstanding and stable commitment to implementing disease management for high-
prevalence, high-cost mental health conditions, as well as a track record of NIH-funded research (e.g., 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
1R01DA029716; 1R01DA026414). Aetna’s commitment to sustaining and expanding mental health care 
management—and the mood disorders CCM in particular—is underscored in the letters from both Dr. Un and 
Aetna’s in-kind contribution outlined in the Budget Justification.  
 
C.2. Most Aetna beneficiaries receive care for bipolar disorder from small or solo practices. Using all 
Aetna Behavioral Health claims data from 2009, we determined practice size based on number of participating 
providers linked to the same tax identification number. Unlike the situation for primary care and multi-specialty 
groups (19, 20, 91), care for bipolar disorder is predominantly provided in small behavioral health practices, 
with the vast majority of individuals with bipolar disorder treated in solo practices (Table 1). Further, among 
Aetna patients with bipolar disorder from solo or small practices, 23% (N=2785) were hospitalized in the past 
year. This distribution convinced us that it is critical to focus the study on smaller or solo practices in order to 
truly reach a vulnerable population who to date may not have access to effective care management. 
 
Table 1: Profile of Aetna Enrollees with Bipolar Disorder by Practice Size 
Practice Size (# Licensed 
Independent Providers)  
# Behavioral 
Health Practices 
(%) 
# 
Psychiatrists 
(%) 
# Patients Treated 
for Bipolar Disorder 
Seen in Given 
Practice Venue Size 
(%)** 
Avg. 
Bipolar 
Patient 
Census 
Single Independent Psychiatrist 
20,318 (84.1%) 
20,318 (45.7%) 
20,203 (94.4%) 
 3.5 
Single Psychiatrist Registered as 
Group* 
  1,250 (5.2%) 
1,250 (2.8%) 
     111 (0.5%) 
 1.3 
Group Practice with 2-3 Psychiatrists 
    967 (4.0%) 
4,428 (10.0%) 
     207 (1.0%) 
 2.3 
Group Practice with 4-10 Psychiatrists 
  1,029 (4.3%) 
6,875 (15.4%) 
     329 (1.5%) 
 2.0 
Group Practice with 11-20 
Psychiatrists 
    380 (1.6%) 
5,089(11.4%) 
     225 (1.1%) 
 2.4 
Group Practice with 21-100 
Psychiatrists 
     214 (0.9%) 
5,645(12.7%) 
     318 (1.5%) 
 4.5 
Group Practice with >100 Psychiatrists 
      14 (0.1%) 
894(2.0%) 
       18 (0.1%) 
18.0 
Total Number of Practices 
24,172 (100.0%) 
44,449 
(100.0%) 
21,411 (100.0%) 
  -- 
*Individual providers could register with Aetna either as an independent practitioner or as a group at their own discretion. 
**Analysis of 20,933 unique patients treated for bipolar disorder; 1,675 (0.8%) received care in >1 venue. 
 
C.3. Aetna national RCT of brief phone-based CCM vs. education control for bipolar disorder (N=457) 
An IRB-approved feasibility RCT of a brief plan-level care management program (CM) was led by Aetna 
Behavioral Health Medical Director Dr. Hyong Un and internal Aetna staff. The study involved 457 adult Aetna 
enrollees or family members with bipolar disorder from 215 distinct practices across all 6 Aetna regions 
randomized to either CM or educational material mailing control (EC).  Inclusion criteria included hospitalization 
for bipolar disorder type I or II and >= 6 months of prior enrollment in Aetna in order to provide sufficient claims 
data. Inability to speak or read English was the sole exclusion criterion to enhance generalizability (<10%, or 
N=48 out of the 505 approached). 
 
EC consisted of 6 biweekly educational mailings about bipolar disorder, and CM consisted of these mailings 
plus 3 semi-structured, monthly telephone patient contacts provided by a care manager stationed at the Aetna 
national CM center in Salt Lake City. Telephone contacts focused on: (a) measurement-based clinical 
assessment, (b) identification and trouble-shooting of patient-identified disease- and treatment-related 
challenges, and (c) motivational interviewing techniques to support treatment engagement.  At the patient’s 
discretion additional sessions were arranged at the end of this 3-month period.  This prototype CM model 
included no provider contacts or participant compensation. Outcomes included several service use and costs 
(inpatient, outpatient, laboratory, pharmacy) from the economic perspective of the insurer.  
 
Of the 457 randomized, 424 (92.8%) completed the protocol. CM participants completed a mean of 4.4 
contacts. Cost data were exported to Dr. Kilbourne for analysis.  Despite the modest intensity, there was a 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
greater reduction in costs among CM vs. EC participants (Cohen’s D=0.21) (Table 2). Importantly, outpatient 
visits were not decreased in the control group, indicating that the protocol did not disrupt usual care. Moreover, 
care managers identified 4 out of 457 patients who had suicidal ideation and, through established protocols, 
alerted the provider and arranged emergency evaluation; and no suicide attempts occurred during the study. 
This study indicates that: (a) participants can be successfully enrolled into a plan-level care manager RCT; (b) 
care management can be successfully implemented with participants across time zones from a national call 
center; and (c) claims data suggest that care management may have had beneficial effects.  
 
Table 2: Changes in Costs among Aetna Patients Randomized to CM or EC: National Data 
 
Care Management (mean $+SD) 
Educational Control (mean $+SD) 
 
6 Mos 
Pre 
6 Mos Post 
Differenc
e 
6 Mos Pre 
6 Mos Post 
Difference 
Aetna BH 
$/member 
5,210+ 
8,370 
934+2,210 
-
4,277+7,926 
6,048+6,347 
3,492+7,621 
-2,556+8,461 
 
C.4. NNDC Multisite mood CCM implemented by existing providers improves outcomes.  In a 
randomized controlled pilot study we determined the effectiveness of a cross-diagnosis mood CCM compared 
to education control in a cohort of outpatients with unipolar depression or bipolar disorder from four sites 
affiliated with the National Network of Depression Centers (NNDC). NNDC is the nation’s largest network of 
academic and community-affiliated mental health and primary care clinics representing a diverse practice size 
and patient population. Existing care managers at four NNDC sites (Ann Arbor, Palo Alto, San Francisco, 
Denver) were trained by study investigators to implement the CCM using Enhanced REP. Sixty patients (mean 
age=46, 40% bipolar diagnosis, 73% female, 17% African-American) with mood disorders were identified using 
chart reviews and randomized to CCM or usual care. CCM included phone-based self-management and care 
management as well as guideline dissemination to providers. Repeated measures analyses comparing 
changes in 3 and 6-month outcomes revealed clinically significant probability of depression remission (>50% 
PHQ-9 score reduction and <=9 on PHQ-9 in 6 months): 50.1% for CCM vs. 19.0% for EC group (OR=9.4, 
p=.04) and improved well-being (positive symptoms based on the Internal State Scale-ISS - Cohen’s D=.36). 
 
Table 3: 6-Month Outcomes Comparing CCM vs Education Control (EC): Repeated Measures Analysis 
Outcomes(N=60) 
CCM (N=29) 
EC (N=31) 
Repeated measures results 
 
Baseline 
3-mon 
6-mon 
Baseline 
3-mon 
6-mon 
Beta 
T(p) 
Effect(D) 
PHQ-9 
13.7 
(7.5) 
9.3 (6.5) 
6.3 (4.2) 
16.8 
(6.5) 
14.0 (6.6) 
11.9 (6.8) 
-
1.24 
-
1.37(.17) 
-.22 
50% drop: PHQ-
9 and PHQ-9 
<=9 
 
 
50.0% 
 
 
19.0% 
OR=9.4; Chi-sq= 4.18; 
p=.04 
ISS-Wellbeing 
13.0 
(7.9) 
15.8 
(8.3) 
20.4 (7.8) 
10.2 
(6.9) 
12.5 (7.8) 
12.4 (8.1) 
2.22 
2.35(.02) 
.36 
 
V. Methodology 
Overview. We will conduct an RCT of the CCM for mood disorders vs. an educational mailing control (EC).  
The population of interest will be Aetna adult enrollees and family members (beneficiaries) across the country 
hospitalized for an episode of unipolar depression or bipolar disorder and belonging to a solo (i.e., single 
provider) or small practice (2-3 providers). Patients will be randomized to CCM or EC. The CCM will be 
implemented using Enhanced REP (76, 77, 79) as we have in prior studies (72, 73, 76). CCM care 
management will be fully remote from practice venues and patients, implemented by Aetna care managers. 
CCM and EC strategies represent add-ons to ongoing practice, which will not otherwise be altered. Main 
outcomes include mood symptoms (e.g., PHQ-9) and health-related quality of life (SF-12), accounting for 
nesting of patients within practices. 
 

)XOO1,+7LWOH,PSOHPHQWLQJ+HDOWK3ODQ/HYHO&DUH0DQDJHPHQWIRU6ROR	6PDOO3UDFWLFHV
3URWRFRO+80
3URWRFROYHUVLRQY
3,$P\.LOERXUQH3K'03+
&E 6DPSOH UHFUXLWPHQW HOLJLELOLW\ HQUROOPHQW )LJXUH  7KH IROORZLQJ VWHSV ZLOO EH WDNHQ WR HQUROO
SDUWLFLSDQWVSURYLGHLQIRUPHGFRQVHQWDQGUDQGRPL]HWKHP3DUWLFLSDQWUHFUXLWPHQWDQGWUHDWPHQWZLOOXWLOL]H
H[LVWLQJ$HWQDUHVRXUFHV

$HWQDFDUHPDQDJHUVZLOOUHFUXLWSDUWLFLSDQWV E\ILUVWVFUHHQLQJWKHPEDVHGRQQHDUUHDOWLPHLQIRUPDWLRQRI
UHFHQW KRVSLWDOL]DWLRQV DV ZLWK WKH SLORW VWXG\ $W KRVSLWDOL]DWLRQ $HWQD LV QRWLILHG IRU SUHDXWKRUL]DWLRQ
W\SLFDOO\EHIRUHRUZLWKLQKRXUVRIDGPLVVLRQ&DUHPDQDJHUVZLOOEHQRWLILHGDERXWSDWLHQWKRVSLWDOL]DWLRQV
YLDWKH$HWQDFDUHPDQDJHPHQWUHJLVWU\H78066HFWLRQ&FDQGZLOOFRQWDFWWKHSRWHQWLDOSDUWLFLSDQWE\
SKRQH DV LQ WKH SLORW VWXG\ 6HFWLRQ &D &DUH PDQDJHUV ZLOO VFUHHQ IRU HOLJLELOLW\ DQG REWDLQ LQIRUPHG
FRQVHQW DQG DXWKRUL]DWLRQ WR UHOHDVH LQIRUPDWLRQ WR WKH UHVHDUFK WHDP DQG WR FRRUGLQDWH FDUH ZLWK WKHLU
SURYLGHUV7KLVZLOOEHGRQHRYHUWKHWHOHSKRQHDIWHUYHUEDOO\H[SODLQLQJWKHSURWRFROULVNVEHQHILWVDQGVDIHW\
PHDVXUHVYLDZULWWHQVFULSWEDVHGRQSUHYLRXV5&7V%HFDXVHSDWLHQWVKDYHQRWEHHQUDQGRPL]HGDW
WKLVSRLQW$HWQDFDUHPDQDJHUVZLOOEHEOLQGWRWUHDWPHQWDVVLJQPHQWDWEDVHOLQH
3DWLHQWLQFOXVLRQFULWHULDDVGHWHUPLQHGE\WKHFDUHPDQDJHUVLQFOXGH
D
$GXOWSDWLHQWVDJH\HDUVRUROGHUIURPWKHFRQWLJXRXV8QLWHG6WDWHVORZHUVWDWHV
E
&XUUHQWO\FRYHUHGE\$HWQD¶V+02RUSUHIHUUHGSURYLGHUSURGXFWVIRUZKRP$HWQDSURYLGHVPHQWDO
DQGPHGLFDOLQSDWLHQWRXWSDWLHQWDQGSKDUPDF\EHQHILWVIRUDWOHDVWPRQWKV
F
5HFHQWSDVWPRQWKKRVSLWDOL]DWLRQIRUDQDFXWHSV\FKLDWULFRUSDUWLDOKRVSLWDOXQLWZLWKDPDQLFRU
GHSUHVVLYHHSLVRGHDQGFRQILUPDWLRQRIPRRGGLVRUGHUGLDJQRVLVDWGLVFKDUJH
G
$ELOLW\WRVSHDNDQGUHDG(QJOLVKDQGSURYLGHLQIRUPHGFRQVHQW

7KHUHVHDUFKDVVRFLDWH5$ZKRZLOOEHH[WHUQDOWR$HWQDWRPDLQWDLQVHSDUDWLRQRIFDUHPDQDJHPHQWDQG
SURJUDPHYDOXDWLRQDQGZLOO
EH EOLQGHG WR UDQGRPL]DWLRQ
DVVLJQPHQW 6KH ZLOO EH
WUDLQHG E\ $HWQD RQ WKHLU
ZHOOHVWDEOLVKHG
FDOO
LPSOHPHQWDWLRQ
SURJUDP
DFURVV WLPH ]RQHV 6HFWLRQ
&F
DQG
E\
VWXG\
LQYHVWLJDWRUV
LQ
KXPDQ
VXEMHFWV
ULVN
UHGXFWLRQ
SURFHGXUHV XVHG LQ SULRU
VWXGLHV   WKDW ZLOO
PLQLPL]H
ULVN
ZKLOH
QRW
FRPSURPLVLQJ VWXG\ LQWHUQDO
YDOLGLW\

7KH
LQLWLDO
DVVHVVPHQW
GHVFULEHG
EHORZ ZLOO LQFOXGH D EULHI
VXUYH\
EDVHOLQH
IDFWRUV
LQFOXGLQJ
GHPRJUDSKLFV
HPSOR\PHQW ZKHWKHU WKH\
ZHUHWKH$HWQDHQUROOHHRUD
IDPLO\PHPEHUDQGSULPDU\
FOLQLFLDQ ZKR WUHDWV WKHLU PRRG GLVRUGHU LH SUHVFULEHV SV\FKRWURSLF PHGLFDWLRQV &RPRUELGLWLHV ZLOO EH
DVFHUWDLQHGIURP$HWQDH7806FKDUWUHYLHZVFRQGXFWHGE\WKH5$V

&F([LVWLQJ$HWQD,QIUDVWUXFWXUH8VHGLQWKLV6WXG\7KHIROORZLQJFRPSRQHQWVDUHRIVSHFLILFUHOHYDQFH
7KH UHJLVWU\GDWDEDVH FRQVLVWV RIIRXUPDLQFRPSRQHQWV DQG ZLOO EH XVHG WR LGHQWLI\ SRWHQWLDOO\ HOLJLEOH
SDWLHQWV DV ZHOO DV DVFHUWDLQ XWLOL]DWLRQ DQG RWKHU RXWFRPHV GDWD VHH YDULDEOH GHVFULSWLRQV EHORZ 7KH
(OHFWURQLF7RWDO8WLOL]DWLRQ0DQDJHPHQW6\VWHPH7806DSURSULHWDU\FDUHPDQDJHPHQWWRROLVSRSXODWHGE\
Patient
Provider
)LJXUH,PSOHPHQWDWLRQRI0RRG&&0DFURVV+HDOWK3ODQ3UDFWLFHVDQG3DWLHQWV
+HDOWK3ODQ<HDUV
&DUH0DQDJHUWUDLQHGLQ&&0YLD(QKDQFHG5(3
LGHQWLILHVDQGFRQVHQWVHOLJLEOHSDWLHQWVH7806
6WXG\LQYHVWLJDWRUV$HWQD VXSHUYLVH&05$VWDII
3UDFWLFH<HDUV
&DUH0DQDJHU6HQGVSDWLHQWOLVWVWR'DWD$QDO\VW
'DWD$QDO\VW,GHQWLILHVSDWLHQWVIURPVRORVPDOOSUDFWLFHV
5DQGRPL]HVVRORVPDOOSUDFWLFHVWR&&0RU(&
'HYHORSEXVLQHVVFDVH<HDUV
&0SURYLGHV
&&0 SDWLHQWV
/LIH*RDOVSURJUDP
&&0	SURYLGHUV
&OLQLFDO	VHUYLFHDVVHVVPHQWV
*XLGHOLQHUHFRPPHQGDWLRQV
&&0 SDWLHQWV
0RQWKO\WHOHSKRQH
DVVHVVPHQWV
6WXG\5$
&RQGXFWV
RXWFRPH
DVVHVVPHQWV
<HDUV
6WXG\5$
FRQGXFWV
SURYLGHURUJ
DVVHVVPHQWV
<HDUV

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
claims data augmented by customizable care manager input; major claims events, such as hospitalization, 
result in automatic care manager notification. The pharmacy claims database accrues prescription claims 
daily, including drug and dose, and is downloaded to eTUMS on a weekly basis. Laboratory claims data are 
similarly downloaded on a monthly basis; laboratory reports contain the name of the test, while reports from 
larger laboratories also provide results. Thus these registries will provide the data necessary to measure 
guideline-concordant care (e.g., non-filling of mood disorder prescriptions, non-completion of cardiometabolic 
laboratory tests). The fourth component, Aetna claims data, includes use and cost data for Aim 3. 
 
2. The Aetna care management infrastructure is substantial and stable. The masters level Aetna care 
managers who participated in the pilot will serve as the CCM care managers. They have a minimum of 5 years 
of care management experience with Aetna, providing short-term or ongoing care management for an average 
of 300 beneficiaries per year across multiple time zones and patient schedules.  
 
Care manager coverage across time zones: All Aetna care managers function in teams with supervisory 
care manager and medical oversight. They also work staggered hours (e.g., covering 9am-midnight ET M-F, 
10-6pm Sat) in order to make patient calls across different time zones. Biweekly 1:1 supervisor meetings and 
monthly team meetings provide support and training. Quality is monitored via audit tools that assess both calls 
and documentation three times per month. Regular Medical Director Rounds are supplemented by reviews 
when claims or other data indicate that the beneficiary is not doing well.  
 
CM and study staff handling of suicide risk/adverse events: Imminent risk (suicidal or assaultive ideation, 
significant medication toxicity) are managed via protocols used in prior studies (8, 50, 52). In the pilot study 
(Section C.3), care managers identified 4 out of 457 who had suicidal ideation and, through established 
protocols, alerted the provider and arranged emergency evaluation; no suicide attempts occurred during the 
pilot study. Moreover, CCMs have been shown to reduce suicide (94).  Study staff (both University of Michigan 
and Aetna) will be trained to identify and address suicidal ideation in patients. If the patient articulates thoughts 
of death or suicidal ideation, the study team member will ask the patient to elaborate on recent suicidal 
behavior (i.e., ask if he or she has an active plan). Follow-up regarding the situation will be based on the 
patient’s responses and whether the study team member perceives that the patient is in immediate danger 
(e.g., active plan verbalized). Endorsement of the suicidal thoughts on the survey will also serve as an 
emergent/urgent situation.  The study team member will then ask whether there is another individual with the 
patient (if over the phone), and will remain on the phone while instructing the other individual to get help. If no 
one else is present with the patient, the team member direct the patient to his/her local emergency room or, 
refer to a national suicide hotline that can do further assessment and planning, so the patient can get 
immediate help. Adverse event reporting is outlined further in the data safety monitoring plan.  We will have a 
adverse events and serious adverse events management plan for this project, as due to the nature of this 
population, fairly frequent visits to the emergency room that do not require hospitalization and hospitalizations 
due to their mood disorder exacerbation are to be expected.  All adverse events encountered by both 
University of Michigan and Aetna staff will be reported to the UM IRB according to our management plan (see 
Data Safety Monitoring Plan document). 
 
C.5.d. Interventions: EC and CCM. The education control (EC) condition consists of a self-guided 
workbook mailed to patients developed by study investigators (95) along with standard care from the practice, 
but none of the CCM components from the plan-level CM.  
 
Table 4: Mood Disorders CCM Core Elements: Life Goals Collaborative Care 
CCM Core Element 1: 
Life Goals Skills Management 
Enhancement 
10 weekly 30-min telephonic sessions utilizing the web-based Life Goals 
program (52, 95) 
• 
Core modules cover management of depressive and manic symptoms 
• 
Additional web modules utilized as clinically indicated (e.g., wellness, foods and 
moods, exercise, substance use, anxiety, psychosis, anger/irritability, metabolic 
risk) 
CCM Core Element 2: 
Access / Continuity 
Parameters for Care Manager 
12 monthly patient telephone contacts for 1 year (in addition to the skills 
management training above) to trouble-shoot self-management issues and 
summarize clinical status 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
 
Scheduled and Ad Hoc Clinical Assessments and Registry Data Utilization 
• 
Ad hoc contacts at either care manager or participant initiation based on clinical 
or other concerns, including response to participants within one business day. 
• 
“In-reach” to treating clinicians for hospitalization, ER visits, or specialty 
consultation 
• 
Communication of clinical assessment data to mental health provider 
• 
Collaboration with family as permitted 
• 
Resource referral as needed 
CCM Core Element 3: Provider 
Decision Support via 
Depression and Bipolar 
Disorder Clinical Practice 
Guidelines(96, 97) 
Provider contacts; care manager elicits preference for mode of communication 
with provider (e.g., call, email, fax) 
• 
Same content as clinic-based CCM 
• 
Guidelines based on AHRQ depression in primary care and APA bipolar 
guidelines 
• 
Will alert provider that his/her patient is receiving Life Goals and when he/she 
completes participation 
 
The mood disorders CCM intervention (“Life Goals Collaborative Care”) consists of: (a) a web-based 
patient self-management skills enhancement (CCM-1), (b) enhanced information flow and continuity of care via 
a care manager (CCM-2), and (c) decision support, or situation-specific evidence-based clinical practice 
guideline recommendations for providers (CCM-3)- see Table 4 (98). The CCM will be implemented utilizing 
telephonic contact with patients and providers by the Aetna care managers who participated in the Aetna care 
management pilot study, based on a recommended caseload of 80-100 patients each (99). The care managers 
will also use the Life Goals web portal as a guide for each session. 
 
Care managers will implement the three CCM Core Elements as follows:  
CCM-1: Web-based Self-Management Support: The care manager will deliver the Life Goals Program 
adapted for mood disorders via telephone using a website that contains modules, exercises, and other 
information for patients and the CM (52, 95). The Life Goals program utilizes psychoeducation based on Social 
Cognitive Theory, emphasizing Motivational Interviewing and cognitive-behavioral techniques, particularly 
behavioral activation, to address symptom management and problem-solving skills. Ten core modules will be 
delivered over 10 weekly telephonic sessions of 30 minutes (59, 100) using content available from the Life 
Goals website. An additional six modules can be implemented based on patient preferences for other specific 
self-management topics (101), and will be tracked as part of the CCM engagement index (Section C.5.h). Both 
the CM and patients will also have access to additional exercises and tailored modules via the Life Goals web 
portal. For patients with bipolar disorder, at least one of the modules will focus on coping with manic 
symptoms, while patients with unipolar depression will also receive an additional module on depressive 
symptoms (100).  
 
CCM-2: Access/Continuity: This core element consists of three components: 
• 
Patient Clinical Assessment: During the first 10 weeks, each fourth call (monthly) will also include an 
assessment of overall clinical and treatment status (~15 minutes). Assessment will include the 9-item 
Patient Health Questionnaire (PHQ-9)(102, 103) to assess depressive symptoms, the 8-item Internal State 
Scale (ISS; (52, 104-106) to assess manic symptoms and well-being; medication use, adherence (7, 107), 
and side effects. The intake contact includes identification of care providers and recent contacts, and each 
follow-up call will include a focus on maintaining engagement/continuity. These calls will continue on a 
monthly basis for up to one year of participation, with additional phone contacts made to follow up in the 
event of a hospitalization or emergency room visit based on our RCTs (50-53, 108). 
• 
Claims Database/Registry Review: The care manager will also use the eTUMS registry to track inpatient 
hospitalizations, pharmacy and laboratory status. These data, available in a clinically useful time frame 
(Section C.5.c), will assist the patient with trouble-shooting illness management, and assist the care 
manager in identifying opportunities for intervention or quality enhancement. 
• 
Provider communication: Care management strategies to date have under-emphasized provider 
engagement (9). In the mood disorders CCM, the care manager will contact the provider at the time of 
patient enrollment to introduce the CCM and to determine how best to integrate CCM procedures into the 
provider’s practice needs. This initial contact will identify the optimal mode of communication between 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
provider and care manager, as well as delineation by the provider for the care manager of the expected 
care plan going forward (frequency of visits, medications, lab tests such as cardiometabolic monitoring). 
After this initial engagement and care planning contact, the care manager will contact the provider in the 
event of hospitalization, emergency room visit, or development of a new clinical issue, as well as to cue 
them if there is a crisis encounter (e.g., suicidality). 
 
CCM-3: Provider Decision Support: As noted above, the care manager will have obtained clinical 
assessment data each month plus eTUMS-based data on pharmacy refills (weekly) and laboratory utilization 
(monthly). Using these multi-component assessment and treatment data, s/he will communicate situation-
specific treatment options based on clinical practice guidelines for bipolar disorder (96, 98). For instance in the 
case of unipolar depression, non-response or partial response to medication (defined as <50% reduction in 
PHQ-9 score and PHQ-9 <=9) would be followed by the care manager providing a series of treatment options 
based on current guidelines (97, 109). For those with bipolar disorder and presence of subtherapeutic lithium 
levels and manic symptoms, the care manager would also outline options based on clinical practice guidelines 
(110). It is important to note that, as in our prior studies (50-53, 108), the CCM is designed to supplement care 
and does not replace or control provider decision-making. That is: clinical decision-making remains in the 
hands of the provider. CCM therefore enhances care processes by: (a) enhancing patient skills to facilitate 
treatment participation, (b) offering the provider timely information, and (c) outlining situation-specific evidence-
based treatment options. 
 
C.5.e. Implementation Procedures. The study will be managed by the Aetna Joint Operating Committee 
led by Drs. Kilbourne and Un and includes William Gillis, PsyD, Aetna Clinical Specialty Programs Head, and 
Lesa Stevens, RN, Aetna Head of Program Development (Section C.5.k, see support letter from Dr. Un). We 
will implement the CCM using Enhanced REP (76, 77, 79). Our research implementation infrastructure will 
provide, keeping with the theme of “light footprint” of research, the customized package of CCM procedures 
(see Appendix), supported by 16 hours of training at study initiation, followed by ongoing technical 
assistance. CCM Fidelity monitoring will be incorporated into ongoing Aetna CM audits (Section C.5.c). 
 
Packaging includes an implementation manual for all three CCM components developed by Dr. Kilbourne as 
well as the secure web-based self-management program for care managers and patients for the CCM-1 
component (52, 95).  
 
Training for care managers will follow existing Aetna standard procedures for telephonic and web-based 
training. This will consist of 16 hours of telephonic training by Dr. Kilbourne and her study team over 2 weeks 
modeled after our current on-site training package. 
 
Technical assistance will be provided by the Joint Operating Committee and supported by study consultants 
Dr. Greg Simon (telephonic care management) and Dr. John Fortney (implementation processes). Joint 
Operating Committee members will conduct conference calls with the care managers (weekly) and supervisors 
(monthly) to guide the use of CCM components, particularly with regard to interface issues between: care 
managers and patients; care manager and providers/practices; and patients and providers. The Joint 
Operating Committee will have separate quarterly and ad hoc calls with consultants for debriefing and advice.  
 
C.5.f. Data Collection and Measures. Key outcomes data will be ascertained from patients (including 
surveys, medical record- eTUMS reviews, and claims data) and organizational surveys (Table 5).  
 
Patient survey: A concise (15-30 minute) quantitative survey will be collected from patients via phone at 
baseline, 6, 12, 18, and 24 months thereafter by the research associate (RA). If the RA is unable to reach the 
patient via phone for follow-up assessments after multiple attempts, a survey will be mailed to the confirmed 
last known address of the participant as a final attempt to receive this data.  Patients will be asked to return this 
survey without any identifying information on the survey or the envelope.  The survey will contain a subject ID 
number to identify the patient’s survey when it is returned via mail.  Patient outcomes data will be entered by 
the RA into an database on a password-protected research computer within a secure network server at the 
University of Michigan.  
 
Patient utilization and cost data, described below will be ascertained from the Aetna eTUMS and claims 
databases, and exported securely for merging with survey data and analysis at the University of Michigan.  
 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
 
Table 5: Primary and Secondary Outcomes and Measures – Baseline and Follow-ups 
Aims 
Key Measures 
Source 
Aim 1. Primary outcomes 
Mood symptoms: PHQ-9 
Patient survey 
 
Health-related quality of life-SF-12 
Patient survey 
Aim 
1. 
Secondary 
outcomes 
Guideline concordant care- key measures 
Medical record (eTUMS)/claims 
 
Mood tx: % receiving guideline-concordant 
antidepressants (if unipolar depression) or guideline-
concordant anti-manic treatment (bipolar disorder dx) 
in 6-month period 
Medical record (eTUMS)/claims 
 
Cardiometabolic monitoring: % receiving lipid profile, 
fasting glucose or HbA1C 
Medical record (eTUMS)/claims 
 
Hospitalizations (ACSC) 
Aetna claims data 
 
Productivity (Work Limitation Questionnaire) 
Patient survey 
Aim 2 
Organizational data: Practice demographics 
Aetna data 
 
Patient: demographics, comorbidities 
Patient survey 
Aim 3: 
CCM costs, patient inpatient, outpatient, ER, Rx use 
Aetna claims, provider survey 
 
Patient utilities (SF-6) 
Patient survey 
 
The research associate will be blind to random assignment at baseline. In follow-ups, given the nature of 
the CCM intervention (CCM-2), there is the possibility that enrolled patients may inadvertently disclose their 
randomization status with the RA. In pilot studies we found that this is a rare occurrence, but will be mitigated 
through RA training (e.g., minimizing leading questions that would disclose treatment assignment). 
 
Primary Aim 1 (patient-level quality of life, symptoms) include changes in mood symptoms based on the 
PHQ-9 (102, 103) or ISS (52, 104-106) and health quality of life via the SF-12 Mental and Physical Health 
Component Scores (MCS/PCS; 85,113). 
 
Patient-level secondary outcomes include guideline-concordant (quality of) care, hospitalizations, as well as 
self-reported productivity. These measures are based on recommended indicators from the Accountable Care 
Organization published rules for performance measures and shared savings (111). We will apply previously 
established quality measures for mood disorders (98, 111, 112), and cardiometabolic management (113, 114) 
(Table 5). Quality of care data will be gathered from electronic medical records (eTUMS) and hospitalization 
data will be ascertained from Aetna utilization and claims files at the end of patients’ 12 and 24-month 
participation in two prospectively observed 6-month intervals.  Preventable hospitalizations over the 12 and 24-
month period will be defined using the AHRQ Ambulatory Care Sensitive Condition (ACSC) definition (115), 
which identifies conditions for which “good” outpatient care can potentially prevent the need for hospitalization 
or for which early intervention can prevent complications or more severe disease. We will also assess the 
probability of psychiatric rehospitalization separately. Patient-level productivity over the 24-month period will 
be assessed in the survey using the Work Limitation Questionnaire(116).  
 
Patient factors include CCM engagement ascertained from the patient survey. Engagement is based on the 
six-item Preference Index for Decision-Making (API-D; (117, 118)) which has been used in chronically ill 
samples to investigate the desire for collaborative care from clinicians. Other patient factors include 
demographic and clinical information, including co-occurring symptoms (substance use (121, 122), 
comorbidities (from eTUMS). 
 
Aim 3: Utilization, costs, CEA, and business case: Primary utilization measures of interest, including 
outpatient services, inpatient (including length of stay), and ER use, and health care costs will be ascertained 
from Aetna eTUMS electronic record reviews using a standard assessment tool (125). Cost data will be 
estimated for each inpatient, ER, and outpatient visit using Current Procedural Terminology (CPT) codes. A 
relative value unit (RVU) weight will allow for the use of the Medicare Fee Schedule to calculate a standardized 
cost for each service. Costs incurred in different years will be discounted at an annual rate of 3% and adjusted 
for inflation and discounted to the baseline (first) year of the study. Other costs incurred by the participants 
including medications, visit co-pays, transportation, and time involved in the CCM will be ascertained from the 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
patient survey. CCM cost data will be estimated using a time-motion survey developed by study investigators 
(R01 MH 79994) administered monthly to the CMs and quarterly to providers over a 2-year period, which 
ascertains time spent on CCM activities, including self-management, phone contacts, and documentation.  
 
C.5.g. Analyses.  General approach: The principal analyses will be for the primary hypotheses. An intent-to-
treat analysis will be performed for all analyses. Bivariate baseline analyses will first be conducted to see if 
randomization was successful by comparing patient demographics and clinical characteristics (e.g., mental 
health diagnoses) between randomization groups. If there is a lack of equal distribution across groups, these 
variables will be added as covariates to analyses or propensity scoring will be used. Baseline characteristics 
will be compared among those enrolled but dropped out over time to those who remained in the study. Our unit 
of analysis for patient-level outcomes will be patients nested within practices with an indicator for non-solo 
practice provider sites (126, 127) and an indicator for patients diagnosis (bipolar vs. unipolar depression).   
 
Missing Data: We attained subject retention of 80% over a three-year follow-up in one study (50, 51), and 
recent 1-year data (MH0745409) indicate 90% retention. Extent and pattern of missingness will be examined 
for outcome variables as well as for baseline covariates. We expect missingness to be completely at random or 
at random, in which case the proposed analytical methods will give unbiased estimates of treatment 
comparison. We will examine how sensitive our conclusions are to potential non-ignorable missingness using a 
pattern mixture model that will allow us to either model the observed pattern of missingness or change the 
imputations to represent the likely differences in conditional distributions between observed and missing data 
(128, 129). For the latter, we will combine the results from each imputed data using Rubin’s rules (130). 
 
Aim 1 primary outcomes (H1a: symptoms, quality of life): Primary outcomes are continuous, and based on 
pilot data, are expected to be normally distributed. If a continuous outcome exhibits a significant lack of 
normality, other options will be considered including data transformation, categorization, and non-parametric 
analyses. Four linear mixed-effects models accounting for clustering will be run to assess the CCM effects 
compared to EC on changes in PHQ-9, ISS, and SF-12 PCS and MCS scores over 12-months, adding 
baseline values of the outcome measure, the effect of CCM, time, and CCM X time interaction.  
 
Aim 1 secondary analyses: For guideline concordance and hospitalization, we will compare the likelihood of 
receiving the guideline-concordant treatment over 12 months (Table 5) as well as the likelihood of an 
ambulatory care sensitive admission using a generalized linear mixed-effects model (GLM) with logit link. 
Similar to our primary hypotheses, linear mixed-effects models will be conducted to determine the effect of 
CCM vs. EC on changes in the Work and Social Adjustment Scale. 
 
Aim 2 analyses are considered exploratory but will be used to gain insight into the mechanisms by which 
CCM impacts patient outcomes. We will first explore the strength and direction of the association between 
patient-level CCM engagement (API-D scores- see section C.5.f) (119) on changes in outcomes (SF-12, PHQ-
9 and ISS scores) by adding patient-level API-D scores to the aforementioned linear mixed-effects models 
from Aim 1. We will also be run exploratory analyses in which we add key organizational factors (e.g., non-solo 
practice, practice type-primary care vs. behavioral health, presence of non-MD clinical staff, EMR use, 
ownership), and other patient factors (e.g., demographics, comorbidities) in the models specified in Aim 1. In 
secondary analyses we will also incorporate the number of additional optional sessions covering supplemental 
modules completed by the patient (Section C.5.d).  
 
Aim 3: Both a payer and societal-based cost-effectiveness analysis (CEA) will be conducted. A CEA 
from the payer perspective will involve comparison of utilization costs of health service providers including 
CM’s time ascertained from time-motion survey. A societal perspective will add non-health care costs incurred 
by patients including time and transportation. Generalized Linear Models with log link functions will be used to 
correct for heteroscedasticity and reduce the impact of outliers (131). The incremental effectiveness of CCM 
compared to EC will be measured by changes in health utilities, assessed using a method described by Zivin 
(132) and Brazier (133) which translates six of the SF-12 items to changes in health utility (SF-6) based on 
responses to standard gamble questions given by community members regarding all combinations of possible 
health states. Cost-effectiveness ratios will be calculated based on the difference in per patient costs and 
effectiveness of CCM versus EC. To quantify uncertainty around these ratios, a standard nonparametric 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
bootstrapping approach will be employed. For the business case, additional analyses will be conducted in 
which changes over time in utilization and costs of inpatient, ER, and outpatient services (medical, psychiatric) 
will be compared between patients in the CCM or EC arms over a 2-year period, in order to determine the time 
dynamics by which the CCM led to changes in health care costs. Additional cost analyses will take into 
consideration the return-on-investment of the CCM and changes in enrollee turnover (134). 
 
C.5.h. Business plan development (Aim 3). The business case development plan is based on our Enhanced 
REP framework (73) and informed by the work by Leatherman and colleagues (also see Appendix for 
submitted paper on business case). In brief, a business case for the CCM exists if the Aetna realizes a 
financial return on investment or other positive impact on organizational function or sustainability within a 
“reasonable” time frame (134). Other positive impacts beyond the potential cost savings from implementing 
CCM include revenue-based from the payer perspective (i.e., value added to payers when the health plan 
offers CCMs in terms of improved enrollee health and access to care). We chose 2 years as the time frame for 
outcomes in this study because prior research on CCMs has not found a return-on-investment within one year 
(3), and 2 years is below the average tenure of an Aetna enrollee. From a business case standpoint, attrition 
from the health plan may attenuate savings from the initial investment, and our analysis will implicitly account 
for this by only counting within-plan utilization costs. Enhanced REP addresses organizational and financial 
incentives (Table 6) designed to inform the CCM return-on-investment (73) and will lead to the development of 
a reimbursement model for CCMs. Preliminary stages for business case development (134), including (a) the 
implementation of a program (CCM) shown to improve patient health (135), (b) establishment of quality and 
cost indicators aligned with existing national initiatives (e.g., ACOs), and (c) identification of a health plan 
champion and consultant team have been completed. The remaining years (3-5) will be devoted to refining and 
marketing the business plan, which will include a CCM reimbursement strategy. 
 
 
Generalizability of CCM business plan: A manuscript describing a generic CCM business plan/reimbursement 
strategy by study investigators is currently under review. An external consultant team (Bao, Bauer, Fortney, 
Simon, Van Deusen Lukas) was also established to advise on the updated business plan based on study 
findings, and to ensure generalizability to other health plans and national organizations beyond Aetna. They 
will also advise on the financial impact of the CCM on the patient, practice, as well as the health plan, and the 
trade-offs between quality improvement and the business case in order to inform the future implementation of 
CCMs in other health care settings. The consultant team will work with study investigators and Aetna 
leadership particularly in Years 3-5 via regular conference calls to review the business case. Based on these 
calls in Year 5 we will produce a series of papers, presentations, and conference workshops on the CCM 
business case with an eye towards other health plans and state and federal government entities including the 
Centers for Medicare and Medicaid Services and AHRQ. These products will present and critique the results 
Table 6: Enhanced REP Framework Business Case Development Steps by Year and Study Aim 
Enhanced REP 
Phase 
Incentives Components Informing Business Case 
Pre-conditions 
(Years 1-2) 
Identify health plan’s current priorities, and organizational barriers and facilitators to 
implementing CCM (Aim 2). Establish, publicize measureable CCM outcomes (Aim 1), aligned 
with existing measures (e.g., ACO quality, measures: cardiometabolic, mood stabilizer use, 
and hospitalizations (111)) 
Pre-Implementation 
(Years 1-5) 
Identify health plan, provider champions who mobilize support for CCM (Aetna, consultant 
team) 
Identify successfully implemented CCM components across plan (e.g., existing health IT- 
eTUMS) 
Implementation 
(Years 3-5) 
Consultant team reviews EBP business case, holds regular calls to develop business plan 
based on study findings (Aim 3- CEA). Propose reimbursement strategies (e.g., billing codes, 
bundled payments through ACOs- See Appendix) and highlight trade-offs with initial 
investment, long-term ROI 
Maintenance and 
Evolution (Years 4-5) 
Market business plan to sustain CCM beyond health plan to other national leaders in public, 
private sector (Dissemination plan). Leverage CCM with existing federal programs that are 
reimbursable (ACOs, health home models); continue publicizing CCM, find consumer stories 
re: CCM benefits 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
and business case in order to support movement toward wider adaptation and dissemination of the CCM for 
other mental health conditions. 
 
C.5.i. Sample Size and Power. Sample size is estimated based on our primary aim and informed by our 
updated CCM pilot studies that estimate effects on changes in depressive symptoms and well-being across 
mood disorders (e.g., Cohen’s D=.36). Assuming 90% power based on a 0.05 level two-sided test and a 
conservatively assumed ICC of 0.05 at the practice level and an average of 2 patients per practice, a total of 
86 practices for each intervention arm (solo or small group providers) would be required. Assuming 20% 
dropout by 12 months, a projected 172 patients per arm (344 patients total) should give us adequate power to 
detect the expected between-group difference in mean outcome scores. For discrete outcomes including 
guideline concordance, using similar assumptions, the proposed sample size provides >90% power to detect a 
difference in guideline concordance of 46.5% (CCM) vs. 27.1% (EC) (98). 
 
C5.j. Potential Study Limitations. This study involves a number of strengths, including a groundbreaking 
health plan- academic partnership, comprehensive data sources, and emphasis on smaller and solo practices. 
Nonetheless, there are key limitations of this proposed design to consider. The focus on mood disorders 
potentially limits the immediate reach of this novel approach to implement CCMs for other chronic conditions. 
However, mood disorders have been identified by Aetna as priority conditions, and represent the most 
common and costly mental disorders in routine practice (136). In addition, while many persons with mood 
disorders are privately insured under network-model HMOs such as Aetna (137), the potential generalizability 
of this study is restricted to insured individuals. This study nonetheless complements a number of initiatives in 
the public sector that are currently being implemented to increase the uptake of CCMs for mental disorders 
such as health homes (10). Site providers will not be blinded to treatment assignment because CCM 
management and guideline supports require care manager-provider communication. Guideline concordance 
based on CM-provider coordination was found to be a key factor to the improvement of outcomes in the CCM 
(98). Finally, the cost effectiveness analysis is exploratory and not fully powered, but will nonetheless, provide 
valuable information for organizations considering its further adoption. 
 
C.5.k. Timeline and Organization. The first 6 months of the study will involve staff training and finalization of 
recruitment procedures (Figure 3). Once recruitment and enrollment begin in month 6, the CCM will be 
implemented throughout all study locations. Based on pilot studies recruitment is anticipated to last 18 months, 
hence allowing for ample time to recruit patients with either bipolar disorder or depression. Years 3-5 will be 
devoted to follow up data, business case, as well as other dissemination and implementation activities. 
 
Figure 3: Study Timeline 
 
Organizational relationship between investigator team and Aetna group (also see letter from Dr. Un): Aetna is 
providing support for the study organization through in kind contribution of effort by thee key personnel (Dr. 
Gillis, Ms. Stevens, and Ms. Keating) and the establishment of a Joint Operating Committee to facilitate project 
operations, care management infrastructure, and the informatics system. Aetna has used joint operating 
committees successfully in past clinical trials involving academic partnerships (e.g., 1R01DA029716; 
1R01DA026414). Specifically, Dr. Kilbourne and Dr. Un will serve as Co-Chairs of the Joint Operating 
Committee which will include both Aetna leaders and study investigators of the project, and will have oversight 
of all aspects of the project. The Committee will employ three function-oriented subcommittees with one co-
chair from Michigan or Harvard and one from Aetna who will oversee: CCM Implementation (Co-chairs: 
Kilbourne, Gillis), Patient Enrollment and Data Collection (Co-chairs Kilbourne, Eisenberg, and Stevens), and 
Study Month 
1-6 
7-12 
13-18 
19-24 
25-30 
31-36 
 
37-42 42-48 
 
49-54 
55-60 
 
Start-Up 
 
  
 
 
 
 
 
 
 
 
Patient Enrollment 
 
  
 
 
 
  
 
 
 
 
Patient Assessments 
 
 
 
  
 
 
 
 
 
 
Provider/org. Data Collection  
 
 
  
 
 
 
  
 
 
Business Case Development  
 
 
 
 
 
 
 
 
 
Final Analyses & 
Dissemination 
 
 
 
 
 
 
 
 
 
 
 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
Data Management and Analysis (Co-chairs Kilbourne, Kim and Keating). The Aetna co-leaders from each 
subcommittee have oversight over the Aetna personnel responsible for the respective functions.  All Aetna 
data will be available for audit by the Committee, and additionally we will utilize a Data Safety and Monitoring 
Board comprised of scientists separate from this project who will have similar access.  Thus our joint operating 
structure provides both the data monitoring oversight and the line authority control required to ensure the 
successful implantation of this project. Dr. William Gillis, Clinical Specialty Programs Head for the Aetna 
Behavioral Health, will oversee and manage the clinical operations and fidelity of the CCM implementation, 
including care manager and research associate supervision. This will include weekly meetings with the care 
managers’ supervisors to review the pilot’s key operational performance and fidelity metrics to ensure 
adherence to pilot workflows and protocols. He will coordinate with leadership to support overall 
implementation. Ms. Stevens will support Dr. Gillis and have responsibility for planning, execution and 
finalization of the project according to identified timelines and within budget.  Ms. Keating, Head of Aetna 
Behavioral Health Clinical Program Informatics, will be responsible for managing and aggregating the 
necessary outcome measures defined for the study (See Budget Justification). 
 
C.5.l. Dissemination and Generalizability. A dissemination strategy that ensures awareness and knowledge 
of research findings among clinicians, consumers, and policymakers, is an initial step to realizing the promise 
of this project, and fostering the potential generalizability of the findings. Once the intervention is completed 
and if proven effective, the Enhanced REP Framework will be applied to train other Aetna care managers in 
CCM. There are three stages of REP (packaging, training, and technical assistance). The CCM package will 
consist of a web-based therapist manual, administrator overview, and consumer handouts. Care managers as 
part of this study as well as providers with a high uptake score will be encouraged to become mentor-trainers 
who can provide training for other providers in the CCM. A refined technical assistance program based on 
study findings will be incorporated into the business plan for sustaining CCMs across Aetna and elsewhere. 
Study findings as well as the business plan will be disseminated through web sites, press releases, national 
conferences, policy briefs, as well as traditional forms of academic scholarship. If proven effective, the 
telephonic mood disorders CCM program may also potentially be scaled up to larger practices, notably through 
the use of technologies such as telemedicine or web-based portals independent of the care manager’s 
physical location. At least five manuscripts will be produced based on study aims including: 1) description of 
the Aetna CCM, and 2) CCM main outcomes, 3) mechanisms of CCM effects, 4) cost and utilization of CCM, 
and 5) the updated CCM business plan. A policy brief based on study findings will be disseminated to 
consumer and policy groups (e.g., AHRQ, CMS, Depression and Bipolar Support Alliance, National Alliance for 
the Mentally Ill, Primary Care Associations, National Council for Community Behavioral Healthcare), as well as 
to policymakers who have an interest in mental health.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
References 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
 
1. 
U.S. Department of Health and Human Services. U.S. Department of Health and Human Services 
(HHS) Inventory of Programs, Activities and Initiatives Focused on Improving the Health of Individuals with 
Multiple Chronic Conditions (MCC): Compiled by the HHS Interagency Workgroup on Multiple Chronic 
Conditions under the direction of Anand Parekh, MD, MPH, Deputy Assistant for Health (Science and 
Medicine), Office of the Assistant Secretary for Health; September 2011 Contract No.: Document Number|. 
2. 
Miller BF, Kessler R, Peek CJ. A National Agenda for Research in Collaborative Care: Papers from the 
Collaborative Care Research Network Research Development Conference. Rockville, MD: Agency for 
Healthcare Research and Quality,  June 2011. Report No.: 11-0067 Contract No.: Document Number|. 
3. 
Woltmann E, Grogan-Kaylor A, Perron B, Georges H, Kilbourne AM, Bauer MS. Comparative 
effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and 
behavioral health care settings: systematic review and meta-analysis. Am J Psychiatry. 2012;169(8):790-804. 
4. 
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness: the 
chronic care model, Part 2. JAMA. 2002;288(15):1909-14. 
5. 
Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 
1996;74(4):511-44. 
6. 
Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the Chronic Care Model in the new 
millenium. Health Affairs. 2009;28:75 - 135. 
7. 
Bauer MS. The collaborative practice model for bipolar disorder: design and implementation in a multi-
site randomized controlled trial. Bipolar Disord. 2001;3(5):233-44. 
8. 
Bauer MS, Williford WO, Dawson EE, Akiskal HS, Altshuler L, Fye C, et al. Principles of effectiveness 
trials and their implementation in VA Cooperative Study #430: 'Reducing the efficacy-effectiveness gap in 
bipolar disorder'. J Affect Disord. 2001;67(1-3):61-78. 
9. 
Coleman K, Mattke S, Perrault PJ, Wagner EH. Untangling practice redesign from disease 
management: how do we best care for the chronically ill? Annu Rev Public Health. 2009;30:385-408. 
10. 
Collins C, Hewson DL, Munger R, Wade T. Evolving models of behavioral health integration in primary 
care. 2010: Available from: www.milbank.org/reports/10430EvolvingCare/EvolvingCare.pdf. 
11. 
Findlay S. Managed behavioral health care in 1999: an industry at a crossroads. Health Aff (Millwood). 
1999;18(5):116-24. 
12. 
Bauer MS, Leader D, Un H, Lai Z, Kilbourne AM. Primary Care and Behavioral Health Practice Size: 
The Challenge for Health Care Reform. Med Care. 2012. 
13. 
Rittenhouse DR, Casalino LP, Shortell SM, McClellan SR, Gillies RR, Alexander JA, et al. Small and 
medium-size physician practices use few patient-centered medical home processes. Health Aff (Millwood). 
2011;30(8):1575-84. 
14. 
Fisher ES, McClellan MB, Bertko J, Lieberman SM, Lee JJ, Lewis JL, et al. Fostering accountable 
health care: moving forward in medicare. Health Aff (Millwood). 2009;28(2):w219-31. PMCID: PMC2656392. 
15. 
Shortell SM, Casalino LP. Implementing qualifications criteria and technical assistance for accountable 
care organizations. JAMA. 2010;303(17):1747-8. 
16. 
Nutting PA, Miller WL, Crabtree BF, Jaen CR, Stewart EE, Stange KC. Initial lessons from the first 
national demonstration project on practice transformation to a patient-centered medical home. Ann Fam Med. 
2009;7(3):254-60. PMCID: 2682981. 
17. 
Carrier E, Gourevitch MN, Shah NR. Medical homes: challenges in translating theory into practice. Med 
Care. 2009;47(7):714-22. PMCID: 2790523. 
18. 
Rittenhouse DR, Casalino LP, Gillies RR, Shortell SM, Lau B. Measuring the medical home 
infrastructure in large medical groups. Health Aff (Millwood). 2008;27:1246-58 
 
19. 
Rittenhouse DR, Shortell SM, Gillies RR, Casalino LP, Robinson JC, McCurdy RK, et al. Improving 
chronic illness care: findings from a national study of care management processes in large physician practices. 
Med Care Res Rev. 2010;67(3):301-20. 
20. 
Casalino L, Gillies RR, Shortell SM, Schmittdiel JA, Bodenheimer T, Robinson JC, et al. External 
incentives, information technology, and organized processes to improve health care quality for patients with 
chronic diseases. JAMA. 2003;289(4):434-41. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
21. 
Druss BG, von Esenwein SA, Compton MT, Rask KJ, Zhao L, Parker RM. A randomized trial of medical 
care management for community mental health settings: the Primary Care Access, Referral, and Evaluation 
(PCARE) study. Am J Psychiatry. 2010;167(2):151-9. 
22. 
Katon WJ, Lin EH, Von Korff M, Ciechanowski P, Ludman EJ, Young B, et al. Collaborative care for 
patients with depression and chronic illnesses. N Engl J Med. 2010;363(27):2611-20. 
23. 
Bauer MS. Bipolar (manic-depressive) disorder. In: Tasman A, Kay J, Lieberman J, Maj M, First MB, 
editors. Psychiatry. 3rd ed. London: Wiley-Blackwell; 2008. 
24. 
Lopez AD, Murray CC. The global burden of disease, 1990-2020. Nat Med. 1998;4(11):1241-3. 
25. 
Judd LL, Akiskal HS. The prevalence and disability of bipolar spectrum disorders in the US population: 
re-analysis of the ECA database taking into account subthreshold cases. J Affect Disord. 2003;73(1-2):123-31. 
26. 
Merikangas KR, Akiskal HS, Angst J, Greenberg PE, Hirschfeld RM, Petukhova M, et al. Lifetime and 
12-month prevalence of bipolar spectrum disorder in the National Comorbidity Survey replication. Arch Gen 
Psychiatry. 2007;64(5):543-52. 
27. 
Kessler RC, Wang PS. The descriptive epidemiology of commonly occurring mental disorders in the 
United States. Annu Rev Public Health. 2008;29:115-29. 
28. 
Goodwin RD, Davidson KW, Keyes K. Mental disorders and cardiovascular disease among adults in 
the United States. J Psychiatr Res. 2009;43(3):239-46. 
29. 
Katon W, Lin EH, Kroenke K. The association of depression and anxiety with medical symptom burden 
in patients with chronic medical illness. Gen Hosp Psychiatry. 2007;29:147-55. 
30. 
Goldstein BI, Fagiolini A, Houck P, Kupfer D. Cardiovascular disease and hypertension among adults 
with bipolar I disorder in the United States. Bipolar Disord. 2009;11:657-62. 
31. 
Osby U, Brandt L, Correia N, Ekbom A, Sparen P. Excess mortality in bipolar and unipolar disorder in 
Sweden Arch Gen Psychiatry. 2001;58(9):844-50. 
32. 
Angst F, Stassen HH, Clayton PJ, Angst J. Mortality of patients with mood disorders: follow-up over 34-
38 years. J Affect Disord. 2002;68(2-3):167-81. 
33. 
Institute of Medicine, Committee on Quality of Health Care in America. Crossing the quality chasm: A 
new health system for the 21st century. Washington, D.C.: National Academy Press; 2003. 
34. 
Boardman JB. Health access and integration for adults with serious and persistent mental illness. 
Families, Systems, & Health. 2006;21(1):3-18. 
35. 
Busch S, Leslie D, Rosenheck R. Comparing the quality of antidepressant pharmacotherapy in the 
Department of Veterans Affairs and the private sector. Psych Serv. 2004;55:1386 - 91. 
36. 
Druss BG, Bradford DW, Rosenheck RA, Radford MJ, Krumholz HM. Mental disorders and use of 
cardiovascular procedures after myocardial infarction [see comments] 1 31. JAMA. 2000;283(4):506-11. 
37. 
Druss BG, Rosenheck RA, Desai MM, Perlin JB. Quality of preventive medical care for patients with 
mental disorders. Med Care. 2002;40(2):129-36. 
38. 
Hogan MF. The President's New Freedom Commission: recommendations to transform mental health 
care in America. PsychiatrServ. 2003;54(11):1467-74. 
39. 
Leslie DL, Rosenheck RA. Adherence of schizophrenia pharmacotherapy to published treatment 
recommendations: patient, facility, and provider predictors. Schizophr Bull. 2004;30(3):649-58. 
40. 
Leslie DL, Rosenheck RA. Benchmarking the quality of schizophrenia pharmacotherapy: a comparison 
of the Department of Veterans Affairs and the private sector. J Ment Health Policy Econ. 2003;6(3):113-21. 
41. 
Sernyak MJ, Dausey D, Desai R, Rosenheck R. Prescribers' nonadherence to treatment guidelines for 
schizophrenia when prescribing neuroleptics. Psychiatr Serv. 2003;54(2):246-8. 
42. 
Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. Washington, 
D.C.: The National Academies Press; 2009. 
43. 
Peele PB, Xu Y, Kupfer DJ. Insurance expenditures on bipolar disorder: clinical and parity implications. 
Am J Psychiatry. 2003;160(7):1286-90. 
44. 
Bryant-Comstock L, Stender M, Devercelli G. Health care utilization and costs among privately insured 
patients with bipolar I disorder. Bipolar Disord. 2002;4:398-405. 
45. 
Simon GE, Unutzer J. Health care utilization and costs among patients treated for bipolar disorder in an 
insured population. Psych Serv. 1999;50:1303-8. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
46. 
Dilsaver SC. An estimate of the minimum economic burden of bipolar I and II disorders in the United 
States: 2009. J Affect Disord. 2011;129(1-3):79-83. 
47. 
Gilbody S, Bower P, Fletcher J, Richards D, Sutton AJ. Collaborative care for depression: a cumulative 
meta-analysis and review of longer-term outcomes. Arch Int Med. 2006;166(21):2314-21. 
48. 
Badamgarav E, Weingarten SR, Henning JM, Knight K, Hasselblad V, Gano A, Jr., et al. Effectiveness 
of disease management programs in depression: a systematic review. Am J Psychiatry. 2003;160(12):2080-
90. 
49. 
Unutzer J, Katon WJ, Fan MY, Schoenbaum MC, Lin EH, Della Penna RD, et al. Long-term cost effects 
of collaborative care for late-life depression. Am J Managed Care. 2008;14(2):95-100. 
50. 
Bauer MS, McBride L, Williford WO, Glick H, Kinosian B, Altshuler L, et al. Collaborative care for bipolar 
disorder: Part I. Intervention and implementation in a randomized effectiveness trial. Psychiatr Serv. 
2006;57(7):927-36. 
51. 
Bauer MS, McBride L, Williford WO, Glick H, Kinosian B, Altshuler L, et al. Collaborative care for bipolar 
disorder: Part II. Impact on clinical outcome, function, and costs. Psychiatr Serv. 2006;57(7):937-45. 
52. 
Kilbourne AM, Post EP, Nossek A, Drill L, Cooley S, Bauer MS. Improving medical and psychiatric 
outcomes among individuals with bipolar disorder: a randomized controlled trial. Psychiatr Serv. 
2008;59(7):760-8. 
53. 
Simon GE, Ludman EJ, Bauer MS, Unutzer J, Operskalski B. Long-term effectiveness and cost of a 
systematic care program for bipolar disorder. Arch Gen Psychiatry. 2006;63(5):500-8. 
54. 
American Psychiaric Association. Practice guideline for the treatment of patients with bipolar disorder 
(revision). Am J Psychiatry. 2002;159(4 Suppl):1-50. 
55. 
Yatham LN, Kennedy SH, Schaffer A, Parikh SV, Beaulieu S, O'Donovan C, et al. Canadian Network 
for Mood and Anxiety Treatments (CANMAT) and International Society for Bipolar Disorders (ISBD) 
collaborative update of CANMAT guidelines for the management of patients with bipolar disorder: update 2009. 
Bipolar Disord. 2009;11(3):225-55. 
56. 
Kilbourne AM, Biswas K, Pirraglia PA, Sajatovic M, Williford WO, Bauer MS. Is the collaborative chronic 
care model effective for patients with bipolar disorder and co-occurring conditions? J Affect Disord. 
2009;112(1-3):256-61. 
57. 
Datto CJ, Thompson R, Horowitz D, Disbot M, Oslin DW. The pilot study of a telephone disease 
management program for depression. Gen Hosp Psychiatry. 2003;25(3):169-77. 
58. 
Hunkeler EM, Meresman JF, Hargreaves WA, Fireman B, Berman WH, Kirsch AJ, et al. Efficacy of 
nurse telehealth care and peer support in augmenting treatment of depression in primary care. Arch Fam Med. 
2000;9(8):700-8. 
59. 
Ludman EJ, Simon GE, Grothaus LC, Luce C, Markley DK, Schaefer J. A pilot study of telephone care 
management and structured disease self-management groups for chronic depression. Psychiatr Serv. 
2007;58(8):1065-72. 
60. 
Lynch D, Tamburrino M, Nagel R, Smith MK. Telephone-based treatment for family practice patients 
with mild depression. Psychol Rep. 2004;94(3 Pt 1):785-92. 
61. 
Mohr DC, Hart SL, Julian L, Catledge C, Honos-Webb L, Vella L, et al. Telephone-administered 
psychotherapy for depression. Arch Gen Psychiatry. 2005;62(9):1007-14. 
62. 
Oslin DW, Sayers S, Ross J, Kane V, Ten Have T, Conigliaro J, et al. Disease management for 
depression and at-risk drinking via telephone in an older population of veterans. Psychosom Med. 
2003;65(6):931-7. 
63. 
Ransom D, Heckman TG, Anderson T, Garske J, Holroyd K, Basta T. Telephone-delivered, 
interpersonal psychotherapy for HIV-infected rural persons with depression: a pilot trial. Psychiatr Serv. 
2008;59(8):871-7. 
64. 
Simon GE, VonKorff M, Rutter C, Wagner E. Randomised trial of monitoring, feedback, and 
management of care by telephone to improve treatment of depression in primary care. BMJ. 
2000;320(7234):550-4. PMCID: 27299. 
65. 
Simon GE, Ludman EJ, Rutter CM. Incremental benefit and cost of telephone care management and 
telephone psychotherapy for depression in primary care. Arch Gen Psychiatry. 2009;66(10):1081-9. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
66. 
Stein MD, Herman DS, Bishop D, Anderson BJ, Trisvan E, Lopez R, et al. A telephone-based 
intervention for depression in HIV patients: negative results from a randomized clinical trial. AIDS Behav. 
2007;11(1):15-23. 
67. 
Tutty S, Ludman EJ, Simon G. Feasibility and acceptability of a telephone psychotherapy program for 
depressed adults treated in primary care. Gen Hosp Psychiatry. 2005;27(6):400-10. 
68. 
Lynch DJ, Tamburrino MB, Nagel R. Telephone counseling for patients with minor depression: 
preliminary findings in a family practice setting. J Fam Pract. 1997;44(3):293-8. 
69. 
Pariser D, O'Hanlon A. Effects of telephone intervention on arthritis self-efficacy, depression, pain, and 
fatigue in older adults with arthritis. J Geriatr Phys Ther. 2005;28(3):67-73. 
70. 
Davis CL, Kilbourne AM, Pierce JR, Langberg R, Blow FC, Winkel BM, et al. Reduced mortality among 
VA patients with schizophrenia or bipolar disorder lost to follow-up and engaged in active outreach to return to 
care. Am J Public Health. 2012;102:S74-S9. 
71. 
Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 
2008;27(3):759-69. 
72. 
Kilbourne AM, Goodrich DE, Clogston J, Bauer MS, Waxmonsky JA, VanDeusen Lukas C, et al. 
Randomized controlled trial of the Replicating Effective Programs implementation framework: the Recovery-
Oriented Collaborative Care Study. Implement Sci. 2012;In review. 
73. 
Kilbourne AM, Neumann MS, Waxmonsky J, Bauer MS, Kim HM, Pincus HA, et al. Public-academic 
partnerships: evidence-based implementation: the role of sustained community-based practice and research 
partnerships. Psychiatr Serv. 2012;63(3):205-7. 
74. 
Green LW, Kreuter MW, Deeds SG, Partridge KB. Health Education Planning: A Diagnostic Approach. 
1st ed. Mountain View, CA: Mayfield Publishing Co.; 1980. 
75. 
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service 
organizations: systematic review and recommendations. Milbank Q. 2004;82:581-629. 
76. 
Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based 
interventions in health care: application of the replicating effective programs framework. Implement Sci. 
2007;2:42. 
77. 
Neumann MS, Sogolow ED. Replicating effective programs: HIV/AIDS prevention technology transfer. 
AIDS Educ Prev. 2000;12(5 Suppl):35-48. 
78. 
Stetler CB. Role of the organization in translating research into evidence-based practice. Outcomes 
Manag. 2003;7(3):97-103. 
79. 
Kelly JA, Somlai AM, DiFranceisco WJ, Otto-Salaj LL, McAuliffe TL, Hackl KL, et al. Bridging the gap 
between the science and service of HIV prevention: transferring effective research-based HIV prevention 
interventions to community AIDS service providers. Am J Public Health. 2000;90(7):1082-8. PMCID: 1446305. 
80. 
Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191-
215. 
81. 
Rogers E. Diffusion of Innovations. 5th ed. New York, NY: Free Press; 2003. 
82. 
Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within health care organizations: 
literature review and Delphi study. Int J Qual Health Care. 2004;16(2):107-23. 
83. 
Kirsh SR, Lawrence RH, Aron DC. Tailoring an intervention to the context and system redesign related 
to the intervention: a case study of implementing shared medical appointments for diabetes. Implement Sci. 
2008;3:34. PMCID: 2442606. 
84. 
Shaw B, Cheater F, Baker R, Gillies C, Hearnshaw H, Flottorp S, et al. Tailored interventions to 
overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane 
Database Syst Rev. 2005(3):CD005470. 
85. 
Stetler C, Legro M, Wallace C, Bowman C, Guihan M, Hagedorn H, et al. The role of formative 
evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21:S1 - 8. 
86. 
Wensing M, Wollersheim H, Grol R. Organizational interventions to implement improvements in patient 
care: a structured review of reviews. Implement Sci. 2006;1:2. PMCID: 1436010. 
87. 
Yano EM. The role of organizational research in implementing evidence-based practice: QUERI Series. 
Implement Sci. 2008;3:29. PMCID: 2481253. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
88. 
Lukas CV, Holmes SK, Cohen AB, Restuccia J, Cramer IE, Shwartz M, et al. Transformational change 
in health care systems: an organizational model. Health Care Manage Rev. 2007;32(4):309-20. 
89. 
VanDeusen Lukas C, Engle RL, Holmes SK, Parker VA, Petzel RA, Nealon Seibert M, et al. 
Strengthening organizations to implement evidence-based clinical practices. Health Care Manage Rev. 
2010;35:235-45. 
90. 
Gold M, Taylor E. Moving research into practice: Lessons from the Agency for Healthcare Research 
and Quality's IDSRN program. Implement Sci. 2007;2:9. 
91. 
Post EP, Kilbourne AM, Bremer RW, Solano FX, Jr., Pincus HA, Reynolds CF, 3rd. Organizational 
factors and depression management in community-based primary care settings. Implement Sci. 2009;4(1):84. 
92. 
Wang PS, Simon GE, Avorn J, Azocar F, Ludman EJ, McCulloch J, et al. Telephone screening, 
outreach, and care management for depressed workers and impact on clinical and work productivity outcomes: 
a randomized controlled trial. JAMA. 2007;298(12):1401-11. PMCID: 2859667. 
93. 
Wennberg DE, Marr A, Lang L, O'Malley S, Bennett G. A randomized trial of a telephone care-
management strategy. N Engl J Med. 2010;363(13):1245-55. 
94. 
Bruce ML, Ten Have TR, Reynolds CF, III, Katz II, Schulberg HC, Mulsant BH, et al. Reducing suicidal 
ideation and depressive symptoms in depressed older primary care patients: a randomized controlled trial. 
JAMA. 2004;291(9):1081-91. 
95. 
Bauer MS, Kilbourne AM, Greenwald DE, Ludman EJ, McBride L. Overcoming Bipolar Disorder: A 
Comprehensive Workbook for Managing Your Symptoms & Achieving Your Life Goals. Oakland, CA: New 
Harbinger Publications, Inc.; 2008. 
96. 
Dennehy EB, Bauer MS, Perlis RH, Kogan JN, Sachs GS. Concordance with treatment guidelines for 
bipolar disorder: data from the systematic treatment enhancement program for bipolar disorder. 
Psychopharmacol Bull. 2007;40(3):72-84. 
97. 
Depression Guideline Panel. Depression in Primary Care: Volume 1. Dectection and Diagnosis. Clinical 
Practice Guideline, Number 5. Rockville, MD: U.S. Department of Health and Human Services; 1993. Report 
No.: AHCPR Publication No. 93-0550 Contract No.: Document Number|. 
98. 
Bauer MS, Biswas K, Kilbourne AM. Enhancing multiyear guideline concordance for bipolar disorder 
through collaborative care. Am J Psychiatry. 2009;166(11):1244-50. 
99. 
Kilbourne AM, Post EP, Nossek A, Sonel E, Drill LJ, Cooley S, et al. Service delivery in older patients 
with bipolar disorder: a review and development of a medical care model. Bipolar Disord. 2008;10(6):672-83. 
100. 
Ludman EJ, Simon GE, Tutty S, Von Korff M. A randomized trial of telephone psychotherapy and 
pharmacotherapy for depression: continuation and durability of effects. J Consult Clin Psychol. 2007;75(2):257-
66. 
101. 
Insel TR. Translating scientific opportunity into public health impact: a strategic plan for research on 
mental illness. Arch Gen Psychiatry. 2009;66(2):128-33. 
102. 
Saarelainen S, Rytkonen-Nissinen M, Rouvinen J, Taivainen A, Auriola S, Kauppinen A, et al. Animal-
derived lipocalin allergens exhibit immunoglobulin E cross-reactivity. Clinical and Experimental Allergy. 
2008;38(2):374-81. 
103. 
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen 
Intern Med. 2001;16:606-13. 
104. 
Bauer MS, Crits-Christoph P, Ball WA, Dewees E, McAllister T, Alahi P, et al. Independent assessment 
of manic and depressive symptoms by self-rating. Scale characteristics and implications for the study of mania. 
Arch Gen Psychiatry. 1991;48(9):807-12. 
105. 
Bauer MS, Vojta C, Kinosian B, Altshuler L, Glick H. The Internal State Scale: replication of its 
discriminating abilities in a multisite, public sector sample. Bipolar Disord. 2000;2(4):340-6. 
106. 
Cooke RG, Kruger S, Shugar G. Comparative evaluation of two self-report Mania Rating Scales. Biol 
Psychiatry. 1996;40(4):279-83. 
107. 
Sajatovic M, Bauer MS, Kilbourne AM, Vertrees JE, Williford W. Self-reported medication treatment 
adherence among veterans with bipolar disorder. Psychiatr Serv. 2006;57(1):56-62. 
108. 
Bauer MS, McBride L, Shea N, Gavin C, Holden F, Kendall S. Impact of an easy-access VA clinic-
based program for patients with bipolar disorder. Psychiatric Services. 1997;48(4):491-6. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
109. 
Work Group on Major Depressive Disorder. Practice Guideline for the Treatment of Patients with Major 
Depressive Disorder Third Edition. Washington, D.C.: American Psychiatric Association; 2010 Contract No.: 
Document Number|. 
110. 
Disorder WGftToPwB. Clinical Practice Guideline for Bipolar Disorder, Third Edition. Washington, D.C.: 
American Psychiatric Association; 2010 Contract No.: Document Number|. 
111. 
Department of Health & Human Services, Centers for Medicare & Medicaid Services. 42 CFR Part 425. 
Medicare Program; Medicare Shared Savings Program; Accountable Care Organizations.  Washington, D.C.: 
Federal Register; November 2, 2011 [updated November 2, 2011; cited 2012 May 1, 2012]; Available from: 
http://www.gpo.gov/fdsys/pkg/FR-2011-11-02/pdf/2011-27461.pdf. 
112. 
The STABLE National Coordinating Council. STABLE: STAndards for BipoLar Excellence. A 
performance measurement and quality improvement program. Center for Quality Assessment and 
Improvement 
in 
Health; 
2006 
[updated 
2006; 
cited]; 
Available 
from: 
http://www.cqaimh.org/pdf/measures_summary.pdf. 
113. 
Kilbourne AM, Lai Z, Bowersox N, Pirraglia P, Bauer MS. Does colocated care improve access to 
cardiometabolic screening for patients with serious mental illness? Gen Hosp Psychiatry. 2011. 
114. 
American Diabetes Association, American Psychiatric Association, American Association of Clinical 
Endocrinologists, North American Association for the Study of Obesity. Consensus development conference 
on antipsychotic drugs and obesity and diabetes. Diabetes Care. 2004;27:596-601. 
115. 
Prevention Quality Indicators Overview: AHRQ Quality Indicators. Rockville, MD: Agency for Healthcare 
Research and Quality,  July, 2004 Contract No.: Document Number|. 
116. 
Mundt JC, Marks IM, Shear MK, Greist JH. The Work and Social Adjustment Scale: a simple measure 
of impairment in functioning. Br J Psychiatry. 2002;180:461-4. 
117. 
Hill SA, Laugharne R. Decision making and information seeking preferences among psychiatric 
patients. Journal of Mental Health. 2006;15:75-84. 
118. 
Hamann J, Cohen R, Leucht S, Busch R, Kissling W. Do patients with schizophrenia wish to be 
involved in decisions about their medical treatment? Am J Psychiatry. 2005;162(12):2382-4. 
119. 
Ludman EJ, Simon GE, Rutter CM, Bauer MS, Unutzer J. A measure for assessing patient perception 
of provider support for self-management of bipolar disorder. Bipolar Disord. 2002;4(4):249-53. 
120. 
Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized anxiety 
disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092-7. 
121. 
Gordon AJ, Maisto SA, McNeil M, Kraemer KL, Conigliaro RL, Kelley ME, et al. Three questions can 
detect hazardous drinkers. J Fam Pract. 2001;50(4):313-20. 
122. 
Kessler RC, Andrews G, Mroczek D, Üstün TB, Wittchen HU. The World Health Organization 
Composite International Diagnostic Interview Short Form (CIDI-SF) Int J Methods Psychiatr Res. 1998;7:171-
85. 
123. 
Rehm J, Ustun TB, Saxena S, Nelson CB, Chatterji S, Ivis F, et al. On the Development and 
Psychometric Testing of the WHO Screening Instrument to Assess Disablement in the General Population. Int 
J Methods Psychiatr Res. 1999;8  (2):110-22. 
124. 
Ustun TB, Chisholm D. Global "burden of disease"-study for psychiatric disorders. Psychiatrische 
Praxis. 2001;28 Suppl 1:S7-11. 
125. 
Kilbourne AM, Farmer Teh C, Welsh D, Pincus HA, Lasky E, Perron B, et al. Implementing composite 
quality metrics for bipolar disorder: towards a more comprehensive approach to quality measurement. Gen 
Hosp Psychiatry. 2010;32:636-43. PMCID: PMC3086005. 
126. 
Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 
1986;73:13-22. 
127. 
Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 
1986;42(1):121-30. 
128. 
Diggle PJ, Kenward MG. Informative dropout in longitudinal data analysis (with discussion). J Royal 
Stat Soc, Series C (applied statistics). 1994;43:49-94. 
129. 
Little RJA, Rubin DB. Satatistical analysis with missing data. New York, NY: Wiley; 1987. 
130. 
Rubin D. Multiple imputation for non-response in surveys. New York, NY: Wiley; 1987. 

Full NIH Title: Implementing Health Plan-Level Care Management for Solo & Small Practices 
Protocol # HUM00073753  
Protocol version: v5 3.23.15 
PI: Amy Kilbourne, PhD, MPH 
 
131. 
Manning WG, Mullahy J. Estimating log models: to transform or not to transform? J Health Econ. 
2001;20(4):461-94. 
132. 
Zivin K, McCarthy JF, McCammon RJ, Valenstein M, Post EP, Welsh DE, et al. Health-related quality of 
life and utilities among patients with depression in the Department of Veterans Affairs. Psychiatr Serv. 
2008;59(11):1331-4. 
133. 
Brazier J, Akehurst R, Brennan A, Dolan P, Claxton K, McCabe C, et al. Should patients have a greater 
role in valuing health states? Appl Health Econ Health Policy. 2005;4(4):201-8. 
134. 
Leatherman S, Berwick D, Iles D, Lewin LS, Davidoff F, Nolan T, et al. The business case for quality: 
case studies and an analysis. Health Aff (Millwood). 2003;22(2):17-30. 
135. 
Woltmann E, Grogan-Taylor A, Perron B, Georges H, Kilbourne AM, Bauer MS. Comparative 
effectiveness of collaborative chronic care models for mental health. Am J  Psychiatry. 2012;In Press. 
136. 
Simon GE, Katon WJ, VonKorff M, Unutzer J, Lin EH, Walker EA, et al. Cost-effectiveness of a 
collaborative care program for primary care patients with persistent depression. Am J Psychiatry. 
2001;158(10):1638-44. 
137. 
Frank RG, Huskamp HA, Pincus HA. Aligning incentives in the treatment of depression in primary care 
with evidence-based practice. Psychiatr Serv. 2003;54(5):682-7.