The Comparative Impact of Patient Activation and Engagement on Improving Patient-Centered Outcomes of Care in Accountable Care Organizations
Conditions: Diabetes, Cardiovascular Disease
Sponsor: University of California, Berkeley

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
1
 
 
 
 
 
 
 
 
 
The Comparative Impact of Patient Activation and Engagement on Improving Patient-
Centered Outcomes of Care in Accountable Care Organizations 
Contract Number: IHS-1310-06821 
Principal Investigator: Stephen M. Shortell, PhD, MPH, MBA 
University of California, Berkeley 
Study Protocol and Analysis Plan 
January 16, 2014 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
2
PCORI RESEARCH PLAN TEMPLATE 
 
RESEARCH STRATEGY  
 
A. Background  
 
 
The Affordable Care Act (ACA) empowered the Center for Medicare and Medicaid Services (CMS) to create 
Accountable Care Organizations (ACOs) charged with being accountable for both the costs and quality of care for a 
defined group of patients. With CMS’s recent approval of 123 additional shared savings ACOs, there are now more than 
600 ACOs across the United States, both federal and private1, 2. These organizations provide a unique opportunity to 
examine fundamental changes in how health care services are delivered. The ability of ACOs to succeed under the new 
payment models will depend importantly on a re-structuring of the doctor-patient relationship; particularly in regard to 
caring for patients with chronic conditions such as diabetes and cardiovascular diseases (CVD). Responsibility for patient 
care is increasingly being shared among members of primary care teams (physicians, nurse practitioners, physician 
assistants, pharmacists, nutritionists, nurses, and others) and, most importantly, involving patients and their families. In 
fact, engaged patients have been referred to as “...the blockbuster drugs of the 21st century”3. 
 
The development of ACOs provides the opportunity to compare the effectiveness of practices within ACOs who 
are more highly involved in Patient Activation and Engagement activities (PA&E) with those less highly involved in 
regard to patient-reported outcomes of care, patient experience of care, and selected clinical measures. It also provides an 
opportunity to engage patients (a requirement of the ACA legislation) in the research and, in particular, in reviewing the 
results of performance feedback data on subsequent patient outcomes of care. Further, the national network of ACOs that 
has developed across the country and is likely to grow provides a platform to rapidly leverage and disseminate study 
findings. This will be further facilitated by our related ongoing research on ACO developments (Shortell, et al. 2014)4, 
physician practices (Shortell, et al, 2014)5, primary care practices (Rodriguez, Anastario et al. 2008; Rodriguez, von Glahn 
et al. 2013; Rodriguez, Giannitrapani et al. 
2013)6-8 and patient activation and engagement 
(Shortell et al, 2014)9. Preliminary evidence 
from our patient-informed national web-based 
survey of PA&E activities in ACOs, presented 
in Table 1, suggests the potential for much 
greater involvement than currently exists. These 
findings underscore the importance and salience 
of the current proposal. 
 
The central question that this proposal 
addresses is whether patients with diabetes or 
CVD who receive care from primary care 
practices with high involvement in patient 
activation and engagement (PA&E) 
initiatives report better patient-reported 
outcomes (PROs), better experiences of care, 
and selected clinical measures (blood pressure, HbA1c, and LDL-C) compared to patients who receive care from 
primary care practices with low involvement in PA&E activities. In addition, we will examine: (1) whether the impact 
of PA&E initiatives on outcomes is stronger for patients who score higher on the patient activation measure (PAM) than 
those who score lower10; and (2) whether the effect is stronger for patients who improve their PAM scores over time 
compared to patients who do not. We will also assess the characteristics of primary care teams in practices associated with 
both high and low involvement in PA&E activities in regard to the degree of coordination, perceived team effectiveness, 
leadership, and team culture that they exhibit. Finally, we will explore the impact of providing structured feedback to 
practices on the PRO, patient experience, clinical indicators, and teamwork assessment results to both practice teams and 
patient advisory groups on any improvement activities undertaken by the primary care teams. In the sections that follow, 
Table 1. Preliminary Results of the Patient Activation and 
Engagement Survey (n=75 ACOs) 
  
Mean (SD) 
Percentage of PCPs that have received training in 
PA&E methods and techniques 
47.8 (30.7) 
Percentage of PCPs that work with patients/families to 
develop a treatment plan that sets goals for their care 
62.0 (29.6) 
Percentage of ACO's high-risk chronic illness patients 
that participate in peer support groups or group visits 
19.0 (18.9) 
Percentage of PCPs that offer patients/families 
evidence-based decision aids (e.g., paper or online 
tools to help make informed choices among their 
options for treatment) 
47.7 (30.5) 
Percentage of ACO's that formally assess the health 
literacy level of patients/families 
26.7% 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
3
we define each of these concepts, their specific measures, and the role they play in our overall logic model and research 
design. The definition of patient activation used throughout the proposal is the patient’s understanding of their role in the 
care process and having the knowledge, skills, and confidence to take on that role. Patient engagement denotes a broader 
concept, which includes patient activation, the interventions designed to increase it, and the patient behaviors that result 
from it.  
 
CRITERION 1 – IMPACT OF THE CONDITION 
 
We will study patients with diabetes (ICD-9CM 250.XX) and/or CVD (ICD-9CM 393-459). Uncontrolled diabetes 
can result in high-cost complications including retinopathy11, incident myocardial infarction (MI), stroke, congestive heart 
failure (CHF), and nontraumatic lower extremity amputation (LEA)12, contributing to high health care expenditures in the 
United States13.  These and related chronic conditions represent an estimated 75 percent of all U.S. health care 
expenditures14. 
 
It is estimated that 26% of adults over age 18 have multiple chronic conditions (MCC) and this percent has increased 
since 2001. The prevalence of MCC significantly increases with age15.  For adults over 65, as many as 85 percent suffer 
from one or more chronic diseases16. At least 18.8 million Americans have diagnosed diabetes. An additional 7 million 
people are estimated to be undiagnosed17. Diabetes is the sixth leading cause of death and is rising as a cause of mortality 
as well as in prevalence, with an annual total cost of $245 billion for diagnosed cases18. Heart disease (which includes 
CAD and CHF among other conditions) affects about 26.5 million in the US while hypertension affects over 76 million 
Americans19. The cumulative death toll from CVD was nearly 600,000 deaths per year in 201020, with 715,000 heart 
attacks per year in the U.S.21 at an annual cost of $108.9 billion for health care services, medications, and lost 
productivity22. Altogether, CVD accounts for 29 percent of all deaths in the U.S. No other group of chronic conditions 
imposes this level of impact on our population; however many people are still undiagnosed or untreated. Among those 
diagnosed and treated, levels of adherence to medication and lifestyle changes remain low and physicians often do not 
know how to better activate patients in clinical settings.  
 
Diabetes represents one of the largest documented health disparities between Latinos and non-Latino whites23 and 
Latinos often receive suboptimal treatment. For example, Latinos are less likely than whites to receive appropriate HbA1c 
and lipid screening24, 25 and are less likely to achieve treatment goals, including glycemic, cholesterol, and blood pressure 
control26-29. Since Latinos, African-Americans, and certain Asian sub-groups are more likely to have diabetes compared to 
whites, many studies have focused on developing and testing interventions to improve outcomes of diabetes care through 
patient activation and engagement among these vulnerable patient populations30-33.  ACOs can play an important role in 
implementing PA&E initiatives that span clinical and community settings, including community health worker or health 
coach models for diabetes care management.  The care of patients with diabetes and CVD presents many challenges for 
health care delivery systems34. Knowledge is emerging on how to better care for patients with these conditions, but little is 
known about how patients can be better activated and engaged in their own care and what health care providers and care 
teams can do to promote such engagement and with what results. This proposal directly addresses these issues. 
 
B. Significance  
 
CRITERION 2 – ATTEMPTS TO IMPROVE HEALTH CARE AND OUTCOMES 
 
It is increasingly obvious that to improve patient outcomes and experience of care will require enhanced PA&E35. 
There is a small but growing literature suggesting that more activated patients experience better health outcomes36-40. But 
much less is known about how to accomplish this41, 42. For example, national data on trends between 2008 and 2012 
indicate essentially no change in the percentage of patients with blood pressure under 140/90, the percentage of heart care 
patients with LDL-C < 100, the percentage of diabetes care patients with LDL-C < 100, and the percentage of patients 
with diabetes who have blood pressure under 130/80 and HbA1c under 9 percent43. We recognize that for patients with 
diabetes over age 85 that strict adherence to these numbers is controversial and therefore we will exclude patients over 85 
from this study.44, 45 In addition, almost no systematic data are available on patient-reported outcomes of care in terms of 
physical and social functioning and related dimensions. Patients often struggle to live their lives while adhering to 
multiple medications, experiencing side effects, and trying to change behaviors such as smoking, diet, and physical 
activity.  The results from this research will provide important knowledge and guidance for others, particularly in regard 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
4
to what health care organizations and teams might do to strengthen their PA&E efforts and the impact on measures of 
most interest to patients. 
 
SPECIFIC AIMS 
Based on the above discussion, the specific aims of the study are outlined below: 
 
1) To answer the question: Do patients with diabetes and CVD receiving care from practices with higher versus lower 
PA&E activities have better patient-reported outcomes (PRO) of care, patient experience, and clinical outcomes? 
2) To answer the question: Is the relationship of PA&E activities and improved PROs stronger for more highly activated 
patients than for less activated patients at baseline? 
3) To assess the stability of the PA&E-outcome relationship over two measurement periods at the individual- and 
practice-level. 
4) To assess changes over time in patient-reported outcomes of care, patient experience, and clinical measures, as a 
function of changes over time in PA&E activities and patient activation. 
5) To explore key capabilities of practice teams that might account for their greater involvement in PA&E activities and 
the resulting impact on outcomes of care. 
6) To assess the extent to which patient and practice team feedback was used and can help explain patient outcome 
changes over time. 
 
CRITERION 4 – PATIENT CENTEREDNESS 
 
We will include patients and caregivers at the front end of the study, as well as gather ongoing input from them 
throughout the study. The vehicle for this will be patient advisory groups in each of the ACOs under study. We will either 
build on existing advisory groups at a site (where one exists) or ask sites to form new advisory groups to aid in finalizing 
study instruments, interpreting survey results, and providing feedback to practices on their PA&E activities. While we will 
encourage a participatory approach that includes patient input, because we are interested in pragmatic approaches, we will 
allow patient advisory structures to be tailored by each site. In order to understand those differences, we will monitor how 
frequently the advisory groups meet, what they prioritize and discuss, and the actions they take; and we will use meeting 
minutes and one-on-one interviews with patient advisory group members during site visits to assess their awareness and 
nature of involvement in PA&E initiatives. This will include learning about their experiences of their advisory roles, 
convenience and usefulness of patient advisory group meetings, perceived importance of involvement in the research 
project, including practice feedback on the research surveys and related issues. A central role of patient advisory groups is 
to review the study measures and proposed study design in the first few months of the project and then to meet 2-3 times 
annually to provide ongoing project input, to review round one measurement results, and to suggest how those results 
might inform the ACO practices’ ongoing PA&E improvement efforts. 
 
The primary purpose of the proposal is to directly address patient-centeredness of care by comparing practices with 
high involvement in PA&E activities versus those with low or no involvement in regard to PRO and relevant clinical 
measures.  
 
C. Study Design or Approach  
 
RESEARCH PLAN 
 
 
 
The logic model and conceptual framework figure on the following page (see Figure 1) provides the basis for the 
research design and analytic plan.  
 
 
We posit that practices that have more highly developed PA&E and better teamwork will more effectively activate 
patients with diabetes and/or CVD.  In turn, we posit that more highly activated patients will be more likely to achieve 
improvements on patient-reported outcomes, patient care experiences, and clinical measures over time compared to less 
activated patients.  We also will also examine the roles that patient advisory groups and primary care team use of 
performance feedback data play in influencing improvements in patient activation and outcomes of care over time. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
5
Figure 1. Logic Model 
 
 
 
We will engage patients and primary care practice teams within two accountable care organizations (ACOs). In each 
ACO, we will randomly select eight practices: four who score high on PA&E activities and four who score low. We will 
survey practice team members twice: once at baseline and one a year later. We will also conduct two site visits: one near 
the beginning of the study and one during the 13-18 month period of the study. 
 
We will randomly select four high and four low PA&E sites within each ACO using a broad definition of PA&E 
activities including: (1) patient care outreach in regard to disease prevention and health promotion; (2) changes in the 
clinician-patient relationship, particularly in the areas of communication, motivational interviewing, and patient 
involvement in treatment care plans; (3) shared decision-making; (4)  patient self-management of their condition(s); (5) 
end of life/advanced serious illness care patient engagement and family involvement; and (6) patient involvement in the 
overall design of care and in organization-wide efforts to improve the quality of care. See methods section for a 
description and Appendix I for an example of the instrument to be used for determining the “more highly involved” and 
“less highly involved” practices.  
 
We will randomly select 273 patients with diabetes and/or CVD from each of the 16 practices. Given the 16 practice 
sites, we will have data from 4,368 patients. We will follow them as cohorts over a two-year period. See Figure 2 below 
for selection design. Patients will complete the 13-item PAM measure, patient-reported outcomes (PROMIS), and 
experience of care measures between 7-12 months and again between 19-24 months. We will obtain measures of blood 
pressure, HbA1c and LDL-C at 12 months and 24 months from the EHR. We expect a 50% response rate to the first 
survey round and 65% response rate to the second survey round, with 3% undeliverable and 20% attrition over the two-
measurement period, leaving 1102 patients, or 69 patients per practice, available for analysis. The calculations provided 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
6
below for the PROMIS self-reported measures and for blood pressure provide evidence of sufficient statistical power to 
detect meaningful differences between practices on these key variables of interest. 
 
Figure 2. Selection and engagement of ACO practices, patients, and teams 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Study Sites  
 
Advocate Health Care (Advocate) and Advocate Physician Partners (APP) operate both a commercial and a Medicare 
ACO. In 2011, Advocate joined with its largest commercial insurance partner to create AdvocateCare, providing care to 
370,000 covered lives. In 2012, Advocate was chosen to participate in the Medicare Shared Savings Program (MSSP), 
becoming one of the largest Medicare ACOs in the country, serving an estimated 106,000 Medicare beneficiaries.  
 
HealthCare Partners ACO (HCP) was formed in 2010 as one of the early Dartmouth-Brookings demonstration 
ACOs46. They hold both CMS and private payer risk-based contracts collectively accountable for 50,000 lives. They serve 
high proportions of Medicaid patients in low-income communities who tend to face many social and financial barriers to 
self-management in care for diabetes and CVD. Approximately 55% of HCP’s Southern California patients are Latino. 
Table 2 compares key characteristics of the partnering ACOs. Both Advocate and HCP are highly committed to this study 
as indicated by their letters of support and related materials in this proposal. 
 
Table 2. Summary of key characteristics of the two participating ACOs 
 
Advocate ACO 
HealthCare Partners ACO 
ACO Established 
MSSP: July, 2012 
AdvocateCare (Commercial):2011   
MSSP: January, 2014 
Cigna (Commercial): April, 2013 
Anthem (Commercial): July 2013 
Location 
Illinois  
Senior ACO (MSSP): CA, FL, 
NV 
Commercial ACO: Greater Los 
Angeles, CA, NV (Cigna)  
Patient population 
MSSP: 106,000 
Commercial: 370,000 
Senior (MSSP): 56,689 
Commercial: 65,739 
ACO Contract Lives 
MSSP: 106,000 
Commercial: 370,000 
Senior (MSSP): 56,689 
Commercial: 65,739 
Two Accountable Care Organizations 
(Advocate and HealthCare Partners) 
Advocate 
HealthCare Partners 
8 Practices 
4 High PA&E Practices 
4 Low PA&E Practices 
273 Patients with 
diabetes/CVD per 
practice 
4 High PA&E Practices 
4 Low PA&E Practices 
273 Patients with 
diabetes/CVD per 
practice 
273 Patients with 
diabetes/CVD per 
practice 
273 Patients with 
diabetes/CVD per 
practice 
Practice team 
assessment 
instruments 
Practice team 
assessment 
instruments 
Practice team 
assessment 
instruments 
8 Practices 
Practice team 
assessment 
instruments 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
7
 CLINIC/PRACTICE 
SELECTION 
 
Advocate ACO and 
HealthCare Partners ACO 
have 150 and 69 primary 
care practice sites, 
respectively. Using the list 
of PA&E activity domains 
previously described (see 
Appendix I for a draft of 
the checklist instrument), 
we will randomly select 
four practices from each 
ACO that score in the top 
quartile of the PA&E 
activities and four practices 
from each ACO that score 
in the bottom quartile. This 
will maximize the 
difference between those 
practices with high 
involvement and those with 
little or no involvement.  
 
HEALTH CARE TEAM 
SELECTION 
 
We define the 
membership of primary 
care teams based on 
administrative designations 
of team membership. Based 
on our preliminary data, we 
expect a varying number of 
care teams up to a total of 
approximately 160 teams 
from the 16 practice sites. 
Within each of the practices, 
we will randomly select 
patients from teams with sufficient patients with diabetes and/or CVD diagnoses to equal the 273 patients with diabetes 
and/or CVD diagnoses. While the statistical analysis will be at the practice level, we will provide team-level scores to the 
practices for their use. 
 
PATIENT SELECTION  
 
To project the sample size requirements given 16 practices (8 high PA&E and 8 low PA&E), we estimated the design 
effect (DE) to account for the clustering of patients within practice sites. While PROMIS data from participating ACO 
patients are presently unavailable, recent studies of change over time on the PROMIS physical functioning measure 
informed our sample size estimates for the PROMIS measure47. Having been selected using item-response theory (IRT) 
methods and extensive testing, the PROMIS physical functioning measure is a precise measure that is sensitive to small 
changes in physical functioning over time. Given the published estimates and a range of design effect estimates (intraclass 
correlation (ICC)=0.01-0.04) to account for the clustering of patients within practices, a clinically meaningful change (e.g., 
Patient 
Race/Ethnicity 
57% Non-Latino White 
22% African American 
8% Latino 
5% Other/multi-racial 
3% Asian/Filipino 
1% American Indian/Alaska 
Native 
1% Middle East 
1% Eastern Indian 
2% Unknown/Declined 
• 
55% Latino 
• 
25% Non-Latino White 
• 
15% Asian/Pacific Islander 
• 
10% African-American 
Payer Mix 
• 
Commercial: 70% 
• 
Medicare Advantage: 5% 
• 
Advocate Employee: 5% 
• 
Medicare ACO: 20% 
• 
HCP commercial: 68% 
• 
HCP Medicaid: 13% 
• 
HCP Senior: 19% 
ACO Risk 
Arrangement 
• 
Shared savings (no risk) 
• 
Shared savings (no risk) 
Why Selected? 
• 
Named among the nation’s Top 
5 largest health systems based 
on quality by Truven Analytics,  
• 
Largest health system in Illinois 
and one of the largest health 
care providers in the Midwest. 
• 
Operates more than 250 sites of 
care, including more than 150 
primary care sites and 12 
hospitals that encompass 11 
acute care hospitals, the state’s 
largest integrated children’s 
network, five Level I trauma 
centers (the state’s highest 
designation in trauma care), 
two Level II trauma centers, 
one of the area’s largest home 
health care companies, and one 
of the region’s largest medical 
groups. 
• 
Advocate Health Care trains 
more primary care physicians 
and residents at its four 
teaching hospitals than any 
other health system in the state 
• 
Large, well-developed, well -
established integrated 
delivery system with group 
and IPA models 
representative of American 
health care (69 primary care 
sites).  
• 
30+yrs of integrated care and 
health delivery experience 
and refinement: 
directly/indirectly aligns with 
IHA’s Triple Aims. 
• 
Diverse patient populations 
with wide arrays of 
cultural/socioeconomic 
backgrounds and varying 
ranges of health status. 
• 
Innovative use of patient-
centric health /wellness 
programs coupled with 
technology. 
• 
Robust data warehouse. 
• 
Broad dissemination of best 
practices nationally. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
8
a 3-5 point change out of 100) on the PROMIS physical functioning measure can be detected with 50-65 patients per 
practice. Based on prior experience, we expect a minimum 50% response rate for completing the first patient survey and a 
65% response rate for the second survey. We expect a 3% rate for undeliverable addresses and a 20% drop-out rate 
between the two survey periods based on our previous research. Patient-level blood pressure data from adult diabetic 
patients at HCP ACO practices were used to calculate the design effect (ICC=0.01-0.02). Previous research suggests that 
population-level improvements in blood pressure on the order of 3-5 mm Hg are clinically meaningful48-50. Based on a 
two-sided difference of means with a power of 0.90, we estimate that an average of 140 patients per practice are needed to 
detect a 3 mm Hg difference in change in systolic blood pressure over time between the high (n=8) vs. low (n=8) PA&E 
practices. Based on these calculations, we will randomly select 273 patients per practice (n=4,368) with diabetes or CVD 
diagnoses from each of the practice sites and teams in the two ACOs for inclusion in the study. Given the assumptions of 
response rates and attrition noted above, this will yield a final total of 1102 patients with two time points for analysis of 
the PROMIS (69 patients per site). In addition we expect to receive retrospective clinical outcomes data from the EHR for 
most of the sampled patients (n=4,368). Thus, even with slightly lower than expected survey response rates, our study will 
be sufficiently powered to detect clinically meaningful differences between high and low PA&E practices on both patient-
reported (PROMIS) and clinical, e.g., blood pressure, outcome measures.  
 
MEASURES 
Measurement of Site PA&E Activities 
 
We will categorize the more than 150 Advocate ACO primary care practice sites and the 69 HCP ACO practice sites 
as high vs. low PA&E involvement based on a brief 40-item practice assessment of the six domains of PA&E activities 
(see Appendix I).  This list was developed in consultation with a National Advisory Committee to our study of ACO 
involvement in PA&E and the preliminary results of the survey (see page 2) supported by the Gordon and Betty Moore 
Foundation. This checklist instrument, including measures of the degree of implementation, will be completed by the 
person most knowledgeable about each practice’s PA&E activities. Based on past experience this will be the Chief 
Medical Officer or Quality Improvement leader. Each of the ACOs practices will be categorized into quartiles based on 
their overall PA&E score.  We will then sample practices for involvement from the top and bottom quartiles to finalize 
four highly involved PA&E practices and four low involved PA&E practices within each of the two ACOs for 
engagement in the research study.  
 
Patient Activation (PAM) 
 
We will use the 13-item PAM Measure (Hibbard et al, 2005 – see Appendix II)10 This short form has similar 
psychometric properties to the longer 22-item form with Rasch overall person reliability of 0.81 and .79 for patients with 
diabetes and .82 for patients with CVD. Patient activation has been associated with positive health behaviors such as 
aerobic exercise and receiving preventive cancer screenings, as well as more favorable emotional health.51-53 The summed 
response scores are then categorized into a 4-part ordinal variable, representing activation levels. The levels of activation 
are similar to the Transtheoretical Model, where individuals progress through stages before changing their behavior, such 
as quitting smoking.54  Individuals are thought to move through these stages of activation sequentially, although stages 
may change with time and stressful circumstances can lower patient activation.51, 55 An individual in the lowest level of 
activation is a passive participant in health care decisions. An individual in the second level of activation has the 
knowledge and confidence to take a more active role in their health care, but has not yet acted on it. In the third level of 
activation, the patient plays an active role in making health care decisions with their providers. The highest level of 
activation describes a patient who has the knowledge and confidence to take action about their own health care, even 
during times of stress55. We will assess the extent to which practices with high PA&E activities and high team functioning 
more effectively transition patients to higher levels of activation over time. 
 
Patient-Reported Outcome Measure – PROMIS Physical Functioning and Social Functioning 
 
The PROMIS Physical Functioning (Short Form 12a) and Social Functioning (Short Form 8a) will be used to measure 
patient reported outcomes of care (See Appendix II). We will also include the depression and anxiety measures (PHQ-4). 
The PROMIS instruments were developed through extensive psychometric testing, have been extensively studied and 
validated, with reliability coefficients in the range of 0.85-0.95 for the physical functioning measure47 and 0.85-0.94 for 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
9
the social functioning measure56. Previous research indicates that the estimated minimally important difference for the 
PROMIS physical functioning is about 0.20 (small effect size) of the baseline standard deviation57. 
 
Patient Reported Outcome Measure – PACIC-11 
 
The Patient Assessment of Chronic Illness Care scale (PACIC-11) developed and validated by Glasgow et al58 
(Medical Care, 2005; see Appendix II) will be used to assess the extent to which practices engaging in more PA&E 
activities and with higher team functioning will be able to effectively improve patients’ experiences of chronic illness care 
over time. The PACIC-11 has high Cronbach’s alpha internal consistency, reliability of 0.93, and reasonable test-retest 
reliability of 0.58. It also demonstrates convergent validity with the Hibbard PAM measure and several dimensions of the 
Ambulatory Care Experiences Survey (ACES), the precursor to the Clinician Group Consumer Assessment of Healthcare 
Providers and Systems (CG-CAHPS) survey (Rodriguez and Crane 2011)59. We will supplement the PACIC-11 with 
selected items form the Ambulatory Care Experiences Survey (ACES) focused on patients’ experiences of primary care 
teams (Rodriguez, Rogers et al., 2007)60. (See Appendix II) 
 
Clinical Measures 
 
Based on each site’s EHR, we will have de-identified data for blood pressure, lipids, and HbA1c to measure 
percentage of hypertensive patients with blood pressure controlled (<140/90 mm Hg), percentage of patients with diabetes 
with blood sugar controlled (HbA1c <8.0%), and percentage of patients with cardiovascular conditions or diabetes whose 
lipids are controlled (LDL-C <100 mg/dL). The patient survey sampling frame includes “established” patients of each 
primary care physician and team. The adult diabetic and CVD patients will need to have at least two primary care visits in 
the baseline year (2013) to improve the reliability of reports of patients’ experiences of care and to focus on an established 
patient population that has a high propensity to maintain clinical relationships and receive routine primary care over time.  
Since attribution of patients to practices and PCPs is a contentious issue in performance measurement in improvement61, 62, 
restricting the sample to patients with diabetes and/or CVD who have ongoing relationships will also aid in team 
acceptance of the feedback data. 
 
Patient Demographics 
 
We will collect patient demographic information as part of the patient surveys, including patient age, sex, race, 
ethnicity, self-rated English proficiency, preferred language, interpreter use, employment status, marital status, education, 
and military status.  We will supplement these survey data with information about patient age and sex from ACO clinical 
and sociodemographic data submitted to Berkeley as part of the project. 
 
TEAMWORK ASSESSMENT INSTRUMENTS 
 
Primary care team members from each of the 16 practices will complete the following instruments that we refer to as 
“teamwork assessment instruments.”  These are: (1) team culture, (2) team leadership, (3) relational coordination, and (4) 
perceived team effectiveness. These will be completed at baseline (0-3 months) and then again at 18 months. 
Implementation of the teamwork assessments will be overseen by a data coordinator at each ACO paid in part by the grant. 
 
An important component of the project is to provide primary care teams and patient advisory groups with feedback on 
their patient’s reports of physical functioning, social functioning, and experiences of primary care. As primary care 
practices embark on integrating new roles and responsibilities for non-physician clinicians and staff for improving patient 
activation and engagement, supportive practice environments that foster organizational learning and effective teamwork 
are essential.63, 64 Providing primary care teams with performance feedback on patient-reported outcomes, patient 
experience, and clinical quality of care measures is one important way of fostering a shared responsibility for patient care 
among interdisciplinary members65, 66 and supporting a measured approach to team performance improvement efforts.67 
When teams do not receive feedback on their collective work, members often encounter difficulties working together to 
modify existing workflows and procedures68, 69.   Consequently, team feedback on patient experience and PRO data and 
teamwork assessment can aid primary care teams in their process improvement efforts. The limited amount of 
performance feedback available to primary care teams on their patients’ experiences and patient-reported outcome 
measures70 can have important downstream consequences for patient care, including less patient-centered clinician-patient 
communication and self-management support.   

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
10
Team Composition Assessment 
 
There is not a “gold standard” for defining the membership and boundaries of health care teams.  In addition to 
administratively-defined teams (what managers perceive to be the boundaries and membership of primary care teams), we 
will elicit team membership and boundaries from frontline clinicians and staff. We will ask members to state who they 
consider to be members of their team. This will enable us to consider differences in team composition and membership in 
interpreting our teamwork survey results and related measures. We will collect data on team member age, sex, 
race/ethnicity, current position, years working in the practice and then on the team, and hours worked per week in the 
practice setting.   
 
Team Relational Coordination 
 
This is a seven-item measure developed by Gittel et al71 and has been used in several studies of surgical teams, 
primary care practices and other settings. The items covered include: 1) extent of shared goals and values among team 
members, 2) extent of shared information and knowledge, 3) extent of mutual respect, 4) accurate communication among 
members, 4) timely communication among members, 5) frequency of communication among members, and 6) extent of 
problem-solving communication among members. It has known reliability with internal consistency of 0.80. (See 
Appendix III)  
 
Team Participation 
 
Engaging lower status primary care team members in taking on new roles and responsibilities to support patient 
activation and engagement is critical to practice redesign. We will use a validated seven-item scale developed by 
Alexander, et al. (2005)72 with internal consistency reliability of 0.90 which has been found to be positively associated 
with better outcomes of care. (See Appendix III) 
 
Team Effectiveness 
 
We will use select survey items from the Organization and Management of Intensive Care Unit study (Shortell, 
1991)73 and from the Improving Chronic Illness Care (ICICE) study (Shortell, 2004)74 as well as the adapted Baldridge 
Quality Improvement Scale (Shortell, 2004)74. These instrument components have high reliability with Cronbach’s alpha 
of 0.76, 0.95, and 0.86, respectively. 
 
Group and Entrepreneurial Culture of Practices 
 
We will assess culture at the practice level using two domains of the Group Practice Organizational Culture 
instrument75, which focus on measuring the group and entrepreneurial culture orientations of teams. We posit that teams 
with more group and entrepreneurial oriented cultures will be associated with greater patient activation and engagement 
initiatives, patients that are more activated, and patients that will experience better outcomes of care. (see Appendix III) 
 
Leadership Facilitation 
 
We will use a seven-item subscale of the Organizational Readiness for Change (ORCA) to measure the extent to 
which team members believe that leadership and management supports them in their work to improve care for patients. It 
has an internal consistency reliability of 0.94 and has been used in previous studies assessing leadership facilitation of 
team and practice redesign76-78. (See Appendix III) 
 
Feedback and Site Visits 
 
We will work closely with the practice leaders and patient advisory groups in designing and developing feedback 
reports for the primary care teams.  Some approaches which will be considered include65, 66:  1) whether results are 
presented in a blinded or unblinded fashion, 2) how the information is delivered to the team, 3) facilitation and guidance 
for teams as they discuss their results, and 4) a system for disseminating best practices among teams within a practice site.  
We posit that practices with high teamwork assessments at baseline will be much more likely to undertake process 
improvements after receiving feedback on PROs and teamwork compared to practices with low teamwork assessments at 
baseline. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
11
 
The data collected as part of the stakeholder engagement process, patient assessment, and teamwork assessments will 
be supplemented by two site visits to each of the two ACOs at two points in time to gain an insight to the strategic goals 
of the ACOs in improving patient activation and engagement and fostering high functioning interdisciplinary primary care 
teams: (1) during the 0-6 month start-up phase of the study, and again (2) during the period of 18-24 months. The purpose 
of these site visits is to obtain a detailed, nuanced understanding of the barriers and facilitators to improving PA&E 
initiatives and activating patients; to help enrich our understanding of the quantitative analysis; and to develop guidelines 
for others to significantly advance their PA&E efforts. We will interview practice leaders, primary care team members, 
patient advisory group members, and patients to get their views on the above issues. The semi-structured interview 
instruments that we will use will be guided by the interview guides and survey currently being used in our preliminary 
study of ACO PA&E activities (Shortell, et al., 2014)9. For example, we will ask questions about the frequency of 
meetings, the topics and issues discussed, which aspects of the feedback reports they found most useful, the challenges 
and lessons learned, what actions were taken as a result, their satisfaction with their engagement in the study, and their 
suggestions for improvement. 
 
ANALYSIS SECTION 
 
Aim 1.To assess whether patients with diabetes and CVD diagnoses receiving care from practices with higher versus 
lower PA&E activities have better patient-reported outcomes of care, patient experience, and clinical outcomes. 
Analyses Addressing Aim 1.  At each of the two cross-sections for the patient and teamwork assessment surveys, we will 
clean the data, examine their distributions, and conduct psychometric analysis to assess the reliability and validity of the 
measures. After composite variables have been constructed and properties assessed, we will examine the cross-sectional 
relation of PA&E activities, teamwork assessment results, and patient survey results. Using a dichotomized variable 
derived from baseline responses to the PA&E Practice Site Assessment Survey (see Appendix I), we will categorize sites 
as “high” or “low” PA&E based on the response distributions in our data. We will then assess association of key measures 
in two stages. In the first equation, using linear regression models, we will examine the PAM as a function of high vs. low 
PA&E (site level), controlling for practice size, and for patient casemix adjustment variables previously used in research 
that has compared organizations on patient experience and clinical quality measures79, 80. Casemix factors to be considered 
include age, sex, educational attainment, and comorbid health conditions. We will then examine the patient reported 
outcome (PRO) measures, patient experience (PE) and the clinical measures as a function of the PAM, PA&E level (at 
site level),  practice size, and casemix using multilevel regression models to account for the clustering of patients within 
practices.  The simplified equations are: 
ε
+
+
+
=
Casemix
ze
PracticeSi
E
lowPA
Hi
PAM
&
/
    
ε
+
+
+
+
=
Casemix
ze
PracticeSi
PAM
E
lowPA
Hi
PE
&
/
   
ε
+
+
+
+
=
Casemix
ze
PracticeSi
PAM
E
lowPA
Hi
PRO
&
/
  
ε
+
+
+
+
=
Casemix
ze
PracticeSi
PAM
E
lowPA
Hi
es
ClinMeasur
&
/
 
 
 
Aim 2.  To assess whether the relation of PA&E activities and better PROs and clinical outcomes is stronger for more 
highly activated patients than less activated patients at baseline. 
Analyses Addressing Aim 2. To address the extent to which PA&E activities conducted by the practice are more 
influential in improving PROs and patient experience among patients with high PAM levels at baseline, we will extend 
the Aim 1 analyses. Specifically, we will use multilevel regression with site random effects to model changes in PROs  
and the clinical measures over time as a function of baseline PAM scores, practice size, patient casemix, PA&E, and then 
by interacting PA&E and the PAM scores.  The simplified equations are:    
ε
+
+
+
+
+
=
Casemix
PAM
E
PA
PAM
ze
PracticeSi
E
lowPA
Hi
PE
*
&
&
/
   
ε
+
+
+
+
+
=
Casemix
PAM
E
PA
PAM
ze
PracticeSi
E
lowPA
Hi
PRO
*
&
&
/
 
ε
+
+
+
+
+
=
Casemix
PAM
E
PA
PAM
ze
PracticeSi
E
lowPA
Hi
es
ClinMeasur
*
&
&
/
  
 
We will explore differences between the two extreme PAM categories (level 1=lowest activation, 4=highest activation) in 
addition to examining the continuous PAM measures, as we expect the biggest differences over time differences in PRO 
and patient experience between patients with highest and lowest baseline PAM scores. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
12
 
 
Aim 3.  To assess the stability of the PA&E-outcome relationship over two measurement periods at the individual and 
practice-level; 
 
Analyses addressing Aim 3. After both waves of data collection are completed, we will assess the consistency of the 
results at two levels- patient-level and practice-level. We will compare the regression coefficients for models at each wave 
to examine differences in associations of the independent variables with outcomes at each cross-section.  
 
 
Aim 4. To assess changes over time (T1 vs. T2) in patient-reported outcomes of care, patient experience and clinical 
measures, as a function of changes over time in PA&E activities, patient activation, controlling for patient casemix. 
 
Analyses Addressing Aim 4. We will integrate the patient survey results and the patient level results at the practice 
level. We will examine ‘over time’ effects using three approaches: 1) multilevel regression with interaction term as 
specified here, 2) stratified by low vs. high PA&E practices and looking at coefficient differences, 3) lagged regression 
(using T1 to predict T2) vs. change in T1 and T2 predicting T2), etc. The simplified equations below provide a multilevel 
framework for studying the impact of site PA&E activities on patient activation, patients’ experiences, and patient-
reported outcomes over time: 
ε
+
∆
+
∆
=
∆
PAM
E
PA
PRO
&
 
ε
+
∆
+
∆
=
∆
PAM
E
PA
PE
&
 
ε
+
∆
+
∆
=
∆
PAM
E
PA
es
ClinMeasur
&
 
 
 
Given the uncertainty in distribution and variation, we will examine the inclusion of other relevant predictors 
including fixed effects for casemix and practice size. 
 
 
Aim 5. To explore key capabilities of practice teams that might account for their greater involvement in PA&E 
activities to engage patients and the resulting impact on outcomes of care; 
 
Analyses Addressing Aim 5.  For these exploratory analyses, we will use a range of analytic techniques to assess the 
extent to which changes in team factors impact changes in PA&E activities, including multilevel regression modeling of 
the quantitative study data and comparative case study analyses of the multiple data sources.  For example, we will 
triangulate the quantitative results with site visit data about the changes to PA&E activities, including treating PA&E as a 
continuous variable (0 to 40 on our index of activities) occurring at the practices that might explain increases or declines 
in the outcome measures and primary care teamwork. 
ε
+
+
+
=
Casemix
ze
PracticeSi
Team
E
PA&
 
 
 
We will explore an overall teamwork composite, but will also examine differences in effect sizes using the various 
team subscales assessed in the survey (e.g., relational coordination, team participation, team culture). 
 
 
Aim 6. To explore the team feedback process to assess the extent to which feedback was used and could explain 
changes in team-related influences over time. 
 
Analyses Addressing Aim 6.  We will use comparative case analyses to compare data feedback processes and 
teamwork factors at the 16 practices as they relate to improvements in PROs and patient experience over time81, 82. 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
13
CRITERION 3 – TECHNICAL MERIT (Also see Research Plan and Analysis Plan under Section C. Study Design or 
Approach.) 
 
D. Project Milestones  
 
Table 3. Project Milestones and Timeline 
0-6 months 
7-12 months 
13-18 months 
19-24 months 
25-30 months 
31-36 months 
Wave 1 PA&E 
Survey  
Finalize enrollment 
of practice sites 
Site feedback on 
wave 1 PA&E survey 
 
Wave 2 PA&E 
Survey  
Site feedback on 
wave 2 PA&E 
survey 
 
Patient identification 
and selection 
Wave 1 Patient 
survey: 
- Patient Activation 
Measure (PAM) 
- Patient-reported 
outcomes (PROMIS) 
- Patient experience 
Retrospective clinical 
measures from EHR 
(BP, etc.) 
Feedback results of 
wave 1 Patient 
Survey and clinical 
measures to practices 
Wave 2 Patient 
survey: 
- Patient Activation 
Measure (PAM) 
- Patient-reported 
outcomes (PROMIS) 
- Patient experience 
Retrospective clinical 
measures from EHR 
(BP, etc.) 
Feedback results 
of wave 2 
Patient Survey 
and clinical 
measures to 
practices 
 
Completion of 
teamwork assessment 
instruments  
Feedback to practices 
results of teamwork 
assessments 
Completion of 
teamwork assessment 
instruments 
Feedback to practices 
results of teamwork 
assessments 
Feedback overall results to the ACO 
practices  
Site Visit #1  
 
Site Visit #2  
 
 
Formation of patient 
Advisory Groups 
Feedback of 
teamwork assessment 
to patient advisory 
groups 
Feedback of patient 
survey and clinical 
data to patient 
advisory groups 
Feedback of 
teamwork assessment 
to patient advisory 
groups 
Feedback of 
patient survey 
and clinical data 
to patient 
advisory groups 
 
 
Project deliverables 
on results to date 
-Integration of 
baseline patient 
sources (patient 
survey and clinical 
data) and practice 
data sources (PA&E 
and teamwork 
results) 
 
Analysis of Wave 1 
data 
Project deliverables 
on results as of 24 
months 
Integration of baseline patient 
sources (patient survey and clinical 
data) and practice data sources 
(PA&E and teamwork results) 
 
Analysis of Wave 2 data and over 
time changes, report writing, and 
development of papers for 
publication etc. 
 
E. Patient Population  
 
DIVERSE STUDY POPULATION 
 
As described in Table 2 above, both HCP and Advocate ACOs serve a diverse patient population. HCP’s patient 
population includes 55% Latino, 15% Asian/Pacific Islander, 10% African American, and 25% non-Latino white patients. 
Between its Medicare Shared Savings and commercial ACOs, its payer mix is 18.5% Medicare, 13% Medicaid, and 68% 
commercial. Advocate’s large patient population includes 22% African American, 8% Latino, 3% Asian, as well as 1% 
each American Indian/Alaska Native, Middle East, and Eastern Indian. Its payer mix is 70% commercial, 25% Medicare, 
and 4% employee. Thus the proposal provides an opportunity to examine the extent to which PA&E activities may need 
to be customized to meet the needs of diverse populations. 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
14
F. Research Team and Environment  
 
RESEARCH TEAM EXPERTISE 
 
Stephen Shortell, PhD, MPH, MBA (Principal Investigator) 
The PI Stephen M. Shortell, PhD, MPH, MBA, is the Blue Cross of California Distinguished Professor of Health Policy 
and Management and Director of the Center for Healthcare Organizational and Innovation research (CHOIR) at UC 
Berkeley’s School of Public Health. He and his colleagues have received many awards for their research and he is an 
elected member of the Institute of Medicine of the National Academy of Sciences. He has extensive experience in 
managing large-scale studies over time and he also has extensive knowledge of accountable care organizations (which 
serve as the context for the proposed study). As PI he will be responsible for all aspects of the proposal and will play a 
lead role in measuring PA&E activities, team assessment analysis and analysis related to the six aims. He will be the 
primary liaison to Advocate Health System and will participate in site visits. 
 
Hector Rodriguez, PhD, MPH (Co-Investigator) 
Hector Rodriguez, PhD, MPH, is Associate Professor of Health Policy and Management and Associate Director of the 
Center for Healthcare Organizational and Innovation Research (CHOIR) at the University of California, Berkeley, School 
of Public Health. His expertise is in organizational analysis and performance measurement, particularly as it relates to 
ambulatory care and health care disparities.  His current delivery systems research focuses on organizational interventions 
aimed at improving ambulatory care quality, including strategies to enhance primary care team effectiveness.  He has 
published extensively on the measurement of patient care experiences and the impact of delivery system interventions on 
patients’ experiences of care. As Co- investigator he will work closely with Dr. Shortell on all aspects of the study and 
will play a lead role in examining the team assessments and their impact on patient reported outcomes of care. He will 
also take the lead in examining differences by patient race, ethnicity, and language. He will be the primary liaison with 
HealthCare Partners and he will participate in site visits.   
 
Susan L. Ivey, MD, MHSA (Co-Investigator) 
Dr. Susan L. Ivey, MD, MHSA, is an Associate Professor in Community Health and Human Development at UC 
Berkeley’s School of Public Health. She is a board-certified family physician who still practices medicine and who has 
been conducting health services research for 16 years. She is currently the Director of Research at Health Research for 
Action, an affiliated research center at the School of Public Health, UC Berkeley. As Co-investigator she will play a lead 
role in working with the patient advisory groups and tracking their involvement throughout the course of the study. She 
will also be involved in the development of the feedback reports, will assist in analysis of the clinical outcome measures 
and, as needed, will participate in the site visits. 
 
Elliott Fisher, MD, MPH (Co-Investigator) 
Dr. Fisher is the director of The Dartmouth Institute for Health Policy and Clinical Evaluation, and a member of the 
Institute of Medicine of the National Academy of Sciences. Among his research interests is the development and testing 
of approaches to performance measurement and payment reform that can support improved quality of care. He is currently 
studying the development and evolution of ACOs working with Dr. Shortell and the Berkeley team. As Co-investigator he 
will assist in interpreting the results of the patient reported outcome measures and the clinical measures and will 
participate in at least one site visit.  
 
Glyn Elwyn, MD, PhD (Co-Investigator) 
Dr. Elwyn is a physician-researcher, Professor, and Senior Scientist at The Dartmouth Health Care Delivery Science 
Center and The Dartmouth Institute for Health Policy and Clinical Practice, Dartmouth College.  He leads 
interdisciplinary research examining the implementation of shared decision-making, user-centered design of patient 
decision support interventions, and the integration of these into routine health care. As Co-investigator he will participate 
in the development of the site visit interview instruments and in some of the site visits. He will play a lead role in analysis 
and interpretation of the site visit material.  

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
15
Patricia Ramsay, MPH (Project Director) 
Patricia Ramsay, MPH, is the Administrative Director of the Center for Healthcare Organizational and Innovation 
Research (CHOIR) at UC Berkeley’s School of Public Health. Ms. Ramsay has more than twenty years of experience in 
public health and health services research at the UC Berkeley, School of Public Health, and the Departments of 
Epidemiology and Biostatistics and Medicine at UC San Francisco. Her focus has been on study management, data quality 
control, and data analysis of CVD and related chronic illnesses. As project director, she will oversee all aspects of the day-
to-day operation of the study, working closely with Drs. Shortell and Rodriguez and the study team as a whole. She will 
be the primary contact with the survey research firm and will oversee all aspects of data collection, quality control, and 
preparation for analysis. She will participate in the various analyses related to the study aims and will, as needed, 
participate in site visits.   
 
RESEARCH ENVIRONMENT 
 
The University of California, Berkeley, is renowned worldwide for the distinction of its faculty and students, the 
scope of its research and publications, and the quality of its libraries. Its academic departments consistently rank among 
the top five in the country. The faculty, renowned for both teaching and scholarship, includes over 20 Nobel Laureates 
and a number of MacArthur Fellows. In recent fiscal years UC Berkeley has received over $600 million in research 
funding per year.  
 
Since its founding, the UC Berkeley School of Public Health has become one of the world’s preeminent public health 
schools dedicated to the health promotion and protection in human populations. The school’s research excellence is 
strengthened by between $40 - $80 million in extramural funding per year.  The Division of Health Policy and 
Management, the administrative home for Dr.’s Shortell and Rodriguez, is nationally recognized for its contributions to 
health services and health policy research. Five of seven core faculty hold endowed chairs. Its interdisciplinary PhD 
program in Health Services and Policy Analysis is ranked by the National Research Council (NRC) as among the top five 
in the country. Its faculty are frequently called upon for their expertise both in California and nationally where they have 
served on the President’s Council of Economic Advisers and task forces that worked with the Obama administration on 
the Affordable Care Act. 
 
Center for Healthcare Organizational and Innovation Research (CHOIR) 
 
Through practice-based research and dissemination of evidence, The Center for Healthcare Organizational and 
Innovation Research (CHOIR), directed by Dr. Stephen Shortell and Associate Director Hector Rodriguez, aspires to help 
make the U.S. healthcare system among the most responsive in the world, characterized by high quality, efficient, and 
patient-centered care. CHOIR currently has approximately $8 million of extramural funding across 8 concurrent projects. 
CHOIR faculty and staff are located together centrally on the UC Berkeley campus to facilitate communication.  
 
Health Research for Action (HRA) Center at UC Berkeley 
 
Health Research for Action is one of the UCB School of Public Health's initiatives to translate research findings into 
successful resources and programs for the public. The HRA Director of Research is Dr. Susan L. Ivey, MD, MHSA. The 
Center works with communities, foundations, and government agencies to reduce health and health care disparities and 
create more hopeful, empowered communities through research-driven health promotion services and programs. The 
Center’s resources and programs have reached over ten million people in the United States and around the world.  
 
Office 
The offices of the faculty of the SPH CHOIR are situated in University Hall on the UC Berkeley campus. Library and 
conference facilities are also available in University Hall, as well as other support facilities, including mail, fax, 
photocopying, and IT support. 
  
Other 
All the support services of a major University campus are available to this project: library, computer center, materials 
management, accounting, business and financial services, and human resources. 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
16
G. Research Engagement Plan 
 
CRITERION 5 - PATIENT AND STAKEHOLDER ENGAGEMENT 
 
There will be a patient advisory group at each study site that will actively participate in all aspects of the research 
including the development of instruments, assessment and review of findings throughout the study, making 
recommendations for improvement of practices’ PA&E activities, and assisting in dissemination and implementation of 
findings. In addition, at least two prominent patient engagement advocates (Sue Edgman-Levitan and Elizabeth Helms) 
are included on our overall study advisory committee. They will help to insure that the study design, and analysis stays 
focused on the patient-centered study aims. They will also assist in the dissemination of study findings. 
 
Importance of Patient-Centered Approaches:  
 
This proposal reflects the importance of examining patient-centered approaches for their potential to improve patient-
reported outcomes of care and related clinical measures. Patient Centered Outcomes Research (PCOR) recognizes the 
expertise of patients, families, and community health organizations as collaborators in research, and has emerged as an 
increasingly important approach in conducting health services research projects. We can leverage the expertise of patients, 
including underserved patient populations, to help guide development, implementation and feedback of our measures and 
findings; the patient advisory groups will address these issues. We believe this participatory approach will be key to 
successful study implementation and yield important, high quality patient-centered health services research findings.83, 84 
 
Plan for meaningful engagement of patients: 
 
As previously indicated we include patient advisory group input throughout the study.  Our goal is to have the patient 
advisory groups provide input in the first few months of the study with these groups to then meet 2-3 times annually and 
to undertake tasks that will include, but not be limited to: input into research design, instrumentation, review of 
preliminary results, interpretation of those results, and advising on dissemination of results.  
 
Further, this proposal has been informed by years of research involving health care teams and their patients and most 
recently from interviews with patients and caregivers on our related preliminary study of ACO Patient Activation and 
Engagement funded by the Gordon and Betty Moore Foundation. 
 
Experience in Patient and Stakeholder Engagement: 
 
Dr. Shortell is currently the PI on the study funded by the Moore Foundation to assess what ACOs are currently doing 
in regard to PA&E activities, and to identify some of the barriers and facilitators to further involvement. In addition to the 
survey which has been informed by patient input, the study team is making two site visits, interviewing patients and their 
caregivers to learn more about what appears to be working and not working in regard to PA&E. 
 
Dr. Ivey and her colleagues at Health Research for Action have many years of experience using participatory 
processes in research projects. Nearly every research project at HRA contains a community advisory board, and for those 
projects based within clinical sites, includes a group of patient advisors. Our work in this proposal will benefit from this 
experience.  
 
As noted above we will draw on our overall study advisory committee and the site-specific patient advisory groups. 
We will ask for direct input from the site-specific patient groups in the first few months of the study with a goal for these 
advisory groups to meet 2 to 3 times annually and to undertake tasks that will include, but not be limited to: input into 
research design, instrumentation, review of preliminary results, interpretation of those results, and advising on 
dissemination of results.  

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
17
 
DISSEMINATION AND IMPLEMENTATION POTENTIAL  
 
 
Describe the potential for disseminating and implementing the results of this research in other settings.  
 
There is great potential for disseminating and implementing the results of this research in other settings due to the 
established relationships which we have with many stakeholders in the field. This includes, but is not limited to: 1) All of 
the ACOs throughout the country through the ongoing work funded by the Commonwealth Fund; 2) through serving on 
the Advisory Board (Shortell) of the Kaiser-Permanente Institute for Health Policy that convenes roundtables and other 
dissemination forums for the results of such research; 3) in California, through our leadership of the Right Care Initiative 
focusing on reducing strokes and CVD in San Diego, Sacramento, and Los Angeles through greater patient and provider 
engagement; and 4) through our ties to the CDC Million Hearts campaign. All of this is in addition to the usual 
publication of study findings in journals, and at conferences and professional meetings. We look forward to working with 
PCORI on an overall dissemination and implementation strategy and determining the necessary resources involved.  
 
Describe possible barriers to disseminating and implementing the results of this research in other settings.  
 
There are two major barriers to disseminating and implementing the results of this research in other settings. The first is 
the challenge of adapting the study results to new and different contexts; the “we are different – it will never work here” 
challenge. The second is having sufficient time and resources to spread the study results to other settings. If the current 
proposal is funded and the results are promising, we look forward to submitting a subsequent dissemination proposal to be 
considered which will address our approach to these issues. 
 
 
 
 
 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
18
 
REPRODUCIBILITY AND TRANSPARENCY OF RESEARCH  
 
Describe the ability to reproduce potentially important findings from this research in other data sets 
and populations. 
 
As large scale delivery systems begin to form and as electronic health records become more prevalent it becomes more 
possible for organizations like Advocate Health Partners and HealthCare Partners to work with their patient populations to 
improve process and outcomes of care. The instruments from our study can be made available to other delivery systems to 
conduct such research in addition to what we learn about the role of patient activation and engagement in the research 
process. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
19
 
PROTECTION OF HUMAN SUBJECTS 
 
PROTECTION OF HUMAN SUBJECTS – University of California, Berkeley 
Risks to Human Subjects 
Human Subjects Involvement, Characteristics, and Design 
 
This research seeks to determine whether patients with diabetes or cardiovascular diseases (CVD) who receive care 
from provider practices affiliated with accountable care organizations (ACOs) with high involvement in patient activation 
and engagement (PA&E) experience better patient-reported outcomes, better experience of care, and improved clinical 
process measures compared with adult patients with diabetes or CVD who receive care from teams with low involvement 
in PA&E activities. These and related chronic conditions represent an estimated 75 percent of all U.S. health care 
expenditures.14 In addition, we will examine: (1) whether this effect is stronger for patients who score higher on the 
patient activation measure (PAM) than those who do not10; and (2) whether this effect is stronger for patients who 
improve their PAM scores over time than for those who do not. We will also assess the characteristics of practice teams 
associated with both high and low involvement in PA&E activities in regard to their degree of relational coordination, 
perceived team effectiveness, leadership, and team culture that they exhibit. Finally, we will examine the impact of 
feeding back the T1 patient outcome and teamwork assessment results to both caregivers and patient advisory groups on 
subsequent T2 measures of both patient outcomes and teamwork assessments.  
 
We will engage patients and primary care teams within two ACOs. The two selected to participate in this research are 
Advocate Health in Chicago, Illinois and HealthCare Partners in Torrance, California. Within each ACO, we will select 
four practices that are highly involved in PA&E and four that are less or not at all involved. We will make this 
determination based on a broad definition of PA&E activities that will include: (1) patient care outreach in regard to 
disease prevention and health promotion; (2) changes in the clinician-patient relationship, particularly in the areas of 
communication, motivational interviewing, and involvement in treatment care plans; (3) shared decision-making; (4)  
patient self-management of their condition(s); (5) end of life/advanced serious illness care patient engagement and family 
involvement; and (6) patient involvement in the overall design of care and in organization-wide efforts to improve the 
quality of care.  
 
From among those practices we will select 273 patients with diabetes and/or CVD from each of eight practices at each 
ACO – a total of four practices with high involvement in PA&E and four practices with low PA&E involvement. Given 
two ACOs, we will have data from 4,368 patients with these chronic conditions. We will follow them as cohorts over a 
two-year period.  We will obtain measures of blood pressure, HbA1c and LDL-C at 12 months and 24 months from the 
EHR. Patients selected for the study will be mailed a survey including the 13-item PAM measure at baseline (0-3 months). 
Patients will complete patient-reported outcomes (PROMIS) and experience of care measures between 7-12 months and 
again at between 19-24 months. Patients who do not complete and return the survey by mail will be contacted by phone 
(up to 8 attempts) and offered an opportunity to complete the questions by computer-assisted telephone interview (CATI). 
We will survey and follow-up with patients in both English and Spanish, according to their stated primary language 
preference. Patients will be offered a $20 incentive to complete the survey by mail or by CATI. The calculations provided 
below for the PROMIS self-reported measures and for blood pressure provide evidence of sufficient statistical power to 
detect meaningful differences between practices on these key variables of interest. 
 
To project the sample size requirements given 16 practices (8 high PA&E and 8 low PA&E), we estimated the design 
effect (DE) to account for the clustering of patients within practice sites. While PROMIS data from participating ACO 
patients are presently unavailable, recent studies of change over time on the PROMIS physical functioning measure 
informed our sample size estimates for the PROMIS measure47.  Having been selected using item-response theory (IRT) 
methods and extensive testing, the PROMIS physical functioning measure is a precise measure that is sensitive to small 
changes in physical functioning over time.  
 
Quantum Market Research, a survey firm in Oakland, CA, will be hired to administer the patient survey. Given the 
published estimates and a range of design effect estimates (intraclass correlation (ICC)=0.01-0.04) to account for the 
clustering of patients within practices, a clinically meaningful change (a 3-5 point change out of 100) on the PROMIS 
physical functioning measure can be detected with 50-65 patients per practice. Based on prior experience, we expect a 
minimum 50% response rate for completing the first patient survey and a 65% response rate for the second survey. We 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
20
expect a 3% undeliverable addresses and a 20% drop out rate between the two survey periods based on our previous 
research.  Patient-level blood pressure data from adult diabetic patients at HealthCare Partners practices were used to 
calculate the design effect (ICC=0.01-0.02). Previous research suggests that population-level improvements in blood 
pressure on the order of 3-5mm Hg are clinically meaningful.48, 49 Based on a two-sided difference of means with a power 
of 0.90, we estimate that an average of 140 patients per practice are needed to detect a 3 mm Hg difference in change in 
systolic blood pressure over time between the high (n=8) vs. low (n=8) PA&E practices. Based on these calculations, we 
will randomly select 273 patients per practice (n=4,368) with diabetes or CVD diagnoses from each of the practice sites 
and teams in the two ACOs for inclusion in the study. Given the assumptions of response rates and attrition noted above, 
this will yield a final total of 1102 patients with two time points for analysis of the PROMIS (69 patients per site). In 
addition we expect to receive retrospective clinical outcomes data for all sampled patients (n=4,368) from the electronic 
health record. Thus, even with slightly lower than expected survey response rates, our study will be sufficiently powered 
to detect clinically meaningful differences between high and low PA&E practices on both patient-reported (PROMIS) and 
clinical, e.g., blood pressure, outcome measures.  
 
Two site visits to each of the two ACOs will be conducted at two points in time to gain an insight to the strategic 
goals of the ACOs in improving patient activation and engagement and fostering high functioning interdisciplinary 
primary care teams: (1) during the 0-6 month start-up phase of the study, and again (2) during the period of 18-24 months.  
 
Sources of Materials  
 
Patients selected for the study will be mailed a survey including the 13-item PAM measure, patient-reported outcomes 
(PROMIS), and experience of care measures between 7-12 months and again between 19-24 months. Please see Appendix 
III for examples of questions to be included in the patient survey. We will obtain measures of blood pressure, HbA1c and 
LDL-C at 12 months and 24 months from the EHR at Advocate Health and HealthCare Partners. 
 
Advocate Health and HealthCare Partners will be responsible for linking patients’ EHR data to their study ID, and 
thus the data transmitted will not be identifiable to any researchers or staff at UC Berkeley. All such de-identified data 
will be housed at Berkeley on password-protected, secure servers accessible only by Principal Investigator Shortell and 
the Co-Investigators and staff under his direction.  
 
Quantum Market Research (QMR) will be responsible for contacting patients to complete the survey. They will 
receive identifiable patient information such as name, address, and phone number directly from Advocate and HealthCare 
Partners only for the purposes of administering the survey. 
 
Primary care team members from each of the 16 practices will complete the following instruments that we refer to as 
“teamwork assessment instruments.”  These are: (1) team culture, (2) team leadership, (3) relational coordination, and (4) 
perceived team effectiveness. These will be completed at baseline (0-3 months) and then again at 18 months. UC Berkeley 
will manage and administer the staff survey. 
 
Only the Investigators at Advocate and Health Care Partners site will have the key linking survey data to names of 
their own patients completing the survey. No investigators at UC Berkeley or Dartmouth or staff on the project besides the 
ACOs will have access to a key allowing linkage of collected data to identifiable patient information. 
 
Potential Risks  
  
There are no physical, financial, or legal risks to patients selected for participation in this project. The principal risks 
from this study concern the potential for loss of confidentiality.  
 
With respect to the site visits and staff surveys performed at the 16 ACO practices, the information collected through 
the survey and in-depth interviews with physicians and other ACO executives pertains to each organization’s professional 
activities only and does not pertain to any personal matters. The collection of these survey data do not pose any physical, 
financial, or legal risks to the ACO practice team members. 
 
Adequacy of Protection Against Risks 
Recruitment and Informed Consent 
 
Informed consent will be obtained from all selected patients before completion of the survey. A written consent letter 
will be included with the mailed surveys and phone consent will be specifically obtained at the beginning of each follow-
up CATI interview. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
21
Informed consent will also be obtained from the team members surveyed during administration by UC Berkeley 
investigators. 
 
Protections Against Risk  
  
To minimize the risk of loss of confidentiality, all information with the potential to identify study participants will 
remain housed at Advocate Health and Health Care Partners, respectively, only accessible to Advocate Health and 
HealthCare Partners staff. 
 
Potential Benefits of the Proposed Research to Human Subjects and Others 
 
Patients may benefit from the feedback of the time one findings which could enable some of the practice sites to make 
changes which directly benefit some patients subsequent to time 2. Furthermore, the knowledge gained from this study at 
the participating ACOs could potentially benefit all enrolled patients. As large scale delivery systems begin to form and as 
the electronic health record becomes more prevalent, it becomes more possible for organizations to work with their patient 
populations to improve process and outcomes of care. The instruments from our study can be made available to other 
delivery systems to conduct such research in addition to what we learn about the role of PA&E in the research process. 
 
Importance of the Knowledge to be Gained  
 
This research is important as relatively little is known about the impact of actively engaging patients on such 
outcomes of care. Further, there are major gaps in knowledge and understanding of what practice team characteristics are 
associated with achieving better outcomes from more fully engaged patients. Thus the research is directly relevant to 
practices working on PA&E as well as payers and policy makers interested in encouraging engagement. The risk of the 
loss of confidentiality combined with the planned procedures to minimize that risk are reasonable in relation to the 
importance of the knowledge that is reasonably expected to result from this study. 
 
INCLUSION OF WOMEN AND MINORITIES 
Detailed Study Enrollment Table 
 
INCLUSION OF CHILDREN 
Individuals under 18 will be excluded from the proposed research as the chronic adult conditions proposed for study for 
this project are not relevant to children. 
 
PROTECTION OF HUMAN SUBJECTS – Dartmouth College 
Risks to Human Subjects 
Human Subjects Involvement, Characteristics, and Design 
 
The investigators at Dartmouth College will participate in the design of the research and will participate in the site 
visits to Advocate Health care and HealthCare Partners to interview practice staff, as described above. 
Racial Categories 
Not Hispanic or Latino 
Hispanic or Latino 
TOTAL 
  
Female 
Male 
Female 
Male 
  
American Indian/ Alaska 
Native 
11 
11 
0 
0 
22 
Asian 
178 
171 
0 
0 
349 
Native Hawaiian or 
Other Pacific Islander 
11 
11 
0 
0 
22 
Black or African 
American 
356 
342 
0 
0 
699 
White 
913 
878 
702 
674 
3167 
More than One Race 
56 
54 
0 
0 
109 
TOTAL 
1526 
1466 
702 
674 
4368 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
22
 
Sources of Materials 
 
Dartmouth will not be participate in the selection of patients and will receive de-identified data only after the 
conclusion of data collection. No Dartmouth investigators or staff will have any access to identifiable patient data. 
Dartmouth will have also have access only to de-identified practice team staff assessment data. Dartmouth will participate 
in the site visit interviews. 
 
Potential Risks  
 
The information collected by Dartmouth investigators through the in-depth interviews with physicians and other ACO 
executives pertains to each organization’s professional activities only and does not pertain to any personal matters. Data 
collected will be maintained confidentially. The collection of these data do not pose any physical, financial, or legal risks 
to the ACO practice team members.  
 
Recruitment and Informed Consent  
 
Consent will be obtained from all ACO practice staff prior to beginning site visit interviews. 
 
Protections Against Risk 
 
The collection of the site visit interview data does not pose any physical, financial, or legal risks to the ACO practice 
team members. Nevertheless names of interviewees will be kept confidential and will not be made public. 
 
Potential Benefits of the Proposed Research to Human Subjects and Others 
 
Data gained and summarized in feedback reports from the site visits may benefit Advocate Health Care and 
HealthCare Partners by enable some of the practice sites to make changes which directly the delivery of care at these 
organizations. 
 
Importance of the Knowledge to be Gained  
 
Please see above under Protection of Human Subjects – UC Berkeley. 
 
INCLUSION OF WOMEN AND MINORITIES 
 
Please see above under Protection of Human Subjects – UC Berkeley. 
 
INCLUSION OF CHILDREN 
 
Please see above under Protection of Human Subjects – UC Berkeley. 
 
PROTECTION OF HUMAN SUBJECTS – Advocate Health Care and HealthCare Partners 
Risks to Human Subjects 
Human Subjects Involvement, Characteristics, and Design 
 
Protection of human subjects considerations are identical for the two ACOs participating in this project. Advocate 
Health Care and HealthCare Partners will each securely transmit names, addresses, and contact information for adult 
patients with diabetes and/or cardiovascular diseases with at least 2 visits in the baseline year (2013) from each of eight 
primary care practices affiliated with our Accountable Care Organization (ACO) to the patient survey vendor selected for 
the project, Quantum Market Research. The four primary managerial and supervisory staff members at the survey firm 
have all completed the Collaborative Institutional Training Initiative (CITI) modules relating to Human Subjects 
Protection training, as have a number of other QMR staff, including interviewers. Although all QMR employees have 
followed strict procedures with regard to human subjects throughout their careers, this formalizes the training. 
 
The two ACOs will also securely transmit a de-identified patient-level data to the UC Berkeley research team in 
Months 7 and 19 covering 2014 and 2015 period, respectively.  The dataset will include adult patients with diabetes, 
cardiovascular disease, and/or asthma with at least 2 visits in the baseline year (2014) and the data elements include select 
sociodemographic information and clinical care process and outcome data using Healthcare Effectiveness Data 
Information Set (HEDIS) definitions for measures of chronic illness care.  

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
23
 
For analysis of all patient-level data at UC Berkeley, we will create unique subject identifiers that do not include date 
of birth, social security number, or medical record number. The ACOs will be responsible for linking each patient’s EHR 
data to their study ID, and thus the data transmitted will not be identifiable to any researchers or staff at UC Berkeley. All 
such de-identified data will be housed at Berkeley on password-protected, secure servers accessible only by Principal 
Investigator Shortell and the Co-Investigators and staff under his direction.  
 
Advocate Health Care and HealthCare Partners will also provide the UC Berkeley research team with the names, 
emails, and mail contact information for all primary care clinicians and staff of the eight practices so that a survey of team 
culture, leadership, relational coordination, and perceived team effectiveness can be conducted at two points in time 
(between months 0-6 and between months 13-18).  
 
These two ACOs will also work with the UC Berkeley team to coordinate two site visits lasting two to three days for 
the research team to interview key people, some patients, and to observe meetings. Finally, the ACOs will assist with 
recruiting five patient representatives from the target population to serve on an advisory board that will provide 
recommendations and feedback for the ACO practices to consider as they refine and improve their PA&E efforts. 
   
Sources of Materials  
 
Advocate Health and HealthCare Partners will be responsible for linking patients’ EHR data to their study ID, and 
thus the data transmitted will not be identifiable to any researchers or staff at UC Berkeley. 
 
As described above, Quantum Market Research (QMR) will receive identifiable patient information such as name, 
address, and phone number directly from Advocate and HealthCare Partners only for the purposes of administering the 
survey. 
 
Only the collaborating investigators and data coordinator located at each the Advocate and Health Care Partners sites 
will have the key linking survey data to names of their own patients completing the survey.  
 
Potential Risks  
  
As described above, there are no physical, financial, or legal risks to patients selected for participation in this project. 
The principal risks from this study concern the potential for loss of confidentiality.  
 
Adequacy of Protection Against Risks 
Recruitment and Informed Consent 
 
Informed consent will be obtained from all selected patients before completion of the survey. As described above, a 
written consent letter will be included with the mailed surveys and phone consent will be specifically obtained at the 
beginning of each follow-up CATI interview. 
. 
Protections Against Risk  
  
All information with the potential to identify study participants will remain housed at Advocate Health and Health 
Care Partners, respectively, only accessible to Advocate Health and HealthCare Partners staff. 
 
Potential Benefits of the Proposed Research to Human Subjects and Others 
 
Please see above under Protection of Human Subjects – UC Berkeley. 
 
Importance of the Knowledge to be Gained  
 
Please see above under Protection of Human Subjects – UC Berkeley. 
 
INCLUSION OF WOMEN AND MINORITIES 
 
Please see the detailed enrollment table above under Protection of Human Subjects – UC Berkeley.  
 
INCLUSION OF CHILDREN 
 
Individuals under 18 will be excluded from the proposed research as the chronic adult conditions proposed for study 
for this project are not relevant to children. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
24
 
REFERENCES CITED 
 
 
1. 
Lewis VA, Colla CH, Carluzzo KL, Kler SE, Fisher ES. Accountable Care Organizations in the United States: 
Market and Demographic Factors Associated with Formation. Health services research. 2013;48(6pt1):1840-58. 
2. 
Muhlestein D. Continued growth of public and private accountable care organizations. Health Affairs blog. 2013. 
3. 
Kish L. The Blockbuster Drug of the Century: An Engaged Patient. 2012 [updated 28 Aug; cited 2013 12 Dec]; 
Available from: http://www.hl7standards.com/blog/2012/08/28/drug-of-the-century/. 
4. 
Shortell S, Wu F, Colla C, Fisher ES, Lewis VA. Mapping the Unicorn: A Taxonomy of Accountable Care 
Organizations. 2014;Submitted. 
5. 
Shortell S, McClellan SR, Ramsay P, Casalino L, Ryan A, Copeland K. Physician practice participation in 
accountable care organizations: the emergence of the unicorn Submitted. 2014. 
6. 
Rodriguez HP, Giannitrapani KF, Stockdale SE, Hamilton A, Yano EM, Rubenstein LV. Teamlet structure and 
early experiences of medical home implementation for Veterans. Journal of General Internal Medicine. 2013;In press. 
7. 
Rodriguez HP, Anastario MP, Frankel RM, Odigie EG, Rogers WH, von Glahn T, et al. Can teaching agenda-
setting skills to physicians improve clinical interaction quality? A controlled intervention. BMC medical education. 
2008;8:3. Epub 2008/01/16. 
8. 
Rodriguez HP, von Glahn T, Rogers WH, Safran DG. Organizational and market influences on physician 
performance on patient experience measures. Health services research. 2009;44(3):880-901. Epub 2009/08/14. 
9. 
Shortell S, Neuhauser L, Bibi S, Sehgal N, Ramsay P. Assessing the business case for Accountable Care 
Organization involvement in patient activation and engagement. The Gordon and Betty Moore Foundation, 2014. 
10. 
Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and testing of a short form of the patient activation 
measure. Health services research. 2005;40(6p1):1918-30. 
11. 
Harris MI, Klein R, Cowie CC, Rowland M, Byrd-Holt DD. Is the risk of diabetic retinopathy greater in non-
Hispanic blacks and Mexican Americans than in non-Hispanic whites with type 2 diabetes? A U.S. population study. 
Diabetes Care. 1998;21(8):1230-5. 
12. 
Karter AJ, Ferrara A, Liu JY, Moffet HH, Ackerson LM, Selby JV. Ethnic disparities in diabetic complications in 
an insured population. Jama. 2002;287(19):2519-27. 
13. 
Hogan P, Dall T, Nikolov P. Economic costs of diabetes in the US in 2002. Diabetes Care. 2003;26(3):917-32. 
Epub 2003/03/01. 
14. 
Anderson G, Horvath J. The growing burden of chronic disease in America. Public health reports. 
2004;119(3):263-70. 
15. 
Ward BW, Schiller JS. Prevalence of Multiple Chronic Conditions Among US Adults: Estimates From the 
National Health Interview Survey, 2010. Preventing Chronic Disease. 2013;10:E65. 
16. 
Frosch DL, Rincon D, Ochoa S, Mangione CM. Activating seniors to improve chronic disease care: results from a 
pilot intervention study. J Am Geriatr Soc. 2010;58(8):1496-503. Epub 2010/07/29. 
17. 
Centers for Disease Control and Prevention. 2011 National Diabetes Fact Sheet. 2013 [updated 25 Oct 2013; cited 
2013 04 Dec]; Available from: http://www.cdc.gov/diabetes/pubs/estimates11.htm#diagnosed. 
18. 
American Diabetes Association. Data from the 2011 National Diabetes Fact Sheet (released Jan. 26, 2011). 2013 
[cited 2013 04 Dec]; Available from: http://www.diabetes.org/diabetes-basics/diabetes-statistics/. 
19. 
Centers for Disease Control and Prevention. Heart Disease Facts. 2013 [updated 28 Aug 2013; cited 2013 04 
Dec]; Available from: http://www.cdc.gov/heartdisease/facts.htm. 
20. 
Murphy S, Xu J, Kochanek K. Deaths: Final Data for 2010. Hyattsville, MD: 2013. 
21. 
Go AS, Mozaffarian D, Roger VL, Benjamin EJ, Berry JD, Borden WB, et al. Heart disease and stroke 
statistics—2013 update a report from the American Heart Association. Circulation. 2013;127(1):e6-e245. 
22. 
Heidenreich PA, Trogdon JG, Khavjou OA, Butler J, Dracup K, Ezekowitz MD, et al. Forecasting the future of 
cardiovascular disease in the United States a policy statement from the American heart association. Circulation. 
2011;123(8):933-44. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
25
23. 
Wong MD, Tagawa T, Hsieh HJ, Shapiro MF, Boscardin WJ, Ettner SL. Differences in cause-specific mortality 
between Latino and white adults. Med Care. 2005;43(10):1058-62. 
24. 
Nwasuruba C, Osuagwu C, Bae S, Singh KP, Egede LE. Racial differences in diabetes self-management and 
quality of care in Texas. J Diabetes Complications. 2009;23(2):112-8. 
25. 
Mainous AG, Diaz VA, Koopman RJ, Everett CJ. Quality of care for Hispanic adults with diabetes. Family 
Medicine. 2007;39(5):351-6. 
26. 
Harris MI, Eastman RC, Cowie CC, Flegal KM, Eberhardt MS. Racial and ethnic differences in glycemic control 
of adults with type 2 diabetes. Diabetes Care. 1999;22(3):403-8. 
27. 
Brown AF, Gerzoff RB, Karter AJ, Gregg E, Safford M, Waitzfelder B, et al. Health behaviors and quality of care 
among Latinos with diabetes in managed care. Am J Public Health. 2003;93(10):1694-8. Epub 2003/10/10. 
28. 
Heisler M, Faul JD, Hayward RA, Langa KM, Blaum C, Weir D. Mechanisms for racial and ethnic disparities in 
glycemic control in middle-aged and older Americans in the health and retirement study. Arch Intern Med. 
2007;167(17):1853-60. 
29. 
Saydah S, Cowie C, Eberhardt MS, De Rekeneire N, Narayan KM. Race and ethnic differences in glycemic 
control among adults with diagnosed diabetes in the United States. Ethn Dis. 2007;17(3):529-35. 
30. 
Rosal MC, Olendzki B, Reed GW, Gumieniak O, Scavron J, Ockene I. Diabetes self-management among low-
income Spanish-speaking patients: a pilot study. Ann Behav Med. 2005;29(3):225-35. Epub 2005/06/11. 
31. 
Gary TL, Batts-Turner M, Yeh HC, Hill-Briggs F, Bone LR, Wang NY, et al. The effects of a nurse case manager 
and a community health worker team on diabetic control, emergency department visits, and hospitalizations among urban 
African Americans with type 2 diabetes mellitus: a randomized controlled trial. Arch Intern Med. 2009;169(19):1788-94. 
Epub 2009/10/28. 
32. 
Hill-Briggs F, Batts-Turner M, Gary TL, Brancati FL, Hill M, Levine DM, et al. Training community health 
workers as diabetes educators for urban african americans: value added using participatory methods. Prog Community 
Health Partnersh. 2007;1(2):185-94. Epub 2007/01/01. 
33. 
Ivey SL, Tseng W, Kurtovich E, Weir RC, Liu J, Song H, et al. Evaluating a culturally and linguistically 
competent health coach intervention for Chinese-American patients with diabetes. Diabetes Spectrum. 2012;25(2):93-102. 
34. 
Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the Chronic Care Model in the new millennium. 
Health Aff (Millwood). 2009;28(1):75-85. Epub 2009/01/07. 
35. 
Lin GA, Halley M, Rendle KA, Tietbohl C, May SG, Trujillo L, et al. An Effort To Spread Decision Aids In Five 
California Primary Care Practices Yielded Low Distribution, Highlighting Hurdles. Health Affairs. 2013;32(2):311-20. 
36. 
Hibbard JH, Greene J, Tusler M. Improving the outcomes of disease management by tailoring care to the patient's 
level of activation. Am J Manag Care. 2009;15(6):353-60. 
37. 
Hibbard JH, Mahoney ER, Stock R, Tusler M. Do Increases in Patient Activation Result in Improved Self‐
Management Behaviors? Health services research. 2007;42(4):1443-63. 
38. 
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): 
conceptualizing and measuring activation in patients and consumers. Health services research. 2004;39(4):1005-26. 
39. 
Mosen DM, Schmittdiel J, Hibbard J, Sobel D, Remmers C, Bellows J. Is patient activation associated with 
outcomes of care for adults with chronic conditions? The Journal of ambulatory care management. 2007;30(1):21-9. 
40. 
Coulter A. Patient Engagement—What Works? The Journal of Ambulatory Care Management. 2012;35(2):80-9. 
41. 
Frosch DL, Rincon D, Ochoa S, Mangione CM. Activating seniors to improve chronic disease care: results from a 
pilot intervention study. Journal of the American Geriatrics Society. 2010;58(8):1496-503. 
42. 
Hibbard JH. Using systematic measurement to target consumer activation strategies. Medical Care Research and 
Review. 2009;66(1 suppl):9S-27S. 
43. 
California Department of Managed Health Care: Right Care Initiative. Health Plan and Medical Group 
Performance Data (NCQA HEDIS Scores for Right Care Metrics). 2012 [cited 2013 12 Dec]; Available from: 
http://www.dmhc.ca.gov/healthplans/gen/gen_rci_perfdata.aspx. 
44. 
Tinetti ME, McAvay GJ, Fried TR, Allore HG, Salmon JC, Foody JM, et al. Health outcome priorities among 
competing cardiovascular, fall injury, and medication-related symptom outcomes. Journal of the American Geriatrics 
Society. 2008;56(8):1409-16. Epub 2008/07/30. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
26
45. 
Belcher VN, Fried TR, Agostini JV, Tinetti ME. Views of older adults on patient participation in medication-
related decision making. J Gen Intern Med. 2006;21(4):298-303. Epub 2006/05/12. 
46. 
Larson BK, Van Citters AD, Kreindler SA, Carluzzo KL, Gbemudu JN, Wu FM, et al. Insights from 
transformations under way at four Brookings-Dartmouth accountable care organization pilot sites. Health Aff. 
2012;31(11):2395-406. 
47. 
Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The Patient-Reported Outcomes Measurement 
Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 
2005-2008. J Clin Epidemiol. 2010;63(11):1179-94. Epub 2010/08/06. 
48. 
Law M, Wald N, Morris J. Lowering blood pressure to prevent myocardial infarction and stroke: a new 
preventive strategy. Health Technol Assess. 2003;7(31):1-94. 
49. 
Rashid P, Leonardi-Bee J, Bath P. Blood pressure reduction and secondary prevention of stroke and other 
vascular events: a systematic review. Stroke. 2003;34(11):2741-8. 
50. 
Vartiainen E, Sarti C, Tuomilehto J, Kuulasmaa K. Do changes in cardiovascular risk factors explain changes in 
mortality from stroke in Finland? BMJ. 1995;310(6984):901-4. 
51. 
Chubak J, Anderson ML, Saunders KW, Hubbard RA, Tuzzio L, Liss DT, et al. Predictors of 1-Year Change in 
Patient Activation in Older Adults with Diabetes Mellitus and Heart Disease. Journal of the American Geriatrics Society. 
2012;60(7):1316-21. 
52. 
Munson GW, Wallston KA, Dittus RS, Speroff T, Roumie CL. Activation and Perceived Expectancies: 
Correlations with Health Outcomes Among Veterans with Inflammatory Bowel Disease. Journal of General Internal 
Medicine. 2009;24(7):809-15. 
53. 
Greene J, Hibbard JH. Why Does Patient Activation Matter? An Examination of the Relationships Between 
Patient Activation and Health-Related Outcomes. Journal of General Internal Medicine. 2011;27(5):520-6. 
54. 
Johnson SS, Paiva AL, Cummins CO, Johnson JL, Dyment SJ, Wright JA, et al. Transtheoretical Model-based 
multiple behavior intervention for weight management: Effectiveness on a population basis. Preventive Medicine. 
2008;46(3):238-46. 
55. 
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): 
Conceptualizing and Measuring Activation in Patients and Consumers. Health Services Research. 2004;39(4). 
56. 
Farin E, Ullrich A, Hauer J. Participation and social functioning in patients with fibromyalgia: development and 
testing of a new questionnaire. Health and quality of life outcomes. 2013;11:135. Epub 2013/08/07. 
57. 
Hays RD, Spritzer KL, Fries JF, Krishnan E. Responsiveness and minimally important difference for the Patient-
Reported Outcomes Measurement Information System (PROMIS) 20-item physical functioning short form in a 
prospective observational study of rheumatoid arthritis. Annals of the rheumatic diseases. 2013. Epub 2013/10/08. 
58. 
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the 
patient assessment of chronic illness care (PACIC). Medical care. 2005;43(5):436-44. 
59. 
Rodriguez HP, Crane PK. Examining multiple sources of differential item functioning on the Clinician & Group 
CAHPS(R) survey. Health Serv Res. 2011;46(6pt1):1778-802. Epub 2011/11/19. 
60. 
Rodriguez HP, Rogers WH, Marshall RE, Safran DG. Multidisciplinary primary care teams: effects on the quality 
of clinician-patient interactions and organizational features of care. Medical care. 2007;45(1):19-27. Epub 2007/02/07. 
61. 
Selby JV, Schmittdiel JA, Lee J, Fung V, Thomas S, Smider N, et al. Meaningful variation in performance: what 
does variation in quality tell us about improving quality? Medical care. 2010;48(2):133-9. Epub 2010/01/09. 
62. 
Fung V, Schmittdiel JA, Fireman B, Meer A, Thomas S, Smider N, et al. Meaningful variation in performance: a 
systematic literature review. Medical care. 2010;48(2):140-8. Epub 2010/01/09. 
63. 
Nutting PA, Crabtree BF, Miller WL, Stange KC, Stewart E, Jaen C. Transforming physician practices to patient-
centered medical homes: lessons from the national demonstration project. Health Aff (Millwood). 2011;30(3):439-45. 
Epub 2011/03/09. 
64. 
Gabbay RA, Bailit MH, Mauger DT, Wagner EH, Siminerio L. Multipayer patient-centered medical home 
implementation guided by the chronic care model. Jt Comm J Qual Patient Saf. 2011;37(6):265-73. Epub 2011/06/29. 
65. 
Hysong SJ, Teal CR, Khan MJ, Haidet P. Improving quality of care through improved audit and feedback. 
Implementation science : IS. 2012;7:45. Epub 2012/05/23. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
27
66. 
Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Medical care. 
2009;47(3):356-63. Epub 2009/02/06. 
67. 
Rubenstein LV, Chaney EF, Ober S, Felker B, Sherman SE, Lanto A, et al. Using evidence-based quality 
improvement methods for translating depression collaborative care research into practice. Fam Syst Health. 
2010;28(2):91-113. Epub 2010/08/11. 
68. 
Vinokur-Kaplan D. Treatment teams that work (and those that don't): An application of Hackman's group 
effectiveness model to interdisciplinary teams in psychiatric hospitals. Journal of Applied Behavioral Science. 
1995;31(3):303-27. 
69. 
Hackman JR. Leading Teams: Setting the Stage for Great Performances. Boston, MA: Harvard Business School 
Press  2002. 
70. 
O'Toole TP, Cabral R, Blumen JM, Blake DA. Building high functioning clinical teams through quality 
improvement initiatives. Qual Prim Care. 2011;19(1):13-22. Epub 2011/06/28. 
71. 
Gittell JH. Coordinating mechanisms in care provider groups: Relational coordination as a mediator and input 
uncertainty as a moderator of performance effects. Management Science. 2002;48(11):1408-26. 
72. 
Alexander JA, Lichtenstein R, Jinnett K, Wells R, Zazzali J, Liu D. Cross-functional team processes and patient 
functional improvement. Health Serv Res. 2005;40(5 Pt 1):1335-55. 
73. 
Shortell SM, Rousseau DM, Gillies RR, Devers KJ, Simons TL. Organizational assessment in intensive care units 
(ICUs): construct development, reliability, and validity of the ICU nurse-physician questionnaire. Medical care. 
1991;29(8):709-27. 
74. 
Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu S-Y, Mendel P, et al. The role of perceived team 
effectiveness in improving chronic illness care. Medical care. 2004;42(11):1040-8. 
75. 
Kralewski JE, Wingert TD, Barbouche MH. Assessing the culture of medical group practices. Medical care. 
1996;34(5):377-88. 
76. 
Helfrich CD, Blevins D, Smith JL, Kelly PA, Hogan TP, Hagedorn H, et al. Predicting implementation from 
organizational readiness for change: a study protocol. Implementation science : IS. 2011;6:76. Epub 2011/07/23. 
77. 
Hagedorn HJ, Heideman PW. The relationship between baseline Organizational Readiness to Change Assessment 
subscale scores and implementation of hepatitis prevention services in substance use disorders treatment clinics: a case 
study. Implementation science : IS. 2010;5:46. Epub 2010/06/16. 
78. 
Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development 
of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 
2009;4:38. Epub 2009/07/15. 
79. 
Elliott MN, Zaslavsky AM, Goldstein E, Lehrman W, Hambarsoomians K, Beckett MK, et al. Effects of survey 
mode, patient mix, and nonresponse on CAHPS hospital survey scores. Health Serv Res. 2009;44(2 Pt 1):501-18. Epub 
2009/03/26. 
80. 
Zaslavsky AM, Hochheimer JN, Schneider EC, Cleary PD, Seidman JJ, McGlynn EA, et al. Impact of 
sociodemographic case mix on the HEDIS measures of health plan quality. Med Care. 2000;38(10):981-92. Epub 
2000/10/06. 
81. 
Yin RK. Case study research: Design and methods. 3rd ed. Newbury Park: Sage Publications; 2002. 
82. 
Miles M, Huberman A. An Expanded Sourcebook. Qualitative Data Analysis. Second ed. Thousand Oaks, CA: 
Sage; 1994. 
83. 
Concannon T, Meissner P, Grunbaum J, McElwee N, Guise J-M, Santa J, et al. A New Taxonomy for Stakeholder 
Engagement in Patient-Centered Outcomes Research. Journal of general internal medicine. 2012;27(8):985-91. 
84. 
Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: A 
framework for understanding the elements and developing interventions and policies. Health Affairs. 2013;32(2):223-31. 
85. 
Fisher ES, Shortell SM, Kreindler SA, Van Citters AD, Larson BK. A framework for evaluating the formation, 
implementation, and performance of accountable care organizations. Health Aff. 2012;31(11):2368-78. 
86. 
Rich J, Howe J, Larson L, Chuang C. Implementing Interactive Voice Recognition Technology to Activate 
Vulnerable Patients. Journal of the International Society for Telemedicine and eHealth. 2013;1(1):Pages 3-11. 
87. 
Davis SE, Pigford M, Rich J. Improving Informal Caregiver Engagement with a Patient Web Portal. electronic 
Journal of Health Informatics. 2012;7(1):e10. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
28
88. 
Sequist TD, von Glahn T, Li A, Rogers WH, Safran DG. Statewide evaluation of measuring physician delivery of 
self-management support in chronic disease care. J Gen Intern Med. 2009;24(8):939-45. 
89. 
Singer SJ, Burgers J, Friedberg M, Rosenthal MB, Leape L, Schneider E. Defining and measuring integrated 
patient care: promoting the next frontier in health care delivery. Med Care Res Rev. 2011;68(1):112-27. 
90. 
Rodriguez HP, Rogers WH, Marshall RE, Safran DG. Multidisciplinary primary care teams: effects on the quality 
of clinician-patient interactions and organizational features of care. Medical care. 2007;45(1):19-27. 
91. 
Farin E, Ullrich A, Hauer J. Participation and social functioning in patients with fibromyalgia: development and 
testing of a new questionnaire. Health and quality of life outcomes. 2013;11(1):135. 
92. 
Löwe B, Wahl I, Rose M, Spitzer C, Glaesmer H, Wingenfeld K, et al. A 4-item measure of depression and 
anxiety: validation and standardization of the Patient Health Questionnaire-4 (PHQ-4) in the general population. Journal 
of affective disorders. 2010;122(1):86-95. 
93. 
Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The Patient-Reported Outcomes Measurement 
Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 
2005–2008. Journal of clinical epidemiology. 2010;63(11):1179-94. 
94. 
TeamSTEPPS: National Implementation. Agency for Research Healthcare and Quality;  [cited 2014 January 6, 
2014]; http://teamstepps.ahrq.gov/about-2cl_3.htm]. 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
29
 
CONSORTIUM CONTRACTUAL ARRANGEMENTS 
 
Describe the proposed research projects that will be performed by subcontracted organizations. 
Explain the strengths that these partners bring to the overall project. 
 
Dartmouth College (The Dartmouth Institute)  
Our faculty and staff have worked closely with colleagues at The Dartmouth Institute for the past several years on an 
ongoing research project involving ACOs,46, 85 which will facilitate our collaboration on site visits to the clinical sites 
selected for participation in this project. 
 
Elliott Fisher, MD, MPH (Co-Investigator) 
Dr. Fisher   is the director of The Dartmouth Institute for Health Policy and Clinical Evaluation, and a member of the 
Institute of Medicine of the National Academy of Sciences. He is a national leader in the development of accountable 
organizations. Among his research interests is the development and testing of approaches to performance measurement 
and payment reform that can support improved quality of care. He is currently studying the development and evolution of 
ACOs working with Dr. Shortell and the Berkeley team. As Co-investigator he will assist in interpreting the results of the 
patient reported outcome measures and the clinical measures and will participate in at least one site visit.  
 
Glyn Elwyn, MD, PhD (Co-Investigator) 
Dr. Elwyn is a physician-researcher, Professor and Senior Scientist at The Dartmouth Health Care Delivery Science 
Center and The Dartmouth Institute for Health Policy and Clinical Practice, Dartmouth College. He has considerable 
expertise in patient activation and engagement, an area in which he is considered an international expert. He leads 
interdisciplinary research examining the implementation of shared decision-making, user-centered design of patient 
decision support interventions, and the integration of these into routine health care. As Co-investigator he will participate 
in the development of the site visit interview instruments and in some of the site visits. He will play a lead role in analysis 
and interpretation of the site visit material.  
 
Advocate Health Partners and HealthCare Partners 
 
Advocate Health Care (Advocate) and Advocate Physician Partners (APP) operate both a commercial and a Medicare 
ACO. In 2011, Advocate joined with its largest commercial insurance partner to create AdvocateCare, providing care to 
370,000 covered lives. In 2012, Advocate was chosen to participate in the Medicare Shared Savings Program (MSSP), 
becoming one of the largest Medicare ACOs in the country, serving an estimated 106,000 beneficiaries. They are leads in 
providing clinically integrated care. As indicated by their letter of support, they are highly committed to this study.  
 
HealthCare Partners ACO was formed in 2010 as one of the early Dartmouth-Brookings demonstration ACOs 46. They 
hold both CMS and private payer risk-based contracts collectively accountable for 50,000 lives. They serve high 
proportions of Medicaid patients in low-income communities who tend to face many social and financial barriers to self-
management of diabetes and CVD. Approximately 55% of HCP’s Southern California patients are Latino.  HCP and the 
HCP Institute for Applied Research and Education have extensive expertise leading patient-centered HIT development 
and research projects.  For the past 3 years, the HCP Institute through earned, competitive grant funding, has been refining 
the application of remote patient monitoring of chronic disease in older adult patients using interactive voice response 
technology (IVR). Based on a third-party evaluation by the Center for Connected Health at Partners Healthcare in Boston, 
HCP’s IVR system resulted in significant improvements in appropriate care 86. HCP’s implementation of patient-centered 
HIT, particularly focused on integrating informal caregivers 87, are important patient activation and engagement 
innovations to study in the context of improving patient-reported outcomes, as there is considerable variation in the use of 
patient-centered HIT across HCP’s practices. They are highly committed to this study as indicated by their letter of 
support and related materials in this proposal. 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
PCORI Research Plan – PFAs 1-4                                                                                                                                               
 
 
 
30
Both Advocate and HealthCare Partners will subcontract with UC Berkeley to perform the following tasks: 
 
Securely transmit names, addresses, and contact information for 273 adult patients with diabetes and/or cardiovascular 
disease with at least 2 visits in the baseline year (2013) from each of eight primary care practices affiliated with our 
Accountable Care Organization (ACO) to the patient survey vendor selected for the project.   
 
Securely transmit a de-identified patient-level data to the UC Berkeley research team in Months 7 and 19 covering 2013 
and 2014 period, respectively.  These datasets will include adult patients with diabetes and cardiovascular disease with at 
least 2 visits in the baseline year (2014) and the data elements include select sociodemographic information and clinical 
care process and outcome data using Healthcare Effectiveness Data Information Set (HEDIS) definitions for measures of 
chronic illness care. 
 
Provide the UC Berkeley research team with the names, emails, and mail contact information for all primary care 
clinicians and staff of the eight practices so that a survey of team culture, leadership, relational coordination, and 
perceived team effectiveness can be conducted at two points in time (at baseline, between months 0-6, and again between 
months 13-18).  The data manager at each ACO will facilitate the completion of the surveys by coordinating 
implementation with practice managers and ensure that participants have time during work hours to complete the survey. 
 
Provide feedback to the UC Berkeley team on the individual practice feedback reports that will include key comparisons 
from the patient and clinician/staff surveys.  These reports will be shared with primary care clinicians, staff, and patient 
advisory group members after each data collection period.  
 
Work with the Berkeley team to coordinate two site visits lasting two to three days for the research team to interview key 
people, some patients, and to observe meetings. The purpose of these visits is to learn more about the successes and 
challenges of delivering more patient-activated and engaged care and to help the research team better understand the 
quantitative data being collected. 
 
Assist with recruiting six patient representatives from the target population to serve on an advisory board that will review 
study instruments and  provide recommendations and feedback for the ACO practices to consider as they refine and 
improve their PA&E efforts.  The ACOs will provide institutional support to the patient advisory board and will provide 
patient advisors with project stipend support for their participation each year.    
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
 
APPENDIX (Optional) 
(Use continuation pages as needed. You may provide an appendix of up to 10 pages to include additional materials that you  
think may be useful to describe, e.g., survey instruments, interview guides, etc. Note, however, that reviewers are not  
required to include the appendix when they review and assess you application.) 
 
 
Appendix I. Patient Activation and Engagement Site Assessment Survey  
 
For most items below, the response categories will be: 
Yes, fully implemented 
Yes, partially 
implemented 
Yes, but not regularly 
No 
 
Begin survey: 
 
Please indicate the extent to which your practice site uses the following patient activation and engagement 
strategies for patients with diabetes and/or CVD.  
 
A. Patient care outreach in regard to disease prevention and health promotion  
1. Conducts a Health Risk Assessment (HRA) Survey 
2. Feedback to patients on results of their HRA 
3. Ongoing monitoring of HRA results (assess over time changes) 
4. Refer patients to a disease prevention or health promotion program as a result of the HRA 
5. Sponsor or participate in a community-based Healthy Eating Program 
6. Sponsor or participate in a Community-based Physical Activity Program 
7. Sponsor or participate in a Farmers Market 
8. Have a Employee Health Promotion/Prevention/Wellness program 
9. Sponsor or participate in School Health Clinic interventions  
 
B. Changes in the clinician-patient relationship, particularly in the areas of communication, 
motivational interviewing, and involvement in treatment care plans. 
10. HRA results available electronically to care team members (through EHR registry) at the point of care 
11. Clinicians are trained in motivational interviewing techniques 
12. Clinicians consistently use motivational interviewing techniques in communicating with patients, e.g. 
encourage patients to ask questions 
13. Clinicians consistently encourage patients to discuss their work, home life, and social situation 
14. Staff note patient preferences for treatment in the patient’s record 
15. Staff are trained in motivational interviewing techniques 
16. Staff consistently use motivational interviewing techniques in communication with patients, e.g., 
encourage patients to ask questions 
17. Select staff serve as “health coaches” for patients seeking to modify their lifestyle  
18. Patients can routinely provide information on their care and their health via a patient portal (not just 
access) 
19. Telehealth is consistently made available to patients with diabetes 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
20. Telehealth is consistently made available to patients with cardiovascular disease  
 
C. Shared decision-making 
21. Clinicians consistently involve patients in developing treatment goals  
22. Goal-setting for behavioral changes with patients as a result of their HRA 
23. Provide eligible patients with shared decision making videos 
24. Physicians consistently have follow up discussions with patients regarding their treatment options and 
preferences  
25. There is a formal evaluation of the impact of shared decision making on patient care choices, outcomes 
of care, and patient experience with their care 
26. There exists an organized follow up program to assist patients in managing their medications at home, 
e.g., pharmacist-led medication management 
27. Shared medical appointment (group visits) are available for patients with diabetes 
28. Shared medical appointment (group visits) are available for patients with cardiovascular disease  
29. Peer to Peer (Patient to Patient) programs are available for patients with diabetes  
30. Peer to Peer (Patient to Patient) programs are available for patients with cardiovascular disease   
31. Programs to improve family participation and support for patients with diabetes  
32. Programs to improve family participation and support for patients with cardiovascular disease   
 
D.  Patient self-management of their condition 
33. At home monitoring devices and/or tools to assess medication management, blood pressure, blood 
sugar, and lipids are made available to patients  
 
E. End of life/advanced serious illness care patient engagement and family involvement  
34. Clinicians consistently discuss the importance of patient advanced directives (Care for 
Older/Vulnerable Adults) 
35. Clinicians consistently discuss hospice care options with patients (Care for Older/Vulnerable Adults) 
36. Clinicians consistently discuss the availability of both hospital based and community based palliative 
care with patients (Care for Older/Vulnerable Adults) 
 
F.  Patient involvement in the overall design of care and in organization-wide efforts to improve 
the quality of care 
37. Patient advisory councils exist for patients with diabetes 
38. Patient advisory councils exist for patients with cardiovascular disease  
39. Patients consistently participate in quality improvement teams 
40. Patients are involved in helping to govern the clinic/practice  

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix II. Patient Activation, Self-Reported Outcomes of Care, and Patient Experience 
(Number of questions = 65) 
Measure 
Item 
Count 
Example Content 
Internal 
Consistenc
y 
Reliability 
Why Selected (Why alternatives were not?) 
Patient 
Activation 
Measure (PAM-
13) 
13 
• When all is said and done, I am the person who is 
responsible for taking care of my health 
• I know how to prevent problems with my health 
• I am confident that I can follow through on 
medical treatments I may need to do at home. 
0.90+ 
PAM is a widely used measure that has high 
internal consistency, has been shown to be 
associated with a range of patient-reported 
outcomes, and is also useful to guide the 
quality improvement activities of participating 
ACO practices10. 
Patient 
Assessment of 
Chronic Illness 
Care (PACIC-11) 
11 
Over the past 6 months, when I received care for 
my chronic conditions.  How often was I: 
• Given choices about treatment to think about 
• Helped to set specific goals to improve my eating 
or exercise. 
• Helped to plan ahead so I could take care of my 
condition even in hard times. 
0.93 
There are few validated measures of patients’ 
experiences of chronic illness care.  We 
considered using two alternate measures- the 
Ambulatory Care Experiences (ACES) quality of 
chronic illness care composite 88 and the 
recently developed Integrated Patient Care 
measures89 (Singer et. al).  The PACIC was 
preferred because of it has more extensive 
research highlighting the reliability and utility 
of the measure.  PACIC was also preferred 
because validation studies have been 
conducted with the brief measure (PACIC-11) 
and is sensitive to change over time.58  
Patients’ 
Experiences of 
Primary Care 
Teams 
(Ambulatory 
Care 
Experiences 
Survey) 
5 
Screener:  Are there nurses or other providers in 
your personal doctor’s office who play an 
important role in your care? 
• In the last 6 months, how often did these nurses 
and other providers in your personal doctor’s 
office explain things in a way that is easy to 
understand? 
0.70 
Few validated measures of patients’ 
experiences of primary care teams exist.  The 
“care team” composite measure of the 
Ambulatory Care Experiences Survey (ACES) 
survey has been previously used in peer-
reviewed research and has acceptable internal 
consistency reliability.  The recently developed 
Integrated Patient Care measures89 includes 2 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix II. Patient Activation, Self-Reported Outcomes of Care, and Patient Experience 
(Number of questions = 65) 
Measure 
Item 
Count 
Example Content 
Internal 
Consistenc
y 
Reliability 
Why Selected (Why alternatives were not?) 
• How would you rate the coordination of your 
care between these nurses and other providers 
and your doctor? 
• Thinking about these nurses or other providers 
in your doctor’s office, how would you rate their 
knowledge of you as a person, including values 
and beliefs that are important to you? 
items related to care team coordination and 
these questions have similar content as the 
ACES items (Rodriguez, et al, 2007).90 
Ability to 
Participate in 
Social Roles and 
Activities 
(PROMIS Short 
Form 8a) 
8 
• I have trouble doing all of the family activities 
that I want to do 
• I have to limit my regular activities with friends. 
• I have trouble doing all of the work that is really 
important to me (include work and home) 
0.85-
0.94 
The PROMIS social role subscale was selected 
because the composite covers a health domain 
central to patient activation and engagement.  
Further, the measure was developed through 
extensive item preparation and review, EFA-, 
CFA- and IRT-guided item banks help provide 
increased measurement precision and 
flexibility.91 
Patient Health 
Questionnaire 
(PHQ-4) 
Depression and 
Anxiety  
4 
Over the past 2 weeks, how often have you been 
bothered by these problems: 
• Feeling nervous, anxious or on edge 
• Not being able to stop or control worrying 
• Feeling down, depressed, or hopeless 
• Little interest or pleasure in doing things 
0.82 
Compared to the PROMIS measures of anxiety 
and depression, the PHQ-4 is parsimonious, is 
used more extensively in primary care settings 
for quality improvement and research 
purposes.  While the PROMIS measures are 
longer and cover a broader range of symptoms, 
research has not demonstrated the relative 
advantage of the PROMIS measures compared 
to the PHQ-9 or PHQ-4 for use in primary care 
settings.92  

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix II. Patient Activation, Self-Reported Outcomes of Care, and Patient Experience 
(Number of questions = 65) 
Measure 
Item 
Count 
Example Content 
Internal 
Consistenc
y 
Reliability 
Why Selected (Why alternatives were not?) 
Physical 
Function 
(PROMIS Short 
Form 12a) 
13  
• Are you able to do yard work like raking leaves, 
weeding, or pushing a lawn mower? 
• Are you able to bend down and pick up clothing 
from the floor? 
• Are you able to push open a heavy door? 
0.86-
0.95 
Research indicates that the PROMIS physical 
function composite, which is comprised of item 
response theory-based items, can result in 
greater responsiveness and precision across a 
broader range of physical function compared 
to other physical function measures. This can 
reduce sample size requirements and thus 
study costs.93  
Demographics 
11 
Age, gender, race, ethnicity, self-rated English 
literacy, preferred language, interpreter use, 
employment status, marital status, education, 
and military status  
 
Census specifications will be used, when 
applicable. 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix III. Primary Care Teamwork, Relational Coordination, Leadership, and Culture Survey 
(Number of questions= 40 excluding the team composition question) 
Measure 
Level 
Item 
Count 
Example Content 
Internal 
Consistency 
Reliability 
Why Selected (Why alternatives were not?) 
Team Composition 
Assessment 
Team 
1  
Which of the following people do you consider 
to be members of your team? [List of all practice 
members]:  Member selects “Yes, always”, “Yes, 
sometimes”, “No” 
NEW 
Recognizing that individual members 
sometimes have different conceptions of 
their team membership boundaries, we 
aimed to clarify differences in 
administratively-defined team membership 
and individual reports of team composition, 
and consider these differences when 
interpreting the teamwork measures.  
Team Processes / 
Relational 
Coordination  
Team 
7 
Do people on your team communicate with you 
in a timely way about patient care? 
 
When there is a problem with patient care, do 
people on your team blame others or work with 
you to solve the problem? 
0.80 
Adapted: Use a team-based referent vs. a 
focal workgroup (external) 71. We chose the 
RC over other measures of teamwork 
because more extensive validation and 
reliability information exists for RC 
compared to other teamwork measures 
such as the new AHRQ TeamSTEPPS 
measures71, 94. 
Team Participation  
Team 
7 
• I frequently contribute information 
• I frequently interpret information 
• I can comfortably disagree with other 
• I usually propose alternatives 
0.90 
Among teamwork measures assessed in a 
recent systematic review, is one of three 
instruments that satisfied standard 
psychometric criteria and associated with 
non self-reported outcomes.  Staff 
participation and engagement is also central 
to patient activation and engagement.72  
Baldridge Award 
Adapted – Patient 
Centeredness 
Practice 
5 
The practice does a good job of assessing 
current patient needs and expectations; staff 
promptly resolve patient complaints; patients' 
0.86 (2004) 
Used in previous research and found to be 
positively associated with changes to 
improve chronic illness care (Shortell, 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix III. Primary Care Teamwork, Relational Coordination, Leadership, and Culture Survey 
(Number of questions= 40 excluding the team composition question) 
Measure 
Level 
Item 
Count 
Example Content 
Internal 
Consistency 
Reliability 
Why Selected (Why alternatives were not?) 
complaints are studied to identify patterns and 
prevent the same problems from recurring; the 
organization uses data from patients to improve 
services; and the organization uses data on 
customer expectations and/or satisfaction/ 
experiences when designing new services. 
2004).74 
 
 
 
 
Leadership Facilitation 
(Organizational 
Readiness for Change 
Assessment- 
leadership behavior 
and change culture 
subscales) 
Practice 
7 
Leadership and management in your practice... 
…Reward clinical innovation and creativity to 
improve patient care 
…Solicit opinions of clinical staff regarding 
decisions about patient care 
…Seek ways to improve patient education and 
increase patient participation in treatment. 
…Make sure that we have the time and space 
necessary to discuss changes to improve care 
…Strongly support practice change efforts 
…Promote an environment that is an enjoyable 
place to work 
…Create an environment where things can be 
accomplished 
0.94 
We selected the ORCA measure of 
leadership because it is more specific to 
the changes related to PA&E initiatives while 
the AHRQ measure is more general. 
Group Practice 
Organizational Culture  
Practice 
8 
Innovation and Risk-taking in our practice 
1. Successful innovations are highly publicized 
2. Risk taking is encouraged 
3. We consider ourselves to be the testing 
ground for new approaches to engage patients 
in their care 
.80 
Consideration was also given to the 
Competing Values measure of culture but 
being an ipsative scale it is more difficult for 
respondents to complete. We therefore 
decided to use the more straightforward 
measures developed and validated by 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix III. Primary Care Teamwork, Relational Coordination, Leadership, and Culture Survey 
(Number of questions= 40 excluding the team composition question) 
Measure 
Level 
Item 
Count 
Example Content 
Internal 
Consistency 
Reliability 
Why Selected (Why alternatives were not?) 
4. We are given time to generate new ideas and 
innovations 
Group Solidarity in our practice 
1. We have a strong sense of belonging to the 
practice 
2. It would be hard for me to leave my 
colleagues at this practice 
3.Team members openly share their patient care 
challenges and failures with each other 
4. There is a well-defined pecking order among 
team members  
 
Kralewski 75 
Demographics 
Individ-
ual 
6 
Age, sex, race/ethnicity, occupation, years 
working at practice, work hours/week. 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Appendix IV. NATIONAL STUDY OF PHYSICIAN ORGANIZATIONS III—FEEDBACK REPORT EXAMPLE 
 
Comparison of <<Your Practice>> with all other physician practices surveyed 
Note: The “your organization” column reflects your self-reported survey responses 
Care Management Processes Indices Score* 
 
Your 
Organization 
Mean Scores 
All  
Practices  
1-7 
physicians  
8-19 
physicians  
20-99 
physicians 
100+ 
physicians 
Practice provides patient educators (0-4) 
1.5 
1.3 
1.5 
1.9 
2.3 
Practice provides performance feedback to physicians (0-4) 
1.1 
1.0 
1.3 
1.4 
2.0 
Practice uses nurse care managers for chronic disease (0-4) 
0.6 
0.5 
0.6 
0.9 
1.6 
Practice sends patients reminders for preventive or follow-up care 
(0-4) 
1.3 
1.3 
1.3 
1.4 
1.7 
Provider receives clinical practice guideline reminders at point of 
care (0-4) 
1.3 
1.3 
1.3 
1.4 
1.7 
Practice maintains electronic  registry of patients with chronic 
condition (0-4) 
1.1 
0.9 
1.2 
1.4 
1.9 
Overall Care Management Processes Index (0-24) 
6.9 
6.1 
7.1 
8.3 
11.1 
Care Management Process Percent Score** 
28.9% 
26.0% 
30.2% 
34.7% 
46.4% 
The Care Management Processes Indices are based on a “yes” response to a series of six questions regarding use of the specified processes for patients with asthma, congestive 
heart failure, depression, or diabetes. It is only possible to receive all points if your practice treats all conditions. 
**The “Care Management Process Percent Score” takes into account that a practice may not treat all conditions and reflects the points received out of points possible regarding 
conditions treated at your practice,  
 
Enhanced Access Index Score 
 
Your Organization 
Percentage/ Mean Scores 
All  Practices 
1-7 
physicians 8-19 physicians 
20-99 
physicians 
100+ physicians 
Majority of physicians communicate with patients via e-
mail 
18.4% 
19.1% 
13.2% 
13.1% 
31.3% 
Practice allows patients to view medical record online 
24.3% 
18.6% 
27.0% 
36.5% 
56.3% 

 
 
 
 
PRINCIPAL INVESTIGATOR (LAST, FIRST, MIDDLE):Shortell, Stephen M. 
 
 
 
 
 
 
Physicians use “advanced access” scheduling 
21.4% 
16.3% 
27.0% 
32.1% 
43.8% 
Practice is open extended hours or on weekends 
59.3% 
53.3% 
61.6% 
78.1% 
82.5% 
Practice uses group visits 
16.5% 
11.2% 
19.5% 
29.2% 
42.5% 
Overall Enhanced Access Index (0-5) 
1.4 
1.2 
1.5 
1.9 
2.6 
 
Care Coordination/Integration Index  
 
Your Organization 
Percentage/ Mean Scores 
All  Practices 
1-7  
physicians 8-19 physicians 
20-99 
physicians 
100+ physicians 
Majority of physicians use the EMR for progress notes 
64.2% 
55.0% 
75.5% 
87.6% 
95.0% 
Electronic access to view emergency department 
records  and hospital discharge summaries 
75.4% 
71.1% 
88.1% 
80.3% 
86.3% 
Majority of physicians send prescriptions directly to 
pharmacies electronically  
81.5% 
76.2% 
91.2% 
92.0% 
97.5% 
Overall Care Coordination/ Integration Index (0-3) 
2.2 
2.0 
2.5 
2.6 
2.8 
 
Patient-Centered Care Index 
 
Your Organization 
Percentage/ Mean Scores 
All  Practices 
1-7  
physicians 8-19 physicians 
20-99 
physicians 
100+ physicians 
Assess patient needs and expectations 
48.7% 
54.4% 
45.3% 
29.9% 
30.0% 
Promptly resolve patients’ complaints 
47.2% 
52.3% 
39.0% 
35.8% 
31.3% 
Complaints studied to identify patterns and prevent 
recurrence  
35.0% 
38.4% 
28.9% 
27.0% 
37.5% 
Data from patients used to improve care 
30.1% 
30.2% 
28.3% 
27.0% 
37.5% 
Data from patients used to develop new services 
17.4% 
18.1% 
14.5% 
16.1% 
18.8% 
Overall Patient-Centered Care Index (0-5) 
1.8 
1.9 
1.6 
1.3 
1.5