Leveraging Integrated Models of Care to Improve Patient-Centered Outcomes for Publicly-Insured Adults With Complex Health Care Needs
Phase: NA
Conditions: Diabetes, Asthma, Chronic Obstructive Pulmonary Disease, Hypertension, Anxiety, Atrial Fibrillation, Congestive Heart Failure, Depression, Bipolar Disorder, Schizophrenia
Sponsor: University of Pittsburgh

 
 
December 1, 2022 
 
Study Title: Leveraging Integrated Models of Care to Improve Patient-Centered Outcomes for 
Publicly-Insured Adults with Complex Health Care Needs 
 
Unique Protocol Number: 1609-36670 
ClinicalTrials.gov ID: NCT03451630 
 
 
Principal Investigator, Research Team, and Study Site: 
 
Dan Swayze, DrPH, MBA, MEMS 
Principle Investigator 
 
Ellen Beckjord, PhD, MPH  
Co-Investigator 
 
Jonathan Encarnacion, MBA  
Co-Investigator 
 
Megan Hamm, PhD  
Co-Investigator 
 
Chaeryon Kang, PhD 
Co-Investigator 
 
Jordan Karp, MD  
Co-Investigator  
 
Kevin Kraemer, MD, MSc  
Co-Investigator 
 
 
 
Doug Landsittel, PhD  
Co-Investigator 
 
Kelly Williams, PhD, MPH 
Co-Investigator 
 
Margaret Park, MDiv  
Patient Stakeholder Co-Investigator 
 
UPMC Center for High-Value Health Care 
U.S. Steel Tower, 40th Floor 
600 Grant St. 
Pittsburgh, PA 15219 
 
University of Pittsburgh 
4200 Fifth Avenue 
Pittsburgh, PA 15260 
 
 
 
 
 
 
 

2 
 
Contents 
Research Synopsis ......................................................................................................................................... 3 
Background and Significance ........................................................................................................................ 3 
Objectives ..................................................................................................................................................... 4 
Aims .............................................................................................................................................................. 4 
Interventions Compared ............................................................................................................................... 5 
Study Design/Methodology .......................................................................................................................... 6 
Randomization Procedure ............................................................................................................................ 6 
Measures ....................................................................................................................................................... 6 
Study Population ........................................................................................................................................... 9 
Inclusion/Exclusion Criteria........................................................................................................................... 9 
Study Duration/Study Timeline ..................................................................................................................... 9 
Statistical Analysis Plan ................................................................................................................................. 9 
Informed Consent Process .......................................................................................................................... 13 
Privacy and Confidentiality ......................................................................................................................... 13 
Risk/Benefit ................................................................................................................................................. 14 
Compensation for Participation .................................................................................................................. 14 
Data Safety Monitoring ............................................................................................................................... 15 
Conflict of Interest ...................................................................................................................................... 15 
Publication and Presentation Plans ............................................................................................................ 15 
References .................................................................................................................................................. 16 
 
List of Common Abbreviations in Protocol: 
• MCC: Multiple chronic conditions  
• CT: Community Team 
• UPMC ISD: University of Pittsburgh Medical Center Insurance Services Division  
• HTE: Heterogeneity of treatment effects 
• ODP: Optimal Discharge Planning 
 
 
 
 
 

3 
 
Research Synopsis 
Study Population: Medicaid or dual-eligible (Medicare-Medicaid) adults aged 21 years and 
older with multiple chronic conditions (MCC), including at least one physical health condition 
and at least one additional physical or behavioral health condition, beneficiaries/members of 
our partnering health care payer, and identified as having high and/or rising health care needs 
and at risk for unplanned health care use.  
 
Study Design: A comparative effectiveness study using an individual-level, randomized design 
along with a pragmatic, mixed-methods approach to compare three strategies (e.g., in-person 
and telephonic supported care, technology-supported care, and optimal discharge planning 
care) all of which include evidence-based components of integrated care. Quantitative (e.g., 
self-report, claims, process) and qualitative (e.g., interviews) data will be collected across 
multiple timepoints during the study period. 
 
Sample Size: We expect to enroll 1,400 individuals.  
 
Study Duration: The contract period begins January 1, 2018 and concludes November 30, 2024.  
Background and Significance 
Chronic disease is widely recognized as the U.S. public health challenge of the 21st century.1 
Defined as “conditions that last a year or more and require ongoing medical attention and/or 
limit activities of daily living,”2 these diseases comprise a wide range of physical illnesses, such 
as diabetes, arthritis, asthma, heart disease, chronic obstructive pulmonary disease, and 
hypertension, as well as mental health and substance use disorders. Among the Medicaid 
population, 80% of high-need beneficiaries have three or more chronic conditions, and 60% 
have five or more chronic conditions, including a high incidence of behavioral health issues.3,4 
Three in five of the 9 million individuals eligible for both Medicare and Medicaid (e.g. dual-
eligible) services have multiple chronic physical conditions, 20% have more than one 
mental/cognitive condition, and almost two in five have comorbid physical/behavioral health 
conditions.5,6 
Although numerous randomized controlled trials have shown that practice-based integrated 
chronic care models assisting patients and their caregivers in managing their health and health 
care can improve outcomes for various chronic conditions,7-13 including comorbid medical and 
behavioral health issues,14-17 implementation of these models has been limited due to the 
significant upfront financial and infrastructure investments that are typically required.18,19 
Related interventions, such as transitional care,20,21 self-management education,22-24 and 
coordinated care,25 have also been proven effective in improving outcomes for adults with 
multiple chronic conditions (MCC), but there is still considerable uncertainty about how best to 
implement these practices to achieve widespread and significant impact.19,26,27 The fact that 
different approaches may be required in order to optimally support specific patient subgroups 
adds a further layer of complexity.28,29 For example, there is very little information available 
about how to effectively support Medicaid29 and dual-eligible enrollees,30 who are often sicker, 
report lower health and functional status, and are more likely to be disabled than their 

4 
 
Medicare-only counterparts.31 Targeting intensive care management services to these high-
need patients is an increasingly well-recognized best practice.32 
Despite increased understanding of how to promote optimal care for individuals with MCC, 
widespread improvements in health outcomes have yet to be realized, especially for high-risk, 
high-need subgroups. There have been no head-to-head trials comparing the effectiveness of 
system-level interventions in supporting the delivery of evidence-based integrated care for 
adults with MCC in general or the Medicaid and dual-eligible populations to date. The proposed 
study will assess innovative approaches of integrated care that can be leveraged to address this 
gap.  
To this end, we propose to compare the effectiveness of the two primary system-level 
features that drive these programs, namely personalized service design and innovative use of 
technology in delivering four evidence-based, patient-centered components of established 
integrated chronic care models: (1) interdisciplinary care team management;33,34 (2) 
development and monitoring of individual care plans;33,34 (3) patient education and chronic 
disease self-management/self-care support;7,33,35-39 and (4) supporting member linkages with 
medical/behavioral health and social services.33,40,41 The hybrid nature of the proposed 
interventions is both unique and compelling. No single, system-level intervention has yet to 
combine all four components of integrated care that have been proven effective in improving 
outcomes for patients with MCC.  
Objectives 
The goal of this collaborative study is to provide much needed information for adults living with 
MCC and other key stakeholders working to support the health of these adults through health 
care system improvements. The three care strategies, High-Touch (in-person and telephonic 
supported), High-Tech (technology-supported) and Optimal Discharge Planning (ODP) will be 
implemented and evaluated to determine their potential to serve as best practice for 
effectively managing chronic conditions. 
To ensure our study’s focus was patient-centered, the study team worked closely with the 
Patient Partner Co-Investigator and patient, provider, and system-level stakeholders to develop 
all aspects of this study protocol, including the early development efforts of in-person and 
technology-supported care approaches, the research questions, the study outcomes, the 
evaluation procedures, and the dissemination strategies included in this protocol.  
Aims 
This study aims to compare the effectiveness of the approaches on patient-centered 
outcomes and determine which care strategy works best for whom and under what 
circumstances. By observing comparisons on key outcomes with ODP, our findings will enable 
patients and health care systems to understand the relative effectiveness of the care strategies 
vis-à-vis current practice and provide much-needed information for patients with MCC as well 
as health systems striving to support them more effectively and efficiently.  
 
Primary Aim 1: Compare the effectiveness of High-Touch, High-Tech, and ODP on primary 
outcomes including hospital readmission, health status, and patient activation, and on several 

5 
 
secondary outcomes including functional status, quality of life, care satisfaction, emergent care 
use, engagement in primary, specialty, and mental health care, and gaps in care. 
Hypothesis 1a: High-Touch will result in lower readmission rates at 12 months compared to 
High-Tech & ODP. 
Hypothesis 1b: High-Touch will result in higher health status at 12 months compared to 
High-Tech & ODP. 
Hypothesis 1c: High-Tech will result in higher patient activation at 12 months compared to 
High-Touch & ODP. 
 
Primary Aim 2: Examine the differential effects of the interventions for patient subgroups, 
based on age, race, illness complexity, and comorbid behavioral health conditions to evaluate 
heterogeneity of treatment effects (HTE) and determine for whom and under what 
circumstances the interventions are most effective.  
Hypothesis 2a: For all primary outcomes, High-Touch will have greater positive impact than 
High-Tech & ODP for older participants (60 years or more) and for participants with 
comorbid behavioral health conditions. 
 
Primary Aim 3: Examine perceived barriers and facilitators to efficient and effective 
implementation of High-Touch and High-Tech interventions for delivering evidence-based 
integrated care. 
Interventions Compared 
High-Touch. This approach leverages a personalized service design that includes payer-
employed registered nurses and social workers to provide intensive, in-person/telephonic 
support and resources for eligible participants in their homes and/or communities for a 
minimum of four months following initial engagement. The health plan-based CT will focus on 
addressing the full range of health determinants that can impact an individual’s health and 
health care and support information sharing with patients and providers through intervention 
that do not require access to mobile devices or the Internet.  
High-Tech. By leveraging telehealth and remote monitoring technology to support self-
directed care management in real time, this approach uses less in-person resources. CT will 
focus on addressing the full range of health determinants that can impact an individual’s health 
and health care through innovative approaches including virtual visits and access to mobile 
communications/applications for managing individual health and health care. 42,43 42,43 42,43 
Optimal Discharge Planning (ODP). Prior to the use of High-Touch and High-Tech 
interventions within our payer environment, all members with MCC who were hospitalized and 
at risk for rehospitalization were supported through the UPMC Health Plan’s optimal discharge 
planning program, considered to be ODP for this study’s purposes. After initial engagement via 
a phone call to the member, a Care Manager from the CT will have an in-home or telephonic 
visit with each individual to provide evidence-based support, including: detailed disease 
management and medication education; confirmation of and connection to family/caregiver 
support and resources; scheduling an ambulatory follow-up appointment; and a hand-off to a 
health plan-based telephonic care manager, as needed.  

6 
 
Study Design/Methodology 
We have chosen an individual-level randomized design along with a pragmatic, mixed-methods 
approach to compare three integrated care approaches, High-Touch (in-person/telephonic 
supported), High-Tech (technology-supported) and ODP for Medicaid or dual-eligible adult 
members with MCC who reside in in Western, Central, or Eastern Pennsylvania (PA) and are at 
high risk for high utilization, including rehospitalization. 
Quantitative data will be collected from several sources. Self-report data will be gathered 
at baseline and three additional timepoints (3-, 6-, 12-months) using REDCap Cloud, a secure 
data collection platform. Claims/administrative data will be collected from UPMC Insurance 
Services Division (UPMC ISD) to inform several outcomes and covariates. Claims data will be 
extracted at six months and 12 months to characterize participant service use through each 
participant’s enrollment period, allowing for one full year of service-use observation.  
Qualitative telephonic interviews will be conducted with a sample of participants and CT 
staff to examine perceived barriers and facilitators to efficient and effective implementation of 
the High-Touch and High-Tech approaches for delivering evidence-based integrated care at 
three timepoints (Participants: baseline, 3-, and 12-months; CT Staff: baseline, mid- and end-
implementation period).  
Randomization Procedure 
The study will use web-based randomization to one of the three care strategies for those 
individuals who consent to participate in the study. The consent process may take place face-
to-face with an eligible individual, or over the phone, providing that the individual consenting 
has received a paper or electronic copy of the consent form and enrollment materials to 
review. Once a member of the CT staff determines the individual’s eligibility, they will enter in 
key identification information, independently or with support of the research team, and the 
system will then generate a Study ID (numeric identification number). A research team member 
will verify eligibility and the accuracy of the information entered with the CT staff and enrollee, 
and then the REDCap Cloud system will provide an assignment to an intervention arm. 
Randomization will be stratified by gender, type of insurance (Medicaid or Medicare-Medicaid), 
and technology/digital literacy, which will be assessed at time of enrollment and before 
randomization, to ensure that intervention arms are balanced with respect to these important 
variables. Within each stratum, random block sizes of 5 and 10 will be used to maximize 
balance between intervention groups while minimizing the ability to unmask investigators to 
the next treatment assignment, triggering an automated alert to CT staff regarding which 
intervention (care strategy) to implement for each participant and documented accordingly in 
HealthPlaNET, UPMC ISD’s integrated health management software program. If a participant is 
unwilling to be randomized, they will be excluded from the study. 
Measures 
Process Measures 
Participant and payer care team staff perceptions/experience will be assessed via telephonic 
qualitative interviews with a sample of participants and CT staff to examine perceived barriers 

7 
 
and facilitators to efficient and effective implementation of High-Touch and High-Tech and to 
identify strategies for intervention improvement/tailoring and dissemination.  
 
Primary Outcome Measures 
Patient activation will be assessed using the Patient Activation Measure (PAM), a 13-item scale 
that gauges individual knowledge, skills, and confidence essential to managing one’s own 
health.44 In consideration of “straight-lined” answers in the PAM measure, (i.e., all 13 questions 
answered with “strongly disagree”), we plan to include all participant responses and conduct a 
sensitivity analysis. This is recommended by our study investigative team, including our Patient 
Partner Co-investigator.   
 
Health status will be measured using the Health Survey (SF-36), a 36-item scale measuring 
functional health and well-being within eight domains, including physical functioning, physical 
ability to complete tasks, bodily pain, general health, vitality, social functioning, emotional 
ability to complete tasks, and mental health.45 
 
Subsequent 90-day rehospitalization will be measured using an all-cause readmission rate 
existing in UPMC ISD inpatient claims for physical and behavioral health service use within 90 
days of the last hospital discharge prior to study enrollment. All-cause readmission does not 
include observation visits.  
 
Secondary Outcome Measures 
Subsequent rehospitalization will be measured using an all-cause readmission rate existing in 
UPMC ISD inpatient claims for physical and behavioral health service use within 30 days and 
again at 12 months, following the index inpatient admission. Inpatient and ED visits that were 
adjudicated and reclassified as Observation visits will be separately explored at 12 months.  
 
Functional status will be measured using the PROMIS Physical Functional Form – Short For 6b 
which is a brief self-report tool that assesses functional impairment in everyday tasks, 
household chores, and walking using a 5 point Likert scale.46  
 
Quality of life will be measured using the Quality of Life Enjoyment and Satisfaction 
Questionnaire (Q-LES-Q-18), which is an 18-item, self-administered questionnaire monitoring 
enjoyment in several domains (physical health, subjective feelings, leisure time activities, social 
relationships).47 
 
Care satisfaction will be measured using the Patient Assessment of Chronic Illness Care (PACIC), 
a 20-item survey designed to provide an assessment of important aspects of care for chronic 
illness patients, including specific actions or qualities of care that patients report experiencing 
in the service delivery system and their satisfaction level.48 
 
Emergent care use will be measured using existing behavioral and physical health claims data 
from the UPMC ISD data warehouse determining participant frequency of emergent service use 
over 12 months. 

8 
 
Engagement in primary, specialty, mental health care will be measured using existing 
behavioral and physical health claims data from the UPMC ISD data warehouse determining 
participant frequency of non-acute visits for participants over 12 months.  
 
Gaps in care will be assessed using Healthcare Effectiveness Data and Information Set (HEDIS) 
quality metrics that will be compiled using existing behavioral and physical health claims data 
from the UPMC ISD data warehouse. Gaps in care will be assessed for six of the most common 
chronic diseases present among our target population, including: asthma, chronic obstructive 
pulmonary disease, congestive heart failure, coronary artery disease, diabetes, and 
depression.49 
 
Covariates 
Engagement in interventions will be assessed using the automatically generated usage statistics 
and administrative/ utilization data routinely collected for non-research purposes as part of 
insurance claims processing and existing payer-provider service delivery. 
 
Patient characteristics will be obtained through self-report data and existing secondary 
administrative data on age, gender, race/ethnicity, and insurance type. 
 
Technology/digital literacy will be measured using a series of questions from the Health 
Information National Trends Survey (HINTS) to assess comfort and experience with use of 
different. communication channels, including the Internet, to obtain vital health information for 
themselves and their loved ones. 
 
Social support will be measured using the Interpersonal Support Evaluation List (ISEL), a 12-item 
measure designed to assess the perceived availability of three types of social support 
(appraisal‚ belonging‚ and tangible).50 
 
Health literacy will be measured using the first ten items of the All Aspects of Health Literacy 
Scale (AAHLS), a 13-item survey designed to assess functional, communicative, and critical 
health literacy.51 The final three items on the AAHLS are part of an ‘empowerment scale’ that is 
not related to the outcomes of this study. 
 
Illness complexity will be measured using the age-adjusted Charlson Comorbidity Index (CCI) 
which was developed as a prognostic burden of comorbid disease and is currently used by the 
UPMC ISD.52 A score will be computed at baseline based on the previous 12 months of claims, 
adding assigned weights for specific diseases as classified by using International Classification of 
Diseases (ICD) diagnoses on claims records, and age adjusting by adding 1 point for every 
decade over 40 to the CCI score.52 
 
Socioeconomic status will be measured using the Area Deprivation Index (ADI), which uses 17 
different socioeconomic indicators to determine a deprivation score for a given neighborhood. 
Scores are reported in the form of quintiles, with a higher quintile indicating a higher degree of 
deprivation.53-55 

9 
 
Study Population 
Medicaid or dual-eligible adults aged 21 years and older with MCC, including at least one 
physical health condition and at least one additional physical or behavioral health condition and 
are beneficiaries/members of our partnering health care payer. In addition, these individuals 
are identified as high risk for unplanned care use including rehospitalization. Our study 
population is representative of the majority of Medicaid and dual-eligible beneficiaries and 
accords with a national sample.56 
Inclusion/Exclusion Criteria  
Inclusion Criteria 
Individuals who are (1) Medicaid or dual-eligible; (2) age 21 years and older; (3) insured through 
physical and/or behavioral health payers within the UPMC ISD; (4) identified as members high 
risk for high utilization including rehospitalization and have had an inpatient hospital stay within 
the last 30 days; (5) who have at least one physical health condition (e.g. cardiovascular 
disease, hypertension, COPD, diabetes) and at least one additional physical or behavioral health 
condition (e.g. depression, serious mental illness, substance abuse disorder); and (6) who reside 
in Pennsylvania.  
 
Exclusion Criteria 
Individuals who are (1) pregnant; (2) in skilled nursing facilities; (3) on hemodialysis for kidney 
disease; (4) in active cancer treatment; and/or (5) unable to speak, read, or understand English 
or Spanish at the minimum-required level.  
Study Duration/Study Timeline 
January 1, 2018 – April 14, 2018: Pre-Implementation Period 
April 15, 2018 – December 31, 2022: Implementation Period  
January 1, 2023 – November 30, 2024: Post-Implementation Period 
Statistical Analysis Plan 
Sample Size and Power Calculations 
We will evaluate, based on a 20% attrition rate, a total of 1,120 participants at 12-month post-
enrollment data collection time point, which will be the main time point for the comparison of 
primary outcomes. We will use a randomization ratio of 2:2:1 (High-Touch; High-Tech; ODP, 
respectively) and, as such, estimate final sample sizes of n=448 for each active care strategy and 
n=224 for the ODP strategy. All calculations for the power estimates were done in PASS version 
13. For the primary test for the trial, the power calculation is based on the comparison of the 
two primary care strategies (High-Touch and High-Tech) for the main outcome of hospital 
readmissions between the strategies, using a two-sample test of proportions; specifically we 
have 80% power to detect a difference between a readmission rate of 20% in the High-Touch 
care strategy versus 28% in the High-Tech care strategy.21,57,58 Additionally, for the overall 
differences between the three care strategies the given sample sizes yield 95% power; based on 
a chi-squared test.  

10 
 
For the additional primary outcomes of health status (SF-36)45 and patient activation 
(PAM)59, power estimates are based on a two-sample t-test for comparing High-Touch and 
High-Tech and using an Analysis of Variance (ANOVA) for an overall difference across the three 
arms. For SF-36 scores, we hypothesized mean increases of 11, 5 and 0 points in High-Touch, 
High-Tech and ODP, respectively, with a standard deviation of 10 in each arm or care 
strategy.60-63 This yielded over 99% power for the test between the two active interventions 
(e.g. High-Touch and High-Tech), assuming a 5-point increase for the High-Tech group would 
yield 80% power for detecting a difference as small as 1.7, which is a small effect size of 0.17. 
The hypothesized mean increases also yield >99% power for the overall test of mean 
differences across interventions and yield >90% for means as close as 5.6, 5.0 and 4.4. For PAM 
scores, we hypothesized mean increases of 4, 8 and 0.5 points in High-Touch, High-Tech and 
ODP, respectively, with a standard deviation of 20.9 in each arm.44,64 This difference, which 
corresponds to a small effect size of 0.19 yielded 82% power for the test between the two 
active intervention groups (e.g. High-Touch and High-Tech). The hypothesized mean increases 
also yield >99% power for the overall test of mean differences across the groups and yield >80% 
for means as close as 6.5, 8 and 6. 
For the tests of interactions, we have identified, a priori, eight subgroups of interest: those 
aged 60 years and above, constituting 23% of the sample, and those aged below 60; race with 
non-white participants constituting 38% of the sample (n=203 High-Touch/High-Tech, n=100 
ODP), and white participants; Charlson Comorbidity Index  Score of five or above, indicating 
higher illness complexity includes 54% of the sample versus lower illness complexity; and 
positive (opposed to negative) for comorbid behavioral health conditions for 60% of the 
sample. We are only presenting power analysis results for the treatment by age interactions 
since we do not have any a-priori hypotheses about the direction of differential effects for 
treatment by race, treatment by illness complexity, or treatment by comorbid behavioral health 
condition. All such interactions will therefore be labeled as purely exploratory in subsequent 
findings and publications. Power estimates for interactions were calculated based on a 2-way 
factorial design and the average sample size across groups. For readmission rates, we used a 
normal approximation to estimate the standard deviation and applied a 2-way ANOVA as 
similar procedures are not available for dichotomous outcomes without approximating as 
normal. 
For age by treatment interactions, readmission rates are hypothesized to increase in older 
age groups, with greater differences in the treatment effects. We also hypothesized that the 
intervention groups will yield similar rates in the younger group while High-Tech will be closer 
to ODP in the older group.65 More specifically, we have 79.8% power to detect a significant 
interaction with readmission rates of 22%, 40%, and 42% in the older group (for High-Touch, 
High-Tech, and ODP, respectively) and rates of 18%, 16% and 28% in the younger group. For 
PAM scores, we hypothesized that, in the older group, the interventions will work equally well 
and much more effectively than ODP, whereas, in the younger group, High-Tech will be most 
effective, with High-Touch and ODP showing no effect on PAM scores.66 More specifically, we 
have 72.2% power to detect a significant interaction with mean increases in PAM of 8, 8, and 1 
in the older group for High-Touch, High-Tech, and ODP, respectively, and mean increases of 0, 8 
and 0 in the younger group. For SF-36 scores, we assumed a greater overall effect for High-
Touch and a greater effect in younger ages and a slightly larger difference between ages for the 

11 
 
High-Touch group.67 Specifically, we have 99.5% power to detect a significant interaction with 
mean increases in SF-36 of 7, 2, and 0 in the older group for High-Touch, High-Tech, and ODP, 
respectively, and mean increases of 15, 8, and 1 in the younger group.  
 
Quantitative Analysis 
We will conduct descriptive analyses and multivariate modeling of patient self-report and 
administrative data to examine changes in outcomes over time and explore moderating 
variables. If any additional analyses are identified during the study, they will clearly be labeled 
as post-hoc comparisons and interpreted as such. 
Primary outcomes will be analyzed using both self-report and claims data. As such, loss to 
follow-up for a participant will be calculated at the point which they no longer complete their 
self- report questionnaires for a given timepoint and they are no longer Medicaid/Medicare-
Medicaid eligible, as determined by the proxy of 9 months of Medicaid/Medicaid-Medicare 
coverage in the 12-month period prior to the data collection timepoint. Whichever comes last is 
the point at which the participant is lost to follow-up. No surveys are collected during 
Timepoint 2, and as such, loss to follow-up will be calculated at the point which they are no 
longer Medicaid/Medicare-Medicaid eligible, as determined by our proxy described above. 
All inferential modeling will be preceded by descriptive analyses of baseline and outcome 
measures. Summary statistics will include the mean and standard deviation, and the median 
and range for continuous variables, and the frequency and percentage for each categorical 
variable. Results will be presented for the entire sample and stratified by intervention arm. We 
will also statistically assess the balance of key covariates across the interventions via analysis of 
variance for continuous measures and chi-squared tests for categorical measures. For 
continuous variables which are assumed to be normal, we will visually assess the univariate 
distribution via a normal probability plot to determine departure from normality and the 
bivariate correlations between pairs of covariates to assess collinearity; appropriate diagnostics 
for collinearity and model fit of the regression models described below will also be checked in 
each analysis. Transformations will be implemented if required, although slight to moderate 
departures from normality will not be problematic with the large sample sizes. 
Our models will be based on an intent-to-treat principle as we do not expect large drop out 
or non-adherence. We will employ linear regression models for continuous outcomes, logistic 
regression for dichotomous outcomes, and Poisson regression or negative binomial models (in 
the case of over-inflated variance) for count outcomes. We will fit multivariate models with the 
key independent variable of interest being the main effect of treatment with main effects for 
additional covariates included to produce an adjusted estimate of the treatment effect. These 
covariates will be defined a priori and will be divided into two subgroups. The first set of 
clinically-important covariates will be included in all models including age, race, gender, 
insurance type, illness complexity, and socio-economic status. The second set of covariates will 
be defined as engagement in interventions, social support, health literacy, and technology 
literacy. To address Aim 2, separate models will be fit for each a priori defined heterogeneity of 
treatment group (HTE) subgroup of interest. As listed below, we will test the interaction of 
treatment with the covariate defining the HTE subgroup.  
The objective of Aim 2 is to determine whether the intervention works better for some than 
for others. Our pre-specified analysis plan will examine differences in main outcomes for 

12 
 
several subgroups. These subgroups are determined by our stakeholders to be most relevant so 
that end results can meaningfully inform patients about what works best for whom, and in turn, 
better supports individuals with MCC as they navigate the health care system and make 
decisions about their care. Furthermore, in examining differential impact of the interventions 
based on these subgroups, providers, payers, and other decision makers can target resources in 
a manner that best meets the needs of those in services, reduces variations in practice, and 
improves health outcomes. 
We are only powering for the interactions with age, since we do not have any a-priori 
hypotheses about subgroup effects for the other variables. Those additional subgroups will 
clearly be labeled as exploratory and will only be examined further if the overall analysis is 
significant. We will conduct modeling as defined above to examine the HTE and will utilize 
contrasts from these regression models to make inferences about the heterogeneity. We will 
present both the treatment effect estimates, in terms of estimated odds or risk ratios for 
dichotomous or rate outcomes or the model coefficients for the continuous outcomes and 
measures of their variability in the form of confidence intervals. Two-sided tests of level 0.05 of 
the interactions between intervention and HTE subgroup will allow us to test the Aim 2 
hypotheses.  
For all the described models, we will focus on the 12-month outcome. Each of the above 
analyses will also be repeated using all follow-up outcomes collected as a secondary analysis 
using a mixed model with a random intercept to account for within-subject repeated measures 
and associated correlation. Significance tests from all regression models will all be assessed 
using likelihood ratio tests for logistic, Poisson, or, if applicable, negative binomial models or 
partial F-tests for linear regression. 
Additionally, exploratory data analyses will include comparison of the following variables in 
the pre- versus post-COVID-19 time periods: a) demographics and primary outcomes at 
baseline, b) the primary outcomes over time, and c) patterns in missing data. If significant 
differences result, we will repeat the primary analysis by: a) adjusting for the demographic 
variables that can describe pre- and post-COVID-19 participants, b) include indicator variables 
for post-COVID-19 enrollment with and without interactions with time variables, and c) 
statistically account for missing patterns. 
Qualitative Analysis 
Codebook construction will follow standard editing methods.68 Separate codebooks for patients 
and staff interviews will be constructed after completion of the first set of interviews. A system 
of audit trails will be employed to document creation of codes during the iterative process of 
codebook development. Two trained independent analysts from the Qualitative, Evaluation and 
Stakeholder Engagement (Qual EASE) Research Core will code the interviews in Atlas.ti, 
qualitative analysis software. A manual will be created to outline information about each code 
in the codebook. After each transcript is coded, coders will meet to process and adjudicate 
differences until agreement is achieved. Codes determined through this process will be 
recorded in a master file to be used in final analysis. Once all transcripts are coded, Cohen’s 
Kappa scores will be calculated to assess inter-coder reliability with the goal of achieving 
reliability of >0.75 which exceeds substantial coder agreement.69 

13 
 
For each qualitative cohort and for each wave of data collection, a final report will be 
created. We will examine key topics in the interview guides to better understand barriers and 
facilitators to intervention implementation and success and how interventions impact patients’ 
ability to manage their chronic conditions. In addition, for each subsequent wave of data 
collection, we will examine changes and consistency across the waves. Final reports will be 
shared with stakeholder teams to ensure ongoing intervention improvements and aid in 
interpretation of findings. 
Informed Consent Process 
Informed consent can be obtained either telephonically or obtained during the first in-home 
visit using a web-based platform, at the participant’s home or a preferred location within the 
community. For telephonic consents, the care manager will provide the participant with a copy 
of the consent form and study FAQs to review prior to enrollment. CT staff, who have 
completed the University of Pittsburgh Human Research Protection Office (HRPO) human 
subjects research training through the Collaborative Institutional Training Institute (CITI), will 
assist the participant in navigating the web-based consent process, with support provided by 
the research team. The web-based consent process will provide subjects with a clear 
explanation of the objectives, procedures, risks and benefits of the study, a frequently asked 
questions page and their rights as study participants. Research Team members will be available 
via the telephone to answer all questions related to the informed consent process and the 
study prior to the member consenting to participation. Consent will be obtained only from the 
participant. Consent will also be required for participation in the qualitative interviews. 
Participants will retain their right to have their questions answered by the Principal 
Investigator or a member of the Research Team via telephone or an appointment for an in-
home visit. The Research Team believes that informed consent is an ongoing process in any 
study and so will continue to educate participants about the nature of the research and will 
address any questions that arise throughout the course of the study. These efforts will comply 
fully with the Health Insurance Portability and Accountability Act (HIPAA) and the informed 
consent guidelines of the University of Pittsburgh’s HRPO. 
Privacy and Confidentiality 
Self-report and qualitative data collected for this study will be used for research purposes only. 
Access to the data will be restricted to the Principal Investigator, Co-Investigators, and other 
Research Team members trained in UPMC and University of Pittsburgh HRPO Human Subject 
Research Training requirements. Research staff will sign confidentiality agreements as required. 
Identifying links to all data will be maintained at baseline and other data collection time points 
using a secure password-protected server, accessible only by authorized members of the study 
team. Once the data is collected, the file containing the link between identifying information 
and the participant's data will be destroyed. 
Qualitative data will be audio recorded using digital recording devices, transferred to a 
secure server immediately following recording and deleted as soon as it is fully transcribed. 
Interviews will be audio recorded, transcribed, and scrubbed of identifying information. 

14 
 
Participants will be asked not to include any identifying information in their responses, and 
coders will not code any identifying information that may be provided inadvertently.  
All data obtained over the course of the study will be confidential and secure. Paper study 
records will be secured using the “double-lock” method (i.e., in a locked cabinet within a locked 
office). Data stored on computers will be password protected and stored on a secure server 
behind the organization’s firewall. Each Research Team member has a unique network account 
and a secure password that complies with existing UPMC policies and procedures. Only 
members of the team who are authorized by the Principal Investigator will have access to the 
secured files. Identities of participants will not be revealed in the publication or presentation of 
any results from this study. 
Risk/Benefit 
Risk to Participants 
There is no known serious health or psychological risk of participating in this study. Some of the 
items in the self-report measures that ask respondents about their health, attitudes, and 
experiences could be considered sensitive. However, participants can refuse to answer 
questions at any time, which will be made clear during the consenting procedures and at the 
time of administration of all research measures and interviews. Minimal risks of study 
participation may still apply, such as participants feeling inconvenienced by the assessment 
requirements and/or uncomfortable when responding to surveys or interviews. As with all 
research involving personal health information (PHI), there is also a potential risk of a breach of 
confidentiality for all study participants. However, safeguards will be in place to fully decrease 
this risk. Currently, there are no known risks associated with involvement in the study 
interventions, other than possible emotional discomfort associated with the discussion of issues 
related to complex health conditions. In instances where participants appear to be unduly 
distressed about health or other issues in their interactions during the interventions, they will 
be advised to speak directly with Dr. Dan Swayze, Principal Investigator.  
Benefits to Participants  
There is a high likelihood that all study participants will benefit from the evidence-based 
components of integrated care delivered through study interventions. Moreover, greater 
knowledge about the comparative effectiveness of the interventions or care strategies and 
their impact on patient-centered outcomes will eventually benefit all patients with MCC. For 
these reasons, the Research Team notes that the potential benefits of the knowledge to be 
gained from the proposed study outweigh the minimal risks posed to participants. 
Compensation for Participation 
Total possible compensation for the entire study: $190 
 
 
Timepoint 1: $20 for baseline self-report measures  
Timepoint 2: $20 for 3-month self-report measures 
Timepoint 3: $20 for 6-month self-report measures 

15 
 
Timepoint 4: $40 for 12-month self-report measures 
 
Timepoint 1: $30 for baseline interview  
Timepoint 2: $30 for 3-month interview 
Timepoint 3: $30 for 12-month interview 
Data Safety Monitoring 
Dr. Dan Swayze (Principal Investigator) and Dr. Kevin Kraemer (Co-Investigator) will be 
responsible for data and safety monitoring. They will meet with the Research Team to review 
all data protocols and policies, including informed consent and data confidentiality procedures. 
All key personnel will adhere to the National Institutes of Health policy on education in the 
protection of human subject participants in the conduct of research. Additionally, we will 
convene biannual Data Safety and Monitoring Board meetings to ensure any issues related to 
participant or data safety are fully vetted and addressed. The Data Safety and Monitoring Board 
will sunset in Spring 2022, after all enrollment goals are met. This decision follows the 
recommendation of the board chair and approval of the PCORI program officer, given the 
limited time remaining in the study implementation phase and the low-risk nature of the 
intervention. 
Conflict of Interest 
The study investigators report having no conflicts of interest or financial interests related to the 
research conducted under this contract.  
Publication and Presentation Plans 
We will form a Stakeholder Dissemination Committee in Year 2 of the study. The committee will 
include members from the Stakeholder Advisory Board and the Patient Partners Work Group, 
and the committee will meet biannually to aid in the development and execution of novel 
dissemination practices. Committee members will provide a diversity of perspectives in terms 
of organizational, geographic, and system backgrounds. 
 
 
 
 
 
 
 
 
 
 
 
 

16 
 
References 
 
1. 
Centers for Disease Control and Prevention and Health Promotion. The Power of Prevention. 
National Center for Chronic Disease Prevention and Health Promotion;2009. 
2. 
Warshaw G. Introduction: advances and challenges in care of older people with chronic illness. 
Generations. 2006;30(3):5-10. 
3. 
Kronick RG, Bella M, Gilmer T, Somers S. The Faces of Medicaid II: Recognizing the Care Needs of 
People with Multiple Chronic Conditions. Hamilton, NJ: Center for Health Care Strategies, Inc.; 
2007. 
4. 
Kronick RG, Bella M, Gilmer TP. The faces of Medicaid III: Refining the portrait of people with 
multiple chronic conditions. Hamilton, NJ: Center for Health Care Strategies, Inc;2009. 
5. 
Kasper JD, Watts MOM, Lyons B. Chronic Disease and Co-Morbidity Among Dual Eligibles: 
Implications for Patterns of Medicaid and Medicare Service Use and Spending. Henry J. Kaiser 
Family Foundation; 2010. 
6. 
Lin W-C, Zhang J, Leung GY, Clark RE. Twelve-month diagnosed prevalence of behavioral health 
disorders among elderly Medicare and Medicaid members. Am J Geriat Psychiat. 
2011;19(11):970-979. 
7. 
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic 
illness: the chronic care model. JAMA. 2002;288(14):1775-1779. 
8. 
Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new 
millennium. Health Aff (Millwood). 2009;28(1):75-85. 
9. 
Ofman JJ, Badamgarav E, Henning JM, et al. Does disease management improve clinical and 
economic outcomes in patients with chronic diseases? A systematic review. Am J Med. 
2004;117(3):182-192. 
10. 
Renders CM, Valk GD, Griffin S, Wagner E, Van Eijk J, Assendelft WJ. Interventions to improve 
the management of diabetes mellitus in primary care, outpatient and community settings. 
Cochrane Database of Systematic Reviews. 2000;4(1):CD001481. 
11. 
Tsai AC, Morton SC, Mangione CM, Keeler EB. A meta-analysis of interventions to improve care 
for chronic illnesses. Am J Manag Care. 2005;11(8):478-488. 
12. 
Krause DS. Economic effectiveness of disease management programs: a meta-analysis. Dis 
Manag. 2005;8(2):114-134. 
13. 
Weingarten SR, Henning JM, Badamgarav E, et al. Interventions used in disease management 
programmes for patients with chronic illnesswhich ones work? Meta-analysis of published 
reports. BMJ. 2002;325(7370):925. 
14. 
Badamgarav E, Weingarten SR, Henning JM, et al. Effectiveness of disease management 
programs in depression: a systematic review. Am J Psychiat. 2003;160(12):2080-2090. 
15. 
Gilbody S, Bower P, Fletcher J, Richards D, Sutton AJ. Collaborative care for depression: A 
cumulative meta-analysis and review of longer-term outcomes. JAMA Intern Med. 
2006;166(21):2314-2321. 
16. 
Katon W, Unützer J, Wells K, Jones L. Collaborative depression care: history, evolution and ways 
to enhance dissemination and sustainability. Gen Hosp Psychiatry. 2010;32(5):456-464. 
17. 
van Steenbergen-Weijenburg KM, van der Feltz-Cornelis CM, Horn EK, et al. Cost-effectiveness 
of collaborative care for the treatment of major depressive disorder in primary care. A 
systematic review. BMC Health Serv Res. 2010;10(1):19. 
18. 
Druss BG, Walker ER. Mental Disorders and Medical Comorbidity.: Robert Wood Johnson 
Foundation;February 2011. 

17 
 
19. 
Woltmann E, Grogan-Kaylor A, Perron B, Georges H, Kilbourne AM, Bauer MS. Comparative 
effectiveness of collaborative chronic care models for mental health conditions across primary, 
specialty, and behavioral health care settings: systematic review and meta-analysis. Am J 
Psychiat. 2012;169(8):790-804. 
20. 
Naylor MD, Brooten DA, Campbell RL, Maislin G, McCauley KM, Schwartz JS. Transitional care of 
older adults hospitalized with heart failure: a randomized, controlled trial. J Am Geriatr Soc. 
2004;52(5):675-684. 
21. 
Coleman EA, Parry C, Chalmers S, Min S. The care transitions intervention: results of a 
randomized controlled trial. Arch Intern Med. 2006;166(17):1822-1828. 
22. 
Lorig KR, Sobel DS, Stewart AL, et al. Evidence suggesting that a chronic disease self-
management program can improve health status while reducing hospitalization: a randomized 
trial. Med Care. 1999;37(1):5-14. 
23. 
Lorig KR, Ritter P, Stewart AL, et al. Chronic disease self-management program: 2-year health 
status and health care utilization outcomes. Med Care. 2001;39(11):1217-1223. 
24. 
Wheeler JR. Can a disease self-management program reduce health care costs?: The case of 
older women with heart disease. Med Care. 2003;41(6):706-715. 
25. 
Peikes D, Chen A, Schore J, Brown R. Effects of care coordination on hospitalization, quality of 
care, and health care expenditures among medicare beneficiaries: 15 randomized trials. JAMA. 
2009;301(6):603-618. 
26. 
Kodner DL. All together now: a conceptual exploration of integrated care. Healthc Q. 2009;13:6-
15. 
27. 
McDonald KM, Sundaram V, Bravata DM, et al. Closing the quality gap: a critical analysis of 
quality improvement strategies (Vol. 7: Care Coordination). AHRQ Technical Review No 9. 2007. 
28. 
Alliance for Health Reform. High and rising costs of health care in the US - The challenge: 
Changing  the trajectory. 2012. 
29. 
Bodenheimer T. Strategies to Reduce Costs and Improve Care for High-Utilizing Medicaid 
Patients: Reflections on Pioneering Programs. Robert Wood Johnson Foundation;2013. 
30. 
Lukens E, Murphy L, Blum J. Medicare Advantage Special Needs Plan / Six Plans’ Experience with 
Targeted Care Models to Improve Dual Eligible Beneficiaries’ Health Outcomes. Prepared for 
Association for Community Affiliated Plans. Avalere Health;2007. 
31. 
Medicare Payment Advisory Commission. June 2007 Data Book, Section 3.  
http://www.medpac.gov/chapters/Jun07DataBookSec3.pdf. Accessed October 6, 2014. 
32. 
Robert Wood Johnson Foundation, The Atlantic Philanthropies, Center for Health Care 
Strategies. Super-Utilizer Summit: Common Themes from Innovative Complex Care Management 
Programs. October 2013. 
33. 
Essential Hospitals Institute, America's Essential Hospitals. Facilitators and Barriers to Providing 
Patient-Centered Chronic Disease Care to Patient Populations at Risk for Health and Health Care 
Disparities in Safety Net Settings. With support from PCORI Addressing Disparities 
Program;2013. 
34. 
Nolte E, McKee M. Caring for People with Chronic Conditions: A Health System Perspective 
Maidenhead. In: Nolte E, McKee M, eds: Open University Press; 2008:1-14. 
35. 
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic 
illness: The chronic care model, part 2. JAMA. 2002;288(15):1909-1914. 
36. 
Bayliss EA, Steiner JF, Fernald DH, Crane LA, Main DS. Descriptions of barriers to self-care by 
persons with comorbid chronic diseases. Ann Fam Med. 2003;1(1):15-21. 
37. 
Baumann LC, Dang TTN. Helping patients with chronic conditions overcome barriers to self-care. 
Nurse Pract. 2012;37(3):32-38. 

18 
 
38. 
Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 
1996:511-544. 
39. 
Clark NM, Becker MH, Janz NK, Lorig K, Rakowski W, Anderson L. Self-management of chronic 
disease by older adults: a review and questions for research. J Aging Health 1991;3(1):3-27. 
40. 
Institute of Medicine, Committee on Quality of Health Care in America. Crossing the Quality 
Chasm: A New Health System for the 21st Century. Washington, DC: The National Academy 
Press;2001. 
41. 
Wagner E. Chronic disease management: what will it take to improve care for chronic illness? Eff 
Clin Pract. 1998;1(1):2. 
42. 
California Healthcare Foundation. How Smartphones are Changing Health Care for Consumers 
and Providers. 2010; 
http://www.chcf.org/~/media/MEDIA%20LIBRARY%20Files/PDF/H/PDF%20HowSmartphonesCh
angingHealthCare.pdf. Accessed December 18, 2013. 
43. 
Hurling R, Catt M, De Boni M, et al. Using internet and mobile phone technology to deliver an 
automated physical activity program: randomized controlled trial. J Med Internet Res. 
2007;9(2):e7. 
44. 
Hibbard JH, Greene J, Tusler M. Improving the outcomes of disease management by tailoring 
care to the patient's level of activation. Am J Manag Care. 2009;15(6):353-360. 
45. 
McHorney CA, Ware JE, Lu J, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). 
Med Care. 1994;32(1):40-66. 
46. 
Jensen RE, Potosky AL, Reeve BB, et al. Validation of the PROMIS physical function measures in a 
diverse US population-based cohort of cancer patients. Qual Life Res. 2015;24(10):2333-2344. 
47. 
Ritsner M, Kurs R, Gibel A, Ratner Y, Endicott J. Validity of an abbreviated quality of life 
enjoyment and satisfaction questionnaire (Q-LES-Q-18) for schizophrenia, schizoaffective, and 
mood disorder patients. Qual Life Res. 2005;14(7):1693-1703. 
48. 
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and 
validation of the patient assessment of chronic illness care (PACIC). Med Care. 2005;43(5):436-
444. 
49. 
National Committee for Quality Assurance. HEDIS & Performance Measurement.  
http://www.ncqa.org/tabid/59/Default.aspx. Accessed November 29, 2016. 
50. 
Cohen S, Mermelstein R, Kamarck T, Hoberman HM. Measuring the Functional Components of 
Social Support. Social Support: Theory, Research and Applications: Springer; 1985:73-94. 
51. 
Chinn D, McCarthy C. All Aspects of Health Literacy Scale (AAHLS): developing a tool to measure 
functional, communicative and critical health literacy in primary healthcare settings. Patient 
Educ Couns. 2013;90(2):247-253. 
52. 
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic 
comorbidity in longitudinal studies: Development and validation. J Chronic Dis. 1987;40(5):373-
383. 
53. 
Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvantage and 30-day 
rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161(11):765-774. 
54. 
Singh GK. Area deprivation and widening inequalities in US mortality, 1969-1998. Am J Public 
Health. 2003;93(7):1137-1143. 
55. 
HIPxCHANGE. Area Deprivation Index.  https://www.hipxchange.org/ADI. Accessed November 
28, 2016. 
56. 
Kaiser Family Foundation. Getting into Gear for 2014: Findings from a 50-State Survey of 
Eligibility, Enrollment, Renewal, and Cost-Sharing Policies in Medicaid and CHIP, 2012–2013. 
Washington, DC: Kaiser Commission on Medicaid and the Uninsured;2013. 

19 
 
57. 
Bielaszka-DuVernay C. The'GRACE'model: in-home assessments lead to better care for dual 
eligibles. Health Affairs (Project Hope). 2011;30(3):431. 
58. 
Maeng D, Starr A, Tomcavage J, Sciandra J, Salek D, Griffith D. Can telemonitoring reduce 
hospitalization and cost of care? A health plan's experience in managing patients with heart 
failure. Popul Health Manag. 2014;17(6):340-344. 
59. 
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure 
(PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 
2004;39(4p1):1005-1026. 
60. 
Ware JE, Kosinski M, Bjorner JB, Turner-Bowker DM, Gandek B, Maruish ME. User's manual for 
the SF-36v2 Health Survey. 2nd ed. Lincoln, RI: Quality Metric Inc.; 2008:81-84. 
61. 
Health Quality Ontario. In-home care for optimizing chronic disease management in the 
community: an evidence-based analysis. Ont Health Technol Assess Ser. 2013;13(5):1-65. 
62. 
Leading Age CAST. Telehealth nd Remote Patient Monitoring (RPM). Washington, DC: 
LeadingAge Center for Aging Service Technologies;2013. 
63. 
Druss BG, Rohrbaugh RM, Levinson CM, Rosenheck RA. Integrated medical care for patients with 
serious psychiatric illness: a randomized trial. Arch Gen Psychiatry. 2001;58(9):861-868. 
64. 
Hibbard J, Tusler, M. Assessing activation stage and employing a 'next steps' approach to 
supporting patient self-management. Journal of Ambulatory Care Management. 2007;30:2-8. 
65. 
Harrison PL, Hara PA, Pope JE, Young MC, Rula EY. The impact of postdischarge telephonic 
follow-up on hospital readmissions. Popul Health Manag. 2011;14(1):27-32. 
66. 
Ryvicker M, Feldman PH, Chiu Y-L, Gerber LM. The role of patient activation in improving blood 
pressure outcomes in black patients receiving home care. Med Care Res Rev. 2013;70(6):636-
652. 
67. 
Ko Y, Coons S. An examination of self-reported chronic conditions and health status in the 2001 
Medicare Health Outcomes Survey. Curr Med Res Opin. 2005;21(11):1801-1808. 
68. 
Crabtree BF, Miller WL. Doing Qualitative Research. Sage Publications; 1999. 
69. 
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 
1977;33(1):159-174.