PACT to Improve Health Care in People With Serious Mental Illness (SMI-PACT)
Phase: NA
Conditions: Schizophrenia Spectrum & Other Psychotic Disorders
Sponsor: VA Office of Research and Development

 
 
Study Protocol and Statistical Analysis Plan 
 
 
PACT to Improve Health Care in People with Serious Mental Illness (SMI-PACT) 
 
ClinicalTrials.gov: NCT01668355 
Funder: Department of Veterans Affairs HSR&D Quality Enhancement Research Initiative 
ID: SDP 12-177 
PI: Alexander S. Young, MD MSHS; Department of Veterans Affairs, VISN-22 MIRECC 
 
Version: May 31, 2017 
 
 
 
 
 

 
 
STUDY PROTOCOL 
 
People with mental illness have substantially worse health outcomes than people without mental 
illness.  Several factors contribute to this.  At the patient level, psychiatric symptoms, cognitive disabilities, and 
decreased motivation impair patients’ ability to self-manage medical illness and navigate the healthcare 
system. At the provider level, clinicians often have limited experience, discomfort, and a lack of familiarity with 
evidence-based practices for this population.  At the organizational level, systems often lack protocols for care 
management, shared treatment arrangements, and effective partnerships between primary care (PC) and 
mental health (MH) staff. 
The Department of Veterans Affairs (VA) has undertaken major initiatives to improve primary care, in 
terms of both efficiency and patient-centeredness, via the Patient Aligned Care Team (PACT) model, and in 
terms of its capacity to handle mental health issues, via Primary Care-Mental Health Integration (PC-MHI).  
Primary care in VA has undergone significant transformation under the PACT model, which is derived from the 
Patient Centered Medical Home (PCMH) concept.  Implementation of PACT is continuing, and active efforts 
are ongoing to augment and refine the model.  In addition, since 2008, the PC-MHI initiative has addressed 
Veterans’ mental health conditions by 1) co-locating mental health clinicians in primary care, and 2) making 
care management services available in primary care for common psychiatric disorders. PC-MHI has focused on 
patients with depression and anxiety, with a goal of managing these patients within primary care, and 
providing specialty mental health care for patients who need it most, i.e., those with serious mental illness. 
The next step is to transform the health care of Veterans with serious mental illness (SMI), a population 
that dies many years prematurely, mostly due to medical illnesses.  To address these medical issues, primary 
care clinicians have, in some VAs, been co-located within specialty mental health.  However, this co-location 
has not been implemented widely, and researchers have found that it has inconsistent effects on care processes 
and outcomes. Treatment processes, in particular, need to be improved to address these patients’ multiple 
needs.  Medical care management models do this by improving access to preventive services and treatments for 
people with SMI.  The research evidence for medical care management is strong in that it improves medical 
treatment and outcomes, and can reduce treatment costs in people with SMI.  There is also growing evidence 
supporting the efficacy of the collaborative care model (a cornerstone of PACT) to improve treatment for 
people with SMI. PACT and collaborative care both include care coordination, team-based care, patient-
centered care, an emphasis on access, and use of clinical data to proactively manage populations.  It is likely 
that PACT can be tailored to people with SMI by applying the evidence on medical care management and 
collaborative care.  Our group has experience implementing and evaluating care management and collaborative 
care in people with SMI, and the current project capitalizes on our experience. 
Using existing evidence, we adapt PACT to meet the needs of individuals with SMI (“SMI-PACT”), and 
to implement and evaluate this adapted model in a site-level controlled trial.  This will be a hallmark study for 
the Mental Health QUERI, and will be supported by this QUERI’s SMI Health Workgroup.  This project 
addresses Goal 1 of Mental Health QUERI for FY12-FY15 to “support and enhance implementation of evidence-
based practices, as well as promising and emerging clinical practices that address high priority system needs, 
for Veterans with mental health conditions, including SMI Health.”  This QUERI Step 4, Phase 1-2 trial 
involves researchers, policy makers, and staff at sites in VISN22.  Our specific aims are: 
1. 
Using applicable evidence, adapt the PACT model to the needs of patients with SMI, and partner with 
one VA healthcare center to implement SMI-PACT. 
2. 
In a site-level controlled trial, evaluate the effect, relative to usual care, of SMI-PACT on: 
a.  provision of appropriate preventive and medical treatments; 
b.  patient health-related quality of life, and satisfaction with care; and 
c.  medical and mental health treatment utilization. 
3. 
Using mixed methods, conduct a formative evaluation of usual care and SMI-PACT implementation to  
 
strengthen the intervention, and to: 
a.  assess acceptability of the SMI-PACT model, and barriers and facilitators to its implementation;  
b.  investigate the relationships between organizational context, intervention factors, and patient and 
 
provider outcomes; and 
c.  identify factors related to successful patient outcomes. 
 
Design Overview 
This is a site-level, controlled trial conducted at three VA healthcare centers. VISN22 has expressed 
strong support for the project, along with the capacity and commitment to be involved in an early 

 
 
implementation study. VISN22 has an organizational structure that includes a Mental Health Product Line 
manager who oversees mental health services across the VISN and who will serve as a change agent.  In 
VISN22, the VA Greater Los Angeles Healthcare System (GLA VA) will serve as the intervention site and the 
San Diego VA and Las Vegas VA will continue with care as usual and serve as control sites.  All facilities in 
VISN22 have adopted the PC-MHI model and have implemented PACT, and both GLA VA and LB VA currently 
have primary care available in their mental health clinics.  These structural features serve to support SMI-
PACT.  At the intervention site, one full-time primary care physician will lead a teamlet that will be adapted for 
the SMI population.  The control sites will continue with care as usual. 
We will recruit patients with SMI from the specialty mental health practices at both sites.  All enrolled 
patients will complete a survey at baseline, 7 months (mid-study), and 15 months (final) conducted in-person 
by a trained Research Assessor (RA).  The RA will enter the data via online survey software during the 
interview.  A subset of patients at the intervention site will also complete a 20-minute qualitative interview 
following the 15-month survey.  Clinicians and administrators will be recruited from specialty mental health 
and PACT at both sites.  All enrolled providers will complete a 20-minute survey and 20-minute qualitative 
interview at baseline, 7 months, and 15 months.  The survey will be delivered via a web link to the online survey 
software and the interviews will be conducted in-person by the Evaluation Lead.  All survey data will reside on 
a secure VA server, behind the VA firewall.  All qualitative interviews will be recorded digitally, and the 
recordings and the transcripts will be kept on a secure VA server, behind the VA firewall. 
At the intervention site, one SMI-PACT teamlet will be formed. Working with GLA VA leadership from 
primary care and mental health, it was decided that the SMI-PACT team will fall under the primary care service 
line.  Consistent with the PACT model, the SMI-PACT teamlet will be staffed by one primary care physician, 
one registered nurse care manager, one licensed or vocational nurse, and one clerical associate.  Inter-
professional team members (who work across several teamlets) include representatives from pharmacy, 
dietetics, and rehabilitation.  The teamlet will also have representation from a psychiatrist.  Psychiatrists are 
discipline-specific team members.  This linkage to specialty mental health is essential. The liaison role for the 
psychiatrist ensures that there are PC-MH linkages to address the interrelationship between patients’ physical 
and mental illnesses.  At the control sites, patients will continue to receive care as usual, meaning PC-MHI and 
PACT.  
 
Sample  
 
Patient subjects:  We will enroll a random sample of patients who have diagnoses of schizophrenia, 
schizoaffective disorder, bipolar disorder, recurrent major depression with psychosis, or severe chronic post-
traumatic stress disorder (160 intervention, 160 control).  Patients will not be excluded based on comorbid 
mental or medical diagnoses.  However, there is a subset of individuals at mental health clinics who are 
psychiatrically stable and advanced enough in their recovery that they do not require enhanced supports to 
make effective use of PC.  This will be assessed at baseline by each patient’s clinician, using the Milestones of 
Recovery Scale (MORS) scale.  This scale rates an individual’s ability to self-manage their care.  Patients who 
rate at “early recovery” or “advanced recovery” on this scale will remain with standard PACT, and are not 
eligible for SMI-PACT. 
 
Staff subjects: At each site this will include: members of PACT, members of SMI-PACT (intervention 
site) or mental health integration (control site), providers from the mental health clinic, administrators who 
oversee the mental health clinic (e.g., psychiatry, psychology, nursing, social work, clerks), and administrators 
who oversee primary care. 
 
Settings 
Outpatient Mental Health Clinics:  The sites offer comprehensive mental health services to patients.  
The clinics are multidisciplinary in nature, staffed by psychiatrists, psychologists, nurses, social workers, and 
psychiatric technicians.  Severity of psychiatric illness covers the full spectrum ranging from adjustment 
disorders to severe psychoses and mood disorders.  A full range of both short- and long-term treatment 
services is available: individual, group, and family therapy, psychiatric consultation and evaluation, 
psychological assessment, medication management, social work assessment, and case management. 
Primary Care:  The sites each have PACT established in primary care (e.g., teamlet formation, 
communication SOP in place, daily huddles) with PC-MHI.  As well, both sites have PC co-located in their 
outpatient mental health clinics. 
 

 
 
Recruitment 
Immediately following funding, we will form an “Implementation Team” at the intervention site and 
begin to have monthly in-person meetings.  The Implementation Team will be comprised of (a) PC 
administrators, MH administrators, and systems redesign experts who have been involved in improving clinic 
performance in the recent past and have influence to help with the reorganization necessary to form and tailor 
the SMI-PACT teamlets, and (b) the study PI and Evaluation Lead.  Many of these individuals who have been 
involved in improving clinic performance have already been identified and have been involved in the revision 
of this proposed project so that it meets the needs, capacity, and plans already established for the intervention 
site. The model and project implementation plan has been discussed with the primary care, PACT, and mental 
health leads; the systems redesign lead; the mental health reorganization workgroup; and the hospital director. 
We have collaborated on design of the model, and worked together to consider where it would be located; 
under what service it would fall; how patients would be recruited, assigned, and followed; and how the PACT 
components would be met.  The Implementation Team, and particularly the primary care / PACT lead, has 
begun to identify providers who might have interest the joining a SMI-PACT teamlet.  Interested providers and 
clerks will join the Implementation Team meetings in Year 1 to further their knowledge about the model, 
including roles and functions; answer questions; and to include them in the ongoing discussions of how to 
tailor and implement the four change components, described below.  Ultimately, the Implementation Team will 
name the final teamlet composition based on provider interest and expertise, and clinic resources. The control 
site PI will introduce the study to providers and administrators at his site. 
 
We will draw a random sample of patients using a visit-based sampling method that has previously 
worked well in this population.  A partial HIPAA waiver will be obtained to allow a panel list to be generated for 
each clinic.  At both sites, patients are eligible during their first visit during the 6-month sampling period.  As of 
that visit, patients have a random determination of inclusion or not, based on proportion expected to be 
required at that site.  If the patient is selected for inclusion, then the patient will be asked if they would like to 
participate in the research project, and participate in the informed consent process.  The consent process will 
include use of a brief consent quiz to ensure potential participants fully understand the study.  
 
Intervention Condition: SMI-PACT Care Model 
 
Consistent with the PCMH model and the VA’s PACT adaptation, SMI-PACT creates a system of care 
that maximizes the delivery of preventative and evidence-based medical care for Veterans served in specialty 
mental health.  The PCMH design principles are best summarized in the Group Health PCMH study described 
by Reid and colleagues. Although that description was of PCMH implementation for traditional primary care, 
many of the change components translate logically to SMI-PACT and will be the basis for this study.  The 
implementation process will be guided by the CFIR process components: planning, engaging, executing, and 
reflecting and evaluating. 
 
Planning:  In the planning period, the behavior and tasks for implementing SMI-PACT are developed 
and the quality of the model is assessed and reviewed.  The project has already started by gaining support from 
management at VACO, VISN, and medical center levels.  The VISN22 MIRECC has close ties to the VISN 
leadership and medical center primary care and mental health leadership.  These regional collaborative 
learning sessions are three-day knowledge-based activities, held approximately every 6 months, to train and 
support health care professionals who are implementing the Veteran-centric PACT model throughout VHA.  
Each learning collaborative session presents the most recent examples of implementation from that region as 
well as current theory and VACO priorities.  Attendance at a learning session during the planning phase will 
ensure that the study team is familiar with the most up-to-date information regarding PACT implementation. 
Also during the planning period, the intervention site’s Implementation Team will begin meeting in-
person to discuss the components of SMI-PACT and to begin tailoring the model to their site.  Specifically, 
while the three pillars that comprise the PACT model (access, care management and coordination, and practice 
redesign) are established, strategic planning will shape the implementation of the change components based on 
local structures, priorities, and patient needs.  Minutes will be taken during the Implementation Team 
meetings and will be reviewed and approved the Team. During this period, the staff at both sites will be 
enrolled and the Evaluation Lead will conduct a site visit at each site to complete a diagnostic analysis.  This 
analysis will include a web-based survey and interviews with staff regarding baseline knowledge, attitudes, 
beliefs and practice patterns (see Formative Evaluation below).  At the intervention site, there will be 
additional questions regarding facilitators and barriers to SMI-PACT implementation.  The information gained 
from the diagnostic analysis will be promptly analyzed, summarized, and conveyed to the Implementation 

 
 
Team to inform strategic planning and tailoring of the model. 
 
The Implementation Team will work specifically on how to implement four overarching change 
components:  Structural Changes, Point-of-Care Changes, Patient Outreach Changes, and Management 
Changes; and they will strategize as to how to address patient-, provider-, and organizational-level barriers and 
facilitators.  Probable modifications to PACT to accommodate the needs of the SMI population are as follows: 
 
Structural Changes:  1) A teamlet will be formed with a suggested composition of one FTE primary care 
provider (physician or nurse practitioner), one FTE registered nurse care manager, one FTE vocational nurse, 
and one FTE clerical associate.  Inter-professional team members (who work across several teamlets) include 
representation from clinical pharmacy, dietetics, and rehabilitation.  Each teamlet will include liaison from the 
three psychiatrists who are providing specialty mental health care to the patients.  2) Each teamlet member will 
function to his/her highest capability.  3) Teamlet members will be located, when possible, in the same clinic to 
facilitate communication and coordination.  4) Teamlet size will be smaller (n=500), in line with the VHA 
Handbook 1101.02 directives for specialty PACT, and therefore will be able to accommodate an increase in the 
standard visit length from 20 to 30 minutes.  This extra time is needed to allow for increased discussion of 
complex medical comorbidities.  5) A call system will be implemented to direct calls for patients in SMI-PACT 
to the team’s clerical associate who will be the first point of contact.  6) Time will be dedicated for “desktop 
medicine” to respond to calls, contact other clinics and labs, and respond to secure messaging (VA secure email 
is being deployed nationally for patients).  7) Clerical associates will schedule routine services based on 
algorithms for appropriate care using Clinical Dashboard software (e.g., mammograms, colorectal cancer 
screens, influenza vaccinations). 
 
Point-of-Care Changes:  1) The registered nurse will be the medical care manager for the panel of 
patients and will be the “communications hub.”  This provider will be in close communication with the clerical 
associate (who receives all calls) and will facilitate coordination within the team.  2) Patients will be active 
collaborators in their treatment plan, which will be jointly “owned” by the PACT teamlet and patient.  3) If 
there is a conflict between the patient’s psychiatric illnesses and treatments and the patient’s medical illnesses 
and treatments (e.g., weight gain from antipsychotic medications), the PACT provider will work with the 
psychiatrist to resolve the conflict.  4) Patients will be encouraged to use secure email to contact the team, book 
appointments, and change appointments.  5) Patients will be encouraged to use phone or secure email as 
alternative or to complement in-person visits with goal of 1/3 phone, 1/3 email, 1/3 in-person visits. 
 
Patient Outreach Changes:  1) Scripted messages will be delivered to each patient on the panel to 
describe the new care model, clinicians on their team, their role in supporting this change, and potential care 
improvements. 2) Patients who are assigned a new PC clinician as part of this reorganization will receive a 
welcoming call within one month of the new assignment to encourage a strong clinician-patient working 
relationship.50  3) Teamlets will routinely review Chronic Disease Panel reports from the Clinical Dashboard to 
track and address any deficiencies.  Teamlet staff will contact patients to address issues (e.g., outstanding 
retinal exam, elevated LDL-C).  4) Patients will be encouraged to join “group visits” comprised of individuals 
with similar medical concerns.  5) There will be outreach to pharmacists for consultation on patients who are 
not well-controlled on their medications.  6) Lab tests will be systematically reviewed and results reported to 
patients either via secure email from the clerical associate (normal results) or via phone from the nurse 
(abnormal results). 
 
Management Changes:  1) Time will be dedicated to a daily “huddle” for the teamlet to discuss 
outstanding issues, problem solve, plan, and divide up responsibilities.  2) The Clinical Dashboard will be 
available to all teamlet members including the clerk.  3) A Memorandum of Understanding (MOU) will be 
established between the SMI-PACT teamlet and other PACT teams regarding assignment of patients based on 
their psychiatric diagnosis and functioning.  4) Same day and walk-in appointments will be available. 
 
Engaging:  During the period of engagement, the project begins to attract and involve appropriate 
individuals in the implementation and use of the intervention through social marketing, education, modeling, 
and training.  In addition to teamlet members, we will invite PACT experts and clinicians to several 
Implementation Team meetings to serve as SMI-PACT opinion leaders, redesign models, and expert resources.  
These meetings will continue to focus on the adaption and implementation of the model. 
Once the teamlet is finalized and strategic planning regarding implementation is completed, the Study 
PI will train staff of the mental health and primary care clinics in the SMI-PACT model, protocols to be 
implemented, and a discussion of barriers and facilitators.  We will use relevant training materials from PACT, 
which have been shared with us by Dr. Davies who leads the PACT collaborative learning sessions nationally.  
The care manager for the intervention site will be trained in treatment guidelines for the prevention and 

 
 
medical care targets of the intervention. We will send the SMI-PACT registered nurse care manager and 
primary care provider to a PACT collaborative learning session during the engaging period.  At this learning 
session, these two attendees will work to understand how to integrate access to specialty care and discipline-
specific providers in Medical Center PACT sites of care (1 of the 5 learning objectives of these sessions).  Since 
the staff will have worked for a period on planning for their site, they will be able to focus more specifically on 
methods related to tailoring PACT for specialty populations. 
In addition to the training during this period, study staff will also meet with the SMI-PACT teamlet, 
PACT leadership, and the systems analyst who manages the Clinical Dashboard for the site.  At this time we 
will also finalize the MOU between SMI-PACT and PACT, regarding patient referrals between the groups.  
Communication routines within the SMI-PACT teamlet itself, and between the teamlet and its inter-
professional (clinical pharmacy, dietetics, and rehabilitation) and multidisciplinary partners (e.g., psychiatrist) 
will be established, practiced, adjusted, and routinized.  Throughout this period, the Evaluation Lead will take 
ethnographic field notes as part of the formative evaluation. 
Throughout the engaging period and the remaining implementation stages, the Implementation Team 
will work to foster a supportive organizational culture for the SMI-PACT team and address environmental 
constraints to model implementation.  The Implementation Team will develop marketing handouts for patients 
regarding the new care model, contact information, and some “tips” including names and contact information 
for teamlet members, encouragement to use phone/email, and availability of same-day appointments.  Team 
members will be educated regarding the importance of this activity, their role in the project, and tools they can 
use to motivate providers and facilitate implementation. SMI-PACT opinion leaders will be activated. 
 
Executing:  In this period, the intervention is implemented. During this period, the Implementation 
Team will focus on barriers and facilitators to implementation, with an eye to sustainability.  Teamlet providers 
will be encouraged to take advantage of local PACT learning collaboratives to support implementation and 
change efforts.  The registered nurse care manager will oversee day-to-day SMI-PACT teamlet implementation 
activities to ensure that care is coordinated, continuous, and appropriate.  In order to do this, he/she will rely 
on Chronic Disease Panel reports from the Clinical Dashboard to monitor progress towards the prevention and 
medical care targets.  The care manager will use supportive strategies and reminders to encourage teamlet 
clinicians and teamlet clerks to address access delays, outstanding preventive and medical care issues, and 
teamlet processes (e.g., cost tracking, team huddles, and minutes for any meetings).  The SMI-PACT care 
manager will report any barriers to implementation that occur between Implementation Team meetings.  The 
PI will assist in problem-solving and use of quality improvement techniques, but ultimately the 
Implementation Team will guide implementation of the model and be responsible for addressing barriers.  
Throughout implementation, periodic and targeted feedback to the Implementation Team will be provided and 
continued tailoring of the model will occur accordingly.  The Team will make use of available financial incentive 
and performance rewards for clinics and providers to motivate implementation. 
 
Reflecting and Evaluating: Damschroder and colleagues explain that in this final stage quantitative and 
qualitative feedback about the progress and quality of implementation is provided. In the proposed 
demonstration trial, we will undertake an iterative process of reflecting and engaging, whereby we will provide 
periodic and targeted feedback to the Implementation Team in addition to regular personal and team 
debriefings about progress and experience.  This feedback will be based in part on mid-implementation data 
collected from patients (survey) and providers (survey and interviews). 
 
Usual Care Condition 
The comparison condition will consist of treatment as usual. The control sites will continue with use of 
PC-MHI and PACT.  
 
Data Collection, Variables, and Measures 
Aim 2 Data Collection 
For the impact evaluation, data will be obtained from patient interviews, chart review, Data 
Warehouses, and national VA data systems.  During the intervention, current data on utilization of services is 
provided to the intervention site to inform implementation.  This is done using the Dashboard, which draws 
from the Regional Data Warehouse, plus chart reviews at baseline and mid-study.  At the end of the 
intervention, data on utilization history is required for both intervention and control sites, and this will be 
obtained using chart review and the VA Informatics and Computing Infrastructure (VINCI).  SAS Programmers 
from the HSR&D Center of Excellence for the Study of HealthCare Provider Behavior, including the data 

 
 
analyst on this project, have been working with the VINCI group since September 2011 to determine how this 
new resource for access to VA data will be able to serve our research project needs.  If VINCI is not fully able to 
meet the project’s needs, data will be obtained from the VA Decision Support System (DSS) and Corporate 
Data Warehouse (CDW).  Table 4 below summarizes the measurement plan for the impact evaluation. 
 
Patient Survey. Baseline, 7-month, and 15-month follow-up research interviews provide data for intervention 
impact and cost determination.  At both sites, RAs will conduct in-person interviews with patients.  They will 
be trained in person to a high level of reliability by VISN22 MIRECC staff who are expert trainers.  The study PI 
and control site PI will oversee data collected at their site.  It will not be possible to blind RAs to the fact that 
clinics are intervention or control.  To reduce bias, RAs will have minimal contact with staff involved with 
implementation. 
 
Demographics and psychiatric illness. At baseline, demographic data including age, race, and ethnicity 
and psychiatric illness history will be collected from the patient. 
 
Chronic medical illnesses.  The Chronic Conditions Survey will be completed at baseline and assesses 
presence or absence of 17 of the most common chronic illnesses.  It has been used in several community studies 
with patients with SMI. 
 
Severity of comorbid medical illness.  For each patient we will calculate their Charlson Age- 
Comorbidity index score.  Each patient’s index score is the total of the patient’s comorbid conditions weighted 
plus age.  A higher score indicates a more severe comorbid condition.  For example, diabetes is weighted a 1 
while moderate to severe liver disease is given a 3. Age is added as 0 for anyone 40 years old or younger, and 
each decade over 40 adds 1 point to the index score. 
Patient Ability and Engagement in Care.  The Milestones of Recovery Scale (MORS) assesses the 
extent of a patient’s mental health recovery and has good test-retest reliability and strong validity. A MORS 
score ranges from 1 (extreme risk) to 8 (advanced recovery) and is based on three underlying dimensions: 1) 
patient’s level of risk, 2) patient’s strength of engagement with healthcare services (but not quantity of service), 
and 3) the patient’s level of skills and supports.  Scores on the MORS are responsive to change over time, 
reflecting the recovery process. At baseline, all patients will be assessed by the patient’s clinician using the 
MORS.  Those patients who have a baseline MORS score of 7 or 8 will continue to be assigned to PACT.  Their 
MORS score indicates that they have the ability to navigate PC, advocate for themselves, and manage their own 
care.  Those individuals whose MORS score is 6 or below will be managed in SMI-PACT. 
 
The patient survey will also include the measures described below in Table 5. 
 
Aim 2a Measures 
 
Prevention composite score: We will calculate composite scores for preventive services based on the 
Office of Quality and Performance Technical Manual. Obesity screen data will be drawn from VINCI.  The 
remaining measures that compose the prevention score will be gathered via chart review at baseline, mid-study 
and follow-up.  We will construct a chart abstraction instrument for use with CPRS abstractions, test it for 
reliability and validity, and train RAs to a high level of reliability.  The instrument will be developed and 
administered using the web-based survey system. 
 
Table 4.  Measurement of Outcomes for Impact Evaluation (Aim 2)
Dependent Variable 
Measures 
Source of Data 
Aim 2a:  Provision of appropriate preventive and medical treatment services 
Composite Prevention 
• Pneumococcal immunization age 65 and older, ever received
• Influenza vaccination 50-64 years of age 
• Influenza vaccination 65+ years of age 
• % of women age 50-69 screened for breast cancer 
• % of women age 21-64 screened for cervical cancer in the past 
three years 
• % of patients receiving appropriate colorectal cancer screening 
• % of patients screened for obesity  
Data Warehouses; VINCI; 
Chart Reviews 

 
 
Composite Diabetes Mellitus  
• LDL-C < 100 mg/Dl
• HbA1c Annual 
• HbA1c >9 or not done (poor control) in past year 
• Retinal Exam, timely by disease 
• LDL Measured 
• BP < 140/90 
• Nephropathy screening test or evidence of nephropathy  
Data Warehouses; VINCI
Composite Cardiovascular 
• Hypertension and BP < 140/90
• LDL-C Measured 
• LDL-C < 100 mg/Dl   
Data Warehouses; VINCI
Aim 2b:  Patient mental and physical quality of life, and satisfaction with care 
Health Related Quality of Life 
SF-12V mental and physical component summary scores
Patient survey
Satisfaction with Care 
CSQ-8 
Patient survey
Aim 2c:  Healthcare utilization and costs 
Healthcare Utilization 
• SURF 
• Pharmacy and lab data 
Patient survey; VINCI
Healthcare Costs 
• Treatment utilization
• Physical Resources 
• VA unit costs 
• Non-VA unit costs 
• SURF; Time Logs; VINCI
• VA accounting records 
• DSS-NDE; Salary tables 
• Appropriate pricing 
schedule (e.g., Medicare fee 
schedule) 
 
 
Diabetes Mellitus and Cardiovascular composite scores:  We will calculate composite scores for 
diabetes and cardiovascular disease based on the Office of Quality and Performance Technical Manual. The 
data that compose the composite calculations will be drawn from VINCI. 
 
Composite calculations:  All composite scores are as follows:  for the numerator, each of the clinical 
indicators that contribute to a composite is scored as 1 or 0 (pass or fail, respectively).  The composite 
numerator is the sum of scores for all measures that contribute to the composite.  The denominator is a count 
of all the measures that contribute to the composite.  Composite scores of the quality of preventive and chronic 
care have been shown to be reliable and valid and useful as a comprehensive assessment. 
 
Aim 2b Measures 
 
Health-related functioning: The Veterans version of the widely used SF-12 will be used to assess self-
perceptions of general health functioning across multiple dimensions (including general, physical, and mental 
health functioning). The SF-12V has good internal consistency, stability, and concurrent validity in outpatients 
with SMI. 
 
Satisfaction with Care: The Client Satisfaction Questionnaire (CSQ-8) is a brief, eight-item self-report 
questionnaire used to measure patient satisfaction with services. The questionnaire has been widely used in 
research and service settings and demonstrates high internal consistency as well as consumer acceptability. 
 
Aim 2c Measures 
 
Healthcare Utilization:  The Service Use and Resources Form (SURF) will be completed by patients at 
all three time points. SURF has been successfully used to assess costs and service utilization in numerous 
multi-site studies with patients with SMI, including CATIE. For the present study, we will use the SURF 
questionnaire to collect information on inpatient and outpatient services utilization for psychiatric and medical 
issues, including type of facility and type of provider contact, for visits both inside and outside VA settings.  
Additional data on services utilization, lab tests, and pharmacy records will come from the VA Decision 
Support System National Database Extracts (DSS NDEs), administrative databases that include information 
for all healthcare use within the VA healthcare system.  We will also compare DSS records of outpatient events 
and inpatient admissions to data from the SURF, and integrate information from both sources when there are 
obvious omissions in data from the SURF. 
 
Healthcare Costs:  Using microcosting methods that are appropriate for the VA context, we will 
document the cost of the SMI-PACT intervention in addition to the costs of all other health care utilized by 
patients during the course of the study, and will similarly document all healthcare costs for patients assigned to 

 
 
usual care.  We will document both the direct cost of the SMI-PACT intervention as well as the overall 
incremental resource costs (i.e., the additional costs) of SMI-PACT relative to usual care (the alternative 
minimal intervention) over a 15 month period.  The direct costs of SMI-PACT include the value of all resources 
used, whether or not some of these costs are redundant in usual care.  The direct cost estimate will provide an 
upper bound estimate of SMI-PACT program implementation cost.  Some of these implementation costs might 
be offset by a reduction of expenditures on redundant services and by preventing the need for other more 
expensive healthcare services in the future.  
 
Intervention Costs:  The total direct cost of SMI-PACT will be estimated by multiplying the quantities of 
all resources used during the delivery of this intervention by the unit costs (i.e., prices) of each type of resource 
and then summing all over all resource categories.  Enumerated resources will include labor costs as well as all 
intervention-related diagnostic (i.e., lab) tests, office and other medical supplies, equipment depreciation, and 
all overhead expenses.  Transportation, phone calls, and other ancillary services needed in order to deliver the 
SMI-PACT intervention will be included in cost estimates. We will estimate the costs of these resources using 
standard methods for approximating of opportunity costs, including salaries plus fringe benefit costs for labor, 
depreciation plus rental cost methods for estimating the cost of capital equipment, time-discounting for 
adjustment of future costs to reflect present value, and imputed overhead expenses.  Average cost per patient 
will be estimated by dividing total direct costs by the number of patients. 
   
Tracking resource use: Resources used in SMI-PACT will be relatively easy to track, as they will be 
comprised mostly by the time of personnel who are involved directly in intervention planning and operation.  
Each provider on the SMI-PACT team and the SMI-PACT clerk will keep a written record (i.e., a time log) of 
time spent in service of a patient (e.g., in clinical encounters and in completing paperwork/CPRS notes), and 
will also record the mode of contact (telephone or in-person), patient involvement (with or without patient), 
and other persons contacted on behalf of the patient.  Time spent in training sessions and in daily huddles will 
be tracked on a time log maintained by the clerk.  Salary estimates plus fringe benefits expenses will be 
obtained from programs and used in estimating personnel costs.  Study interventions also will require use of 
physical resources—handouts, office space and office supplies, expenses tracked using study receipts and VA 
accounting records.  Lab tests recorded for dates when the patient had an SMI-PACT encounter will also be 
included in the direct costs of the intervention, as such tests are conceptually integral to the delivery of PACT 
services.  Lab tests and their costs are recorded in DSS NDEs. 
Other Healthcare Costs:  For each patient,  we will include in our overall cost estimate costs for other 
inpatient, outpatient, emergency department, pharmaceutical and labs utilization during the 15 months 
starting with the intervention baseline and during the 6-months prior to the intervention baseline.  To avoid 
double-counting, we will exclude from these costs the direct costs of the intervention.  The costs of most 
outpatient encounters and inpatient admissions will be estimated by multiplying visit counts or bed days by a 
unit cost amount (or price) that is representative given the type of service provided, the duration of the service, 
and the provider type.  Prices will be obtained from VA administrative sources unless a service is not available 
within the VA healthcare system.  Costs of non-VA services will be derived from an appropriate pricing 
schedule, such as the Medicare fee schedule.  For prescription medication costs and lab tests, we will use 
Decision Support System (DSS) cost estimates. DSS estimates are based on VA acquisition costs and workload.  
Prescription medication costs are adjusted for the pharmacy dispensation costs.  We will check pharmacy and 
lab cost data for outliers, which can occur due to key entry errors or unusual fluctuations in acquisition prices.  
Usual Care Costs: We will assess the costs of usual care over the 15 month period beginning with study 
enrollment using utilization information from the SURF and DSS coupled with information on unit costs.   
 
Aim 3 Data Collection 
As with other studies of the PCMH model, mixed methods evaluation is necessary in order to 
understand the process of implementing a new care model that entails major shifts in roles, responsibilities, 
structures, and clinical care.  In order to understand this process, it is necessary to evaluate the pre-
implementation conditions at both the intervention and control sites.  Usual care must be thoroughly 
characterized in order to establish the magnitude and dimensions of the changes that occur with the 
introduction of a new care model (e.g., SMI-PACT).  In this evaluation, usual care will be assessed at baseline, 
mid-, and post-implementation via quantitative measurement of the presence of Chronic Care Model 
components (described above), and qualitative assessment of structures and existing delivery models of care, 
as well as knowledge of and attitudes toward the medical home model (and the specific PACT model). 

 
 
Subsequently, implementation processes and patient- and provider-level factors will be evaluated in numerous 
ways.  Table 5 summarizes the measurement plan for the formative evaluation. 
 
Table 5.  Mixed Methods  Evaluation Measures (Aim 3)
Domain of Inquiry 
Measures 
Source 
Usual care versus SMI-
PACT (organizational 
contexts) 
 Assessment of Chronic Illness Care (ACIC) 
 Patient Assessment of Chronic Illness Care (PACIC) 
 Semi-structured qualitative interview at 3 time-points 
(baseline, mid-study, final follow-up) 
 Providers 
 Patients 
 Providers 
SMI-PACT 
implementation 
 Rogers Adoption Questionnaire 
 Degree of implementation of model components 
 Providers 
 Providers 
Acceptability 
 Semi-structured qualitative interview at 3 time-points 
 Semi-structured qualitative interview at follow-up 
 Providers 
 Patients 
Barriers and facilitators 
to implementation 
 Ethnographic field notes (throughout implementation) 
 Implementation Team meeting notes 
 Observational logs 
 Semi-structured qualitative interview at 3 time-points 
 Evaluation Lead 
 Research Assistant 
 Providers 
 Providers 
Provider outcome 
 Maslach Burnout Inventory 
 Providers 
Patient factors 
 Behavior and Symptom Identification Scale (BASIS) 
 Ambulatory Care Experiences Survey (ACES) 
 Interpersonal Support Evaluation List (ISEL) 
 Medication Possession Ratio (MPR) 
 Patients 
 Patients 
 Patients 
 VINCI 
 
Aim 3 Measures 
Usual Care Versus SMI-PACT 
 
The 28-item Assessment of Chronic Illness Care (ACIC) assesses strengths and weaknesses in care for 
chronic illness, specifically community linkages, self-management support, decision support, delivery system 
design, information systems, and organization of care.  Scores range from 0-11 with categories defined as 
follows: 0–2 (little or no support for chronic illness care); 3–5 (basic or intermediate support for chronic illness 
care); 6–8 (advanced support); and 9–11 (optimal, or comprehensive, integrated care for chronic illness). 
Subscale scores for the six areas are derived by summing the response choices for items in that subsection and 
dividing by the corresponding number of items. In the original study, teams were encouraged (but not 
required) to complete the ACIC together.  However, in this study we will ask providers to complete it 
individually and results will be shared with the Implementation Team to confirm and discuss findings, with 
particular attention to components that appear to need strengthening. 
 
The 20-item Patient Assessment of Chronic Illness Care (PACIC) assesses the extent to which patients 
with chronic illness receive care that aligns with the Chronic Care Model, i.e., care that is patient-centered, 
proactive, planned and includes collaborative goal setting, problem-solving, and follow-up support. In the 
original study, the PACIC demonstrated moderate test-retest reliability (r = 0.58 during the course of 3 
months) and was correlated moderately, as predicted (r = 0.32-0.60, median = 0.50, P < 0.001) to measures of 
primary care and patient activation.  In a subsequent study of over 4000 patients with chronic conditions, 
PACIC scores were positively associated (odds ratios ranging from 1.20 to 2.36) with measures of use of self-
management resources, self-management behaviors such as regular exercise, patient rating of their health care, 
and quality of life. 
 
SMI-PACT Implementation 
 
The 20-item Rogers’ Adoption Questionnaire assesses three factors that potentially affect rate of 
adoption: complexity, relative advantage, and observability.  Each scale has good internal consistency 
(Cronbach Alpha =.83, .88, .77 respectively). 
 
Degree of implementation of SMI-PACT model components will be rated as to the degree of 
implementation (structural changes, point-of-care changes, patient outreach changes, and management 
changes).  Ratings will be made on a 4-point scale (not at all implemented, partially implemented, mostly 
implemented, fully implemented).  This rating scale is consistent with that used in the ACIC, described above. 
 

 
 
Provider Outcome 
 
The Maslach Burnout Inventory (MBI) is a standard tool used to measure aspects of workplace stress 
and has three subscales:  emotional exhaustion, depersonalization, and personal accomplishments.  The MBI 
has been used in several studies of the PCMH.  In the Group Health demonstration, provider burnout was 
reduced at 12 months and continued to decrease at two years, particularly as gauged by emotional exhaustion. 
 
Patient Factors 
The Behavior and Symptom Identification Scale – Revised (BASIS-R) will be administered at all three 
time-points.  It measures psychopathology, and provides covariates for analyses.  We will use the following 
scales: psychosis (4 items), depression/daily functioning (6 items), interpersonal functioning (5 items), and 
alcohol/drug use (4 items). The BASIS-R is widely used, and research supports its reliability and validity. 
 
The Ambulatory Care Experiences Survey (ACES; Short Form) is a brief patient-completed 
questionnaire that evaluates patients' experiences with a specific physician and that physician's practice.  The 
ACES uses the Institute of Medicine definition of primary care as its underlying conceptual model for 
measurement, but was designed to evaluate any sustained clinician-patient relationship, irrespective of clinical 
specialty.  It is sensitive to changes in patients’ experiences of care over time and demonstrates excellent 
psychometric properties. As a result, it has become a national standard for measuring patients' experiences 
with individual clinicians and their practices.  A short-form version of ACES is available and has been widely 
used since 2003.  The Short Form will be used to collect data on 5 domains:  access, quality of doctor-patient 
interactions, shared decision-making, coordination of care, and helpfulness of physician office staff.  
 
The Interpersonal Support Evaluation List (ISEL) is a widely used 30-item instrument that assesses 
perceptions of social support, including: belonging, appraisal help, tangible support, and self-esteem support. 
The ISEL has been subjected to extensive reliability and validity testing and has shown to be internally 
consistent and valid with the general population as well as with people with SMI.  
 
A Medication Possession Ratio (MPR) will be calculated from 6 months of pharmacy data to assess 
medication adherence.  MPR assesses the extent to which dispensed medications provide coverage for a given 
interval.  It has been shown to be a valid measure of adherence in people with SMI. 
 
Aim 3 Qualitative data collection 
 
Semi-structured qualitative interviews will be conducted at baseline, mid-study, and final follow-up 
with intervention and control site staff.  Baseline assessment will include an examination of usual practices, 
and knowledge, attitudes, and behaviors regarding medical care of patients with SMI.  Staff at intervention 
sites will be asked about their expectations for SMI-PACT, and anticipated barriers and facilitators to 
implementation.  The mid-study and final follow-up will interviews will assess: (1) usual care versus SMI-
PACT; (2) barriers and facilitators to implementation of SMI-PACT (intervention staff); (3) provider 
perceptions of acceptability of SMI-PACT (intervention staff).  Members of the Management Team will also be 
interviewed periodically to capture their perceptions of intervention characteristics, outer setting, inner setting, 
and process (i.e., key CFIR constructs).  Semi-structured interviews will also be conducted with a subsample of 
patients at final follow-up to assess perceptions of acceptability of and satisfaction with SMI-PACT. 
 
Ethnographic field notes will be taken by the Evaluation Lead throughout implementation to capture 
aspects of the inner setting and otherwise unmeasured aspects of usual care.  The care manager on the SMI-
PACT teamlet will be trained to record a more moderate version of field notes in the form of observational logs, 
and will record noteworthy occurrences and events related to barriers and facilitators to implementation.  
Notes and minutes will be recorded for all demonstration-related meetings (including trainings) and 
conference calls. In addition, substantive emails and other communications regarding SMI-PACT 
implementation will be archived and analyzed. 
 
STATISTICAL ANALYSIS PLAN 
 
Formative Evaluation 
Investigators from 9 different previous PCMH evaluations convened and offered suggestions for future 
PCMH studies: 1) look critically at models being implemented and identify aspects requiring modification;  2) 
embed qualitative and quantitative data collection to detail the implementation process;  3) capture details 
concerning how PCMH components interact with one another over time;  4) understand and describe how and 
why physician and staff roles do or do not evolve;  5) identify the effectiveness of individual PCMH components 

 
 
and how they are used;  6) capture how primary care practices interface with other entities such as specialists, 
hospitals, and referral services; and  7) measure resources required for initiating and sustaining innovations. 
Our mixed methods evaluation will follow these suggestions. 
In the Planning stage, pertaining to suggestion #1, we will conduct a careful diagnostic analysis of 
treatment as usual at both intervention and control sites. This baseline analysis will incorporate:  1) 
quantitative measurement of (a) chronic care model implementation using the ACIC, (b) degree of 
implementation of PACT model components, (c) adoption factors using Rogers’ Adoption Questionnaire, and 
(d) provider burnout using the MBI; and, 2) qualitative assessment via semi-structured interviews and site 
visits.  As Nutting and colleagues noted in the PCMH National Demonstration Project, it is critical to assess 
which components of the model are in place at baseline in order to define what changed as a result of 
implementation of an alternative care model. In addition, providers at the intervention site will be asked about 
their expectations of the SMI-PACT intervention.  Patients will complete a baseline assessment of the extent to 
which they receive chronic illness care (using the PACIC), and will provide information regarding their 
symptoms (using the BASIS-R), their experiences of care (using the ACES), and their social support networks 
(using the ISEL).  These measures are included in the patient survey described above.  For each patient we will 
calculate their MPR to assess medication adherence.  These measures of patient factors enables analyses of 
patient-level covariates that may help to determine which patients are best served by SMI-PACT versus PACT. 
In the Engaging and Executing stages, assessment of degree of implementation will continue on a 
quarterly basis, and observational logs and data collection of documents (minutes, notes, emails, etc.) will be 
ongoing.  Particular attention will be paid in the Engaging stage to suggestion #7 with regard to measuring 
resources required for initiating innovations.  In the Executing stage, we will focus on suggestions #1, 3, 4, and 
5.  Assessment of how different model components interact, physician and staff role changes, and effectiveness 
of individual components will be accomplished via mid-implementation semi-structured interviews at the 
intervention sites, along with the quarterly ratings.  Furthermore, usual care at the control site will be re-
investigated mid-implementation in order to capture any global changes in PACT implementation or co-located 
primary care at the site (e.g., changes that might be relevant to PACT for special populations). 
We consider the Reflecting & Evaluating stage to occur, for the most part, concurrent to the other 3 
phases.  In other words, within each stage, reflecting and evaluating will occur, with periodic and targeted 
feedback to the Implementation Team.  For example, we will use results from the ACIC and PACIC to identify 
strengths and weaknesses in SMI-PACT model implementation, and will tailor implementation efforts 
accordingly.  This continual bi-directional feedback and tailoring model is consistent with the PCMH National 
Demonstration Project (NDP) approach. In the summative evaluation at final follow-up, reflecting and 
evaluating will include re-measurement of CCM implementation (ACIC and PACIC), adoption (Rogers 
Adoption Questionnaire), burnout, and patient factors (BASIS-R, ACES, ISEL), along with interviews with 
providers and other key stakeholders.  In addition, a subsample of patients (n=30) from the intervention site 
will complete a brief interview regarding their experiences in SMI-PACT.  This interview will be designed to 
assess acceptability of and satisfaction with care delivery under SMI-PACT.  We feel that it is important to have 
a qualitative assessment of satisfaction (along with the CSQ described above) due to the finding in the NDP 
that patient ratings of satisfaction did not increase, and in some domains decreased slightly. Qualitative 
investigation of patient satisfaction may yield insight into non-measured aspects of their experiences and will 
be triangulated with the quantitative data from the CSQ. 
 
Data Analysis Plan 
 
In this section we describe our analytic methods for key hypotheses derived from our specific aims.  We 
first present the outcome analyses (Aim 2), and then the mixed methods evaluation analyses (Aim 3). 
 
Aim 2.  In a site-level controlled trial, evaluate the effect, relative to usual care, of SMI-PACT on: 
a.  provision of appropriate preventive and medical treatments; 
b.  patient health-related quality of life, and satisfaction with care; and 
c.  medical and mental health treatment utilization 
 
 
The main data analytic strategy for the site-level controlled trial will be a generalized linear mixed 
model (GLMM).  The advantage of the GLMM is that it allows us to capitalize on the information provided by 
the longitudinal structure of the data, and that it provides a unified framework that will allow us to integrate 
person level time varying and time invariant covariates if necessary.  Additionally, the GLMM is robust to 

 
 
missing data under Rubin’s “missing at random” assumptions and will allow us to include cases with missing 
test occasions while maintaining unbiased parameter estimates. 
 
The main outcome is service utilization and will be modeled as a within subject design, the baseline 
measurement as a person level covariate, and PACT participation as a between subject variable.  To model this 
dichotomous outcome variable, we will use the logistic link function in the GLMM to model each participant’s 
likelihood of service utilization.  We hypothesize that at the 15-month follow-up, the treatment group will show 
significantly higher likelihood of service utilization than the control group. 
 
The outcomes measured by the Quality of Life, Patient Satisfaction, and health functioning composite 
scales will be modeled using the same strategy, but will use the canonical link function to appropriately model 
the continuous nature of these variables. 
 
To complement these analyses we also plan to do a range of exploratory analyses.  We will explore if 
patient demographics are associated with the efficacy of PACT by evaluating if modeling these covariates as 
associated with the individual trajectories over time will improve model fit.  
 
Aim 3. Using mixed methods, conduct a formative evaluation of usual care and SMI-PACT implementation to  
strengthen the intervention, and to: 
a.  assess acceptability of the SMI-PACT model, and barriers and facilitators to its implementation; 
b.  investigate the relationships between organizational context, intervention factors, and patient and 
provider outcomes; and 
c.  identify patient factors related to successful patient outcomes. 
 
 
To assess implementation of the care model, we will collect both quantitative and qualitative data.  
Qualitative methods enable the researcher to gain a thorough understanding of the context and interpretation 
of variables.  Qualitative methods are strongly recommended for evaluation of PCMH demonstrations.  
 
All semi-structured interviews will be digitally recorded and sent to Keystrokes, a medical transcription 
company, via a secure website for verbatim transcription. We have used Keystrokes in two previous studies, 
with great success.  Transcripts will be reviewed and edited for accuracy by Drs. Hamilton and Cohen.  Analysis 
of the qualitative data will be conducted using ATLAS.ti, a qualitative data analysis software program that 
allows for fluid “interaction” of data across types and sources.  Initially, a top-level codebook will be developed 
for the baseline interviews based on the semi-structured interview guide. Using a constant comparison analytic 
approach, this codebook will be elaborated upon based on emergent themes, and it will be adjusted as each 
round of interviews is reviewed.  Interviews will be compared within each clinic, across clinics, and over time.  
Additional sources of qualitative data (i.e. meeting minutes, field notes, and archival information) will also be 
included in the data set and will be coded separately and in relation to the interview data.  These multiple 
approaches and groupings are easily facilitated within the software program, which has the capacity to group 
data in multiple ways and which allows the qualitative researchers maximum flexibility in negotiating a 
complex narrative dataset. 
 
In the pre-implementation transcripts, we will identify commonly shared knowledge, attitudes, and 
beliefs related to PACT structures, processes, and effectiveness and the potential for SMI-PACT’s effectiveness. 
We will synthesize this information with survey data to create baseline summaries of care as usual, and to tailor 
our marketing and implementation strategies for use at the intervention site.  In mid-study interview data from 
the intervention site, we will identify factors facilitating and impeding implementation of the care model, and 
strengths and weaknesses of the model as implemented.  In conjunction with mid-implementation survey data, 
we will assess the extent to which components of the model are being implemented, and which components are 
efficient and easy to incorporate into routine care.  We will explore whether particular components appear to 
be of limited value in improving care and examine clinic and provider characteristics associated with varying 
levels of care model implementation and effectiveness.  In mid-implementation interview data from the control 
site, we will explore changes in PACT implementation and co-location of primary care, i.e., changes in usual 
care. In patient interview data collected at final follow-up, we will examine which characteristics of care were 
most (and least) salient for patients, and then look for degree of convergence between patients’ and providers’ 
perspectives on core elements of care within the SMI-PACT model.  In addition to identifying themes and 
patterns qualitatively, we will examine statistical associations between care model components and important 
process and outcome variables such as treatment appropriateness and improvement in patient outcomes.  
Tables generated in ATLAS.ti can be exported into SPSS in order to facilitate the integration of qualitative and 
quantitative data. 

 
 
The quantitative measures that assess implementation (i.e., from the formative evaluation) of the care 
model consist of three general types:  1) those that will primarily be used to inform the tailoring of the care 
model implementation (ACIC, PACIC);  2) those that will describe the level of, and factors that impact, 
implementation of the care model (Rogers’s Adoption Questionnaire, quarterly ratings of degree of 
implementation); and  3) provider and patient factors that may relate to implementation (MBI) and outcomes 
(BASIS, ACES, ISEL, MPR), respectively.  For the first category, we will compile descriptive statistics and 
prepare feedback reports to use for tailoring of implementation.  We will also use the second category of these 
measures descriptively, correlating them with the change scores of the outcome variables (treatment 
appropriateness, patient outcomes, and service utilization). Although correlations do not allow for 
determination of causality, they will allow us to assess the relationship between improvement in outcomes and 
1) greater implementation of multiple SMI-PACT model components; and 2) less perceived complexity and 
greater relative advantage and observability of the care model (Rogers Adoption Questionnaire).