Evaluation of Veteran Directed Home and Community Based Services
Conditions: Functional Impairment, Caregiving
Sponsor: VA Office of Research and Development

 
 
 
 
ID: SDR 16-194 
Evaluation of Veteran-Directed Home and Community Based Services 
Protocol Date: 04/23/2018 

Version 04/23/2018 
 
1
 
 
 
 
 
 
  
 
 
Evaluation of Veteran-Directed Home and Community Based Services 
 
 
Funding Agency: HSR&D 
SDR 16-194 
Principal Investigator/Study Chair: James Rudolph, MD 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Version 04/23/2018 
 
2
 
 
 
 
 Abstract 
 
Background 
In the next decade, there will be a doubling of older Veterans for whom the VA has a 
requirement to provide long-term care. The Veteran’s Health Administration Office of 
Geriatrics and Extended Care (GEC) is seeking to preserve Veteran independence in the 
community through the Veteran-Directed Home and Community-Based Services Program 
(VD-HCBS). VD-HCBS is a person-centered consumer-directed program that provides 
Veterans the opportunity to self-direct their long-term services and supports (LTSS) and 
continue to live independently at home. GEC is planning to expand VD-HCBS to an 
additional 90 VAMCs over the next 3 years. This project will conduct a mixed-methods 
evaluation of the Veteran and Caregiver experience with the VD-HCBS program, and 
identify implementation factors that help facilitate successful program rollout and 
administration.   
 
Objectives 
This study will 1) examine the Veteran experience with VD-HCBS by analyzing operational 
data on satisfaction, unmet needs, and quality of life and b) interviewing Veterans to 
capture the experience in their own words; 2) Understand the effect of VD-HCBS on 
Caregiver wellbeing  through phone surveys aligned with the Caregiver Support Program 
and 3) improve the process of future VD-HCBS implementation by interviewing VA and 
ADNA coordinators about the process of implementing the VD-HCBS.    
 
Methods 
There are three primary methods for this proposal request:  a secondary analysis of 
operations data related to VD-HCBS,a telephone survey with VD-HCBS caregivers and 
semi-structured qualitative interviews with Veterans or Caregivers and VD-HCBS program 
coordinators. For the secondary data analyses, we will obtain survey data from Veterans 
through a Data Use Agreement with the VA Office of Geriatrics and Extended Care (GEC).  
This operational data set is collected through the VD-HCBS program and the VA-CARES 
program as part of routine care. This data will allow us to conduct analyses on Veteran 
satisfaction, needs (met and unmet) and quality of life.. A telephone survey will be 
conducted with VD-HCBS caregivers to understand their experience as a caregiver in the 
VD-HCBS program. Additionally, qualitative interviews will be conducted with Veterans 
selected at random regarding their experience with the VD-HCBS program and how the 
program is meeting their needs. In addition, qualitative interviews will be conducted with 
VA based and ADNA based coordinators of the VD-HCBS program to gain insight into the 
implementation of the program and will help identify facilitators and barriers that impact 
implementation of VD-HCBS for VAMCs and ADNAs. 

Version 04/23/2018 
 
3
 
Impact 
This study will evaluate the impact of the VD-HCBS program and provide key insights into 
the implementation and effectiveness of this program. This evaluation may provide 
insight to VA policymakers and VA care providers that helps them ensure that community 
based LTSS options are adequately administered and help refine policies regarding the 
mission and operations of VD-HCBS to align with VHA and GEC goals.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Version 04/23/2018 
 
4
 
 
 
 
List of Abbreviations 
ACL -- Administration for Community Living  
ADNAs -- Aging and Disability Network Agencies  
BIC -- Bayesian Information Criterion 
CCDE -- Cash and Counseling Demonstration and Evaluation 
CDW -- Corporate Data Warehouse 
CFIR -- Consolidated Framework for Implementation Research 
CITI -- Collaborative IRB Training Initiative 
COIN -- Centers of Innovation 
CREATE -- Collaborative Research to Enhance and Advance Transformation and Excellence 
Initiative 
ERIC -- Expert Recommendations for Implementing Change 
FMS -- Financial Management Service 
GEC -- Office of Geriatrics and Extended Care 
GLMMs -- Generalized Linear Mixed Models 
HSR&D/QUERI -- Health Services Research & Development/Quality Enhancement Research 
Initiative 
IIR -- Investigator Initiated Research 
IRB -- Institutional Review Board 
LTC -- Long-Term Care 
LTSS -- Long-Term Services and Supports 
NRCPDS -- National Resource Center for Participant-Directed Services 
OIG -- Office of Inspector General 
P1a -- High Priority Veteran 
PCAFC -- Program of Comprehensive Assistance for Family Caregivers 
PEPReC -- Program Evidenced-based Policy Resource Center  
QCA -- Qualitative Comparative Analysis 
QUAN+QUAL -- Quantitative/Qualitative 
SCI -- Spinal Cord Injury  
TBI -- Traumatic Brain Injury 
USH -- Undersecretary for Health 
VA-CARES -- VA Caregiver Evaluation Center 
VAMCs -- VA Medical Centers 
VD-HCBS -- Veteran-Directed Home and Community Based Services  
VHA -- Veterans Health Administration  
VINCI -- VA Informatics and Computing Infrastructure 
VPN -- Virtual Private Network 
 
 
 

Version 04/23/2018 
 
5
 
 
 
 
1.0 Study Personnel 
 
 
 
Principal Investigator/Study Chair: James Rudolph, MD Providence VA Medical Center 
(8/8ths) 401-273-7100 Ext. 6406 
 
Investigator: Kali Thomas, PhD Providence VA Medical Center (8/8ths) 401-273-7100 
Ext: 6268 
 
Investigator/ Durham Local Site Investigator: Nina Sperber, PhD Durham VA Medical 
Center (8/8ths) 919-286-0411 Ext. 5655 
 
Collaborators: VHA Office of Geriatrics and Extended Care 
Program Evidenced-based Policy Resource Center (PEPReC) 
The VA Caregiver Support Program 
U.S. Department of Health and Human Services Administration for 
Community Living 
The Lewin Group 
Boston College 
 
2.0 Introduction 
 
In the next decade, the number of older high priority (P1a) Veterans will double. The VA has a 
statutory obligation to purchase or provide institutional care to all P1a Veterans1. At a cost of 
$110,000 per Veteran per year for institutional care, the ability of the VA system to honor its 
statutory obligations will be challenged with current resources. As stated by the 2013 Office of 
the Inspector General report, 2 there is a rapid need to rebalance the focus from institutional 
programs to non-institutional programs. In FY14, VA’s Office of Geriatrics and Extended Care 
(GEC) spent 74% of its budget on institutional care and 26% on non-institutional long-term 
services and supports (LTSS).3 For comparison, state Medicaid programs, which are under a 
statutory obligation to provide institutional care, devote the majority (51%) of LTSS spending to 
non-institutional care.4 In an effort to meet this impending demand, GEC has been working 
toward developing a dynamic system of LTSS that shifts the delivery of long-term care (LTC) 
from institutions to the community. 
 
As part of this rebalancing plan, GEC has implemented many innovative programs to postpone 
or prevent institutionalization.5,6 Recognizing the need to increase home and community-based 
services (HCBS) utilization for high-need Veterans and following the reported success of the 
Cash and Counseling Demonstration and Evaluation (CCDE),7,8 GEC leadership implemented a 

Version 04/23/2018 
 
6
program for Veterans to self-direct their care thereby enabling them to maintain community 
residence and avoid institutionalization: Veteran-Directed Home and Community Based 
Services (VD-HCBS). In 2008, a partnership was formed between VA and the Administration for 
Community Living (ACL) to implement the VD-HCBS program, capitalizing on the home care 
service delivery experience and skills of ACL’s Aging and Disability Network Agencies (ADNAs) 
and the VHA’s commitment and resources. Through this partnership, VAMCs purchase VD-HCBS 
care management from ADNAs on behalf of Veterans. GEC is planning an expansion of VD-HCBS 
to an additional 90 VA sites over the next 3 years. 
 
Description of the VD-HCBS Program. 
The research project described in the protocol is an evaluation of the VD-HCBS program. Like 
the participant-directed programs developed for Medicaid programs (i.e. Money follows the 
Patient, or Cash and Counseling), 8,9 VD-HCBS maximizes Veteran-centered care and 
independence. VD-HCBS provides Veterans with a monthly financial allotment based on an 
assessment of the extent of their need for assistance with personal care due to injury or the 
impact of disease. Veterans have the opportunity to spend their allotment on services that best 
meet their needs, such as personal care workers whom they hire and manage themselves. 
Personal care workers may be family members who otherwise may not be able to afford to stay 
out of the workforce to care for the Veteran. Allotted funds may also be spent on supplies, 
home modifications, and/or adaptive equipment not available through other VA programs.  
 
ADNAs play a key role in identifying Veteran’s needs, coordinating Veteran services, financial 
reporting, and coordinating with VA. An ADNA coordinator works with the Veteran to 1) 
develop a person-centered plan for the Veteran’s care, 2) identify risks to Veterans’ health and 
safety and include in the plan how Veterans and their caregivers can mitigate those risks, 3) 
monitor spending and service utilization and 4) provide ongoing options counseling and support 
to Veterans. In addition, the Veteran (or appointed representative) and the ADNA 
representative work closely with a Financial Management Service (FMS) to manage 
employment taxes and insurance and process payroll and timesheets. The ADNA bills the VA for 
their services and oversight of VD-HCBS recipients.  
 
Previous VD-HCBS Evaluations.  
In an evaluation of the VD-HCBS conducted by Boston College’s National Resource Center for 
Participant-Directed Services (NRCPDS), survey responses from VD-HCBS Coordinators at 
VAMCs indicated that VD-HCBS is filling a niche for Veterans at risk of nursing home 
placement.10 The survey results further pointed out that, for many, collaboration between 
VAMCs and ADNAs has been enhanced by mutual benefit; but in some cases, these 
partnerships were complicated and difficult. The evaluation report suggests that role 
clarification and coordination efforts were critical to maximize the roles of the VA and ADNA. In 
complementary work conducted by investigators at the Providence VAMC Center of Innovation 
for LTSS (LTSS COIN), the majority of VD-HCBS coordinators at ADNAs interviewed suggest that 
the experience of collaborating with VAMCs was positive and interviews identified several key 
mechanisms for facilitating a successful partnership (e.g., including frequent communication, 
training, designated VAMC staff personnel, and active involvement of the VAMC).11 Veterans 

Version 04/23/2018 
 
7
currently enrolled in VD-HCBS who participated in the NRCPDS’ qualitative “Voice of Veterans” 
study called the program “life changing.” While the Veterans interviewed said they would 
recommend VD-HCBS, they also made a number of suggestions to improve the program. 
Preliminary analysis of these data reveals program implementation issues from the perspective 
of users, as well as concern about consistency across programs, access for more Veterans, and 
sustainability of VD-HCBS. 
 
Importance of Caregiving in maintaining Veteran Independence. Because Veterans eligible for 
VD-HCBS are vulnerable with functional deficits and significant caregiving needs, the caregiver 
is an indispensable partner in maintaining community independence. The emotional, physical, 
and financial tolls of informal caregiving are well documented in the civilian12-14 and Veteran 
populations.15-17 According to the 2015 National Survey of Caregiving,18 one in five Caregivers 
report high levels of physical strain resulting from caregiving, two in five consider caregiving to 
be emotionally stressful, and about one in five reports experiencing financial strain. Given the 
relationship between Caregiver burden, stress, and well-being and the ability for the care 
recipient to remain in the home, the impact of VD-HCBS on Caregivers merits attention. 
Previous literature has suggested that consumer directed programs like VD-HCBS benefits 
consumers (see, for example, Tilly, et al.19) and Caregivers.20-22 Caregiver benefits include 
decreased worry, physical, and financial strain, along with the ability to enjoy respite time.23, 24 
Focus groups with 23 Caregivers of Veterans enrolled in VD-HCBS were conducted by our 
partners at NRCPDS as part of their Voices of Veterans study. Analysis of these focus groups 
revealed major sources of stress and elucidated mechanisms behind stress relief and support 
experienced in VD-HCBS. For example, participants commented on the program’s ability to 
enable Veterans to remain home, provide stress relief, and afford spousal Caregivers more time 
with their Veteran loved one spent as a “couple” resulting in reciprocal, mutual benefits. While 
the Voices of Veterans study was the first to include VD-HCBS Caregivers, there has been no 
other work, to date, that has specifically evaluated the impact of the VD-HCBS program on 
Caregivers as measured by improvement in their well-being.  
 
SIGNIFICANCE 
Knowledge to be gained. Proponents of VD-HCBS suggest that expansion of this program could 
be one promising solution to meet the increased need for LTSS while containing costs and 
promoting Veteran-centered care. In addition, it provides an innovative approach to HCBS 
delivery to Veterans living in rural areas or far from VAMCs. Several VAMCs have documented 
the success of their programs both in terms of Veterans’ satisfaction and reduced NH 
utilization, 25-27 However, uptake of the program varies greatly across the country: only 51 
VAMCs offer VD-HCBS and the numbers of Veterans enrolled at each site range from one to 
100+. There has been no formal evaluation of implementation or effectiveness of the program 
and therefore the evaluation we are proposing is significant because it will be the first major 
evaluation of the effect and implementation of the VD-HCBS program.  
 
Impact on Veterans and the VA. Veterans enrolled in VD-HCBS represent a vulnerable 
population, with most having functional or cognitive deficits, needing >20 hours of home care, 
and meeting eligibility for long term care. At a cost of $110,000 per Veteran per year for long-

Version 04/23/2018 
 
8
term care and a doubling of the number of Veterans eligible for VA-paid long-term care, GECs 
and Veterans have a mutual interest in preserving Veteran independence in the community. As 
a result, VD-HCBS has tremendous opportunity to improve the fiscal outlook of GEC’s LTSS 
while delivering the highest level of Veteran-centered care. This proposed VD-HCBS evaluation 
leverages the utilization analysis of PEPReC, by systematically capturing perspectives of 
Veterans and Caregivers to provide a comprehensive view of the value of VD-HCBS to all 
stakeholders.  
 
Alignment with VA Priorities. VD-HCBS expansion is consistent with the VA and GEC Mission 
and Vision. Importantly, VD-HCBS highlights the efforts of GEC to rebalance toward home and 
community-based services (HCBS) as required by the OIG. Access to HCBS is the priority goal of 
both the VA Secretary and the VA Undersecretary for Health (USH). Because VD-HCBS is built on 
a participant-directed, patient-centered framework, the program is consistent with improving 
the Veteran Experience (VA Secretary Priority #1; USH Priority #2). The expansion of VD-HCBS, a 
best practice in HCBS (USH Priority #3), enhances strategic partnerships (VA Secretary Priority 
#5 & USH Priority #4) and builds a culture of improvement (VA Secretary Priority #4) in a high 
performing network (VA USH Priority #4). Evaluation of VD-HCBS aligns with the “MyVA” 
initiative which is Veteran-centric and objectives to provide access to consistent programs 
throughout the VHA system.28 
 
INNOVATION 
The large-scale expansion of VD-HCBS presents an unparalleled opportunity to understand and 
measure change in the VA. In concert with the outcomes of VD-HCBS, our implementation 
evaluation will provide insight into GEC programs at VAMCs (educating GEC leaders), but also 
other VACO program offices looking at implementation. With the diverse team assembled and 
the comprehensive evaluation proposed, we will produce evidence to inform VA program 
implementation and dissemination, a critical component of the VA’s priorities to develop high-
performing, highly-reliable, and consistent networks. Through collaborations with GEC, PEPReC, 
ACL, and NRCPDS, we seek to leverage the investment in VD-HCBS to describe the value of 
participant-based programming within VA and the nation. The findings from this study will have 
implications for future VA-Community partnerships, and can be informative for the 
organizations providing this contracting service as well as for the providers enrolling in the 
network. 
 
The individual components of our evaluation are innovative. Large scale measurement of 
Veteran independence, unmet needs, and satisfaction of a large program is unique in the VA. 
Additionally, including mixed methods will allow a comprehensive understanding of how 
Veterans are impacted by VD-HCBS. The inclusion of the Caregiver survey results will provide a 
first look at how this program may enable Veterans to remain in their homes through improving 
Caregiver well-being.  
 
3.0 Objectives 
 

Version 04/23/2018 
 
9
In this evaluation, we will test the impact of the VD-HCBS program on Veterans and their 
Caregivers while simultaneously gathering information on its delivery and implementation.  
 
Specific  Objectives: 
 
 
Objective 1: To describe the impact of VD-HCBS on Veterans’ satisfaction, unmet need 
for services, quality of life, and independence. Using mixed methods, this objective will 
provide a comprehensive view of the value of the VD-HCBS program to vulnerable 
Veterans and create a model system for describing the value of GEC programs.   
 
Hypothesis: Our working hypothesis is that Veterans participating in the VD-
HCBS program will experience improvements in satisfaction, unmet need for 
services, quality of life, and feelings of independence compared to before their 
participation in the program.  
 
Objective 2: To understand the effect of VD-HCBS on Caregivers’ well-being. As Veterans 
lose function, Caregivers are critical to community living. We will conduct a telephone 
survey with VD-HCBS cargivers to compare Caregivers’ financial strain, depressive 
symptoms, caregiving stress, health status, and positive caregiver experiences to a 
propensity matched population to determine the impact of the VD-HCBS program on 
Caregivers.  
 
 
Hypothesis: Our working hypothesis is that Caregivers of Veterans participating 
in VD-HCBS will experience reductions in financial strain, depressive symptoms 
and caregiving stress, and overall improvements in their health and caregiver 
experiences. 
 
Objective 3: To examine the implementation of the VD-HCBS program expansion. VD-
HCBS is a complex program in a complex VAMC environment. Utilizing constructs from 
the Consolidated Framework for Implementation Research (CFIR) and a published 
typology from the Expert Recommendations for Implementing Change (ERIC) project, 
we will interview VA and ADNA VD-HCBS coordinators to systematically identify 
common contextual factors and implementation strategies.   
 
This proposed VD-HCBS evaluation leverages the information that will be obtained by GEC in 
this expansion by systematically capturing perspectives of Veterans and Caregivers to provide a 
comprehensive view of the value of VD-HCBS. Additionally, the large-scale expansion of VD-
HCBS presents an unparalleled opportunity to understand and measure change in the VA. In 
concert with the outcomes of VD-HCBS, our implementation evaluation will be critical to 
determining how the program was executed. With the diverse team assembled and the 
comprehensive evaluation proposed, we will produce evidence to inform VA program 
implementation and dissemination, a critical component of the VA’s priorities to develop high-
performing, highly-reliable, and consistent networks. Through collaborations with our 

Version 04/23/2018 
 
10 
operation partners we seek to leverage the investment in VD-HCBS to describe the value of 
participant-based programming within VA and the nation. 
 
4.0 Resources and Personnel 
 
Providence VAMC: 
 
James Rudolph, MD is a VA Geriatrician and health services researcher who directs the 
LTSS COIN. Dr. Rudolph will serve as the administrative and coordinating principal 
investigator and will oversee the management of our partners. He will supervise the 
program manager for the evaluation and reporting. Dr. Rudolph will be contact PI and 
be responsible for submitting all necessary documents to HSR&D, including IRB 
approvals, and annual progress reports. 
 
Kali Thomas, PhD is a Research Health Science Specialist with a dual appointment in the 
LTSS COIN and the Brown School of Public Health. Dr. Kali Thomas will lead the data 
collection and analyses of Objectives 1 and 2. She will supervise the work of all IPAs and 
coordinate the Veteran interviews, recruiting, consent, and the analysis of Veteran 
experience and Caregiver surveys. 
 
Vince Mor, PhD Vincent Mor, PhD is a national expert in health services research 
specializing in aging, chronic disease and LTC. He is the PI of the LTC CREATE at the 
Providence VAMC and also funded through an IIR investigating concurrent Hospice care. 
Dr. Vince Mor will provide insight and guidance on the analyses.  
 
Edward Miech, Ed.D.  Dr. Miech will direct the analytic strategies of the implementation 
analyses in year 2 and 3 (AIM 3). Dr. Miech is a member of the Implementation Core of 
the PRIS-M QUERI, a Research Scientist at the Indianapolis VA HSR&D Center for Health 
Information & Communication, and a Research Scientist at the Center for Health 
Services Research at the Regenstrief Institute.  He has experience in program evaluation 
and mixed methods, with special expertise in Configurational Comparative Methods and 
Qualitative Comparative Analysis (QCA). Dr. Miech leads the national VA QCA 
Workgroup, has given several national conference presentations on the application of 
QCA to implementation research and health services research, and helped organize a 5-
day seminar on QCA held in Indianapolis in September 2017 that was attended by 32 
health services researchers from around the United States.     
 
The following members of the Boston College National Resource Center for Participant-
Directed Services (NRCPDS) will be retained on VA independent practice agreements 
(IPA) through the Providence VAMC. 
 
Kevin J. Mahoney, PhD serves as the Founding Director of the National Resource 
Center for Participant-Directed Services (NRCPDS) since its inception in 2009. Dr. 

Version 04/23/2018 
 
11 
Mahoney will advise the team on the delivery of participant-directed services, 
including the complexity of the VA-ADNA relationship in delivering VD-HCBS . Dr. 
Mahoney will not have access to identifiable or sensitive data. He will not have 
access to the Providence research server.   
 
Ellen K. Mahoney, PhD is Associate Professor and Interim Associate Dean for 
Research in the Connell School of Nursing at Boston College,  Dr. Mahoney will 
recruit Veterans/caregivers, obtain consent, conduct interviews and participate 
in the analysis of interviews for Objective 1. 
 
Merle Edwards-Orr, PhD has been working in self-direction for over a decade 
and has led NRCPDS’ work with VD-HCBS for three years. In this role, he works 
closely with VHA and ACL and with local staff at VAMCs and ADNAs to support 
their starting a VD-HCBS site.  Dr. Edwards-Orr will advise the team on the 
function of the VD-HCBS program and advise on the implementation analysis for 
Objective 3. Additionally, he will participate in some of the coordinator 
interviews which will provide more insight into the implementation process and 
will take an active role in the analyses.  
 
 
Durham VAMC: 
 
Nina Sperber, PhD, is a Core Investigator in the Center for Health Services Research in 
Primary Care at the Durham VAMC and Associate Director of the Qualitative Methods 
Core. Dr. Sperber will be the Local Site Investigator for the Durham VA and will be 
responsible for the implementation evaluation of Objective 3. She will oversee the 
recruiting, consenting, and interviewing of VD-HCBS and ADNA stakeholders and will be 
the primary coordinator of the analyses that will examine the integration of contextual 
factors, implementation strategies, and Veteran outcomes. 
 
Courtney Van Houtven, PhD is a health economist for the VA Caregiver Support 
Evaluation Center based out of the Durham VAMC, Project Director of VA-CARES and 
Associate Professor at Duke University whose research focuses on how family caregiving 
effects healthcare utilization, expenditures, health and work outcomes of care 
recipients and caregivers.  Dr. Van Houtven will advise the team and collaborate on 
analyses evaluating caregiver outcomes. 
 
Salt Lake City VAMC HSR&D Centralized Transcription Service Program:  
 
Susan Zickmund, PhD is the Director of HSR&D Centralized Transcription Service 
 
Program will be overseeing the Veteran and Coordinator transcriptions conducted by 
 
the Providence and Durham sites.  She will serve as the local site investigator for Salt 
 
Lake City. 
 
Data Use Agreements: 

Version 04/23/2018 
 
12 
For this analysis, we will enter into three data use agreements in order to complete our 
secondary data analysis aims. We will enter into a Data Use Agreement with the Office of 
GEC for the use of the ADNA assessments and The Lewin Group readiness assessments both 
of which are reported to GEC as part of routine care/service. Only the ADNA Assessments 
contain PHI. The second data use agreement will be with the VA Caregiver Support 
Program. The caregiver survey data will be used as a comparison group for our research 
analysis of the Caregiver Survey (Objective 2).  Finally, we will enter into a data use 
agreement with PEPReC for the VD-HCBS program analysis data including VAMC 
randomization criteria, the utilization outcomes analysis, and the economic outcomes 
analysis data. This will allow us understand the context of implementation in programs that 
are ‘successful’ in implementation and outcomes development of VD-HCBS.  
 
 
5.0 Study Procedures 
 
Objective 1 –  
The purpose of Objective 1 is to understand the impact of the VD-HCBS program on 
similar outcomes for Veterans using a mixed qualitative and quantitative method 
design30. The quantitative methodology (Objective 1a) will be a secondary data analysis 
of the ADNA assessments. These data will be obtained via a Data Use Agreement (DUA) 
with the VHA Office of Geriatrics and Extended Care. These assessments contain surveys 
that are administered at each Veteran’s initial in-home visit, and quarterly thereafter as 
part of the national roll-out and operation of the VD-HCBS program. PHI is included in 
this data set including name, social security number, and date of birth., address, phone 
number, email (if available), and caregiver contact information. All secondary survey 
data will be stored on the restricted access Providence VA research server, in a 
restricted folder in accordance with VA policy.   
For the qualitative aim (Objective 1b), we will be conducting semi-structured telephone 
interviews with a random sample of Veterans (n=64)  newly enrolled to VD HCBS (to 
include the sites involved in the roll out and all other sites with VD HCBS) to gain further 
insight into the ways in which the VD-HCBS program has impacted their lives. Veterans 
will be interviewed at two time points in 12 months after initiation of VD-HCBS, for a 
total of 128 interviews. If a Veteran is unable to communicate on the telephone, we will 
instead contact their caregiver and invite them to share their own experiences with the 
program.  These interviews will be recorded for transcription purposes. Participants will 
be sent an information consent sheet, however consent for the interviews and the 
recording will be acquired verbally as detailed in the interview guides. Following the 
completion of the interviews and the transcription, the interviews will be coded as 
described in the data analysis section.    
Recruitment Methods:  
All quantitative data will be obtained from the data owners (VHA Office of GEC), 
following an approved Data Use Agreement in accordance with VHA policy.  As 

Version 04/23/2018 
 
13 
part of standard operations, the ADNA meets with the Veteran initially and every 
90 days thereafter.  During these meetings it is customary for the ADNA to ask 
about current services, as well as, unmet needs and satisfaction with the 
program.   
The VD-HCBS program currently enrolls ~825 new Veterans annually at existing 
sites.  At newly opened sites, there is a period between readiness review and 
enrolling Veterans that delays early recruitment.  As a result, we anticipate that 
~825 Veterans among the new sites will be enrolled annually.  With 4 routine 
ADNA assessments per Veteran, we expect to receive up to 6,600 surveys. 
A total of 64 respondents consisting of Veterans or Caregivers will be recruited 
for qualitative portion of this objective.  Recruitment packets will initially be sent 
using via US Mail to the Veterans’ listed residences in the ADNA assessment data 
or should the addresses not be listed, by using the listed SSN in the ADNA 
assessment data to access the national Electronic Health Record (EHR) by way of 
CAPRI/VistAWeb and/or JLV. We will randomly select four Veterans enrolled in 
VD-HCBS from each of the 16 VAMCs chosen for the implementation analysis 
described in Objective 3. Veteran selection will be done by inputting all case 
numbers at a VAMC into a random number generator. This packet will include an 
introduction letter, refusal postcard, and the CIRB approved information consent 
sheet. The letter will inform potential participants about the interview, the goal 
of the interviews, that the interviews will be recorded for transcription purposes 
and will provide contact information for a Providence-based project staff 
member to address any questions or concerns they may have prior to 
participating.  They may also use this telephone number to indicate their refusal 
to participate.  
The CIRB approved information consent sheet for Veterans to keep, will describe 
the procedures and risks and contain contact information should they have any 
questions.  
Also included will be the refusal to participate postcard. Veterans will be asked 
to complete and return this postcard if they do not wish to participate in the 
study. Receipt of this postcard by the project staff will terminate all future 
contact with the Veteran regarding this project.    
For Veterans whom we do not receive a refusal postcard or telephone call, we 
make no more than three attempts to contact them via telephone between the 
hours of 10am and 7pm, in their representative time zones.  
If it is discovered during these attempts that the Veteran is unable to participate 
due to medical limitations such as being non-verbal, aphasic, or cognitively 
impaired, the project team will exclude the Veteran from participation and 
instead will contact the Veteran’s Caregiver as listed in the ADNA survey. At this 
point, a packet including the introduction letter, refusal postcard, and the CIRB 
approved Caregiver information consent sheet will be sent to the named 

Version 04/23/2018 
 
14 
caregiver. Receipt of this postcard by the project staff will terminate all future 
contact with the Caregiver regarding this project.   
For Caregivers whom we do not receive a refusal postcard or telephone call, we 
make no more than three attempts to contact them via telephone at the number 
listed in the ADNA survey between the hours of 10am and 7pm, in their 
representative time zones. 
If the caregiver agrees to participate in this project, the caregiver will be asked 
questions regarding their own experiences with the program as described in the 
Caregivers interview guide and this data will be analyzed separately. 
In order to ensure that the appropriate number of Veterans and Caregivers are 
enrolled, for every instance where we receive a refusal or are unable to make 
contact with the Veteran or Caregiver, a new Veteran will be randomly selected 
and the recruitment process will repeat. 
 
Informed Consent Procedures: 
For Objective 1a, the secondary data analysis portion of this aim, we are 
requesting waiver of informed consent and a HIPAA waiver 
For Objective 1b, the Veteran or Caregiver Interviews portion of this objective, a 
CIRB approved information consent sheet appropriate for the respondent will be 
mailed to the Veterans with the recruitment packet described above, or as 
described above to the Caregiver. A waiver of written consent will be requested 
for all interviews because interviews will be conducted over the phone and are 
considered minimal risk. Verbal informed consent will be obtained in 
conformation with the policies established by the CIRB Institutional Review 
Board in accordance with the Department of Health and Human Services 
Regulations regarding the protection of human subjects (45 CFR 46.116). The 
verbal consenting procedure is detailed in each of the interview guides.  
  
Privacy and Confidentiality: 
All secondary data records and all data containing PHI will be stored exclusively 
on the Providence Research APP26 server.  Please see the Data Assurances 
section of the protocol for a description of the security measures in place for all 
electronic data. 
To help mitigate the risks to Veteran interviewees in Objective 1, telephone 
interviews will be conducted in private settings as individual interviews with the 
interviewer and an additional member of the project team. At no time will any 
content of interviews be discussed with or released to clinicians of Veterans, or 
ADNA program coordinators.  

Version 04/23/2018 
 
15 
Identifiers, including first and last name and the name of their associated VAMC, 
will not be elicited from interview respondents during interview recordings. After 
transcription, the transcripts will be read by a project team member to ensure 
that any identifiable or sensitive data that participants volunteer during the 
interviews is removed prior to analysis. Interview responses will be aggregated 
so individual sites and interviewees cannot be identified. No individually 
identifiable information will be published or disclosed, unless required by law.  
To protect participants against risk after interview data have been collected, we 
will take a variety of measures to ensure confidentiality. In all records, 
participants will be assigned unique identifiers. The crosswalk linking unique 
identifiers to participants, the verbal consent log, as well as all interview 
recordings and transcripts will be stored electronically in a password protected, 
restricted-use folder on a Providence VA server in a restricted access service 
directory, in a restricted folder with access limited to project staff. Any hard 
copies of any of these files will be stored at the VA in a locked file cabinet in the 
locked office of study personnel. When the data are reported, neither VAMCs 
nor interviewees will be identified in any way, to preserve anonymity of 
respondents. 
Please see the Data Assurances section of the protocol for a description of the 
security measures in place for all electronic data. 
 
Objective 2  
 
The second objective of this evaluation is to understand the effect of the VD-HCBS program on 
Caregivers’ well-being. This will be done using telephone surveys with caregivers. Each 
Caregiver will participate in two 20-minute phone surveys held 12 months apart. These calls will 
not be audio recorded, but caregiver responses will be entered into the project database using 
a unique coded ID. A crosswalk to the database will be maintained separately in a restricted 
access folder on the Providence VA server.  We will examine the outcomes for Caregivers of 
Veterans enrolled in the VD-HCBS program and compare this to a control group of Caregivers 
using data previously collected as part of the VA CARES evaluation.  
  
Data for the outcomes for Caregivers of Veterans enrolled in the VD-HCBS program will be 
collected via telephone survey and data from the control group will be acquired following an 
approved Data Use Agreement with the VA Office of Geriatrics and Extended Care. We will test 
our working hypothesis through analysis of the telephone surveys and compare them to 
Caregivers who participated in a previous partnered evaluation of the Caregiver Support 
Program. With the documented relationship between Caregiver burden, stress, strain and 
quality of life and the ability for the care-recipient to remain in the home, it is imperative to 
understand the impact on caregivers as part of the causal chain of program impact for 
Veterans. When the proposed studies from this objective are completed, it is our expectation 

Version 04/23/2018 
 
16 
that the impact of the VD-HCBS program on Veterans’ Caregivers will be better understood.  
Our proposal for analysis of the Caregiver Survey Data is describe in Section 6.0 (objective 2). 
 
Recruitment Methods: 
This objective will be completed in two separate stages. The first stage will be 
data collection of caregiver Survey data via telephone surveys.  Caregivers of 
Veterans receiving VD-HCBS will be given a sealed packet during the ADNA’s 
initial visit or will be placed within the Veteran interview packet to be mailed as 
described above. The sealed packet will be labeled “To the Veteran Caregiver” 
and will include  information about the project, the phone survey, the goal of the 
project, and provide contact information of a Providence-based project staff 
member to address any questions or concerns they may have prior to 
participating. Also included in these letters will be a copy of the informed 
consent sheet for caregivers to keep and will contain contact information should 
they have any questions either before or after the survey. In addition, within this 
introductory packet will be an opt-in postcard that caregivers will be asked to 
complete and return if they wish to participate or be contacted again regarding 
this study. A draft of this opt-in postcard is included in the application (Opt-in 
Postcard included in submission). . Data for the outcomes for caregivers of 
Veterans enrolled in the VD-HCBS program will be obtained from the VHA Office 
of Geriatrics and Extended care following an approved Data Use Agreement in 
accordance with VA Policy. We estimate that there will be approximately 4500 
Caregivers’ data included in this data set. Data for the control group will be 
obtained from the VHA Caregiver Support Program following an approved Data 
Use Agreement in accordance with VA Policy. We estimate that there will be 
approximately 158 Caregivers (92 approved for the Caregiver Support Program 
and 66 not approved) with pre and post survey data included in this data set.  
 
Informed Consent Procedures: 
 
The control group data used for this   objective is will be obtained through a DUA 
for existing data.  We are requesting waiver of informed consent for the control 
group and a HIPAA waiver for this objective.  
For the Caregiver telephone survey, a CIRB approved information consent sheet 
will be included in the recruitment packet left with the Veteran and Caregiver as 
detailed above. A waiver of written consent will be requested for the survey 
because the survey will be conducted over the phone and are considered 
minimal risk. Verbal informed consent will be obtained in conformation with the 
policies established by the CIRB Institutional Review Board in accordance with 
the Department of Health and Human Services Regulations regarding the 

Version 04/23/2018 
 
17 
protection of human subjects (45 CFR 46.116). The verbal consenting procedure 
is detailed in the survey document.   
 
 
Privacy and Confidentiality: 
Please see the Data Assurances section of the protocol for a description of the 
security measures in place for all electronic data. 
 
 
Objective 3  
 
In Objective 3 we will identify factors impacting VD-HCBS program implementation activities 
from multiple stakeholder vantage points (VA and ADNA) and throughout the implementation 
process by identifying specific strategies for effective program implementation within a range 
of contexts. Therefore, we will: 
 
Objective 3a) elicit implementation strategies used based on a compilation published by 
the Expert Recommendations for Implementing Change project;  
 
Objective 3b) examine how contextual factors impact implementation using the 
Consolidated Framework for Implementation Research;  
 
Objective 3c) examine the relationship between implementation strategies, contextual 
factors and outcomes at individual and institutional levels.   
 
As part of this objective, we will conduct individual interviews with VA and ADNA coordinators 
from 16 purposefully selected VA medical centers with recently established and existing  VD-
HCBS programs. The 32 coordinators will include 16 ADNA Coordinators and 16 VA coordinators 
from 16 VD-HCBS program sites.  Whenever possible, each coordinator will be interviewed at 
two time points (64 total interviews). The first set of interviews with the coordinators, will be 2 
to 4 months after GEC allows the program to begin enrolling Veterans (completion of the Lewin 
Group readiness review). The second set of interviews will be 6 months after the initial 
interview to obtain follow-up data about implementation process, compared to the initial start-
up time, implementation strategies that they have employed, and their evaluation of 
implementation success (See Interview Guides).  If it is not possible to interview sites at 2-
months after GEC allowed to begin enrolling Veterans (completion of the Lewin Group 
readiness review), we will conduct one interview at least 6-months after readiness review.    
From the VA coordinators, we will request a local VAMC leader (n=16) and a VISN GEC leader 
(n=16) for one-time interviews (32 interviews). Additionally, we will ask the GEC VISN leads 
when we interview them to recommend local GEC contacts. 
 

Version 04/23/2018 
 
18 
Pair Interview Dropout.  Our primary interest is in triangulating data from the VA and ADNA 
coordinators at a case level.  However, there will be dropout and turnover in the coordinator 
positions.  When there is dropout, we will seek to recruit another individual from the site to 
complete the pairwise analysis. If this is not possible, we will exclude that program site in the 
case-level analysis and continue to recruit until we have 16 program sites, with participation 
from both the VA and ADNA coordinators.  We anticipate dropout of 25% in the pairwise 
analysis and are requesting permission to perform additional interviews at 4 sites (8 interviews) 
 
Including the additional interviews to allow for the projected drop out rate, we estimate that a 
Total of 104 interviews will be conducted with a total of 72 individuals.  
 
Recruitment Methods: 
We will purposefully select 16 VAMCs based on criteria related to quality of VA-ADNA 
collaboration and standard practice, identified as factors impacting implementation in 
our prior evaluations.  For each case, we will conduct individual interviews with VA VD-
HCBS coordinators and associated ADNA coordinators plus additional key decision 
makers identified by coordinators and GEC points-of-contact for the VISNs that cover 
the VD-HCBS programs in our sample. Coordinators will be those reported by The Lewin 
Group, the contractor responsible for Technical Assistance, as the contact person for the 
VD-HCBS program. Additional respondents will be identified when coordinators are 
asked to name their local VAMC leader and a VISN GEC leader as described above.  Once 
potential participants have been identified within eligible VAMCs and ADNAs, they will 
be sent an introductory letter to their work email address. The letter will inform them 
about the interview, the goal of the interview, that the interview will be recorded for 
transcription purposes only, that they may contact a Providence-based project staff 
member to address any questions or concerns they may have prior to participating or to 
schedule an interview, and that unless a refusal is received they will be contacted at 
their work telephone number a week after the email is sent to schedule an interview. 
Participants will also be informed that they may be required to do this interview on off-
duty time unless their supervisor approves on-duty time. If they are willing to 
participate, a phone call will be scheduled.  
 
Informed Consent Procedures: 
When Coordinators or key informants respond to the invitation letter or 
subsequent recruitment phone call and state that they want to participate, a 
member of the project team will send them a CIRB approved information 
consent sheet about the interview. This information consent sheet will include 
information about their rights as study participants.  At the time of the interview, 
the interviewer will confirm that the participant received the written information 
about the study and consent process. Then the interviewer will reiterate key 
parts of the informed consent information that were sent via email when the 
interview was scheduled, including the purpose of the study, the key informant’s 

Version 04/23/2018 
 
19 
rights as a study participant, and the confidentiality of the interview. The 
interviewer will invite the participant to ask questions about the consent 
information. The interviewer will then request verbal consent to proceed with 
the interview and ask permission to audio-record the interview for transcription 
purposes. If the participant verbally consents, the interview can be conducted. 
Interviewers will keep a verbal consent log.    
We are requesting a waiver of documentation of consent for these telephone 
interviews.  
Privacy and Confidentiality: 
To help mitigate the risks to interviewees in Objective 3 telephone interviews 
will be conducted in private settings as one-on-one interviews with the 
interviewer. A research associate may be present as well to take notes, and this 
individual will be identified and introduced to the respondent. The information 
gained during the interviews will not be used to evaluate quality of care. Neither 
the employee’s supervisors nor peers nor the clients will have access to the data. 
Data will remain confidential. Identifiers, including first and last name and the 
name of their associated VAMC, will not be elicited from interview respondents 
during interview recordings. After transcription, the transcripts will be read by a 
project team member to ensure that any identifiable or sensitive data that 
participants volunteer during the interviews is removed  prior to analysis. 
Interview responses will be aggregated so individual sites and interviewees 
cannot be identified. No individually identifiable information will be published or 
disclosed, unless required by law.  
To protect participants against risk after interview data have been collected, we 
will take a variety of measures to ensure confidentiality. In all records, 
participants will be assigned unique identifiers. The crosswalk linking unique 
identifiers to participants, the verbal consent log, as well as all interview 
recordings and transcripts will be stored electronically in a password protected, 
restricted-use folder on a Providence VA server in a restricted access service 
directory, in a restricted folder with access limited to project staff. Any hard 
copies of any of these files will be stored at the VA in a locked file cabinet in the 
locked office of study personnel. When the data are reported, neither VAMCs 
nor interviewees will be identified in any way, to preserve anonymity of 
respondents. 
Please see the Data Assurances section of the protocol for a description of the 
security measures in place for all electronic data. 
 
6.0 VD-HCBS Contextual Information.  GEC, ACL, and The Lewin Group regularly host training 
and check-in activities via conference call or webinar. GEC, ACL and The Lewin Group have 
given approval for an evaluator to join the conference calls and webinars for the purpose of 

Version 04/23/2018 
 
20 
collecting information about the specific implementation issues that are discussed as VD-
HCBS programs are developing and operating.  The evaluator will take notes on the overall 
tone, topics, and attitude of these calls. Notes will not include quotations, names, or 
information on individual sites. Information gained from these calls will be for the sole 
purpose of identifying patterns of implementation experiences. 
7.0 Data Analysis 
Objective 1: 
Quantitative Analysis of Veteran Survey Data -  We will analyze the quantitative data from 
the Veteran Assessment survey data by calculating descriptive statistics and means for the 
survey items. We will compare the responses at baseline to responses at 3 months, and to 
responses at 12 months to gauge early and long-term change in the various domains. We 
will also calculate differences in change by Veteran characteristics, including care 
arrangement, functional impairment, socioeconomics, and demographics. This descriptive 
pre-post analysis will provide a first look at the impact of the VD-HCBS program on 
Veterans. Lastly, we will use finite mixture models31 to identify subgroups of Veterans that 
have similar trends in outcomes. Finite mixture models are clustering methods based upon 
a statistical model with stochastic elements.32 Formally, they can be written as 𝑝(𝑌𝑖) =
∑
𝜋𝑗𝐹(𝑌|𝐺𝑖= 𝑗)
𝐽
𝑗=1
, where Yi is a vector of outcomes for Veteran i, πj is the proportion of 
subgroups j among Veterans, Gi is an unknown group classification for Veteran i, F is the 
distribution of the outcome at the that subpopulation, and J are the number of 
subpopulations. We will examine different possible values for the number of subgroups, 
and select the number with the highest Bayesian Information Criterion (BIC).33 Our initial 
model for F will be multivariate Gaussian distribution, which may require appropriate 
transformations of some of the outcomes (e.g., log). If the Gaussian model will not provide 
a good fit for the data, we will examine other possible models such as a mixture of t 
distributions, or a mixture of skewed-Normal distributions. After the model identifies the 
possible subgroups, we will attempt to describe them by examining the trends in the 
observed outcomes (e.g., Veterans with increasing outcome trends overall, patients with 
moderately increasing outcomes at 3 weeks but stable at 12 months). These subgroups will 
enable us to identify the possible patterns that exist among Veterans exposed to the VD-
HCBS program. Following this identification, we will examine qualitative (see below) or 
quantitative variables that are possibly associated with these patterns, so that we could 
provide further understanding on the possible effectiveness of the program.  
 
Qualitative Analysis -  The team will read interview transcripts multiple times and 
independently make initial codes, then resolve coding differences through a process of 
deliberation and consensus.34 In subsequent and repeated bi-weekly meetings we will 
refine code definitions and reach consensus about the interpretation of the material, with 
our decisions recorded in an audit trail. Continuing this process, codes will be clustered into 
related categories using conventional content analysis35 to generate themes that represent 
successive levels of interpretation. Subsequent analysis will connect common codes and 
themes across Veterans and VAMCs. In the search for alternative interpretations and to 

Version 04/23/2018 
 
21 
provide analytic rigor about the validity of findings, findings will be challenged by team 
members to ensure comparability of perspectives.36 Dr. Thomas will oversee the analysis 
and the team will report on the process during bi-weekly meetings. 
 
Connect Qualitative and Quantitative Data - We will use evidence from both the 
quantitative and qualitative analysis to portray the impact of the VD-HCBS program on 
Veterans. We will analyze the quantitative and qualitative data separately and then 
integrate the results to present a comprehensive picture of how the program has impacted 
Veterans, both in terms of satisfaction and their life experience.37 Specifically, we will use 
our qualitative findings to provide depth of understanding and quantitative data to provide 
breadth of understanding Veterans’ experiences with the VD-HCBS program. With this 
parallel mixed method design, we will be able to provide explanation and description of 
Veterans’ experiences and the ways by which the VD-HCBS program impacts their lives. 
 
Please see the Data Assurances section of the protocol for a description of the 
security measures in place for all data. 
 
Objective 2: 
Quantitative Analysis of Caregiver Survey Data -  The primary comparison of interest will be 
the change in outcomes from the baseline survey to the follow survey between VD-HCBS 
Caregivers and VA CARES participants. Before testing for change over time, we will examine 
the frequency distributions for all study variables and appropriate transformations will be 
carried out whenever violations of normality are determined. Descriptive statistics and 
bivariate correlations will be computed. Each variable will be tested for leptokurtosis to 
ensure that there is a sufficient range of scores. Variables found to be significantly 
leptokurtic will be converted into dichotomous outcomes.  
 
An important issue for longitudinal follow-up analyses is the problem of missing data. Data 
may be missing for one of two reasons: 1) the Veteran of a Caregiver who participated in 
baseline is no longer receiving VD-HCBS at the 6-month follow-up period; 2) a Caregiver 
participant is lost to follow-up because of non-participation. In order to avoid biasing the 
result by including only Caregivers who complete both surveys, we will first examine 
whether there are indications that the data are missing completely at random. Specifically, 
we will compare baseline scores for Caregivers that complete both surveys to those that did 
not complete the follow-up survey. If no significant differences are observed between 
completers and those that “dropped out” on the baseline variables, then we will proceed 
with data analysis of the Caregivers who completed both rounds of the survey. However, if 
we determine that the data are missing at random, then we will impute the missing data 
using a multiple imputation technique. If necessary, we will perform sensitivity analysis to 
examine the plausibility of the missing at random assumption using a pattern mixture 
approach.38  
 

Version 04/23/2018 
 
22 
In order to compare the VD-HCBS Caregivers to the control group of Caregivers from the VA 
CARES evaluation (non-approved Caregivers, n=66), we will attempt to identify VD-HCBS 
Caregivers that are similar to the control group participants in terms of baseline survey 
variables. Identifying matching individuals on the entire set of covariates may be complex, 
so we will rely on the propensity score,39 a technique used in non-randomized studies to 
balance groups based on observed differences. Because the number of participants in our 
control group is significantly smaller than the one expected in VD-HCBS, we will use 
matching with replacement in which controls can be used multiple times to match within a 
specified caliper. Caregivers in VD-HCBS that will not have a matching VA CARES within a 
caliper will be removed from the analysis, because the positivity assumption does not seem 
to be valid for them. This procedure will include many Caregivers in VD-HCBS, which will 
improve the precision of point estimates in this group while maintaining only VD-HCBS 
Caregivers that are relatively similar to control Caregivers. Having a large number of 
Caregivers from VD-HCBS increases the power of the study, but it is dominated by the 
precision in the control group. To estimate the propensity score, we will use a logistic 
regression model 𝑃(𝐶𝑖|𝑋𝑖) =
exp⁡(𝛽′𝑋𝑖)
1+exp⁡(𝛽′𝑋𝑖), where Ci is indicator variable for whether 
Caregiver i is from the VD-HCBS program or the control group, and Xi are the set of baseline 
covariates for caregiver i. Following the propensity score calculation, we will match 
Caregivers in the control group to VD-HCBS Caregivers using the estimated propensity 
scores. We will then examine whether the distributions of the baseline covariates and 
second order interactions are similar across the two groups. Differences in distributions will 
be examined graphically and via appropriate statistical tests. If some variables are found to 
differ between groups, we will update the propensity score model with higher order 
interactions. We will repeat this procedure until balance is obtained on all baseline variables 
and their interactions. 40  
 
Because the matching is not exact, there might be minor imbalances between the 
covariates among Caregivers in VD-HCBS and the control group. To address these minor 
imbalances, we will rely on general linear regression models to estimate differences in the 
outcomes between Caregivers who are participating in the VD-HCBS program and those 
who were part of the VA-CARES evaluation. For continuous outcomes, we will use multiple 
linear regression 𝑌𝑖= 𝑋𝑖𝛽+ 𝜏𝐶𝑖+ 𝜀𝑖, where Yi is the outcome for Caregiver i, Xi are the 
covariates, Ci is an indicator variable for Control/VD-HDBS, τ is the treatment effect and 
𝜀𝑖~𝑁(0, 𝜎2). For the ordinal variable, we will use the proportional odds model 𝑙𝑜𝑔𝑖𝑡(𝑌𝑖<
𝑗) = 𝛼𝑗+ 𝑋𝑖𝛽+ 𝜏𝐶𝑖, where αj is the log-odds of the outcome variable j, and the rest of the 
variables are as previously described. To obtain interval estimates we will use either the 
variance estimates provided by Abadie & Imbens41 or the bootstrap procedure.42  
 
Dependent Variables – We will examine Caregivers’ perceived financial strain, CESD-10 
score (screener for depressive symptoms), burden, self-reported health, and positive 
aspects of caregiving as described in the table below. 
 
Dependent Variables for Objective 2 
Measures of 
Measurement 

Version 04/23/2018 
 
23 
Independent Variables- 
To examine 
improvements in the 
various outcomes, the 
models will examine 
Veteran and Caregiver 
characteristics that may 
be related to 
improvement as well as 
their baseline values on 
each of the measures of 
interest. Most of these 
characteristics will be 
measured during 
baseline surveys of 
Caregivers and a few will be drawn from the Veteran Assessment Data.  For example, the 
models will include Veterans’ demographic and health/function characteristics, month of 
enrollment, and the Veterans’ monthly budget amount. We will also examine Caregiver 
characteristics such as their relationship with the Veteran, whether or not the Caregiver was 
employed, lived with the care recipient, as well as their age, sex, race, education, marital 
status, and whether or not there are dependent children in the household.  
 
 
Power-  
 
Without any adjustment for covariates and assuming that the standard error is equal to 
10 (the observed standard deviation on 
the Positive Aspects of Caregiving Scale in 
the non-approved “control group” from 
the VA Cares evaluation, n=55 
respondents for this scale), we should 
have over 80% power to identify a 
difference of 4 points on the Positive 
Aspects of Caregiving Scale (range 9-45) 
with approximately 1,000 Caregivers in 
VD-HCBS (see Table 4). If we used a 
different control group, (i.e., the approved Caregivers in the VA CARES evaluation, n=92 
respondents), we can identify a 3.1 point difference on this scale. Using regression 
adjustments, we are expecting to be able to identify even smaller effects. Therefore, we 
expect to have a large enough sample size to detect a meaningful difference in at least 
one of the outcomes for this objective 
 
Objective 3: 
Interest 
Caregiver 
Perceived Financial 
Strain 
3 Item Impact on Finances subscale from the Caregiver Reaction 
Assessment 
Scores range 3-15 where higher scores indicate higher strain. 
Caregiver 
Depressive 
Symptoms 
Center for Epidemiologic Studies Depression 10 item Scale (CE SD 10). 
Scores range 0-30 where higher scores indicate more depressive 
symptoms. 
Caregiver Burden 
 
Zarlt Caregiver Burden. Defined as the level of stress felt by a 
Caregiver. The scale covers the areas most often mentioned by 
Caregivers as problems, including health, psychological well-being, 
finances, social life, and the relationship shared by the Caregiver and 
care recipient. 
Scores range 0-48 where higher scores indicate higher burden. 
 
Caregiver’s Self-
Reported Health 
Status 
Single item from the Health and Retirement Study (2012) 
Responses are “Poor”, “Fair”, “Good”, “Very Good” or “Excellent” 
Positive Aspects of 
Caregiving 
Nine items assessing positive aspects of caregiving as captured by 
Tarlow and colleagues’ reliable and well-validated measure.  
Scores range from 9-45 where higher indicates more positive aspects 
of caregiving experienced. 
Estimated Power for Aim 2 Caregiver Analysis by Effect 
Size and Number of Surveys 
VD-
HCBS 
Caregiver 
Surveys 
(n) 
 
Effect Size 
3 
pts 
3.5 
pts 
4 
pts 
4.5 
pts 
500 
56% 
70% 
80% 
89% 
1,000 
58% 
71% 
82% 
90% 
2,000 
59% 
73% 
83% 
91% 

Version 04/23/2018 
 
24 
Analysis - Our primary unit of analysis for this implementation evaluation will be the VA-
ADNA institutional pairs implementing this program. Thus, this will be a case-oriented 
analysis. Below we describe a three-stage team approach to analyze data, adapted from 
Damschroder43 and we will repeat these stages at each of the two data collection phases, 
ultimately integrating data at case-level to describe program implementation from two to 
eight months after randomization. 
 
Coding - First, open-ended data from the interviews will be reviewed and coded by a team 
of two researchers. They will code transcribed data according to CFIR and ERIC constructs 
using directed content analysis,44 in which codes are developed a priori; additional data 
derived codes will be established for text that cannot be coded with the a priori codes. 
Thus, this will be primarily a deductive approach, and we will additionally create codes that 
are inductively derived from data. The researchers will first independently read through a 
subset of interviews (about three to five) to apply initial codes and then come together to 
compare codes and arrive at agreement through deliberation and consensus.45 The 
researchers will then develop a final list of codes and apply these codes to all responses.  
 
Memos- Researchers will aggregate individual interviews from VA and ADNA key informants 
into case memos and identify whether CFIR constructs are manifested as facilitators or 
barriers and what implementation strategies are used. A case memo is a Word document in 
which quotes from individual interviews within each case are collated and organized by 
construct. To determine whether CFIR constructs manifest as facilitators or barriers, 
researchers will independently rate coded text according to strength and valence on a 5 
point scale, with -2 reflecting a statement about a strong barrier to +2 reflecting a 
statement about a strong facilitator. They will again use a team approach to resolve 
differences through deliberation and discussion and arrive at final, agreed upon ratings. An 
overall rating for each construct will be established based on average of ratings for 
individual quotes under that construct. The memos will also include a summary statement 
to describe what was coded (e.g., how did key informants describe the innovation source) 
and whether implementation context or strategy can be ascribed to a specific entity (i.e., 
solely VA or ADNA) or if it is a shared strategy, and they will include a rationale for ratings 
(e.g., the program was mandated without buy-in from local entities, thus a strong negative 
rating). We will build on the case memos developed during the first data collection period 
with data from the second to describe implementation strategies and any updates from the 
first interview for cases that have two interviews. 
 
Qualitative Comparative Analysis -  The objective of this stage will be to consolidate 
collected implementation data to assess how combinations of strategies and barriers relate 
to outcomes using Qualitative Comparative Analysis (QCA).46 At the individual level, 
outcomes will be high report of unmet needs for personal care or very good to excellent 
satisfaction, analyzed as part of Objective1 and, for use in QCA, transformed to reflect 
presence or absence of these values. At the institutional level, outcomes will be the number 
of Veterans receiving care through VDHCBS as a proportion of the total number of Veterans 
receiving Home and Community Based Care one year after program start. QCA is a 

Version 04/23/2018 
 
25 
quantitative, case-oriented method that uses applied set theory and Boolean algebra to 
examine multifactorial causality (i.e., when several variables together have a joint effect on 
an outcome). In essence, it offers a systematic way to compare cases for factors related to 
success of an outcome, distinguishing between necessary and sufficient conditions, in 
projects with small or medium-N. It is a well-established quantitative approach that has 
been in use since the 1980s, especially in political science, to answer research questions like 
"What combinations of conditions are directly connected to the outcome of interest, such 
that cases with those specific combinations also always had the outcome present?" 
Although quantitative, QCA is a form of case-oriented research that requires close 
familiarity with the qualitative dataset of interest and theoretical framework to interpret 
results. As Baptist and Befani say in their primer on QCA, “Put simply, QCA allows evaluators 
to answer the question ‘what works best, why, and under what circumstances’ quite 
literally, in a way that emerges directly from other empirical analysis; that can be replicated 
by other researchers, and is generalizable to other contexts.”47 Because findings from QCA 
will be generalizable, even though we will be evaluating a subset of cases, they could be 
used to inform future expansion of VD-HCBS or assistance for struggling entities. 
 
 
8.0 Protection of Human Subject Data 
Overview of Risks- The primary risk in this study is a breach of confidentiality.  In the case of 
Veterans and Veterans’ Caregivers, a breach of confidentiality could cause an unauthorized 
release of protected health information. In the case of VA Employees and ADNA 
coordinators, a breach in confidentiality could affect their employment. The project staff 
are aware of these risks and have outlined precautionary measures to reduce this risk as 
much as possible in the project. 
 
Data Assurances - All study staff will be monitored by the Dr. Rudolph and Local Site PIs for 
consenting, protocol, and information security adherence. Intensive training (including all 
VHA mandated trainings) at study initialization will be conducted along with annual 
refresher courses. Training certificates for all study staff members will be kept on file with 
the PVAMC Research Office. Dr. Rudolph will terminate access to study records when a user 
no longer requires access to them. 
 
If there is a theft, loss, or other unauthorized access of study data or storage devices 
associated with this study, this will be reported according to VA regulations. The local 
ISO/PO will be informed if there is any theft, loss, or other unauthorized access of sensitive 
data or storage devices and non-compliance with security controls. 
 
Electronic Health Data/Secondary Data Storage (Obtained through DUAs) 
 
All electronic health data, including all Personal health information and HIPAA identifiers 
from the secondary data used in objectives 1 and 2 will be stored and accessed exclusively 
on the Providence VA APP 26 research sever in a restricted access folder limited to 

Version 04/23/2018 
 
26 
approved project members. Data related to this project containing no PHI or HIPAA 
identifiers will be stored on the Providence research server on 
(\\R04pronas21.v01.med.va.gov\RESEARCH_PROTOCOLS\Rudolph\VD-HCBS-R, or 
VHAPROAPP26 H:\Research_Protocols_COIN\Rudolph).  Please see section “Description of 
Providence Research Computing Infrastructure” for further information.  
 
Any paper data generated in connection with this research study will be stored at the 
Providence VA Medical Center in a restricted access building, in a locked office, in a locked 
cabinet/drawer. In there is a theft, loss or other unauthorized access of study data or 
storage devices associated with this study, this will be reported according to VA regulations. 
All electronic data collected as a part of this study will be stored an analyzed on the 
Providence VA Research server. Data will only be disseminated in aggregate form and will 
not include any identifiable information. All records including qualitative interview 
recordings will be maintained in accordance with the Department of Veterans Affairs record 
Control Schedule 10-1. Should an inappropriate disclosure of the data occur, the Privacy 
Officer and ISO will be notified immediately. 
 
 
 
 
Qualitative Interview Data Storage: 
 
There are two types of qualitative interview data to be collected as a part of this project. 
These are led by separate Investigators and will use different data collection methods. 
 
The list of interview participants for Objective 1 and 3 will be maintained exclusively on the 
Providence VA server in a folder 
(\\R04pronas21.v01.med.va.gov\RESEARCH_PROTOCOLS\Rudolph\VD-HCBS-R ) with access 
restricted to project team members. The Providence VA server complies with all VA Data 
Security regulations. Only research team members with the proper permissions will have 
access to this list. Hard copy and electronic interview notes will be labeled only with a 
coded identification number. The crosswalk of names to identification numbers will be 
stored in hard copy only in a separate, secure file cabinet in Building 32 on the Providence 
VAMC campus accessible only to the project team as needed to complete their project 
duties.  
 
As described above in the methods section and noted in the information consent sheet, 
interviews will be recorded exclusively for transcription purposes. Veteran Interviews will be 
conducted by Ellen Mahoney of Boston College a Providence VA IPA employee but recorded 
by a Providence VA study member also attending the call.  All participants will be reminded 
that they are being recorded and whom is on the call. No name, PHI, PPI or HIPAA 
identifiers will be collected in these interviews. After obtaining the appropriate permissions 
and sanctuary to insert the recorder via its USB port to a VA computer, a Providence VA 

Version 04/23/2018 
 
27 
study member will upload the audio to a restricted file on the protected Providence server 
(\R04pronas21.v01.med.va.gov\RESEARCH_PROTOCOLS\Rudolph\VD-HCBS-R). 
 
Alternatively, if the Providence staff is unavailable to record, the interview will be recorded 
on the recorder by Dr. Mahoney, and Dr. Rudolph will physically claim the recorder and 
transport it to the Providence VA in a HIPAA compliant locked bag.  The interviews will be 
uploaded as stated above. All recorders will be kept in a locked office in a locked room in 
both the Providence and Boston locations. 
 
Dr. Nina Sperber at the Durham VAMC will be conducting the Coordinator/ key informant 
interviews. An approved digital recording device will be used to audio-record the key 
informant interviews. No PHI, PPI or HIPAA identifiers will be collected as a part of these 
interviews.   The recordings will be recorded directly into a restricted access folder on the 
Providence server, or will be recorded onto a VA approved recording device and uploaded 
immediately to a restricted-use folder on the Providence VAMC server and then deleted 
from the recording device.  
 
For all transcriptions not completed by a project team member, copies of the files will be 
accessed by the HSR&D Centralized Transcription Service Program (CTSP), Salt Lake City, UT, 
one of our local sites.  The CTSP is a national VA service, housed behind the VA firewall in 
the Veterans Health Administration Salt Lake City’s (VHASLC) Informatics, Decision-
Enhancement and Analytic Sciences (IDEAS 2.0) Center of Innovation (COIN).  All leadership 
and staff are VA employees. All data will reside on the Providence Server and no data will be 
copied onto the local VA SLC files.  Their service is approved by the VA SLC IRB and there is 
no outside contracting.  Data analyses for the qualitative aims will be done using NViVO 
and/or Atlas software. All records including qualitative interview recordings will be 
maintained in accordance with the Department of Veterans Affairs record Control Schedule 
10-1. Should an inappropriate disclosure of the data occur, the Privacy Officer and ISO will 
be notified immediately 
 
 
Description of Providence Research Computing Infrastructure: 
 
The VA Providence computer system is highly secure, and accessible only to authorized users. The 
Providence research computing infrastructure consists of a Windows server accessed via 
remote desktop sessions and client Windows PCs. Network security is provided by a 
combination of VA firewalls, local network access controls, and continuous auditing and 
monitoring for security breaches. Access from systems external to the VA’s intranet and 
COIN server is limited to encrypted channels, e.g. VPN. Person-identifiable as well as 
partially-de-identified data housed on the Providence research server are restricted to the 
system analyst at IRM and to the project’s approved staff and investigators. No data are 
allowed to be transferred to other computers (desktops or laptops) unless they are stripped 
of all identification information. VA employees have signed a VA privacy agreement to 

Version 04/23/2018 
 
28 
ensure confidentiality of all sensitive information, and its violation may result in criminal 
charges and a fine from $5,000 to $20,000. 
 
In summary, the Providence VAMC research computer system is highly secure, and 
accessible only to authorized users.  Within the group of authorized users, access to project 
data is restricted to individuals who are authorized to work on that specific research 
project. All identifiable data including secondary data, digital audio files and text documents 
will be stored in restricted folders.  
 
Violations of data policy or approved use of data will be subject to full penalty of law, which 
may include suspension of access privileges, reprimand, suspension from work, demotion, 
removal, and criminal and civil penalties. 
 
Upon completion of the research project, the study principal investigator in conjunction 
with the VA Information Security Officer (ISO), and in accordance with VA policy, will ensure 
that, study data containing sensitive, confidential information will be returned to the VA, 
sanitized and removed from all servers, desktops, removable storage devices, etc. 
 
Transfer of Data:   
Secondary electronic data will be to be transferred to the Providence VA following an 
approved Data Use Agreements with the data owners.  Following an approved DUA, data 
will be transferred to the Providence VA via an approved method of transfer such as a 
secured share or VA encrypted email. Project staff will work with VA Information Security 
Officers and VA Privacy Officers to ensure that all measures are taken to ensure the security 
of the data.  
 
In the case that interviews are recorded by project investigator Ellen Mahoney, the digital 
recorder containing the interview will need to be physically transported by Dr. Rudolph to 
the Providence VA for the files to be uploaded. Dr. Rudolph will physically claim the 
recorder and transport it to the Providence VA in a HIPAA compliant locked bag.  The 
interviews will be uploaded to the Providence Research Server upon his arrival back that the 
Providence VAMC.    
 
 
 
 
Reporting:  
Any Unanticipated Problems Involving Risks to Subjects or Others, Serious Adverse Events, 
Protocol Deviations, Apparent Serious Noncompliance, and Information Security Incidents 
will be reported as outlined in VHA Handbook 1058.01.  All staff will be instructed to notify 
Dr. Rudolph immediately if a problem arises.  The ISO and Privacy Officer will be notified 
within one hour of improper use or disclosure of data.  
 

Version 04/23/2018 
 
29 
9.0 Communication Plan / Project Management 
Dr. Rudolph will serve as the administrative and coordinating principal investigator and will 
oversee the management of the project team. He will supervise the program manager and 
research assistants for the evaluation and reporting. Dr. Rudolph will be contact PI for the 
CIRB submission and will be responsible for submitting all necessary documents to VA 
HSR&D.   
 
Dr. Rudolph will meet weekly with the entire research team. Project sub-teams may be 
organized in addition when data analyses are active.  Minutes will be taken during all 
meetings and will be circulated to the group with a list of action items for team members to 
accomplish during an allotted time frame.   
 
This project is funded for 3 years. The following Gantt chart outlines the timeline for the 
expected objectives.  
 
 
 
VD-HCBS Project Gantt Chart:  
 
 
FY 17 
FY 18 
FY 19 
FY 20 
Tasks and Relevant Objective 
Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 
DUA with GEC for the Veteran 
Experience Survey (1) 
x 
X 
 
 
 
 
 
 
 
 
 
Analyze Veteran Experience Survey 
(1) 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
Interview Veterans (1) 
 
x 
x 
x 
x 
x 
x 
x 
x 
x 
Interview Caregivers 
 
 
 
 
 
x 
x 
x 
x 
x 
 
Conduct qualitative Veteran 
Interview analysis (1) 
 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
Conduct mixed methods analysis (1) 
 
x 
x 
x 
x 
x 
x 
x 
x 
Obtain DUA with GEC and VA CARES 
for Caregiver Survey (2) 
 
 
x 
x 
 
 
 
 
 
 
 
Obtain VD-HCBS Caregiver Survey 
Data (2) 
 
 
 
 
x 
X 
 
 
 
 
 
Obtain VA CARES Data for 
Comparison (2) 
 
 
 
 
x 
x 
 
 
 
 
 
Analyze Caregiver survey (2) 
x 
X 
x 
x 
x 
x 
x 
Pilot implementation interview 
guides (3) 
x 
Pilot ERIC questionnaire and email 
protocol (3) 
x 
x 
Select sites for in-depth interviews 
x 
x 
 
x 
 
x 
 
x 
x 

Version 04/23/2018 
 
30 
(3) 
Collect data via participant 
observation and interviews   (3) 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
Prepare and disseminate reports 
and presentations (1-3) 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
x 
 
 
10.0 
Study Closure 
The CIRB will be notified upon study closure of the local and main sites (Forms 117a and 117b 
will be completed and sent to the IRB).  Any remaining staff’s folder access will be revoked.  
Should any outstanding analysis on de-identified data remain, this will be addressed in the CIRB 
forms 117a and 117b.   All records will be maintained in accordance with the Department of 
Veterans Affairs record Control Schedule 10-1. 
11.0 
References 
 
1. Uniform Geriatrics and Extended Care Services in VA Medical Centers and Clinics: VHA 
HANDBOOK 1140.11. Washington, DC: Department of Veterans Affairs; November 4, 2015. 
2. VA Office of the Inspector General. Audit of Selected Non-Institutional Purchased Home Care 
Services. Washington, DC: Office of Audits and Evaluation; 2013. 
3. Department of Veterans Affairs’ Office of Budget: FY2015 Congressional Submission. 
Washington, DC: Veterans Affairs; 2014. 
4. Eiken S, Sredl K, Burwell B, Saucier P. Medicaid Expenditures for Long-Term Services and 
Supports (LTSS) in FY 2013: Home and Community-Based Services were a Majority of LTSS 
Spending. Cambridge, MA: Truven Health Analytics; June 30, 2015. 
5. Edes T. Innovations in homecare: VA home-based primary care. Generations: Journal of the 
American Society on Aging 2010;34:29-34. 
6. Shay K, Yoshikawa TT. Overview of VA healthcare for older veterans: Lessons learned and 
policy implications. Generations: Journal of the American Society on Aging 2010;34:21-8. 
7. Carlson BL, Foster L, Dale SB, Brown R. Effects of Cash and Counseling on personal care and 
well-being. Health Serv Res 2007;42:467-87. 
8. Simon-Rusinowitz L, Loughlin DM, Ruben K, Garcia GM, Mahoney KJ. The benefits of 
consumer-directed services for elders and their caregivers in the cash and counseling 
demonstration and evaluation. Public Policy & Aging Report 2010;20:27-31. 
9. Doty P, Mahoney KJ, Simon-Rusinowitz L. Designing the Cash and Counseling Demonstration 
and Evaluation. Health Serv Res 2007;42:378-96. 
10. Mahoney E, Kayala D. Veteran-Directed Home and Community-Based Services: A Program 
Evaluation. Boston, MA: Trustees of Boston College, National Resource Center for Participant-
Directed Services; 2013. 
11. Thomas KS, Allen SM. An Interagency Partnership to Deliver Veteran-Directed Home and 
Community-Based Services: Interviews with Aging and Disability Network Agency Personnel 
Regarding their Experience with Partner Veteran Affairs Medical Centers. Journal of 
Rehabilitation Research and Development In Press. 

Version 04/23/2018 
 
31 
12. Rabow MW, Hauser JM, Adams J. Supporting family caregivers at the end of life: "they don't 
know what they don't know". Jama 2004;291:483-91. 
13. Zarit SH. Family care and burden at the end of life. CMAJ : Canadian Medical Association 
Journal 2004;170:1811-2. 
14. Schulz R. Caregiving as a risk factor for mortality: The caregiver health effects study. JAMA : 
the journal of the American Medical Association 1999;282:2215. 
15. Calhoun PS, Beckham JC, Bosworth HB. Caregiver Burden and Psychological Distress in 
Partners of Veterans with Chronic Posttraumatic Stress Disorder. Journal of Traumatic Stress 
2002;15:205-12. 
16. HWANG SS, CHANG VT, ALEJANDRO Y, et al. Caregiver unmet needs, burden, and 
satisfaction in symptomatic advanced cancer patients at a Veterans Affairs (VA) medical center. 
Palliative & Supportive Care 2003;1:319-29. 
17. Beckham JC. Caregiver burden in partners of Vietnam War veterans with posttraumatic 
stress disorder. Journal of consulting and clinical psychology 1996;64:1068. 
18. Institute TNAfCAPP. Caregiving in the US2015. 
19. Tilly J, Wiener JM, Cuellar AE. Consumer-directed home and community services programs 
in five countries: policy issues for older people and government. Washington, DC: The Urban 
Institute; 2000. 
20. Foster L, Randall, B., Phillips, B. & Carlson, B. The Effects of Cash and Counseling on the 
Primary Informal Caregivers of Children with Developmental Disabilities. 2005. 
21. Foster L, Brown R, Phillips B, Carlson BL. Easing the burden of caregiving: the impact of 
consumer direction on primary informal caregivers in Arkansas. The Gerontologist 2005;45:474-
85. 
22. Foster L, Randall, B., Phillips, B. & Carlson, B. How Cash and Counseling Affects Informal 
Caregivers: Findings from All Three States. 2005. 
23. Foster L, Dale SB, Brown R. How Caregivers and Workers Fared in Cash and Counseling. 
Health Services Research 2007;42:510-32. 
24. Sussman T. The Influence of Community-Based Services on the Burden of Spouses Caring for 
Their Partners with Dementia. Health & social work 2009;34:29. 
25. Veterans-Directed Home and Community-Based Services Program: San Diego Veterans 
Independence Services At Any Age (SD-VISA) Fact Sheet. San Diego, CA: Veterans 
Administration San Diego Health System; October 2015. 
26. Abdella V. Veteran-Directed Home and Community Based Services: Getting Started and 
Keeping Growing. National n4a Concerence; 2015; Philadelphia, PA. 
27. Snider-Meyer JM, Minkin AJ, Olson DA, Gresser SK, Smith HM, Kier FJ. Comparison of a 
Veterans-Directed Home Care Program vs Community Nursing Home Placement. Fed Pract 
2013;30:24-30. 
28. Affairs USDoV. MyVA Integrated Plan (MIP)2015 July 30, 2015. 
29. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. 
Contemporary clinical trials 2007;28:182-91. 
30. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method 
designs in implementation research. Adm Policy Ment Health 2011;38:44-53. 
31. Fruhwirth-Schnatter S. Finite mixture and Markov switching models 2006. 
32. Izenman A. Modern multivariate statistical techniques 2008. 

Version 04/23/2018 
 
32 
33. Schwarz G. Estimating the dimension of a model. The Annals of Statistics 1978;6:461-4 
34. Hill CE KS, Thompson BJ, Williams EN, Hess SA, Ladany N. Consensual Qualitative Research: 
An Update. Journal of Counseling Psychology 2005;52:196-205. 
35. Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qualitative 
health research 2005;15:1277-88. 
36. Crabtree BF, Miller WL. Doing Qualitative Research in Primary Care: Multiple Strategies. 2nd 
ed. Newbury Park, CA: Sage; 1999. 
37. Carpenter J, Kenward M. Multiple imputation and its application: John Wiley & Sons; 2012. 
38. Little R, Rubin, DB. Causal Effects in Clinical and Epidemiological Studies Via Potential 
Outcomes: Concepts and Analytical Approaches. Annual Review of Public Health 2000;21:121-
45. 
39. Stuart EA. Matching Methods for Causal Inference: A Review and a Look Forward. 2010:1-21 
40. Abadie A, Imbens GW. Large Sample Properties of Matching Estimators for Average 
Treatment Effects. Econometrica 2006;74:235-67. 
41. Austin PC, Small DS. The use of bootstrapping when using propensity-score matching 
without replacement: a simulation study. Statistics in Medicine 2014;33:4306-19. 
42. Sobo EJ, Simmes DR, Landsverk JA, Kurtin PS. Rapid Assessment with Qualitative Telephone 
Interviews: Lessons from an Evaluation of California’s Healthy Families Program & Medi-Cal for 
Children. American Journal of Evaluation 2003;24:399-408. 
43. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using 
the consolidated framework for implementation research (CFIR). Implementation Science 
2013;8:1-17. 
44. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qualitative health 
research 2005;15:1277-88. 
45. Hill CE KS, Thompson BJ, Williams EN, Hess SA, Ladany N. Consensual Qualitative Research: 
An Update. Journal of Counseling Psychology 2005;52:196-205. 
46. Cragun D. Qualitative Comparative Analysis: A Hybrid Method for Identifying Factors 
Associated With Program Effectiveness. Journal of mixed methods research 2015. 
47. Baptist CB, B. Qualitative Comparative Analysis - A Rigorous Qualitative Method for 
Assessing Impact2015. 
 
 
 
 
 
 
 
 
 
 
 
 

Version 04/23/2018 
 
33