Overview of Mixed Methods to Assess Short-Term Impacts of 

VA Caregiver Support on Veterans and Caregivers

-Courtney Harold Van Houtven, PhD

-Director, The VA Caregiver Support Evaluation Center (VA CARES 

Evaluation Center)

-Core Investigator, Durham VA HSR&D COIN
VETERANS HEALTH ADMINISTRATION

Outline of Today’s Talk

I. VA Caregiver Support Program Overview

II. VA CARES Evaluation Center 

– Evaluation Plan 

 

2 
I. VA Caregiver Support Program Overview 

 

Background slides provided by Ms. Laura Taylor, CSP

Care Management & Social Work Services

Department of Veterans Affairs

 
VETERANS HEALTH ADMINISTRATION 4 

Creation of the VA Caregiver Support Program 

4

P.L. 111 – 163 – the Caregivers and Veterans Omnibus Health Services Act of 2010 

was signed into Law by President Obama on May 5, 2010. Title One –Sections 101-104 

outlined specific new services to be provided for Caregivers of Veterans. 

The law established:

1. Program of Comprehensive Assistance for family Caregivers for Caregivers of 

eligible Veterans injured in the line of duty on or after September 11, 2001

2. Program of General Caregiver Support Services for Caregivers of all Veterans in 

need of a Caregiver

Organizationally, the Caregiver Support Program Office is within Care Management 

and Social Work Services, Patient Care Services. 
VETERANS HEALTH ADMINISTRATION

Caregiver Support Program

5 

Mission Statement: To promote the health and well-being of family Caregivers who 

care for our nation’s Veterans, through education, resources, support, and services.

• Allow Veterans to remain at home in the community

• Address specific needs of Family Caregivers with a menu of 

programs/services

• Promote Veteran & Caregiver health and well-being

• Provide one location to obtain needed information 

• Provide training & information on common conditions

• Reduce isolation with professional & peer support

• Provide options to give Caregivers respite

• Sensitize health care providers to the Caregivers’ role
VETERANS HEALTH ADMINISTRATION

Caregiver Support Coordinators

6 

• At all VA medical centers (n=225)

• Clinical experts on Caregiver issues 

• Knowledgeable about VA and non-VA resources 

• Organize Caregiver focused activities and services

• Assist with application for Comprehensive 

Assistance Program
VETERANS HEALTH ADMINISTRATION

Outreach to Caregivers

 - VA Caregiver Support Website

7 

• Caregiver Support Coordinator 

locator

• Contact info for Support Line

• “Caregiver Toolbox”

• Caregiver “Connections” with stories 

from peer Caregivers

• Link to application for new 

benefits/services with live chat 

support

• Videos of Family Caregivers and 

Veterans

www.Caregiver.va.gov
VETERANS HEALTH ADMINISTRATION

Outreach to Caregivers

 - Caregiver Support Line (CSL)

8 

• Staffed by licensed social workers

• 24/7 coverage provided by 

Veteran’s Crisis Line

• Follow-up provided by local 

Caregiver Support Coordinators
VETERANS HEALTH ADMINISTRATION

Program of Comprehensive Assistance for Family 

Caregivers - Eligibility Criteria

Veterans OR Service member undergoing medical discharge, who incurred or aggravated a 

serious injury in the line of duty on or after September 11, 2001 (includes Traumatic Brain 

Injury, psychological trauma or other mental disorders).

and

 

Veteran or Service member requires on-going assistance from a Caregiver for

a minimum of 6 months to: 

1. Assist with the management of personal care functions required in everyday living; 

2. Significantly enhance the Veteran’s or Service member’s ability to live at home 

safely; and 

3. Support the Veteran or Service member’s potential progress in rehab. 

**Note: there are additional criteria beyond what is listed here.

9 
VETERANS HEALTH ADMINISTRATION

Primary Family Caregiver Benefits

10

• Stipend 

– Direct payment to primary Family Caregiver

– Based on wages of a home health aide in the geographic area where the 

Veteran resides 

• $613-$2340 depending on Tier

• Healthcare Coverage

– Civilian Health And Medical Program of the Department of Veterans Affairs 

(CHAMPVA) 

• Travel, Lodging and Per Diem

– Integrated into existing beneficiary travel program

• Mental Health Services

– Individual/Group psychotherapy and counseling

• Standardized Core Curriculum and Ongoing Support and Certification
VETERANS HEALTH ADMINISTRATION

Program of Comprehensive Assistance for Family 

Caregivers - Current Data (May 2011 – April 2014)

11

• Applications filed: ~28,000

• Approved Applications: ~15,000

• New Healthcare Coverage enrollees: ~4,000

• Caregiver Demographics:

• 91% women

• Spouses (78%) and Parents (13%) 

• 26-40 yrs old (53%) and 41-64 years old (35%) 
II. Overview of Mixed Methods to Assess Short-Term 

Impacts of VA Caregiver Support on Veterans and 

Caregivers
VETERANS HEALTH ADMINISTRATION

Two year partnered evaluation with CSP, QUERI and 

Durham VA HSR&D 

HSR&D Durham 

• Courtney Van Houtven 

• Susan Hastings 

• Eugene Oddone 

• Matt Maciejewski 

• Nina Sperber

• Maren Olsen

• Darryl Wieland 

• Karen Steinhauser

• Corrine Voils 

• Cristina Hendrix

• Karen Stechuchak 

• Merritt Schnell

• Jennifer Lindquist

• Valerie Smith

• Bradley Dokter

• Avi Alkon

• Kate Miller

CSP – VACO

• Margaret Kabat

• Margaret Campbell-Kotler

• Jennifer Henius

QUERI

• Amy Kilbourne

• John Midolo

• Linda McIvor

13
VETERANS HEALTH ADMINISTRATION

Objective

To provide a rigorous evaluation of the short-term impacts of CSP on 

Caregivers and Veterans, to help CSP achieve its mission of providing the 

most effective support to Caregivers and Veterans that they can within the 

guidelines of the law. 

 

14
VETERANS HEALTH ADMINISTRATION

Evaluation Questions

 

AIM 1: Does Caregiver support reduce veteran avoidable use?

AIM 2: How does Caregiver support affect Caregiver wellbeing?

 

AIM 3: How do Caregivers use and value components of the comprehensive program 

and the general program?

 AIM 4: What is the value of services offered (comparison of services and costs)?

Results from this evaluation will begin to inform CSP about its return on investment 

and provide information on best practices for improving and better targeting its 

programs. 

15
VETERANS HEALTH ADMINISTRATION

Conceptual Model

CG & Veteran 

Baseline 

Characteristics

CSP

•General

•Comprehensive

CG Activities

1. Clinical

2. Psychological

3. Support seeking

4. Quantity of care

Caregiver 

Outcomes

Veteran 

Outcomes

Figure 1. Conceptual Model of CSP

VHA

Outcomes

16
17
VETERANS HEALTH ADMINISTRATION

AIM 1: Does Caregiver support reduce veteran avoidable use?

Study design 

• Pre-post cohort design 

• Non-equivalent control groups 

– Increase internal validity and increase the likelihood that we identify “true” program 

effects that are unbiased by confounding from other contemporaneous change that 

affect outcomes

18
VETERANS HEALTH ADMINISTRATION

• Treatment group

– All Veterans whose Caregivers were in The Comprehensive Program Jan. ‘14 

• Control group

– Veterans ‘administratively ineligible’ for The Comprehensive Program Jan. ‘14

Aim 1 Sample Inclusion

19
VETERANS HEALTH ADMINISTRATION

Aim 1 Outcomes–Utilization expected to be sensitive to CSP

• Potentially avoidable utilization 

– Potentially avoidable hospitalizations (PAH) and emergency department (PAED)

• Mental health outpatient care

• Home and Community-Based Services

• Home-Based Primary Care (HBPC) 

• Skilled home health

• Unskilled home health (Homemaker Home Health Aide Services)

• Adult day health 

• Respite care

• VHA perspective – VA-provided and VA-financed health care

20
VETERANS HEALTH ADMINISTRATION

Aim 1 – Longitudinal outcomes in 6 month intervals

Time in Months

Application Date 

(May ‘11-Jan ‘14)

PA ED use PA ED use PA ED use

t-12 t

0 t-6 t+6 t+12 

PA ED use

t-9 t-3 t+3 t+9 

PRE POST 

21
VETERANS HEALTH ADMINISTRATION

Aim 1 – Covariates

Variable Descriptions

DEMOGRAPHICS Insurance OTHER COVARIATES

Race Tricare Date of Injury

White Other private Date of Separation from Military

Black Medicare Branch of Military Service 

Other Race Medicaid Army

Unknown No non-VA insurance Navy

Ethnicity HEALTH STATUS Air Force

Hispanic TBI Diagnosis1 Marines

Marital Status SCI Diagnosis1 Medical Center where PCP located

Married Polytrauma Diagnosis1 Polytrauma Care Used

ACCESS TO CARE Mental Health Diagnoses Use of PRC PNSs

Eligibility for LTC PTSD Use of PSCT

 Service Connectedness Depression Use of POC

Low (<10%) Substance Use Disorder

Medium Low (11% to 49%) Anxiety Disorder Other?

Medium High (50% to 69%) Adjustment Disorder

High (70% and above) TBI & Mental Hlth Diagnosis

Means Test SCI & Mental Hlth Diagnosis

Musculoskeletal

22
VETERANS HEALTH ADMINISTRATION

Aim 1 Methods – (1) Propensity Score Methods

• Concern that control and treatment groups may be different at baseline

• Use propensity score to construct inverse probability of treatment weighting 

(IPTW) (Austin, 2011; Rosenbaum, 1987)

• Apply weights to create a pseudo-population 

– Distribution of the observed baseline covariates is independent of treatment. 

– Weights allow creation of an analytic sample that is more comparable between the two 

groups 

• Obtain Average Treatment Effect among the Treated (ATT)(Morgan & Todd, 2008) 

in our multivariable outcomes models

23
VETERANS HEALTH ADMINISTRATION

Aim 1 EXAMPLE OF POTENTIAL FINDINGS --

SIMULATED RESULTS 

24

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

12 6 0 6 12

Pr(Any PA ED use)

Application Date

Impact of CSP on Potentially Avoidable ED Visits

Participant

Non_Participant

Months
VETERANS HEALTH ADMINISTRATION

Aim 1 EXAMPLE OF POTENTIAL FINDINGS --

SIMULATED RESULTS 

25

0

1

2

3

4

5

6

7

8

9

10

12 6 0 6 12

Number of Visits

Application Date

Impact of CSP on Mental Health Outpatient Visits

Participant

Non_Participant

Months
VETERANS HEALTH ADMINISTRATION

AIM 2: How does Caregiver support affect Caregiver well-being?

CG & Veteran 

Baseline 

Characteristics

CSP

•General

•Comprehensive

CG Activities

1. Clinical

2. Psychological

3. Support seeking

4. Quantity of care

Caregiver 

Outcomes

Veteran 

Outcomes

Figure 1. Conceptual Model of CSP

VHA

Outcomes

26
VETERANS HEALTH ADMINISTRATION

Aim 2 - Caregiver Activities

27

Contents Source

Info 

About: 

CG or V

Caregiver Activities (clinical; psychological; support seeking; quantity)

A. Medication Management/

Adherence Voils et al. 2012 V

B. Coping Skills Pruchno & Resch 1989b CG

C. Self-care (Health Behaviors 

& Exercise) REACH I 2001 & REACH II 2004 CG

D. Community Engagement FACES 2009 CG

E. Communication with 

Providers Adapted from CAHPS© 2013 CG

F. Continuity of Care CAHPS© 2013 & REACH II 2004 CG, V

G. Time use (Quantity of 

Caregiving and Work 

Shocks

Health and Retirement Study 2012 & 

FACES 2009 CG
VETERANS HEALTH ADMINISTRATION

Aim 2 – Caregiver Outcomes

28

Contents Source

Info 

About: 

CG or V

Caregiver Outcomes

A. Perceived Financial Strain Given et al. 1992 CG

B. Positive Aspects of 

Caregiving Tarlow et al. 2004 CG

C. CAHPS© CAHPS© 2013 CG, V

D. Zarit Burden Bédard et al. 2001 & Zarit 1980 CG

E. CESD-10 Andresen et al. 1994 CG

F. Relationship Satisfaction Lawton et al. 1989 & Struchen et al. 2002 CG

G. Health Shocks Health and Retirement Study 2012 CG
VETERANS HEALTH ADMINISTRATION

Aim 2 – Veteran Outcomes

29

Contents Source

Info 

About: 

CG or V

Veteran Outcomes

A. Veteran Functioning Scoring Guide, Caregiver Support Program V

B. Veteran non-VA Health 

Care, incl mental health Adapted from Bhandari & Wagner 2006 V

Other Key Characteristics

C. Demographics, 

Socioeconomics Health and Retirement Study 2012 CG, V

D. Use and value of CSP 

programs/services Created by study team CG
VETERANS HEALTH ADMINISTRATION

AIM 2: How does Caregiver support affect Caregiver wellbeing?

30

Sample for cross-sectional survey

Objective (1)

Describe differences in Caregiver activities, skills, and outcomes between participants 

and non-participants
VETERANS HEALTH ADMINISTRATION

AIM 2: EXAMPLE OF POTENTIAL FINDINGS --SIMULATED 

RESULTS 

31

0

5

10

15

20

Days Exercise per Week Current Financial Situation

Strong=7 Weak=1

CESD-10 Satisfaction with VA

Healthcare for Vets

Differences in Caregiver Well-being by Participation in 

CSP

Participants

Non-Participants
VETERANS HEALTH ADMINISTRATION

AIM 2: How does Caregiver support affect Caregiver well-being?

Objective (2)

Describe the impact of The Comprehensive Program on Caregiver activities, skills, 

and outcomes by comparing changes in outcomes for new participants and nonparticipants 

Sample

32

Comprehensive

Program
VETERANS HEALTH ADMINISTRATION

Aim 2 – Objective (2) Sample 

Summer 2014

Application 

Date 

Administratively ineligible

In process

33

Summer 2015

Administratively ineligible

Newly enrolled
VETERANS HEALTH ADMINISTRATION

AIM 2: EXAMPLE OF POTENTIAL FINDINGS – 

SIMULATED RESULTS 

34

-5

0

5

Days Exercise per Week Current Financial Situation CESD-10 Satisfaction with VA Healthcare

for Vets

Impact of CSP on Caregiver Well-being

Participants

Non-Participants

No Change
VETERANS HEALTH ADMINISTRATION

Aim 2 -Survey Administration 

• Recruiting

– Letter mailed to all eligible Caregivers with 

• generic web link to survey 

• ~$2-$3 token incentive (i-tunes credit or Forever stamps)

– Email sent to Caregivers with individualized web link to survey

• +85% of Caregivers provided email addresses

– Will send reminder postcard at 2 weeks

• Caregiver can call by phone if do not want to complete by web 

35
VETERANS HEALTH ADMINISTRATION

AIM 3: How do Caregivers use and value components of the 

Comprehensive program and the General program? 

Study Design: Sequential quantitatively-driven mixed method design 

 (QUAN  qual) 

(1) Quantitative: Survey of Caregivers (Aim 2) will provide a population level picture 

of which services Caregivers use the most and regard as most helpful.

(2) Qualitative: Individual semi-structured interviews to understand in-depth 

contextual information about experiences using services

(3) Integrate findings at results phase. Qualitative to enhance understanding and 

interpretation of quantitative results (Morse & Niehaus 2009) 

36
VETERANS HEALTH ADMINISTRATION

AIM 3: How do Caregivers use and value components of the 

Comprehensive program and the General program? 

Sample

• Quantitative: Aim 2 survey sample of treated/enrolled Caregivers 

– 13,775 Comprehensive Program Caregivers

– 3,496 General Program Caregivers

• Qualitative: Purposeful sampling to select Caregivers from among 10 local 

programs with attention to geography, program size, services offered, etc.

– 50 Comprehensive Program Caregivers

– 50 General Program Caregivers

37
A. Use and Value of CSP Programs/Services

1. Do you use [insert service from list below]? (yes/no)

a. Building Better Caregivers 

b. Caregiver Support Lline; 

c. Self-care courses; 

d. Peer support program 

e. Direct counseling in person from a 

CSC at your usual VAMC

f. Direct counseling on the phone from 

a CSC at your usual VAMC;

g. Referral for other services from a 

CSC? 

h. Training from a CSC online

i. Training from a CSC face to face

j. Training from a CSC Online? 

2. How often would you say you access this service? 

a. Once a day

b. Once a week

c. Once a month

d. Once every few months

e. Once a year

3. On a scale of 1 to 10, with 1 being not helpful at all and 10 being very helpful, how helpful is [services in 

q. 1] program for you?

4. [ASK ONLY COMPREHENSIVE CAREGIVERS] On a scale of 1 to 10, with 1 being not at helpful and 

10 being very helpful, how helpful would you say are the following parts of the Caregiver Program?

a. Stipend

b. Travel expenses when Veteran

receives care

c. Health insurance for you 

(CHAMPUS)

d. Mental health services for you

e. Caregiver training provided by 

Easter Seals

f. Respite care 

5. How has the stipend impacted your life? [Text Box]

6. Please state the extent to which you agree or disagree with the following statements? (5 response 

options, 1= strongly disagree to 5= strongly agree). The caregiver support program has helped me:

a. To navigate the VA health care system

b. To schedule appointments for the Veteran

c. To schedule appointments for myself

d. As a point of entry to the VA when the Veteran 

was discharged from the military.

38

INITIAL DRAFT – TO BE REFINED
VETERANS HEALTH ADMINISTRATION

Aim 3 Analysis Plan - Quantitative 

• For each component and service of CSP 

– Mean frequency of use

– Mean level of perceived helpfulness

– Mean agreement that the CSP has helped them with caregiving activities. 

• Calculate cross-sectional differences by Caregiver criteria, including 

socioeconomics and demographics. 

39
VETERANS HEALTH ADMINISTRATION

Aim 3 Qualitative 

• 30 minute in-depth interviews with each Caregiver by phone

• We will ask participants to reflect upon their experiences with the program and 

discuss: 

– What portions of the program they used and under what contexts and conditions

– What they found more and less helpful

– Needs not addressed by the program 

• Prior to fielding the interviews, we will consult with key stakeholders at Central 

Office, including program designers, to refine our questions.

• All interviews will be audio recorded and transcribed. 

40
VETERANS HEALTH ADMINISTRATION

Aim 3 Analysis Plan - Qualitative 

• Directed approach to content analysis

• Will develop codes based on predetermined categories, such as those derived 

from interview questions. 

– Also allow emergent themes. 

• Use theme matrix technique to identify patterns in the data based on program or 

Caregiver characteristics. 

– See if they might be differences

41
VETERANS HEALTH ADMINISTRATION

Aim 4 -What is the value of services offered?

This aim is provisional and formative. 

Study Design

• Hybrid approach: (a) data from Aims 1-3; (b) budget data, and (c) a brief 

survey of Caregiver Support Coordinators on time use

– Model the effect of The Comprehensive Program on Veteran health care costs

– Quantify delivery costs of CSP programs and services and compare them to program 

components considered of very high value

• Achieving this aim will allow for an initial assessment of the relative value 

of programs and services and return on investment for VHA 

42
VETERANS HEALTH ADMINISTRATION

Aim 4 -Proposed metrics to frame value

Costs: 

• Direct input costs: cost of programs and services 

• Indirect input costs: cost of CSC time spent on specific tasks

• Direct VHA costs: Veteran resource utilization costs

Utility:

• Caregiver satisfaction with VHA, CSP and its benefits and services

• Caregiver evaluation of relative usefulness of national programs (eg Caregiver 

Support Line; Caregiver training) 

• Caregiver evaluation of relative usefulness of Comprehensive Program services 

(e.g. Stipend, health insurance, mental health care)

43
VETERANS HEALTH ADMINISTRATION

Aim 4 -What is the value of services offered?

44
VETERANS HEALTH ADMINISTRATION

Questions and Comments? 

Courtney H. Van Houtven, PhD MSc. 

Director, VA CARES Evaluation Center

Core Investigator, Durham COIN, HSR&D, Durham VAMC 

courtney.vanhoutven@va.gov

courtney.vanhoutven@duke.edu 

45
VETERANS HEALTH ADMINISTRATION

Draft Interview Protocol

• Please tell me about your experience using the ____________________ program.

• When did you start using it? How were you referred? 

• What were the circumstances that led to you enrolling in the program?

• Which specific program components have you used?

• Did you have experience using (INSERT SERVICE, for example Caregiver support 

hotline)? If so, would you tell me about your experience with that.

• Tell me about those parts of the program that you found most helpful about the 

______________________ program? 

• Tell me about those parts of the program that you found least helpful?

• Tell me about any needs you had as a Caregiver that were not covered by the 

program. 

• Tell me about any parts of the program that were available but you could not 

access?

• Note: We will probe all themes elicited spontaneously, during the interview. 46
VETERANS HEALTH ADMINISTRATION

Menu of VA Services

47

In-Home Care

Skilled Nursing

Home Health Aide

Home Based Primary Care

Veteran Directed Home & 

Community Based Care

Respite Care

Services to Address Mobility Issues

Equipment

Home Modification

Automobile Modification

Education and Training

Financial Support

Aid and Attendance

Caregiver Stipend (Post 9-11) 

Information and Referral

Caregiver Web site: www.Caregiver.va.gov

Caregiver Support Line: 1-855-260-3274 

 

 

Caregiver Support

Support Groups

Caregiver Support Coordinators

Peer Support Mentoring Program

Caregiver Education and Training

Building Better CaregiversTM 

Standardized Core Curriculum 
VETERANS HEALTH ADMINISTRATION

 

Program of Comprehensive Assistance for Family Caregivers 

Caregiver Education and Training

 

Standardized Core Curriculum 

• Developed in collaboration with Easter Seals 

• In-person, workbook/DVD, or web-based

• Includes modules on:

 Caregiver Self-Care

 Home Safety

 Basic Caregiver Skills (vital signs, etc)

 Providing Personal Care

 Managing Challenging Behaviors

 Resources (legal, financial, VA, community, advocacy)

Training Data Update:

• More than 18,400 Family Caregivers have completed training to date 

48
VETERANS HEALTH ADMINISTRATION

On-going Support to Veteran & Caregiver

• In-Home initial visit by VA Clinicians

• Well-being checks every 90 days or as clinically indicated

• Home visits are supportive and instructive

• Evaluation of Veteran and Caregiver’s physical and emotional 

well-being 

• Recommendations for additional training, support, 

equipment, etc. 

49