HEALTHCARE AND HUMAN SERVICES POLICY, RESEARCH, AND CONSULTING―WITH REAL-WORLD PERSPECTIVE.

Exploratory Study of the Global Outcomes of the Older Americans Act 

Programs and Services

Older Americans Act Programs and Services Final Report

Prepared for: Department of Health and Human Services, Administration on Aging

Submitted by: The Lewin Group, Inc.

March 2013
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services <Table of Contents

Executive Summary i

Table of Contents

Executive Summary ...................................................................................................... 1

Overview of Study Design Process.......................................................................... 1

Organization of Report............................................................................................. 1

Older Americans Act Title III Overview .................................................................... 1

Table 1: OAA Title III Services .......................................................................................................1

Literature Review..................................................................................................... 3

Methods and Overall Search Results.............................................................................................3

Elements and Relationships to Consider .......................................................................................4

Evaluation Design Options and Issues/Considerations .................................................................5

Findings and Lessons Learned from Past HCBS Studies.................................................................6

Available Data Sources and Data Elements ..................................................................................7

Study Design Framework ........................................................................................ 8

Logic Model ...................................................................................................................................8

Figure 1: Logic Model ....................................................................................................................9

Research Questions.......................................................................................................................9

Defining the Scope of the Study ..................................................................................................10

Proposed Study Design ......................................................................................... 11

Figure 2: Proposed Study Design Image......................................................................................12

Proposed Study Design Description.............................................................................................13

Limitations and Recommendations........................................................................ 14

Limitations...................................................................................................................................14

Recommendations.......................................................................................................................14

Conclusion............................................................................................................. 16

Appendix A: Literature Review ................................................................................. 17

Appendix B: Study Framework and Design........................................................... 108

Appendix C: Limitations and Recommendations.................................................. 130
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 1

Executive Summary

Overview of Study Design Process

In the process of creating this study design, the research team identified a number of 

strengths and limitations to conducting a global outcomes evaluation of the Older 

Americans Act (OAA). This report proposes a study design of OAA programs. This 

study will assist AoA in identifying actions needed to better position the states for 

participation in a nationwide evaluation. 

Organization of Report

This report is organized as an executive summary that references appendices that are 

the major sections of the project: Appendix A – Literature Review; Appendix B – Study 

Design; Appendix C – Limitations and Recommendations.

Older Americans Act Title III Overview

Older American Act programs and services represent a significant federal investment in 

developing a comprehensive, coordinated, and cost-effective system of home and 

community-based services (HCBS) that enables adults to live independent and healthy 

lives in their homes and communities. The services under OAA Title III are described in 

the table below:

Table 1: OAA Title III Services

Title Services

Title III-B: 

Supportive 

Services

and Senior 

Center 

Programsi

Title III-B funds a broad array of services that enable seniors to remain in their 

homes for as long as possible. These services include but are not limited to:

► Access services, such as transportation, case management, and information 

and assistance; 

► In-home services, such as personal care, chore, and homemaker 

assistance; and

► Community services such as legal services, mental health services, and 

adult day care.

This program also funds multi-purpose senior centers that coordinate and 

integrate services for older adults such as congregate meals, community 

education, health screening, exercise/health promotion programs and 

transportation.

 

i Administration on Aging. Supportive Services and Senior Centers Programs. Retrieved from: 

http://www.aoa.gov/AoARoot/AoA_Programs/HCLTC/supportive_services/index.aspx.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 2

Title Services

Title III-C: 

Nutrition 

Servicesii

The purpose of the OAA Nutrition Program is to:

► Reduce hunger and food insecurity

► Promote socialization of older individuals

► Promote the health and well-being of older individuals and delay adverse 

health conditions through access to nutrition and other disease prevention 

and health promotion services.

Congregate Nutrition Services and Home-Delivered Nutrition Services provide 

meals and related nutrition services to older individuals in a variety of settings 

including congregate facilities such as senior centers; or by home-delivery to 

older individuals who are homebound due to illness, disability, or geographic 

isolation. Services are targeted to those in greatest social and economic need 

with particular attention to low-income individuals, minority individuals, those in 

rural communities, those with limited English proficiency and those at risk of 

institutional care. Nutrition Services Programs help older individuals to remain 

independent and in their communities.

The OAA authorizes and provides appropriations to the Administration on 

Aging (AoA) for three different nutrition programs under Title III:

► Congregate Nutrition Services (Title III C1)

► Home-Delivered Nutrition Services (Title III C2)

► Nutrition Services Incentive Program (NSIP).

Title III-D: 

Disease 

Prevention 

and Health 

Promotion 

Servicesiii

Title III-D of the OAA provides grants to States and Territories based on their 

share of the population aged 60 and over for education and implementation

activities that support healthy lifestyles and promote healthy behaviors. Health 

education reduces the need for more costly medical interventions. Priority is 

given to serving elders living in medically underserved areas of the State or 

who are of greatest economic need. 

Title III-E: 

National 

Family 

Caregiver 

Support 

Programiv

The National Family Caregiver Support Program (NFCSP) provides grants to 

States and Territories, based on their share of the population aged 70 and 

over, to fund a range of supports that assist family and informal caregivers to 

care for their loved ones at home for as long as possible.

The NFCSP offers a range of services to support family caregivers. Under this 

program, States shall provide five types of services:

► information to caregivers about available services, 

► assistance to caregivers in gaining access to the services, 

 

ii Administration on Aging. Nutrition Services (Title III C). Retrieved from: 

http://www.aoa.gov/AoARoot/AoA_Programs/HCLTC/Nutrition_Services/index.aspx.

iii Administration on Aging. Disease Prevention and Health Promotion Services (Title III D). Retrieved 

from: http://www.aoa.gov/AoARoot/AoA_Programs/HPW/Title_IIID/index.aspx#purpose.

iv Administration on Aging. National Family Caregiver Support Program (Title III E). Retrieved from: 

http://www.aoa.gov/AoARoot/AoA_Programs/HCLTC/Caregiver/index.aspx.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 3

Title Services

► individual counseling, organization of support groups, and caregiver training, 

► respite care, and 

► supplemental services, on a limited basis

These services work in conjunction with other State and Community-Based 

Services to provide a coordinated set of supports. Studies have shown that 

these services can reduce caregiver depression, anxiety, and stress and 

enable them to provide care longer, thereby avoiding or delaying the need for 

costly institutional care.

Under this project, the Administration on Aging (AoA) seeks to study the impact of 

OAA programs and services, specifically services provided under Title III, on key 

outcomes, including HCBS use, health care use, community tenure, and long-term 

services and supports (LTSS) expenditures. OAA services and programs are diverse, 

often integrated and/or provided in combination with other services, funded through 

multiple funding streams, and administered and delivered by different state and locallevel agencies with varying data collection capacity. Therefore, studying OAA-funded 

programs and services is inherently challenging.

Prior to developing this design, The Lewin Group (Lewin) conducted a literature 

review, developed and presented a study framework, and completed a study design. 

These steps are discussed below.

Literature Review

The development of a comprehensive OAA Global Outcomes Study Design required 

background research on prior studies about the impact of HCBS on the four outcomes 

of interest in this project: (1) HCBS Use; (2) Health Care Use; (3) Nursing Home (NH)

Admission/Community Tenure; and (4) Cost Savings. To better understand potential 

study techniques and key variables related to these outcomes, Lewin conducted a 

review of the literature (Appendix A).

Methods and Overall Search Results

In order to conduct a thorough literature review, the Lewin team used articles from the 

Administration on Aging’s (AoA) literature collection of studies compiled by S3, 

previous literature reviews by the Lewin team, and a targeted updated search to 

identify new and additional relevant materials. After identifying approximately 900 

articles, the Lewin team applied inclusion/exclusion criteria to identify: 

► 19 key studies examining impacts of OAA services or new and useful findings on 

outcomes of interest related to OAA services, including HCBS use, health care use, 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 4

community tenure, NH admissions and costs used to inform the Findings and 

Lessons Learned Section;

► 100 additional studies and informational materials providing background 

information about OAA programs and services, key variables, and related 

outcomes; and, 

► 21 key studies and papers from the program evaluation and social research 

literature. These studies, in combination with the 19 key studies, informed the 

methods for this design. 

Elements and Relationships to Consider

LTSS presents a complex and fragmented delivery and financing system. A 

combination of federal, state and local sources fund a wide range of services to older 

persons residing in diverse communities and that require these services. To 

supplement these services, many older people rely on care provided by informal 

caregivers, caregivers paid out of pocket, or local agencies that provide services. Such a 

dynamic, multi-layered system requires tremendous coordination and constant 

adjustments to the changing policy environment and needs of the population. To study

such a complex system requires understanding the challenges of defining constructs 

(e.g., funding, mix of services, unmet needs of individuals, etc.) and the means to 

measure these constructs (e.g., self-report, use of administrative records, standardized 

scales, the pros and cons of various proxy measures, and other considerations). Despite 

the inconsistencies in definition, we found a series of broadly defined variables that 

demonstrated statistical significance, or were previously found to have demonstrated 

significance in the studies’ background research, with the four key outcomes of interest 

to this review: 1) HCBS service use, 2) health care use, 3) community tenure/NH use, 

and 4) LTSS expenditures.

We chose Andersen’s Behavioral Model of Health Services Use (Andersen, 1995) to 

classify the variables into the domains listed below. This model has been widely used 

to determine variables tested in medical care studies (Andersen, 1995; Miller and 

Weissert, 2000). This model highlights predisposing, enabling, and need characteristics 

as variables that contribute to use of services, including HCBS, health care, and nursing 

home use. The literature review presented findings from numerous studies on how the 

characteristics, or variables, contribute to use of and/or cost of services. For each 

domain, we examined the impact the variables had on the aforementioned four key 

outcomes of interest.

► Predisposing

 Demographic variables including: Age, Sex/Gender, Race/Ethnicity, 

Education, Geographic Location, Living Alone, Marital Status, Having 

Children, and Spousal Use of Services
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 5

► Enabling

 Demographic variables including: Income/Wealth, Home Ownership, 

Payment Method

 Informal Support: Availability and Use of Informal Support

 Caregiver Characteristics & Supports: Available Support for Caregiver,

Caregiver Age, Caregiver Burden, Caregiver Awareness of Services

 Policy/Funding/Supply Level: State/Medicaid HCBS Funding, Proportion of 

LTSS Spending on HCBS, Number of Home Health Agencies, County Supply 

of NH Beds, NH Diversion/Pre-admission Screening

► Need

 Health characteristics variables including: Perceived Unmet Need, General 

Poor Health, Alzheimer's/Cognitive Impairment/Dementia, ADL Limitations, 

IADL Limitations, Comorbidities, MH Health Status 

► Service Use, Mix and Spending Variables

 Acute: Hospital Admissions/Use, Emergency Department Use

 Post-Acute: Case Management, Timing of Care, NH Use (short-stay)

 LTSS: HCBS Use, Volume of Services Received, Combination/Bundle of 

Services, Use of State/Medicaid/Medicaid Wavier Services, Use of OAA III-B, 

Use of OAA III-E, Use of Title III-C, Timing of Care, Case Management, IADL 

Assistance, NH Use (Long-Stay)

Lewin suggests that these variables be taken into consideration as part of the final study

design.

Study Design Options and Issues/Considerations

Determining the outcomes attributable to a program or service presents a central 

challenge for any impact study. A study must not only use the best and most-efficient 

design, but it also must address the needs and values of persons served by the program 

and prove useful to informing policy and practice. Our review of previous HCBS 

studies and pertinent materials from the general evaluation literature identified several 

designs to consider in studying OAA programs and services:

► Randomized field experiments, considered the gold standard in research design for 

proving impact, offer a design most suited for new program models or service 

packages. This approach may be less appropriate for established and scaled up 

programs where no suitable control group exists or it might be unethical to 

withhold services from a control group. The literature indicates that randomized 

experiments are less informative when studying complex social interventions, such 

as OAA services and programs, with substantial variability in local applications.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 6

► Non-randomized comparison groups measure outcomes for two or more groups,

but do not randomly assign participants to service receipt. Several HCBS studies 

have constructed comparison groups using different populations and analytical 

methods. Significant advances have been made in analytic methods that address 

vulnerabilities of comparison group designs and researchers have argued that 

results of this design can be extremely rigorous and emulate results of randomized 

experiments when certain criteria are met (Shadish & Cook, 2009). 

Various methodologies create comparison and treatment groups when using nonrandomized comparison designs, such as regression discontinuity and propensity score 

matching. These methods control for systematic differences in the comparison groups 

that would otherwise bias the results in a non-randomized design. Additionally, 

depending on the specifics of the research question and the context of the study, a 

variety of analytic techniques, such as survival analysis and structural equation 

modeling, can be applied to understand the impact and pathway of the outcome.

The literature frequently recommends combining more than one analytic 

methodology. More than one strategy may be needed to answer all study research 

questions. Also, because all methods have limitations, synthesizing findings across 

multiple measures and methods may be needed to gain a full understanding of the 

program.

In addition to the complex challenges associated with designing a valid and reliable 

impact study, we must consider the dynamic nature of the LTSS delivery system. At 

times programs expand based on new policy guidance, while, at other times, they 

contract in response to budgetary constraints. Further, each State and locality has 

different funding systems, policies, practices, and client needs that must be accounted 

for in any design. Lewin compiled a list of key questions to ask states in order to 

develop a study design that could best take into account the variation within states. 

This list is included in the Study Design (Appendix B).

Findings and Lessons Learned from Past HCBS Studies

Our synthesis of previous HCBS studies suggests several key findings, lessons learned, 

and recommendations to consider in studying the impact of OAA and non-OAA HCBS 

programs. These findings suggest that designing a study that produces credible, solid 

evidence of results will be of critical importance to AoA, the Aging Network, and 

service recipients and families who rely on OAA and non-OAA programs and services. 

Increasingly, stakeholders seeking continued or expanded funding for health and social 

programs must be able to “make the business case” for the investment, with rigorous 

research demonstrating success in improving peoples‘ lives and achieving more 

effective/appropriate use of economic resources. In addition to providing evidence of 

an impact, a study design that allows for exploration into the pathways through which 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 7

the impact is made (e.g., case management, targeting of services, individual belief about 

HCBS, etc.) will allow policy leaders to make informed decisions about how to allocate 

future funding in a way that results in the greatest possible impact on outcomes. The 

primary findings are listed below:

► OAA and non-OAA services work better when offered as part of a comprehensive 

system to meet individual needs.

 Coordinated systems of care at the national, state, and agency level are 

important considerations when evaluating the effects of OAA and non-OAA 

social support activities;

 New studies highlight the effectiveness of moderate services, including OAA 

home-delivered meals and IADL help;

 Availability of services and resources in the community play an important role 

in affecting OAA and non-OAA HCBS outcomes.

► For OAA and non-OAA services, the greatest HCBS impacts have been on 

community tenure, rather than health care utilization.

► Community tenure is associated with family caregiver availability and individual 

characteristics of both the caregiver and the consumer.

► An assessment of HCBS outcomes must consider the effect of OAA services, 

controlling for the impact of non-OAA services.

► Service use is affected by many individual characteristics, some of which are 

difficult to measure (e.g., changing needs over time).

► Virtually every study concluded that further research is needed to develop 

understanding of the complexities inherent in the LTSS delivery system.

Available Data Sources and Data Elements

Our review identified numerous data sources that have been used in recent related 

HCBS studies, the types of data available from each source, and examples of studies 

using the data. These include OAA-specific, state, national and qualitative data sources. 

In addition, we provide examples of surveys of HCBS programs that could potentially 

be used to capture some types of program information that have been identified in the 

literature as important to consider, but are not included in existing data sets.

A high-quality study design hinges on valid and reliable data. Regardless of how well a

study is designed, the outcome is only as good as the data used. While available data 

sources provide a cost-effective and efficient way for collecting data, all sources 

reviewed were found to have limitations in terms of the measures/indicators collected, 

the quality of the data, the frequency of the collection, or comparability of the 

measures/indicators over localities and states. Many studies have used supplemental 

surveys, questionnaires or interviews to collect additional data that complement the 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 8

data collected through regular reporting systems to allow for more robust evaluation 

results. While supplemental data collection methods may be costly, they are likely the 

only way to ensure the inclusion of high quality data on certain key variables (e.g., 

availability of informal caregivers, level of unmet need, functional status, etc.).

Study Design Framework

The Study Framework and Design (Appendix B) describes Lewin‘s proposed approach 

for understanding the impact of OAA programs and services on the aforementioned 

outcomes of interest. Building off of the literature review, this document includes a 

discussion of the challenges and trade-offs that must be considered in the selection of 

variables, research questions, study design options and secondary data sources, in order 

for AoA to endorse a rigorous impact study of OAA programs and services.

This design focuses on the impact of services provided through OAA-funded HCBS 

programs authorized under Titles III-B, C, D, and E of the OAA. These programs 

include a range of supportive services, nutrition services, health promotion and disease 

prevention programs, as well as services for family caregivers. While the mix and type 

of services offered differ by state and locality, the vast majority of OAA funding is used 

for the provision of nutritional services.

Logic Model

To best understand how OAA program activities relate to short and long-term 

outcomes, a logic model was developed in collaboration with ACL staff (Figure 1). The 

logic model depicts the inputs, activities, outputs, and outcomes of OAA-funded HCBS 

programs and services. We have included “non-OAA funded activities” (e.g., informal 

services, state-funded HCBS, private pay) as these activities often occur in combination 

with OAA-funded activities and impact the key outcomes of interest for this project.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 9

Figure 1: Logic Model

Research Questions

Based on the expected outcomes depicted in the logic model, we propose the following 

research questions. What is the impact of OAA-funded HCBS programs and services 

on:

► Community tenure

► Health care utilization

► Costs of care for older adults (e.g., LTSS, health care costs)

► Physical, mental, and emotional health and wellness (e.g., preventive measures) of 

care recipients and caregivers

► Unmet needs among older adults

► Caregivers (e.g., strain, burden, depression, health, etc.)

► Coordination of services (e.g., care management)
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 10

In addition to answering questions about the impact of service use on desired 

outcomes, we recommend that AoA consider a design that measures the strength of 

association of covariates/intervening variables on the key outcomes including, but not 

limited to, mix, type, and intensity of services. This step would answer the following 

sample questions:

► What is the impact of OAA services alone or in combination with services paid for 

by other sources? 

► What is the impact of service mix and intensity on outcomes of interest? [if possible, 

we will isolate OAA services]

► What subgroups had the most favorable outcomes? (e.g., health conditions, 

demographics, functional status)

Defining the Scope of the Study

A critical methodological decision hinges on how non-OAA funded services are 

factored into the study design. For example, does the design consider the OAA service 

package alone or in combination with a similar service package that is funded by 

alternative sources? This decision will impact the comparison group, sample size, 

research questions and generalizability of the findings.

Another important issue discussed frequently in the literature, and one that we are 

confronted with in this design, is the level of exposure to HCBS services (i.e., mix, 

duration, intensity and timing). Consideration of this step will ensure that individual 

variations are factored into the final models. Several studies that were reviewed 

developed exposure algorithms that may be applicable for use in the final models to 

address this issue (NYSOA, 2010).

Furthermore, an important issue raised in the literature is the impact of certain policy or 

other contextual factors on individual outcomes (e.g., nursing home admission or HCBS 

use). These measures include the amount of state spending on Medicaid HCBS, the 

number of nursing home beds in a geographic area, availability of Waiver slots, the 

number of home health agencies in a particular area, and the use of pre-admission 

screening prior to NH use.

Several of the key variables proposed for the OAA study are well established measures 

(e.g., sex as male/female/unknown). However, the literature points to several 

limitations of many of the measures (e.g., informal care measures, ADL/IADL 

limitation measures), which must be considered prior to including them in the study.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 11

Proposed Study Design

We suggest that the design approach with the greatest potential for delivering 

conclusive and actionable results would be a quasi-experimental design that includes: 

1) a retrospective study component drawing on existing individual data from federal 

sources; and 2) a prospective study component using data to be collected during the 

study period. Evaluators of HCBS and other health and social service programs usually 

rely on quasi-experimental techniques as the best impact study design and most 

powerful alternative to random experiments. We know from the literature that 

demonstrating impacts on HCBS expenditures can take several years, with a lag 

between increased spending on HCBS expansion and savings from this investment

(Kaye, LaPlante, & Harrington, 2009). Thus, we recommend a study period of seven 

years, a plausible option given the retrospective/prospective design option. This study 

period time can be modified depending upon time, data, or funding constraints.

To reduce bias and error in assessing program outcomes, we recommend incorporating 

multiple methods, measures, and data sources. In using a quasi-experimental design 

with a matched comparison group, a suggested method includes propensity score 

matching for constructing a matched comparison group in the final design. This will 

allow comparisons of older adults who had an outcome of interest with older adults 

who are matched on certain key characteristics relevant to the outcome, to determine 

whether OAA services contributed to the difference in outcome. The matching process 

will allow us to control for factors that are believed to contribute to different outcomes 

(e.g., gender, diagnosis, multiple chronic conditions, etc.).

The key to unbiased inference in this approach, as in any quasi-experimental design, is 

proper specification: choosing a set of key factors to be controlled so that uncontrolled 

variables are approximately random, i.e., not correlated with the controlled variables or 

the treatment (Achen, 1986). Knowledge about the factors that affect receipt of services 

and factors correlated with outcomes (i.e., community living tenure and healthcare 

utilization) identified in the literature review are essential to the success of the design.

The research team proposes using this design in a study of 3 states. A primary purpose 

of the study will be to identify a core set of variables that all states should collect. States 

have been granted great flexibility regarding program administration and data 

collection. As a result, states lack uniform and comprehensive data. Conducting this 

study, as discussed in the Recommendations section, can address that issue. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 12

Figure 2: Proposed Study Design Image
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 13

Proposed Study Design Description

The above graphic depicts the proposed design for the Study of the Global Outcomes of 

Older American Act Programs and Services. The goal is to measure the 

impact/association of OAA service use by older adults on four key outcomes (health 

care utilization, HCBS expenditures, NH admissions and community tenure). Further, 

the design allows for the examination of various service use trajectories over time (e.g., 

HCBS user in a NH, NH user exits NH, etc.) and the extent to which OAA programs 

and services are associated with these trajectories. The details of the study design are 

discussed below.

Intervention: Receipt of OAA services, alone or in combination with other HCBS.

Target Population: The target population for this study is older adults age 60 and above 

(those eligible for OAA services). In the selection of the cohorts in stage 1, the sample 

will be selected and then connected back to those who received OAA services in stage 2 

of the design. Those who have received OAA services are the treatment, or intervention, 

group and those who are not matched to OAA service use are the comparison group.

Pre-Stage: Three states will be selected through an RFP process to participate in this 

study design. The states will have the data capacity and availability to participate in the 

stages outlined below.

Stage 1: The proposed design takes place over a six-year study period and consists of 

three stages. In Stage 1, (depicted in the blue box in the middle), the study sample is 

selected using MDS, Medicaid/Medicare claims data, and OAA or state-funded 

services data, that cover a one-year period of time (2009-2010) and an entire state (e.g., 

Georgia). Three cohorts of older adults will be identified from this dataset: Cohort 1 will 

be a sample of older adults who have been admitted to nursing homes; Cohort 2 will be 

a sample of older adults enrolled in any HCBS [Medicaid, state-funded, or OAA]; and, 

Cohort 3 will be a group of individuals who were not admitted to a nursing home and 

did not receive HCBS. This group will be matched to older adults in Cohorts 1 and 2 on 

certain key variables to ensure comparability (propensity score matching). Cohort 3 will 

serve as the comparison group. 

Stage 2: During Stage 2 of the study (represented in the purple box on the left), the 

HCBS service use by Cohort will be measured. This will be accomplished through the 

analysis of Medicare/Medicaid Claims, POMP, State OAA administrative records, HRS 

and any other available sources. The study period covers 2006-2008. Data will be 

analyzed by service type, mix, and intensity of service use. A group of individuals will 

be non-users.

Stage 3: In the final stage of the study, Stage 3, (illustrated in the green box on the 

right), using the same combination of data sources, three additional years of service use 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 14

and client disposition—status post-intervention will be examined. In sum, the study 

will have a longitudinal data base (2006-2012 or the most current year available to us) 

that includes three unique cohorts of individuals: nursing home users at baseline, 

HCBS users at baseline, and non-users at baseline. This combination will allow for both 

within-group and between-group data modeling. Results will demonstrate if receipt of 

OAA services, whether alone or in combination with other services, will directly 

impact: health care utilization, HCBS expenditures, nursing home (NH) admissions and 

community tenure. 

Limitations and Recommendations

The Administration on Aging (AoA) and The Lewin Group (Lewin) hosted an Expert 

Group meeting, conducted several key informant interviews with additional 

stakeholders and held calls with states about their Older Americans Act (OAA) data 

systems. This feedback yielded a better understanding of the limitations of this study

design and the recommendations to address them (Appendix C).

Limitations

The following major limitations were identified in those discussions: 

► State Older Americans Act (OAA) Data: A select group of states (FL, GA, MA, 

MN, and OH) were interviewed about their OAA data collection policies and 

procedures. These interviews highlighted some of the limitations of state OAA data, 

along with identifying states with more comprehensive data systems from our small 

sample. Specific limitations that impact the study design included:

 The absence of Social Security Numbers (SSNs), which provide the most 

reliable method to match to Medicare and Medicaid claims.

 Lack of information regarding the reasons for termination of services, which

limits the ability to measure community tenure as an outcome.

► Informal Caregiver Data: As discussed in the expert group meeting and key 

informant interviews, difficulties will be encountered obtaining caregiver

information, due largely to the non-existence of such data.

► Private Pay Data: This is a limitation for both the treatment and comparison 

groups. States do not uniformly collect information on private pay services received 

by OAA users. Therefore, this information would not be readily available for the 

proposed treatment or comparison groups.

Recommendations

While this study can be conducted, it will require complicated data gathering, cleaning 

and merging to overcome many of the data limitations. It will also be heavily 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 15

dependent on the strength of the chosen states’ OAA data system and timeline when 

the states migrated to a statewide data system. Based upon the limitations of data 

sources available, and feedback from expert group participants and key informants, the 

recommended approach is outlined below:

Step 1: Issue RFP to conduct study

AoA issues an RFP to carry out the proposed study. The awarded research team would 

work with AoA to prepare for and conduct the study.

Step 2: Convene Technical Working Group

Within three months of the contract begin date, AoA in collaboration with the awarded 

research team establishes a Technical Working Group (TWG). A TWG can assist the 

research team in identifying the minimum key variables necessary for a 

retrospective/prospective design.

Step 3: Issue RFP to states

Prior to the completion of step 2, AoA issues an RFP identifying up to three states 

willing to participate in organizing existing data at the state level. Proposed criteria for 

this RFP are outlined in the Limitations and Recommendations document.

Step 4: Assist states / Collect available secondary data

Step 4 has two concurrent components that will begin upon awarding three states 

grants to conduct the study: (1) AoA and the research team will work with the three 

selected states in gathering and preparing their OAA administrative data; and (2) while 

the selected States are preparing and assisting in the collection of data, the research 

team should assist the states as necessary. In addition, the research team will begin 

preparations for linking the state-level data with other national datasets.

Step 5: Expand work to include primary data collection

Step 5 includes expanding the work described in steps 2-4 in the selected states to 

include primary data collection, as funds and time allow. Primary data collection will 

strengthen the data by gathering detailed information on such key aspects of caregiving 

as informal supports. This step could be accomplished through either a state-wide 

random selection of OAA participants, or targeted data collection that occurs with select 

AAAs in the chosen states.

Step 6: Design and field the study

The research team selected in Step 1 should work with AoA and the three participating 

states in finalizing the study design and fielding the study with data collected at the 

state and federal level. The proposed design in the Study Framework and Design

should be the foundation for this design, and modified as appropriate given available 

state secondary data and any additional primary data.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Executive Summary

Executive Summary 16

Step 7: Reconvene TWG and present findings

Once the exploratory study is conducted in the three selected states reconvene the TWG 

and present results for feedback and discussion. TWG members, in collaboration with 

AoA and the research team, will define a core set of variables that AAAs will employ to 

ensure consistent data collection across agencies.

Step 8: Training & technical assistance for data collection

With changes to SPR data requirements, many states will require intensive T&TA as 

they make updates to their current data systems. However, some states will be more 

prepared than others or already collect the core variable requirements, while others will 

require a complete overhaul.

Conclusion

Conducting this study can position AoA to design and implement a more 

comprehensive evaluation of OAA services. As all states begin collecting newly 

required data elements, AoA might consider how this new data can be used in a 

national evaluation, or an evaluation conducted in a random selection of states. This 

evaluation could result in definitive conclusions providing a full picture of the impacts 

of OAA Title III services to date. Specifically, AoA will better understand the impact of 

OAA services across the board in promoting positive outcomes for service recipients, 

including increased community tenure and decreased health care use. This evaluation 

will also examine the potential cost savings of OAA service use. Additionally, it will 

enhance capacity for ongoing research and provide information about the programs 

that AoA and the Aging Network can use in future program planning.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

i

Table of Contents

Study Design Options................................................................................................. 19

Findings from Early HCBS Evaluations .................................................................... 38

Theoretical Considerations ........................................................................................ 40

Findings and Lessons Learned from Past HCBS Studies ....................................... 45

Conclusion................................................................................................................... 68

Appendix A-A: Available Data Sources and Data Elements................................... 69

Appendix A-B: Detailed Key Variable Table............................................................. 77

Appendix A-C: Detailed Bibliography....................................................................... 84

Tables

Table 1: Data Sets with OAA/Aging Network Specific Data......................................................69

Table 2: Other State Specific Data Sets...................................................................................72

Table 3: National Data Sets and Other Sources.......................................................................73
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 17

Appendix A: Literature Review

Introduction

This report presents The Lewin Group’s findings from the review of materials (Subtask 

2.1 of the project to design an Exploratory Study of the Global Outcomes of Older Americans 

Act Programs and Services). In this overview, we describe our methods and present 

findings from our review of materials. These include findings related to: (a) a study 

design that may be suitable for assessing OAA program impacts, taking into account 

the complexity in measuring impact; (b) findings and lessons learned from past HCBS

studies; (c) elements and relationships to consider in developing a program model and 

planning a study of HCBS impacts (including HCBS goals, policies, strategies, impacts, 

and research questions); and (d) available data sources identified for us in examination 

of these issues (Appendix A-A).

Methods and Overall Search Results

We found relevant studies, reports, and other published materials through a variety of 

methods. There were two phases to our search:

1. We utilized the collection of articles previously compiled for AoA by S3 and 

other materials provided by AoA. We augmented these study reviews with a 

targeted search for new and additional studies (published and unpublished) on 

the impacts of HCBS on community tenure, health care utilization, and/or 

economic/systems outcomes, through:

a. A search on PubMED and Google Scholar using key search terms related 

to this literature review, and

b. Browsing research articles and abstracts in the Clearinghouse for Home 

and Community Based Services (http://www.hcbs.org/).

2. Additional studies were obtained through other sources (e.g., email lists, 

reviewing sources cited in reviewed studies, other studies known to the research 

team, and recommendations from the expert group, key informants, and state 

OAA representatives).

From this, we scanned approximately 900 articles/studies and narrowed the review to 

the studies below. Inclusion criteria included:

► Materials providing background information about OAA programs and services.

► General review of HCBS studies that included extracting relevant information to 

inform this report: previous evaluations of OAA services and studies of HCBS use 

on community tenure, health care utilization, and/or economic outcomes were 

used. These studies were most directly relevant to this project. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 18

► In-depth review of studies where we extracted information on the data sources, 

detailed findings for all outcome measures, and the authors’ recommendations for 

future research. 

► Two researchers read the full text of the studies and reviewed each entry. For 

evaluation methodologies, we paid particular attention to 19 studies that specifically 

examined impacts of OAA services or that provided new and useful findings to help 

inform a study design. The in-depth reviewed studies included: 

 Studies that specifically examined OAA programs and services, including 

research conducted as part of the Administration on Aging’s Performance 

Outcomes Measurement Project (POMP), and related studies by the same 

authors that specifically examined outcomes of OAA services, along with other 

HCBS, in numerous states (e.g., Florida, New York, Georgia, and Rhode 

Island). 

 Findings from the Community Tenure study, which examined receipt of OAA 

services, along with other HCBS, and has been nationally recognized for its 

methods. This 5-year prospective study tracked older adults in Kansas who 

had applied for nursing facility admission and received a pre-screening 

admission assessment under the State’s Client Assessment Referral and

Evaluation (CARE) Program. The study tracked CARE participants’ 

community tenure and outcomes when community tenure ended (Chapin et 

al., 2009). 

 Select studies of non-OAA HCBS that could be particularly useful in 

developing a study design. These were recent studies that: (a) measured the 

impacts of HCBS on community tenure and/or health care utilization; (b) used 

rigorous methods and large samples so that the results are reasonably 

generalizable; (c) examined a complex set of factors to rule out alternative 

explanations for program results; (d) examined relationships or factors not 

previously examined in the literature. 

► Over 100 articles presenting background research, prior HCBS studies and 

evaluations, and research on factors predicting HCBS use, health care use, and 

nursing home utilization. These include meta-analyses that examined predictors of 

nursing home admission (Gaugler et al., 2007) or other adverse outcomes (Miller & 

Weissert, 2000). Other studies examined targeted groups experiences with the 

outcomes of interest, such as people with dementia, older adults in rural areas, and 

African American women.

► 21 relevant materials from the general program evaluation/social research 

literature. These materials were used to enhance understanding of evaluation design 

options suggested by the HCBS studies and to identify other potential design 

options that may be appropriate to consider. These evaluation design-related 

materials included: evaluation literature known to Lewin and AoA, materials 

recommended in OMB guides, and additional items found through targeted 

searches for materials about specific methodologies that appeared most relevant. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 19

There were many studies that, while informative, did not meet our inclusion criteria for 

further review. These include:

► Studies of particular program strategies or interventions (e.g., adult day care, 

assistive technology, care management, participant direction, food assistance 

programs);

► Studies related to the direct service workforce, studies of informal/family 

caregivers;

► Overview articles/issue briefs;

► State specific program overviews;

► Research on demographic trends and predictors of health outcomes;

► Studies related to needs assessments;

► Studies related to medical, exercise or therapeutic home-based interventions;

► Studies not written in English and/or not conducted in the United States

We did not apply exclusion criteria based on the time in which the study was 

conducted or published to capture the early HCBS findings from the Channeling 

Demonstration.

Study Design Options

In this section, we summarize our findings from the reviewed HCBS studies and 

program evaluation literature on possible study design options for AoA to evaluate the 

impact(s) of the Older Americans Act (OAA) and services on older adults and 

caregivers. Additionally, we must account for the combination of OAA programs and 

services with non-OAA HCBS programs and services.

Strategies in Evaluating Aging and Disability Programs

A study that seeks to evaluate the effectiveness of a social program, frequently referred 

to as a program evaluation, is different from other types of research (Krause, 1996; 

Rossi, Freeman, & Lipsy, 1999). In what is sometimes described as “pure” research, 

social scientists start with hypotheses, gather data, and then try to form generalizations 

about the results. A search for theory is the primary function, and this quest for theory 

requires the use of research procedures in a fashion that often differs from what 

generally occurs within program evaluations. For example, correlations that might 

produce a scientific journal article could easily be meaningless to an agency director 

who is trying to decide whether to spend additional dollars on a particular program. 

While the major focus of studies evaluating program outcomes is to determine whether 

outcomes can be attributed to the program (GAO 1991; GAO 2012), they are also 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 20

concerned with identifying any unintended outcomes and understanding how the 

program works with other programs (Treasury Board of Canada, 1998). 

DIFFERENCE BETWEEN PROGRAM EVALUATION AND OTHER RESEARCH

Program evaluation is the use of social research procedures to systematically investigate the 

effectiveness of social intervention programs.—Rossi, Freeman, & Lipsey, 1999

Program evaluation, or “evaluation research,” refers to the research procedures and techniques 

used to examine the effectiveness of social programs. Program evaluation, in other words, is a 

process that generates the information used to describe what a program is doing and how well 

it does it.” — Krause, 1996

GAO (2012) recommended that five key steps, similar to those presented by Krause 

(1996), should be taken into consideration before data are collected in a program 

evaluation: 

1. Clarify understanding of the program’s goals and strategy.

2. Develop relevant and useful evaluation questions. 

3. Select an appropriate evaluation approach or design for each evaluation 

question. 

4. Identify data sources and collection procedures to obtain relevant, credible 

information. 

5. Develop plans to analyze the data in ways that allow valid conclusions to be 

drawn from the evaluation questions. 

Each of these steps is crucial and should not be overlooked. As a tool for understanding 

how program activities relate to short-term and long-term outcomes, evaluation 

materials typically recommend using a program logic model (GAO, 2012). A review of 

materials, along with discussions with program stakeholders, is essential to gathering 

the information needed to develop a successful program logic model and evaluation 

design. Content analysis or a summary of the content of existing materials, such as that 

undertaken in this report, has several applications in program evaluation (GAO, 1989). 

These include identifying program goals, describing program activities, and 

determining program results.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 21

Research Designs

Research designs fall into several groups discussed in depth in this section: 

randomized; non-randomized comparison group; time series/longitudinal; and 

alternate causal explanation elimination.

A frequent recommendation in the evaluation literature is to combine two or more 

research designs. GAO (2009) recommended, “Since all evaluation methods have 

limitations, our confidence in concluding that an intervention is strengthened when the 

conclusion is supported by multiple forms of evidence” (pg. 28-29). Some of the HCBS 

studies cited used more than one type of design; for example, a survival analysis study 

was used with a matched comparison group in the Kansas Community Tenure Study 

(Chapin et al., 2002).

The Treasury Board of Canada (1998) advised that using more than one evaluation 

strategy is desirable to increase support for inferences about program impact. This is 

because, “Generally speaking, no single evaluation strategy is likely to yield enough 

evidence to answer unambiguously the questions posed by the evaluation” (Treasury 

Board of Canada 1998, p. 28). Because an evaluation study generally addresses several 

issues at a time, more than one strategy may be needed to answer all research questions 

(Treasury Board of Canada, 1998). Trochim (2006) noted that the currently prevailing 

philosophy in science recognizes that all observation is fallible and has error and that all 

theory is revisable. This view emphasizes the importance of multiple measures and 

observations, each of which may possess different types of error and the need for 

triangulation (i.e., synthesizing findings) across these multiple error-containing sources 

to try to gain a full understanding of the program. 

Shadish and Cook (2009) described pattern matching designs, which “counter the 

unfortunate notion that researchers should choose from a small and fixed set of 

designs” (pg. 623). While researchers often focus on minimizing threats to validity, 

pattern matching designs “attend to a less often noticed piece of advice, to predict a 

diverse pattern of results whose strong testing might require multiple nonrandomized 

designs, each with different presumed biases” (pg. 623). Shadish and Cook (2009) 

quoted Campbell & Stanley’s 1963 advice that “the more numerous and independent 

the ways in which the experimental effect is demonstrated, the less numerous and less 

plausible any singular rival invalidating hypothesis becomes” (pg. 623). They 

recommended that when an ideally implemented experiment or quasi-experiment is not 

feasible, researchers assemble more than one design that predicts a pattern of causal 

results, because fewer alternative explanations are plausible if the results match the 

predicted pattern. 

An example discussed by Shadish and Cook (2009) was measuring the same outcome 

via a randomized and nonrandomized comparison group design then comparing 

results. They referred to Cook et al. (2008), who suggested a stringent set of seven 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 22

criteria for good comparisons of results obtained from a randomly formed and a nonrandomly formed control group. 

1. The studies compare a randomly formed control group and a non-randomly 

formed control group.

2. The randomized and nonrandomized experiment both estimate the same 

estimator (e.g., average effect of treatment on the treated or intent to treat).

3. The randomized and nonrandomized groups should differ from each other only 

in method of assignment.

4. The person estimating results from the nonrandomized study should not know 

the results from the randomized experiment.

5. The randomized experiment should be an exemplar of its kind, not subject to 

large attrition or partial treatment problems.

6. The quasi-experimental design should similarly be an exemplar of its kind, 

without attrition or partial treatment problems, with focal local controls and 

good pre-test measurement of variables related to treatment and outcome.

7. A defensible standard for what counts as a match in randomized and 

nonrandomized results is used. “This is difficult both because reasonable people 

might disagree on substantive criteria that would make a difference to policy 

decisions and because statistical criteria will inevitably be subject to power 

problems” (pg. 622). 

Cook and colleagues (2008) showed that when most or all of the seven criteria were 

met, results from different kinds of nonrandomized experiments matched results from 

randomized experiments. This was true for regression discontinuity designs, welldesigned nonrandomized experiments with focal local controls and stable matching, 

and statistical analyses such as propensity scores. 

Randomized Field Experiment

Many HCBS studies, using randomized assignment to test the effectiveness of a 

particular program, or of HCBS overall, have encountered serious implementation 

problems that lead to inconclusive results. The early HCBS studies revealed many 

limitations to the use of randomized experiments in the study of HCBS programs. These 

include: lack of clarity over whether the actual experimental intervention was likely to 

have its intended effect, small estimates of results due to similarities in services received 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 23

by treatment and comparison group members, and lack of generalizability of the results 

(Kane, 1988; Brown & Phillips, 1986). 

Another limitation of randomized experiments is that results could show relatively 

small estimates of program impacts, even if the program is very effective, if only a 

fraction of the treatment group actually participated in program services or if a 

substantial proportion of the control group received similar services from existing 

agencies; this was an issue in the Channeling demonstration (Brown & Phillips, 1986) 

and in the Cash & Counseling demonstration (Applebaum, Seidl, & Austin, 1980). 

Another noted challenge to this design is that treatment and control groups may differ

despite random assignment, as was the case in the evaluation of Georgia’s Alternative 

Health Services program (Skellie, Mobley, & Coan, 1982). Several studies performed 

secondary analyses of Channeling demonstration data to address research questions not 

answered in the original demonstration project. Although their data is now outdated, 

these studies provide examples of the use of more sophisticated models to gain a better 

understanding of how and why specific program effects may or may not occur in HCBS 

programs (Rabiner, Stearns, & Mutran, 1994; Liu, McBridge, & Coughlin, 1990; Greene, 

Lovely, & Ondrich, 1993; Green, Ondrich, & Laditka, 1998).

A recent article in an evaluation blog discussed the debate about the use of randomized 

experiments in evaluating social programs: “The promotion of experimental designs 

often has a polarizing effect: Proponents sometimes act as if it is the cure for all 

evaluative ailments; opponents sometimes act as if it is the root of all evil.” /“Any 

design should be selected because it is the best way to answer a particular question, and 

the question to be answered should be directly related to the stage of the organization 

or program being tested. Not all questions in the field are best answered through an 

experimental design approach. But some are.” (Buteau, February 17, 2012).

A 2009 GAO report described several circumstances in which random experiments are 

generally not well-suited. In particular, GAO (2009) advised: “the evaluation literature 

cautions that as social interventions become more complex, representing a diverse set of 

local applications of a broad policy rather than a common set of activities, randomized 

experiments may become less informative... In these cases, aggregating results over 

substantial variability in what the intervention entails may end up providing little 

guidance on what, exactly, works” (pg. 25).

However, Xu et al. (2010) recently recommended a different use for randomized 

experiments: to determine the effectiveness of increasing the volume of various types of 

formal HCBS on health outcomes and expenditures. This was based on their finding 

that receiving a greater volume of attendant care, homemaking services, and homedelivered meals was associated with a lower risk of hospitalization. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 24

A 1990 paper in Evaluation Review (Dennis, 1990) noted that randomized field 

experiments can and often do encounter threats to the validity of their inferences about 

program results. In addition, because of their difficulty and expense, the author 

recommends that randomized field experiments “should be considered only when no 

simpler alternatives for evaluating the intervention can be found” (pg. 350). 

However, several developments in the analysis of randomized experiments in recent 

years have led to more sophisticated randomized experiments (Shadish & Cook, 2009). 

Developments in analytical methods and computer software have occurred for coping 

with partial treatment implementation and potential attrition and the use of nested 

models. In nested models (often called group-randomized or cluster-randomized 

models), individuals are “nested” within aggregated groups, which are themselves 

nested within conditions. Examples include students nested within classrooms, clients 

nested within psychotherapy groups, or workers nested within worksites. 

RECENT ADVANCES IN RANDOMIZED FIELD EXPERIMENTS

In recent years, more sophisticated randomized experiments have developed, as a result of 

developments in analytical methods and computer software for coping with implementation 

problems. One example is nested models, in which individuals are nested within aggregated 

groups, which are themselves nested within conditions. (Shadish & Cook, 2009)

Non-Randomized Comparison Group Designs

A non-randomized, often called “nonequivalent,” comparison group design measures 

outcomes for two or more groups but does not randomly assign participants to receive 

services or not (Shadish & Cook, 2009). The GAO (1991) described three different ways 

to form a non-randomized, or nonequivalent, comparison group for a program 

evaluation: matched comparison group; regression discontinuity/biased assignment, 

and naturally occurring comparison groups. These options are discussed below.

Matched Comparison Groups

A few of the studies reviewed in-depth for this report used matched comparison 

groups. These studies used a number of different populations from which to form the 

comparison group and different statistical techniques to match the groups, as 

summarized in Table 1 below.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 25

Table 1 : HCBS Studies With Non-Randomized Comparison Groups

Study Comparison Group 

Populations Analytical Methods 

Felix et al., 

2011

Arkansas Community 

Connector participants 

People receiving 

Medicaid in nearby 

counties where the 

Community Connector 

program was not 

operating

This longitudinal study used propensity score 

matching to compare the two groups. The 

Arkansas Community Connector Program used 

specially trained community health workers to 

identify people living in the community who have 

unmet LTC needs and who may be at risk for 

entering NHs in three disadvantaged Arkansas 

counties and connect them to Medicaid HCBS.

Shapiro & Loh 

2007; Shapiro, 

Loh & Mitchell 

2009

HCBS users in Florida

People on waitlist

This study of five HCBS programs in Florida used 

propensity score matching to compare 

outcomes for HCBS users matched with 

individuals placed on the waitlist when program 

capacity has been reached. The user/waitlisted 

status was not expected to be determined by 

unobservable characteristics that potentially affect 

outcomes because, in Florida, persons with 

equivalent need for services and frailty are 

prioritized on a first-come, first-serve basis. Any 

differences in frailty that arise from group 

membership are accounted for through regular 

assessments that both users and waitlisted 

persons complete. The authors developed an 

algorithm that determined the differences in 

Medicaid expenditures between the two groups, 

producing an estimate of cost-savings. Second, 

the study used a linear panel model in which the 

difference-in-difference estimator was employed. 

This estimator represents the difference in the 

average outcome in the treatment group before 

and after treatment, minus the difference in the 

average outcome in the control group before and 

after treatment.

Shapiro & Loh,

2010

HCBS users in Florida 

(individuals who applied 

for and received any 

HCBS through DOEA 

during the month)

Individuals not receiving 

services that month, 

including those on waitlist 

and those who stopped 

Building on findings from Shapiro, Loh, & Mitchell 

(2009), this study of HCBS in Florida compared 

outcomes for three match comparison groups, as 

well as a naturally occurring comparison group 

(individuals in nursing homes). Three-stage-least 

squares estimators were used to produce the final 

estimates. The authors implemented several 

layers of controls to reduce potential bias due to 

differences between the groups. Multiple 

regression models were used to control for 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 26

Study Comparison Group 

Populations Analytical Methods 

receiving HCBS

Individuals who did not 

apply for services

Individuals in nursing 

homes

demographic differences and differences in ADL 

scores. They then estimated a set of fixed effect 

models to control for unobserved factors of 

Medicaid expenses that may correlate with the 

HCBS service status. The authors noted that 

employing propensity score matching using the 

CIRTS database may introduce selection bias. To 

account for this, they compared those in the 

CIRTS database with nursing home residents who 

were not in the CIRTS database in the study 

period.

Chapin et al., 

2002; 2003; 

2009; 

Macmillan et 

al., 2007

Older persons in Kansas 

who received a CARE 

assessment and were 

diverted from NH

Those who received an 

assessment and were not 

diverted and entered a 

NH

This 5-year longitudinal study, tracked a cohort of 

older adults in Kansas for 18 months and 24 

months who received CARE Assessment. 

Quantitative data analyses compared those who 

were diverted and those who were not diverted. 

MATCHED COMPARISON GROUPS EXAMPLE

A study of HCBS users in Florida constructed 4 comparison groups by matching data from the 

state CIRTS system with MDS data (Shapiro & Loh, 2010):

►Individuals who applied for and received any HCBS during the month (MDS II & 

CIRTS match)

►Individuals not receiving services that month, including those on waitlist and those 

who stopped receiving HCBS (MDS II & CIRTS match)

►Individuals who did not apply for services (MDS, no CIRTS match)

Four techniques for improving the similarity between comparison groups are: (1) 

difference-in-difference design, (2) the use of instrumental variables, (3) propensity 

score matching, and (4) case matching (Duignan, 2009). Currently, the most popular 

method for matching the two groups to be as similar as possible is propensity score 

matching (Shadish & Cook, 2009). The Money Follows the Person (MFP) evaluation 

design proposed two comparison groups (Brown et al., 2008). One group comprised 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 27

Medicaid enrollees during a previous period who appear to meet MFP eligibility 

criteria and transitioned from institutional to community-based care. The other group 

included institutionalized Medicaid enrollees in the pre-MFP period who have the same 

observable characteristics as MFP participants, but did not transition to the community.

Many studies across a number of fields have tested the ability of matched comparison 

group designs to produce unbiased estimates by comparing them with estimates from 

randomized experiments. These studies generally suggested that matched comparison 

group designs can often produce misleading results. In 2003, the U.S. Department of 

Education discussed this literature and reported:

“There is persuasive evidence that the most common comparison-group 

designs produce erroneous conclusions in a sizeable number of cases. A 

number of careful investigations have been carried out—in the areas of 

school dropout prevention, K-3 class-size reduction, and welfare and 

employment policy—to examine whether and under what circumstances 

comparison group designs can replicate the results of randomized 

controlled trials. ...These investigations have shown that most 

comparison-group designs in education and other areas produce 

inaccurate estimates of an intervention's effect.....In a sizeable number 

of cases, the inaccuracy produced by the comparison-group designs is 

large enough to result in erroneous overall conclusions about whether 

the intervention is effective, ineffective, or harmful.” (p. 3)

However, more recent research suggests that valid results from statistically matched 

comparison group designs are possible. Shadish & Cook (2009) noted that significant 

advances have been made in addressing vulnerabilities in the design and analysis of 

nonrandomized comparison group experiments. 

WHEN CAN MATCHED COMPARISON GROUP DESIGNS APPROXIMATE RESULTS 

FROM RANDOMIZED DESIGNS?

“Evidence suggests that nonequivalent comparison group designs can approximate answers 

from randomized designs when they use focal local controls, careful matching on stable 

covariates, and measure a rich set of pretest predictors of treatment and outcome that can be 

used in statistical adjustments such as propensity score analysis.” – Shadish & Cook, 2009, 

pg. 625

Several of the evaluation design articles we reviewed emphasized the importance of 

selecting comparison group members who are local to the participant group, not a 

national sample.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 28

Regression Discontinuity Design

For many evaluations, the evaluator does not form the treatment and comparison 

groups (GAO, 1991). Rather, some people have naturally been exposed to the program 

and others have not. 

NATURALLY OCCURRING COMPARISON GROUP

This design may be used when the evaluator does not form the treatment and comparison 

groups. Rather, some people have naturally been exposed to the program and others have not. 

Although this somewhat limits the evaluator’s options, the general logic of the design is the 

same. (GAO, 1991)

An example in the studies we reviewed was Medicaid participants living in the 

community and in nursing homes. The regression discontinuity design, noted by the 

GAO (1991) and described below, can be used in situations where people are naturally 

assigned to either receive or not receive services based on known criteria. To our 

knowledge, no studies in the long-term services and supports field have used this lesser 

known approach to forming a comparison group, nor have we seen any HCBS studies 

mentioning this approach.

The Research Methods Knowledge Base website (Trochim, 2006, Design, QuasiExperimental Design) describes regression-discontinuity design as “an important and 

often misunderstood alternative to randomized experiments, because its distinguishing 

characteristic--assignment to treatment using a cutoff score on pretreatment variable--

allows us to assign to the program those who need or deserve it most.” Thus, the 

regression-discontinuity design does not require assignment of potentially needy 

people to a no-program comparison group in order to evaluate program effectiveness. 

REGRESSION DISCONTINUITY DESIGN

In this design, participants are assigned to treatment using a cutoff score on a measured 

variable. This allows the program to be assigned to people who need or deserve it most.

(Trochim, 2006)

Regression discontinuity studies have taken advantage of many naturally occurring cases of 

assignment to treatment using a cutoff. Examples include students to remedial writing training 

if they scored lower than a cutoff on a measure of writing skills and villages to receive social 

welfare assistance if they scored lower than a cutoff on a measure of village development.

(Shadish & Cook, 2009)
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 29

In regression discontinuity design, the choice of cutoff value is usually based on one of 

two factors: (1) solely on the basis of program resources that are available (e.g., if a 

program has the capability of handling 25 people and 70 people apply, one can choose a 

cutoff that distinguishes the 25 most needy persons from the rest); or (2) substantive 

grounds, such as an indication of severity of illness (Trochim, 2006). To interpret the 

results, one must know the nature of the assignment variable, who received the 

program, and the nature of the outcome measure.”

According to Shadish et al. (2011), regression discontinuity design, sometimes called a 

cutoff-based experiment or cut-off based assignment, is close behind preference to 

random assignment in estimating unbiased treatment effects. The theory of regression 

discontinuity design is that it yields an unbiased estimate, like randomized 

experiments, because the assignment variable is completely known and measured. In a 

regression discontinuity design, a program effect is suggested when a “jump” or 

“discontinuity” in the regression lines is observed at the cutoff point (Trochim 2006). 

This is because units become more similar as they approach the cutoff from either side. 

The Coalition for Evidence-Based Policy describes how this design addresses the 

challenge of self-selection and motivation bias: 

“Among comparison-group studies, regression-discontinuity designs are 

particularly well-suited for constructing program and comparison 

groups with similar motivation. Such designs compare a program group 

comprised of individuals just above the threshold for program eligibility, 

with a comparison group of individuals just below (e.g., families earning 

$19,000 per year versus families earning $21,000, in an employment 

program whose eligibility cutoff is $20,000). Because program 

participation is not determined by self-selection, and the two groups are 

very similar in their eligibility score, there is reason to believe they are 

also similar in motivation.

The most common implementation problems in studies using this design occur when 

assignment to condition does not adhere strictly to the cutoff and when persons 

manipulate their assignment scores to receive or avoid treatment (Shadish et al., 2011). 

However, the authors noted, new analytic methods have been developed for modeling 

nonlinearities in regression discontinuity designs and coping with violations of 

assignment by cutoff score.

As with matched comparison group designs, several studies in the research methods 

literature have compared results from randomized experiments and regression 

discontinuity designs. Shadish et al. (2011) recently reviewed these studies and found 

that, as expected, most analyses suggested that bias was least when restricting the 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 30

sample to participants closest to the cutoff. Unlike most of the studies comparing 

matched comparison groups to randomized experiments, results from four studies 

reviewed by Shadish et al. (2011) generally supported the hypothesis that regression 

discontinuity design produces causal estimates similar to those from randomized 

experiments. Some of the comparisons did not yield the same results, but the authors 

noted this may have been because the studies used different statistical methods to 

identify the regression discontinuity design effect estimate, with different estimates 

producing different results. Shadish et al. (2011) corrected problems found in past 

studies by randomly assigning 588 participants to be in a randomized experiment or a 

regression discontinuity design in which they were otherwise treated identically. The 

authors compared results for the two designs estimating both the same and different 

parameters, using three different methods of modeling nonlinearities. Results suggested 

that estimates from regression discontinuity designs approximated the results of 

randomized experiments reasonably well. They noted that, in cases when the 

assignment variable moderates the effect of treatment, results for people at the cutoff 

would not represent results for people at other points on the assignment variable.

Also, Shadish et al. (2011) raised the question of how to interpret results of comparisons 

between results from randomized field experiments and quasi-experimental designs. 

They found that neither the randomized experiment results nor the regression 

discontinuity design results would adequately represent the interaction between 

treatment and assignment variable. In this sense, the regression discontinuity design 

has the advantage in that a graphical display of the relation between assignment and 

outcome is likely to make the potential interaction more salient than it would be in a 

randomized experiment in which the researcher might not even think to test the 

interaction. Thus, the authors suggested that, in comparing randomized experiments 

and regression discontinuity design, the matter is less about one design being better or 

worse than the other across the board and more a matter of understanding the 

competing strengths and weaknesses of the two designs in representing the treatment 

effect in the presence of the interaction.

Further, Shadish et al.’s (2011) study questioned what it means to claim that estimates 

from randomized experiments are the gold standard in social research. This term can 

imply either that the randomized experiment is generally the best design for causal 

inference or that the failure of a non-randomized estimate to match the randomized 

estimate is cause for questioning the nonrandomized one. The authors noted that 

widespread agreement that the first claim is true has perhaps led to uncritical 

acceptance that the second claim necessarily follows. However, the authors noted, 

differences between the two estimates could be due to sampling error, differences in 

implementation, or differences in the parameters estimated. Thus, they suggested, past 

authors of studies comparing randomized experiments with nonrandomized designs 

may have overstated the case when they interpreted their results as suggesting that 

nonrandomized designs may not well approximate results from randomized 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 31

experiments. The authors recommended that randomized experiments are still 

preferable to regression discontinuity design, because they have more power and fewer 

assumptions. They posit that if researchers do need to use a regression discontinuity 

design, they can be reasonably confident that their results are “an accurate estimate of

the effects of treatments” (Shadish et al., 2011, pg. 190).

Time Series / Longitudinal Models

An alternative type of quasi-experimental design is a time series, or historical, design, in 

which “the basis for contrast is the same participants before program implementation” 

(GAO, 1991, p. 52). Similarly, the Treasury Board of Canada (1998) noted that a 

comparison group could be either a non-randomized “constructed group,” which was 

not exposed to the program (or was exposed to a different level of the type of services), 

or a “reflexive group,” namely the experimental group itself before exposure to the 

program. A historical design is an extension of a basic pre-program/post-program 

design that incorporates many measurements over time.

According to the Treasury Board of Canada (1998), an advantage of historical designs is 

that they are useful for analyzing programs that have time-dependent effects. “The 

authors state, “The longitudinal aspect of these designs can be used to address several 

questions: Is the observed effect lasting or does it diminish over time? Is it immediate or 

delayed, or is it seasonal in nature? Some type of historical design is called for 

whenever these types of questions are important” (pg. 49).

A frequently mentioned limitation in these studies was limitations in the available data. 

Another limitation in some studies was that other competing explanations may exist 

that were not accounted for in the study. The Treasury Board of Canada (1998) states, 

“Other threats remain--those related to history for example--because time series designs 

cannot eliminate the possibility that something other than the program caused a change 

between measurements taken before and after exposure” (p. 48). In addition, 

“Numerous data problems may exist with historical designs. In particular, the time 

series available are often much shorter than those usually recommended for statistical 

analysis (there are not enough data points); different data collection methods may have 

been used over the period being considered; and the indicators used may have changed 

over time.” Moffitt (1991, p. 61) discussed the challenge of data consistency in time 

series designs: “When measurements are made repeatedly, definitions and procedures 

may change. Care must be taken to see that time series are free of definitional and 

measurement changes, because these can be mistaken for program effects.”

However, recommendations have been made to strengthen time series designs. A more 

complete set of measures over time “allows the evaluator to eliminate many of these 

threats by analyzing pre- and post-program trends” (Treasury Board of Canada, 1998, 

p. 47). Another recommendation is to add one or more comparison groups (Treasury 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 32

Board of Canada, 1998). Because both the experimental and comparison groups should 

experience the same external factors, an observed change being caused by anything but 

the program is unlikely. “As with any design using a non-equivalent comparison 

group, however, the groups must be similar enough in terms of the characteristics of 

interest” (Treasury Board of Canada, 1998, p. 48) “With adequate time series data, this 

design can be fairly rigorous, ruling out many threats to internal validity, particularly 

maturation and testing effects” (Treasury Board of Canada, 1998, p.48). 

Survival Analysis

Survival analysis is a statistical approach that evaluates whether an event occurs, when 

it occurs, and what factors contribute to it occurring (Singer & Willett, 1991). It provides 

a hazard rate (what is the distribution of risk of the outcome over time?) as well as a 

survivor function (what is the overall chance of an outcome occurring?). Also, the 

method is able to identify time periods on a continuum when individuals are most at 

risk of an outcome and compare patterns of risk for different subgroups. Survival 

models relate the time (survival time) that passes before some event occurs to one or 

more covariates (predictor or confounding variables) that may be associated with the 

survival time. 

This methodology is versatile, in that it can use prospective or retrospective data, 

measure the outcome in either continuous or discrete intervals and be applied to 

various panel study designs (Singer & Willet, 1991). The only requirement is that data is 

collected over time, at intervals that are meaningful to the outcome and covariates being 

studied and precisely documents when the outcome event occurs. 

Several recent studies have used various types of survival models to assess the impacts 

of HCBS on time to nursing home admission or hospitalization. One such model is the 

Cox proportional hazard model introduced by Cox in 1972. The Cox proportional 

hazards model is the most widely used method of survival analysis to study an 

individual’s survival time to a particular outcome (New York State Office for the Aging, 

2010). Many of the studies we reviewed used this method.

► The Cox proportional hazard model was used in the Kansas Community Tenure 

Study (Macmillan et al., 2007), which has been recognized nationally for its methods. 

This study tracked a cohort of NH applicants who received a CARE assessment and 

were diverted from NHs for five years to identify duration of community tenure.

► The New York State Office for the Aging, 2010 used the Cox proportional hazards 

model to study time to nursing home placement for participants in Aging Network 

Services in New York. 

► As part of AoA's POMP, Brock et al. (2007) examined the effect of the receipt of 

OAA services on the potential delay in NH placement among OAA service clients 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 33

age 60 and older. The authors conducted time-to-event analyses (time to NH 

placement) using the Cox proportional hazard model.

► In the Advanced POMP, Westat conducted time-to-event analyses (time to NH 

placement) using Cox proportional hazards regression models applied to client data 

from the participating states (Karuza et al., 2011).

► Miller et al. (1998) used a Cox proportional hazards model in a longitudinal, 

prospective study on the effect of health care system characteristics on time to 

nursing home admission for persons with Alzheimer’s disease. The authors noted 

that the Cox model can allow HCBS researchers to examine the influence of 

predictor values most proximal to the event of nursing home admission. The study 

used data from 28 states from a longitudinal survey.

► Xu et al. (2010) used the Cox proportional hazards model in a longitudinal study 

with a single cohort on interval from the initial HCBS placement until each hospital 

admission, by volume of Indiana Medicaid waiver HCBS. The authors noted that a 

strength of this method is that the methodology examines whether specific 

components of the HCBS program are associated with hospitalization.

COX PROPORTIONAL HAZARDS MODEL

This is the most widely used method of survival analysis to study an individual’s survival time to 

a particular outcome. Past studies have used this method to estimate:

►Duration of community tenure (Macmillan et al., 2007)

►Time to nursing home placement (New York Office for the Aging, 2010; Brock, 

2007; Karuza et al., 2011; Miller et al., 1998)

►Time to each hospital admission (Xu et al., 2009)

Survival models are able to take into account the effect of time-varying covariates 

(e.g., deteriorating cognitive functioning) as well as censored data (i.e., data on study 

participants who have not experienced the outcome by the time study participation 

ends) (Singer & Willett, 1991). Additionally, the survival model can analyze an event as 

a risk factor for a subsequent event (e.g., if someone has two ER visits, are they at higher 

risk for the second ER visit because of the first). Each study participant has their own 

pattern of experiences. However, caution must be used, as pooled repeat spells can 

create bias.

As noted by Alecxih, Lutzky and Corea (1996), the effect of unique state factors--such as 

instituting certificate of need requirements or policies to promote HCBS, can be 

estimated by using a pooled time-series analysis technique. This methodology explains 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 34

changes in the dependent variable within each state (e.g., state changes in the number of 

HCBS care clients) by within-state changes or national regulatory changes. At the same 

time, the methodology holds constant all the many factors that vary substantially across 

states, but remain relatively fixed over time (e.g., geographic characteristics, the 

structure of the economy, racial and ethnic composition of the population). Pooled timeseries analysis has two major advantages over analysis of time-series data for a single 

state. First, 50 times as many observations are available to provide information on the 

effects of the independent variables. Second, each state serves essentially as a 

comparison state for all other states, allowing researchers to more definitively assess the 

extent to which changes observed in one state after a change in its programs or policies 

are unique to that state or are also observed in other states that had no program or 

policy change.

DISCRETE TIME SURVIVAL ANALYSIS/POOLED LOGISTIC REGRESSION

Studies have used this approach to measure:

►Odds of six outcomes (ER use, hospitalization, caregiver burden, death, NF use, and 

permanent NF placement), by dates of major policy and budget changes in 

Michigan (D’Souza et al., 2009)

►Risk of permanent NH relocation, by state, county, and person-level characteristics 

that change over time (Muramatsu et al., 2008)

►Risk of first long-term NH admission, by states’ spending on HCBS 

(Muramatsu et al., 2007)

Several studies used discrete time survival analysis. D’Souza et al. (2009) noted that 

this method is procedurally equivalent to pooled logistic regression.

► D’Souza et al. (2009) conducted a discrete time survival analysis of adjusted odds of 

six outcomes (emergency room use, hospitalization, caregiver burden, death, NH 

use, and permanent NH placement), controlling for demographics, functional status, 

and cognitive status. Using dates of major policy and budget changes in Michigan, 

the authors defined four distinct time periods to examine trends and outcomes. A 

longitudinal sampling scheme with adequate frequency provided the ability to 

analyze multiple outcomes over time while controlling for many covariates. 

► Muramatsu et al. (2008) performed a discrete time survival analysis of the risk of 

permanent NH relocation (NH death), to examine how it is associated with state 

spending on HCBS, taking into consideration the timing of the NH relocation and 

state, county, and person-level characteristics that change over time. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 35

► Another study by Muramatsu and colleagues (2007) was a discrete time survival 

analysis of first long-term NH admissions that occurred between 1995 and 2002, 

using Health and Retirement Study panel data. The study examined the effects of 

states’ spending on HCBS on risk of first long-term NH admissions. 

Interrupted and Short Interrupted Time Series

The GAO (1991) stated that the persuasiveness of the argument about the effect of a 

program on participants can be strengthened by using the interrupted series subtype of 

time series design. Also, administering these designs is “relatively easy.”

An interrupted time-series analysis compares trends in repeated measures of an 

outcome for a group before and after an intervention or policy is introduced, to learn if 

the desired change in outcome has occurred (GAO, 2009).

An interrupted time series design has about 100 observations on one unit, with 

treatment introduced at some known time (Shadish & Cook, 2009). Similar to regression 

discontinuity design, an effect is measured as a change in the slope or intercept of the 

time series at the point of treatment introduction. 

Because researchers rarely have the opportunity to gather so many data points over 

time, recent years have seen more interest and progress in the design and analysis of 

short interrupted time series, having for example 10 to 50 time points (Shadish & Cook, 

2009). Practice has moved toward interrupted time series designs with a control series, 

either created from units not participating in the intervention or from nonequivalent 

dependent variables—those that the intervention should not affect but that other 

alternative causes should affect.

During the 2000’s, new analytic models have been developed in the use of pooled time 

series and multilevel models that can often estimate treatment effects when many 

independent short-time series assessing the same intervention on the same outcome are 

available (Shadish et al. 2009). Examples of the use of these designs include 20-40 

schools or students that are each measured 4-8 times.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 36

Exhaustive Alternative Causal Explanation Elimination Design

The Treasury Board of Canada (1998) recommended including a causal model as part of 

a program evaluation design. They state:

“An alternative way of addressing the issues of causal inference involves 

the use of a causal model: an equation that describes the marginal 

impact of a set of selected independent variables on a dependent 

[outcome] variable. While quasi-experimental designs focus on 

comparisons between program recipients and one or more control 

groups, causal models focus on the variables to be included in the 

model--both endogenous (intrinsic to the program) and exogenous 

(outside the program)--and their postulated relationships. In quasiexperimental designs, the program is of central interest; for causal 

models, the program is only one of several independent variables that 

are expected to affect the dependent variable” (pg. 56).

An exhaustive alternative causal explanation design works in situations where an 

outcome is observed, but whether the outcome was caused by the intervention or some 

other factor is unclear (Duignan, 2005-2009). Using this design involves listing all 

possible alternative explanations and then systematically eliminating each of them as 

having caused the improvement. This provides a reasonable basis for believing that the 

intervention is the cause of the improvement. This is sometimes described more as a

“forensic-type” method than the quasi-experimental approaches described above.

Some of the HCBS studies that we reviewed used a cross-sectional statistical model to 

develop causal explanations for factors affecting HCBS outcomes.

► Chen and Thompson (2010) used a structural equation model to test a theoretical 

framework to predict the likelihood of older adults residing in the community.

► Miller (2011) combined data from many sources to conduct a state-level analysis of 

data from 2000 to 2007, using multivariate fixed effects models to examine the 

association between level of HCBS investment, other state and program 

characteristics, and state-level rates of NH use for working-age and older adults.

► Muramatsu and Campbell (2002) used a cross-sectional, hierarchical model. The 

study involved a two-level multinomial logistic regression analysis to examine 

effects of state expenditures on HCBS and the use of formal personal assistance and 

informal caregiving.

Muramatsu and Campbell (2002) noted that multi-level models are a set of related 

analytic approaches for exploring links between macro and micro levels of social 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 37

phenomena. The authors described several advantages of these models, also called 

contextual models, hierarchical linear models, or random coefficient models: 

► First, they address the non-independence of observations engendered by the nesting 

of subjects within state.

► Second, they provide a useful means of linking macro level data to individual 

outcomes. 

► Finally, they explicitly recognize the fact that the states represented in the sample 

are a subset of the 50 states and DC to which the study wishes to generalize. The 

authors used a new approach by putting a relatively well-researched individuallevel model of use of personal assistance into a macro-level (state-level) context and 

by using a multi-level multinomial logistic regression analysis.

The Treasury Board of Canada (1998, p. 57) noted, “Causal models are best suited to 

situations where sufficient empirical evidence has confirmed, before the evaluation, the 

existence of a relationship between the variables of interest... In addition, statistical 

analyses can be used to control for selection or history biases....The main strength of 

simulation is that it allows the evaluator to estimate incremental effects in complex and 

uncertain situations. The main limitation of the technique is that it requires a 

sophisticated understanding of the dynamics of the program, as well as some skill in 

building quantitative models” (p. 97). They added, “Statistical models are often vital in 

identifying incremental effects.” 

The Treasury Board of Canada (1998, p. 106) recommended, “Statistical models are 

versatile and, if properly constructed, will provide very useful estimates of program 

results. On the other hand, statistical models must be appropriately specified and 

validated to provide reliable results, which is not always as straightforward a task as it 

may at first appear. One weakness of statistical models is that the evaluator may not be 

able to draw inferences from them. For example, if the model covers only certain age 

groups or individuals in certain geographic areas, the evaluator may not be able to infer 

from his or her results the program's probable effects in other geographic areas or on 

other age groups.”

Most federal programs are larger programs, which normally produce large data sets 

and can therefore rely on regression-based ‘linear models’ to identify effects (Treasury 

Board of Canada, 1998, p. 105). “Regression analysis often is used “as the final 

confirmation and measurement of a causal relationship between the program and 

observed effects. In fact, it is important that the regression model be based on a priori

reasoning about causality”. One way to determine if a model is probably robust is to 

“specify and calibrate the model using only half the data available and then see whether 

the model is a good predictor of outcomes shown in the other half of the data. If this is 

the case, then the model is probably robust.”
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 38

“In practice, most evaluators will want to use both causal and comparative approaches 

to determine program results” (Treasury Board of Canada, 1998). Quasi-experimental 

designs can be used to construct and manipulate control groups and, thereby, to make 

causal inferences about program results. Causal models can be used to estimate the 

marginal impact of variables that affect program success.” [refers to Bickman 1987 and 

Trochim 1986] (p. 56) “Regardless of the design chosen, it is desirable that the causal 

model approach be incorporated into the evaluation design, to the extent possible, to 

support the credibility of the findings.” (p. 59)

Findings from Early HCBS Evaluations

Many early evaluations of HCBS programs centered on the question, Does HCBS save 

money? For example, a frequently referenced early study is the National Long-Term 

Care Channeling Demonstration (“Channeling”). A recent fact sheet prepared for AoA 

by members of the Lewin team summarized findings and lessons learned from this 

demonstration, based on our review of many Channeling-related studies (The Lewin 

Group, 2012). This large-scale demonstration was a randomized experiment that tested 

the feasibility and cost-effectiveness of a community-based LTC delivery model 

integrating health and social services. The concept involved “channeling” people with 

LTC needs, i.e., matching them with appropriate LTC services (ASPE, 1991). Results 

from the Channeling demonstration did not show any significant differences between 

treatment and control groups in hospital use, nursing home (NH) use, or longevity. 

Several early reviews of the literature on the impacts of HCBS, conducted from 1985 

through 2006, found inconclusive evidence that it reduced nursing home utilization or 

saved costs (Hughes, 1985; Hedrick & Inui, 1986; Kemper, Applebaum, & Harrigan 

1987; Weissert & Hedrick 1994; Doty 2000; Grabowski 2006) (see Text Box). This 

suggests the need to look at studies beyond this time period to understand the most 

recent available evidence. A frequently mentioned conclusion in these studies was that 

efforts to expand community-based LTC would need to be supported on the basis of 

other benefits they provided (e.g., meeting individual preference and improved life 

satisfaction), rather than on grounds of cost-effectiveness (Weisserts & Hedrick, 1994; 

Kemper, Applebaum, & Harrigan, 1987; Kemper, 1988; Thornton, Dustan, & Kemper, 

1988). Nonetheless, policymakers were reluctant to fund HCBS for years after studies 

prematurely suggested that HCBS might not be cost-effective. Only in the past few 

years have studies using data from a longer time period and more sophisticated 

analytical models demonstrated the ultimate cost effectiveness of shifting funding from 

institutional care to HCBS, as our several recent reviews of the literature show. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 39

Findings And Lessons Learned From Early HCBS Literature Reviews

(1985-2006)

Several early reviews of the literature, conducted from 1985 through 2006, found inconclusive 

results and suggested the need to examine the multi-level factors affecting HCBS outcomes.

Hughes (1985) found “contradictory findings” in the community-based LTC literature. She 

suggested that “increased preoperational specification of underlying theory, increased 

sophistication in targeting services to high-risk groups, use of multivariate analysis, and the 

development of more relevant outcomes measures will improve the quality of future study 

findings, thereby contributing to theory and model building in this field.”

Hedrick & Inui (1986) found that, across studies, home care services either had no effect on 

hospitalization or tended to increase the number of hospital days. The cost of care either was 

not affected or increased. The author contended, “The critical need at present is for betterdesigned studies to test the effects of different types of home care, targeted at various types of 

patients, on the outcomes assessed in the existing studies, as well as other important outcomes 

such as family finances, quality of life, and quality of care.”

Kemper, Applebaum, & Harrigan (1987) found evidence that expanding publicly financed 

community services would likely increase costs. They recommended, “Policymakers should 

move beyond asking whether expanding community care will reduce costs to addressing how 

much community care society is willing to pay for, who should receive it, and how it can be 

delivered efficiently.”

Weissert & Hedrick (1994) found an absence of demonstrated cost-effectiveness of 

community-based LTC. They suggested, “It may be time to lay aside studies showing that home 

care is not cost-effective and get started on ways to make it become so.” (Weissert, 1990)

Doty (2000) found that in principle, personal care and waiver services could provide an 

alternative to NH placement, but limitations on the effectiveness of these programs in deterring 

NH admissions were experienced in practice. A major finding from this review was that, in 

contrast to LTSS for elderly persons and people with disabilities, state systems for people with 

intellectual and developmental disabilities had “evolved far beyond the traditional ‘cost-effective 

alternatives to institutional care’ paradigm. In that system, a totally different perspective was 

evident, in which “States do not make those desiring services demonstrate or contend that they 

will otherwise go into institutions; the states are on record as desiring that people are spared 

that debilitating experience.” Instead, states are faced with creating priorities for people 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 40

Findings And Lessons Learned From Early HCBS Literature Reviews

(1985-2006)

waiting for community services, based on factors such as immediacy of need, severity of 

disability, and length of wait.

Grabowski (2006) conducted a review and reported that noninstitutional LTC services were 

generally associated with increased costs, but greater client and caregiver welfare. The review 

found that “most recent evaluations have relied on potentially confounded research designs, 

which leaves open the question of whether the findings relate to the programs or biased 

selection across the treatment and comparison groups” (pg. 3).

A frequently mentioned recommendation in the early HCBS literature was that, to 

demonstrate cost savings, HCBS would need to be better targeted to individuals most 

likely to benefit from the services (Weissert & Hedrick, 1994; Greene, Ondrich, & 

Laditka, 1988). On the other hand, Kemper, Applebaum, and Harrigan (1987), 25 years 

ago, contended that the important issue was not efficiency of targeting, but rather an 

issue of equity: “Who deserved the limited community care that society can pay for?” 

They recommended that the debate about HCBS “should move beyond asking whether 

expanded public financing of community care will reduce costs to addressing how 

much community care society is willing to pay for, who should receive it, and how a 

more efficient long-term care system can be designed” (Kemper, Applebuaum, & 

Harrigan, 1987, p. 98). This was based on their review of previous community-based 

care demonstrations to examine “What have we learned?” Similarly, in 1988, Kane 

recommended the need for a “new paradigm” to replace the quest for cost-effective 

“alternatives” to nursing homes, contending that this quest may be counterproductive. 

A reason is that community-based LTC services include a wide array of service types, 

and research is needed to examine the effective design and targeting of each kind of 

community service.

Theoretical Considerations

Previous studies have used various models, or theoretical frameworks to guide research 

related to the effects of HCBS on community tenure. Several examples are described 

below, including a recent model known as an “outcomes model”, which establishes all 

steps necessary to achieve high-level outcomes for the program. The indicators are then 

monitored and mapped onto the visual model, along with evaluation questions. 

Models and Theoretical Frameworks

A frequently used model is the Andersen model (Andersen, 1995, p. 2-3, cited in Chapin 

et al., 2002). The Andersen model, widely used to understand patterns of health service

use of older adults, was selected for use in the Kansas Community Tenure Study 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 41

because it organizes factors leading to service use, thus matching the study’s structure 

of examining factors predicting NH admission and mobilization of community 

supports (Chapin et al., 2002). In addition, Miller et al. (1998) used Andersen’s 

Behavioral Model of Health Services Utilization (Andersen & Newman, 1973) as the 

conceptual framework for selecting and analyzing model variables in the study. In the 

version employed by Chapin et al. (2002), the model comprises six categories related to 

factors affecting the use of HCBS and NH services:

Building on Andersen’s model, Chen and Adams Thompson (2010) used the Health 

Behavioral Model (HBM) to construct an HCBS model with a longitudinal perspective, 

to predict and explain when use of HCBS facilitated remaining in communities. Their 

home and community based services (HCBS) model included the following categories 

and sub-categories:

D’Souza et al. (2009), in a study of the effects of financial strain on HCBS use and 

outcomes in Michigan, tested a conceptual framework for how budget environments 

affect resources, which may affect program staffing and services, ultimately impacting 

participant outcomes. Elements of the framework included:

► Personal factors

 Predisposing factors

 Social enabling and financial factors

 Disability and perceived unmet need factors

► Service use

 HCBS use

► Remaining in communities

people live and work. Then, people must have the means and know-how to get 

those services....income, health insurance, a regular source of care”

► Need factors: “Perceived need is largely a social phenomenon...Evaluated need 

represents professional judgment about people’s health status and their need for 

medical care”

► Prior Use: Health Service Use, HCBS use, NH service use

► Predisposing factors: factors that affect “the status of a person in the community, 

his or her ability to cope with presenting problems and commanding resources 

to deal with these problems”

► Enabling factors: “Health personnel and facilities must be available where 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 42

The research design for the evaluation of the Money Follows the Person (MFP) grant 

program (Brown et al., 2008) uses a model for MFP transition programs that includes 

the following outcomes and action steps:

► Other factors

 Agency-specific effects

 Medical care environment

► Decline in hours of formal care per participant

► Decline in participant health 

► Adverse outcomes

► Change in medical/functional eligibility, leading to increase in need for formal 

care

► Inflation, budget reductions, and program intake closure, leading to strain on 

program resources

► Decrease in direct services funding

 Purchased services

► Decrease in administrative funding

 Staffing

 Monitoring

 Service coordination

 Outside funding sources
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 43

Visual Outcomes Model

Generally, models used for program evaluation design connect inputs (resources 

invested in the program); program activities (what the program does and who it 

reaches); short-term, medium, and long-term outcomes; and influential factors in the 

environment (GAO, 2012). 

A new, less well known and efficient way of building a model that can be used for 

evaluation planning and other purposes is to use a visual “outcomes model” of the 

project, program, or organization (Duignan, 2009). An outcomes model identifies all 

steps that are believed to be needed to achieve high-level outcomes for the program or 

organization. The indicators to be monitored can then be mapped onto the model, along 

with the evaluation questions that need to be answered. This approach, developed by 

► Outreach program: identify eligible enrollees, provide informed consent

► Institutionalized enrolled: characteristics affect likelihood of transition (age, level 

of need, types of conditions, type of institutional care/setting, length of 

institutionalization), receive transition services (housing assistance, service 

planning)

► Enrollees transitioned receive qualified HCBS, demonstration services, 

supplemental services

► Generate enhanced funds for rebalancing program

► Enrollee outcomes: increase number transitioned, increase length of community 

residence, decrease re-institutionalization, reduce Medicaid and Medicare 

expenditures, receive high quality of care, increase quality of life

► MFP rebalance initiative: Medicaid enrollees transition from institutional care to 

the community and receive HCBS, enhanced funds flow to state to reduce 

institutional care, support transitions, increase HCBS, strengthen infrastructure

► Institutional care: decrease number and types of beds, change budget allocation 

between institutional care and HCBS

► HCBS: Change mix of services, increase number and type of waiver programs

► System-level outcomes: Increase proportion of institutionalized Medicaid 

enrollees transitioned to community, increase average age/acuity level of entry 

to institutional care, increase absolute and relative total use of and spending on 

HCBS

► Decrease absolute and relative use of and spending on institutional care

► Decrease LTC spending per LTC recipient

► Increase ratio of HCBS expenditures to total LTC expenditures
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 44

New Zealand evaluation expert Duignan, is called Duignan’s Visual Monitoring and 

Evaluation Planning and DoView Results Roadmap Evaluation Planning. It provides a 

transparent model that can be used for other purposes, such as strategic planning and 

evidence-based practices, as well as impact evaluation. 

OUTCOMES MODEL

An outcomes model can be used when implementing similar projects in a number of different 

settings. This is more efficient than all places “reinventing the wheel” when it comes to planning, 

implementing, monitoring, and evaluating their particular program. (Duignan, 2009)

Once the outcomes model is drawn, the links between steps and outcomes within the 

model can be examined to see what evidence exists for the link (Duignan, 2008a). 

Any step or outcome can have one or more of the following features; it can be: 

“influencable” by a program, controllable by a program, measurable, attributable to a 

program, and/or accountable. In addition, outcomes models are distinguished by the 

following features (Duignan, 2008a): 

► The steps and outcomes are not limited to those that are controllable or attributable 

to a particular program. They do not have to only include steps and outcomes that 

are measurable.

► They have no artificial limit on their size. The most common mistake in this regard, 

according to Duignan (2008a), is to attempt to fit a model onto a single printed page. 

This creates serious problems, because in all significant policy areas, drawing a 

comprehensive causal map will require more space than just one page. 

► They allow for the possibility that any step or outcome can be causally connected to 

any other step or outcome in the model. The current standard method of visualizing 

causal links (a drawn line and arrow) has major disadvantages when it comes to 

visualizing models that have complex patterns of causality, Duignan observed. 

Often causal links between steps and outcomes are omitted from a diagram because 

the person drawing the diagram cannot see how they can fit an additional causal 

line into the diagram. Thus, models often end up being drawn in a “siloed” manner, 

such that any lower-level step is only allowed to contribute to a single higher-level 

outcome. Duignan noted that this indicates the need for a way of representing 

causal links between steps and outcomes in situations where they cannot be 

visualized as line and arrow links.

► They are not compromised by arbitrary constraints that make them harder to 

interpret. In particular, Duignan noted, evaluation studies sometimes require that 

models consist of a single layer of outputs, above them a single layer of intermediate 

outcomes, and above them a single layer of final outcomes. Duignan stated, “The 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 45

use of this constraint is reinforced by the unexamined acceptance of the one page 

limit on models.” 

► They need to be able to be used in all parts of the decision-making process. While 

such a model could theoretically be developed using any software, a software called 

DoView facilitates this by making an outcomes model easily portable across 

different media, to be used in meetings with stakeholders and for other purposes. It 

is the only software we discovered that is designed for making large models with 

numerous steps showing very complex relationships crossing multiple pages. The 

user can add to each element in the model: notes, links to documents and websites, 

pictures, and questions. The model can be easily converted to poster size, turned 

into a web model, or projected on a screen. 

► They can be divided up into an interconnected set of sub-modules. Each sub-module 

contains a set of conceptually related steps, for example, many types of programs 

can set out steps and outcomes at a national, regional, organizational, and 

individual level.

The findings included throughout this report could be used to develop a generic model 

for HCBS outcomes, “so that the evaluation is not approached as a novel exercise, with 

different evaluation plans laid out in very different formats” (Duignan, 2008b). Also, 

this model would make it easier for those planning evaluations to ensure that they have 

included all of the important elements needed in an evaluation plan. The model could 

be developed through discussions among AoA, stakeholders, and the Lewin study

design team, using software to facilitate representation of the complex relationships 

across levels and outcomes. Some of these items can be measured using existing data 

sets, while some items would require new data collection efforts (e.g., surveys of older 

adults to ascertain their awareness of unmet needs or perceptions of functional 

abilities).

Findings and Lessons Learned from Past HCBS Studies

This chapter synthesizes key findings from our review of past studies of the impacts of 

OAA and non-OAA HCBS programs and services. The complex, fragmented, dynamic 

and unpredictable long-term care delivery system does not lend itself to traditional 

views of cause-and-effect that assume a linear relationship resulting in an additive 

model in which the output of a system is proportional to its inputs. Overlay the 

disjointed long-term care system with myriad service users, and we must consider a 

linked system filled with attributes that can be seen as both cause and effect, as well as 

additive and summative. 

We decided to focus on variables that demonstrated significance with one of the four 

key outcomes of interest in this review: 1) HCBS service use, 2) health care use, 3) 

community tenure/NH use, and 4) cost impacts. These variables are both statistically 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 46

significant independent variables and control variables. It is presumed that the final 

study design will included a broad mix of variables. To best organize these variables, 

we used the Andersen Model, a widely-used framework among the studies analyzed.

Andersen Model

For this literature review, we chose to use the Andersen Health Behavioral Conceptual 

Model to frame our literature analysis. As mentioned previously, this model has been 

widely used to determine variables tested in health care studies (Andersen, 1995; Miller 

and Weissert, 2000). This model highlights predisposing, enabling, and need 

characteristics as variables that contribute to use of services, including HCBS, health care, 

and nursing home use. The exhibit below illustrates the way these variables relate to one 

another. Predisposing characteristics include demographics, social support, and health 

beliefs. Enabling characteristics include familial, facility, market, and policy resources. 

Finally, need includes perception of need, cognitive and physical function, and disease. 

The use of other services are also predictors of one another; for example, prior hospital 

use is a predictor of NH admission (D’Souza, 2009; Gaugler et al., 2007; Goodwin et al., 

2011; Miller & Weissert, 2000; Sands et al., 2008; Xu et al., 2010). This literature review 

presents findings from numerous studies on how the aforementioned characteristics, or 

variables, contribute to use of services. The purpose of this section is to review the most 

important variables to include in a study of this kind.

Key Variables Table

Table 2 shows the key variables identified through the literature review and their 

association with the four outcomes of interest. A full table with sources can be found in 

Appendix A-B. 

Table 2 : Key Variables

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Predisposing Variables

Age √ √ √

Education √ √

Gender √ √ √

Race/Ethnicity √ √ √

Geographic Location √ √ √

Living Alone √ √ √
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 47

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Marital Status √ √ √

Informal Caregiver Availability √ √ √

Having Children √ √

Social Support √

Enabling Variables 

Income √ √ √

Home Ownership √

Payment Method √ √

Caregiver or Spouse Age √

Support for Caregiver √

Caregiver Burden √

NH Bed Supply √

Number of Home Health Agencies √

Medicaid HCBS Spending √ √

State HCBS Spending √ √ √

Pre-admission screenings √

Need Variables 

Alzheimer's/Cognitive 

Impairment/Dementia

√ √ √

Comorbidities/Chronic Medical 

Conditions

√ √ √

Psychiatric Issues/Problematic 

Behavior

√

Worse ADL/IADL Performance in 

general

√

ADL Limitations √ √

IADL Limitations √ √

Self-health Ratings √

Perceived Unmet Need √ √ √

Service Use Variables 

Hospital Admissions/Use √ √ √

HCBS Use √ √ √ √

NH Use √ √
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 48

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Paid Helpers √

Case Management √

Medicaid Waiver √

Timing of Care √

Volume of Services √ √ √

Low-intensity HCBS √ √

Combination of Services √ √

Use of OAA III-B √

Use of OAA III-C √ √ √

Use of OAA III-E/Respite √ √

Predisposing Variables

Age

Many predisposing characteristics have been identified as contributing to HCBS use. 

Prior research has highlighted the effects of demographic variables on using HCBS. 

Some studies find that age is a strong predictor of service utilization (Liu, McBridge, and 

Coughlin, 1994; Mitchell and Krout, 1998). In a study by Alkema et al. (2006), researchers 

found that age was only a significant predictor for one type of referral to service, 

suggesting that it may be more appropriate as a proxy variable, representing increasing 

needs due to disability and chronic care. 

Similarly, increased age is one of the strongest predictors of nursing home admission

(Bharucha et al., 2004; Cai, Salmon, & Rodgers, 2009; Chen & Thompson, 2010; Gaugler et 

al., 2007; Miller and Weissert, 2000; Muramatsu et al., 2007; Ness, Ahmed, & Aronow, 

2004). Friedman et al. (2005) found that, for Programs for All-inclusive Care for the 

Elderly (PACE) participants enrolled from the community, those of greater age were at an 

increased risk of NH admission. Older age has also been associated with an increased 

risk of LTC placement after hospitalization (Goodwin et al., 2011). For those without 

dementia in Florida, Andel, Hyer, and Slack (2007) found that older age was associated 

with increased risk of NH admission. However, it is important to note that some studies 

found no association between age and NH admission (MacMillan, 2007; Sands, 2008). 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 49

Gender

Gender is a statistically significant predictor of HCBS use. One study found that being 

female, along with having lower educational attainment, is associated with using home 

safety services. Women were also more likely to use home modification and 

transportation services (Alkema et al., 2006). Laditka, Laditka, and Davis (2006) 

compared HCBS use by older African American, Hispanic, and Non-Hispanic White 

women and men. They found that, in comparison to non-Hispanic white women, nonHispanic white men were less likely to use Personal Care Aides, Senior Centers, and 

Transportation. Additionally, in comparison to the same group, they use services less 

intensively.

In Miller and Weissert’s (2000) meta-analysis of studies regarding adverse outcomes for 

older adults, they found that females were less likely to be hospitalized.

Female gender is another variable that has resulted in mixed findings for NH admission. 

Gaugler et al. (2007) stated that women appear less likely to enter a NH than men. This is 

supported by Cai, Salmon, and Rodger’s (2009) finding that female gender was associated 

with less risk of NH admission. Conversely, Goodwin et al. (2011) report that female 

gender is a risk factor for LTC placement after hospitalization. Ness, Ahmed, and 

Aronow, in their study of 23 years of NH demographic and payment data, found that 

women were at an increased risk of NH placement from 1977-1999. Karuza and Wu 

(2011) discuss these inconsistent findings, pointing out that Miller and Weissert (2000) 

found that women were at less risk, but of the 47 articles using gender as a variable, 40 of 

them did not show significant findings. Noël-Miller (2010) analyzed the Health & 

Retirement Study – following couples who were living in the community in 1998 over 8 

years. She found that widowed men’s risk of nursing home entry doubled; however, risk 

was unchanged for women after spousal death. 

Race and Ethnicity

Race and ethnicity is also a predictor of HCBS use. This variable has mixed findings. 

One study noted that among predisposing characteristics, age and race/ethnicity have 

been found to be the strongest predictors of service utilization (Liu, Mcbridge, & 

Coughlin, 1994). Mitchell and Krout’s findings (1998) indicate that being African 

American is associated with increased discretionary service use. These services are 

considered to be more based on the choice of the eligible adult/caregiver, whereas less 

discretionary services are considered medical services, such as hospitalization, that are 

not the choice of the consumer. Sands (2008) found that HCBS users were more likely to 

be white than nursing home patients, suggesting that white older adults were more likely 

to remain in the community. Goins, Tincher, and Spencer (2003) examined awareness 

and utilization of HCBS by rural American Indian and White older adults who have co-
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 50

morbid diabetes. They found that American Indian respondents were more likely to both 

be aware of and use HCBS programs in comparison to white respondents. 

Wallace et al. (1999) found that Black women in their study had greater health needs, 

however they use less formal services. Another study by Laditka, Laditka, & Drake 

(2006) found that African American women had higher odds, in comparison to nonHispanic white women, of using Personal Care Aides, Transportation, Adult Day 

Centers, and Information & Referral. They conducted an intensity analysis and found 

that African American women used services more intensively than non-Hispanic white 

women. Casado, van Vulpen, and Davis (2010) found that black race/ethnicity was 

associated with having unmet HCBS needs.

Haller and Gessert (2007) examined the factors associated with medical care utilization

at end of life in NH residents who had severe cognitive impairments. They found that 

non-white race was associated with high medical care utilization within 90 days prior to 

death.

Many studies have found that being white increases the risk of being institutionalized

(Andel, Hyer, & Slack, 2007; Miller & Weissert, 2000; Gaugler et al., 2007, Cai, Salmon, & 

Rodgers, 2009). Stevens et al. (2004) found that African American care recipients were 

placed in NH significantly slower than White care recipients. However, other studies 

have found that being African American or Hispanic is significantly associated with an 

increased risk of NH admission (Muramatsu et al., 2007; Ness, Ahmed, & Aronow, 2004). 

Zhanlian et al. (2011) found that the number of NH residents of color has increased more 

rapidly than their overall demographic increases. They suggest that this may relate to 

unequal access to HCBS alternatives to NH admission. 

Education

Alkema et al. (2008) found that level of education was significantly associated with use of 

any HCBS category – when controlling for other factors, every year increase in education 

resulted in 14% less likelihood of using any HCBS.5 Li (2006) found that less educational 

attainment within rural areas was a barrier to accessing homemaker services.

Some studies have shown that higher education is significantly associated with reduced 

nursing home admission (Muramtasu et al., 2007; Miller, 1998). However, Miller and 

 

5 The authors did not anticipate this finding, given a lack of research on educational attainment’s effect on 

service use. They posit that, given the fact that socioeconomic status is a proxy for education, lower 

income individuals may not have considered using these services (e.g., home modifications) before. 

But upon hearing of their benefit and potential affordability, they were more open to their use. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 51

Weissert (2000) found that education did not appear to be a significant predictor of 

institutionalization.

Geographic Location

Geographic location is an important variable, especially with the increase in 

nonmetropolitan residents who are older adults6. Living in a rural area has been 

associated with service use (McAuley, Spector, Van Nostrand, & Shaffer, 2004; Mitchell & 

Krout, 1998). McAuley et al. (2004) found that using Medicare home health was 

significantly greater for residents in the most rural counties of their study, no matter what 

the individual’s Medicaid coverage was. This was the same for Medicaid-covered 

residents in nonmetropolitan areas. Mitchell and Krout (1998) found that living in a rural 

or in a small town was associated with increased use of the most discretionary services. 

Another study by Sun (2011) reported that rural older adults in Alabama were less likely 

to use HCBS than their urban counterparts. 

Older adults with cognitive impairments living in an urban location had an increased risk 

of high medical care utilization within 90 days prior to death (Haller & Gessert, 2007).

In an analysis of the Longitudinal Study on Aging (1984-1990), Coward, Home, and Peek 

(1994) found that older adults with urinary incontinence, who lived in less urbanized and 

thinly populated nonmetropolitan counties, were more likely to have NH admission than 

older adults with urinary incontinence in other residential areas.

Living Alone

Living alone has also been associated with more service use (Xu et al., 2010; Alkema et 

al., 2006; Wallace et al., 1999). Older adults living alone had a higher probability of using 

transportation services (Alkema et al., 2006) and Xu et al. (2010) found that those living 

alone were more likely to use more than 8 hours of formal attendant care. Weddle et al. 

(2012) found that African Americans who live alone are more likely to use congregate 

meal services. Many studies find that living alone also increases the risk of being 

institutionalized (Cai, Salmon, & Rodgers, 2009; Gaugler et al., 2007; Miller & Weissert, 

2000).

Marital Status

Wallace et al. (1999) found that being widowed was associated with increased service 

use. In two systematic literature reviews, being married was found to be associated with 

 

6 http://www.ers.usda.gov/briefing/population/challenges.htm
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 52

lower odds or reduced risk of NH placement (Gaugler et al., 2007; Miller & Weissert, 

2000). Similarly, Borrayo et al. (2002) report not being married was associated with 

increased likelihood of being in a NF or ALF than using HCBS.

For older adults with dementia, Andel, Hyer and Slack (2007) found that married 

individuals were more likely to be placed in a NH. They found that widowed status 

reduced the risk of NH placement among older adults without dementia. They suggest 

that this may be due to the unique nature of this population. Having limited financial 

resources, a spouse may actually be a burden or an additional strain on resources. 

Another study found that being widowed was associated with a decline in NH 

placements from 1977 to 1999 (Ness, Ahmed, & Aronow, 2004).

Having Children

Gaugler et al. (2007) found that having children lowered the odds of NH admission. 

Muramatsu et al. (2007) also found that living close to or living with children was 

significantly associated with lower NH admissions. Noël-Miller (2010) found that having 

adult children reduced the risk of NH admission for wives, regardless of husbands’ vital 

status. Adult children buffered husbands’ risk only after their wives’ death. 

Informal Support

Older adults who receive unpaid caregiving support, and who have personal care needs, 

are less likely to use discretionary services and more likely to use nondiscretionary 

services than those who do not receive any formal assistance (Mitchell & Krout, 1998). 

Having an informal caregiver was associated with increased risk of hospitalization

(Miller & Weissert, 2000). Van Houtven and Norton (2004) found that informal care 

reduced home health care use.

The presence of an informal caregiver is also a variable that has mixed findings relating to 

NH admission. In their systematic literature reviews, both Miller and Weissert (2000) 

and Gaugler et al. (2007) found that caregiver availability increased risk/was associated 

with greater odds of entering a nursing home. Gaugler et al. (2007) points out that this 

contrasts with prior conclusions on informal care. In an analysis of PACE participants 

who had a caregiver, Friedman et al. (2006) found that caregiver presence did not change 

the older adult’s institutional risk. In their analysis of the AHEAD study, Lo Sasso and 

Johnson (2002) found that receiving frequent assistance from children with their [older 

adults with a disability] basic personal care reduced the likelihood of NH use over a two 

year period by about 60%. Andel, Hyer, and Slack (2007) also found that caregiver 

presence reduced the risk of NH admission for older adults without dementia. Van 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 53

Houtven and Norton (2004) found, in an analysis of the HRS and AHEAD survey, that 

informal care reduced home health care use and delayed NH admission. Finally, on a 

similar note, less or worse social support was associated with increased risk of 

institutionalization (Bharucha et al., 2004).

Spouse Service Use

Having a spouse use services was positively related to older adults using CCRC services

(Krout et al., 2000).

Enabling Variables

Income

Wolinsky and Johnson (1991) point out that income enables service use. Borrayo et al. 

(2004) found that Medicaid-eligible individuals were 26 times more likely to utilize 

Florida’s Aged and Disabled Waiver program than other state-funded services. Another 

study found that concern over the affordability of services is associated with unmet 

HCBS need (Casado, van Vulpen, & Davis, 2010).

In Miller and Weissert’s (2000) review, they found that most studies did not report 

income to be a significant predictor of institutionalization or hospitalization. 

Muramatsu et al. (2007) measured income and non-housing wealth using terciles. 

Income was not associated with risk of NH admissions. However, being in the high 

wealth tercile, as opposed to the low wealth tercile, did reduce the risk of NH admission.

Payment Method

Medicaid and private pay were found to be the most common payment sources in an 

analysis of the National Nursing Home Surveys from 1977-1999 (Ness, Ahmed, & 

Aronow, 2004). In the last year (1999), Medicaid payment was most prevalent among 

residents age 65-74, women, and African Americans. The CARE study, found that being 

a customer identified as potentially lower income at the time of the study was associated 

with higher rates of NH admissions than those not qualifying for Medicaid, and were 

identified as low-income (Macmillan et al., 2007). A higher percentage of those receiving 

Medicaid HCBS/FE services remained in the community in comparison to those who did 

not; however, this was not statistically significant. Muramatsu et al. (2007) caution that 

Medicare status is difficult to evaluate as a predictor of HCBS or NH admission, because 

virtually all people over 65 have Medicare. Borrayo, Salmon, Polivka, and Dunlop (2002) 

found that Medicaid eligibility was associated with higher use of nursing facilities in 

Florida. However, this was also associated with a higher likelihood of HCBS use, in 

comparison to ALF use.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 54

Home Ownership

Home ownership has been associated with a delayed or decreased risk of NH admission

(Cai, Salmon, & Rodgers, 2009; Gaugler et al., 2007).

Caregiver Characteristics, Burden, & Assistance

Casado, van Vulpen, and Davis (2010) reported that if a caregiver has substitute help, 

they report having less unmet HCBS services.

Regarding informal caregiving, enabling characteristics are associated with the 

caregiver’s characteristics. Yaffe et al. (2002) found that caregivers being between the 

ages of 65 and 74 and older than 75 was associated with a greater risk of NH placement. 

If the caregiver receives greater support, Miller and Weissert (2000) found that was 

associated with a decreased risk of NH entry. Caregiver burden is a strong predictor of 

NH entry. Greater caregiver burden has been associated with increased NH placement in 

a number of studies (Gaugler et al., 2009; Miller, Rosenheck, & Schnieder, 2012; Spillman 

& Long, 2009; Yaffe et al., 2002).

Policy Variables: State Spending on HCBS 

Muramatsu et al. (2007) used two variables to look at state HCBS spending and other 

state-level funding: One was per capita HCBS spending (total HCBS 

spending/population of 65 or older), which measures the absolute amount of money 

going towards HCBS; and (2) Percentage of LTC spending going to HCBS rather than 

nursing homes, which measures the extent of the state's LTC system orientation 

towards HCBS as compared with institutions. The study’s interaction effect models 

showed that living in a state with higher HCBS spending reduced NH admission risk 

among childless seniors, but not among those with living children. Controlling for other 

characteristics, only 3% of the variance in NH admissions was between states, 

indicating that state-level factors played relatively small roles. However, the authors 

recommended that the allocation of HCBS resources requires careful consideration 

about fairness for individual older adults and their families, as well as costeffectiveness. They noted, “Targeting HCBS resources to seniors without children 

would penalize those with children such that these seniors would have to continue to 

depend on their children, who would be locked into caregiving roles” (Muramatsu et 

al., 2007). 

Miller et al. (1998) found that a higher percentage of Medicaid LTC spending on HCBS 

(versus institutional LTSS) was significantly associated with a longer time to NH 

admission for unmarried persons in the study, but not married persons. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 55

Results of the Muramatsu et al. (2008) study on state HCBS spending and place of death 

among older Americans suggested that doubling per capita HCBS services would result 

in a 25% reduction in the risk of NH relocation for people on Medicaid. The effect was 

not significant for people without Medicaid. Higher state HCBS spending was 

associated with lower risk of NH relocation, especially among those with Medicaid. 

Recent studies have also examined predictors of low-care NH residents (Hahn et al., 

2011; Thomas & Mor, 2013 manuscript). One study found that a $10,000 increase in perenrollee HCBS Medicaid waiver expenditures was associated with a 3.5 percentage

point reduction in prevalence of low-care resident prevalence in Florida counties (Hahn 

et al., 2011). Another study examined the relationship between OAA Title III state 

expenditures and the prevalence of low-care NH residents in those states (Thomas & 

Mor, 2013 manuscript). They found that increased spending on Title III was associated 

with fewer NH residents who had low-care needs, suggesting that increased state 

spending on this lower-intensity service could decrease low-care NH resident 

prevalence.

Policy Variables: Reduction in Resources

Regarding policy resources, D’Souza et al. (2009) found that reductions in resources for 

Medicaid home care were associated with increased probability of adverse outcomes 

including hospitalization, ER use, and NF placement.

Policy Variables: Home Health Agencies

Miller (1998) found that for married individuals, a greater number of home health 

agencies was associated with a longer time to NH admission. This may be related to 

easier access to services. On a similar note, one study found that having a primary care 

provider is associated with reduced odds of nursing home admission (Goodwin et al., 

2011).

Policy Variables: Nursing Home Bed Supply

Miller and Weissert (2000) report that many studies found a positive relationship 

between nursing home bed supply and risk of nursing home placement. However, 

Miller (1998) found that this variable was not significant relating to NH entry when 

examining NH beds per 1,000 persons 85+. Borrayo et al. (2002) examined predictors of 

NH, ALF, or HCBS use in Florida. They found that individuals who lived in north, 

central, and west Florida were more likely to be living in a NF than in an ALF or using 

HCBS. Those in south Florida were more likely to use HCBS than live in an ALF or NH 

setting. This is interesting to note because of the increased NH bed supply in those 

regions outside of south Florida and the increased HCBS program access in south Florida.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 56

Policy Variables: Pre-admission screenings

Chapin et al. (2009) assessed the CARE program and found that, of the numbers of 

individuals diverted from the NH, there was almost an equal amount of older adults who 

died or who were placed in nursing homes, while 18% of the older adults were still living 

in the community. They suggest that providing HCBS information during a preadmission screening could potentially permanently prevent NH admission.

Need Variables

Awareness of Need/Self-health Ratings

Older adults displaying characteristics of need is connected to HCBS use, whether it be 

awareness of need, ADL or IADL limitations, or poorer health overall. Perceived unmet 

need is important to both discretionary and nondiscretionary service use among families 

providing care to dependent older adults (Kosloski & Montgomery, 1994; Mitchell & 

Krout, 1998). Chen and Thompson found that awareness of need was associated with 

HCBS use (2010). Casado, van Vulpen, and Davis (2010) report that a lack of awareness, 

reluctance, and unavailability of HCBS were some of the main reasons for unmet HCBS

needs.

Self-health ratings are associated with an increased risk of NH admission (Cai, Salmon, & 

Rodgers, 2009; Muramatsu et al., 2007). Muramatsu et al. (2007) found that worse selfrated health status was significantly associated with increased risk for NH admissions. A 

lower self-perceived health was associated with an increased risk of long-term stay 

nursing home admission (Cai, Salmon, & Rodgers, 2009). 

General Poor Health

Poorer health and a greater number of illnesses were strong predictors of CCRC onsite 

services for respondents in Krout et al.’s 2002 study. In one study, greater overall care 

needs were associated with unmet HCBS needs (Casado, van Vulpen, & Davis, 2010). 

Among a study of African Americans in an urban congregate meal program, Weddle et 

al. (2012) found that nutrition risk was a significant influence on use of the service. 

Alkema et al. (2006) found that older adults with incontinence were three times more 

likely to use any service. Those who had sensory impairments had over five times 

greater odds of utilizing HCBS.

Variables that affected hospitalization and other adverse outcomes include worse 

performance on physical function measures not based on ADL, greater illness severity, and 

prior hospital use (Miller & Weissert, 2000). 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 57

Banaszak-Holl et al. (2004) found that models that excluded functional impairment 

showed that chronic medical conditions and dementia significantly affected the risk of 

institutionalization. Even in those models that controlled for functional impairment, 

dementia still had a strong effect, but chronic medical conditions did not. Friedman et 

al. (2005) found that IADL dependence and bowel incontinence were associated with a 

higher risk of nursing home admission or PACE participants who joined PACE from 

the community. Borrayo et al. (2002) reported that older adults with more chronic 

diseases were more likely to be in a NF or using HCBS than in an ALF. Andel, Hyer, and 

Slack (2007) report that, among older adults without dementia, diabetes, incontinence, and 

stroke also contribute to an increased risk of NH admission.

Overall, Miller and Weissert (2000) highlight that worse performance on any measure of 

ADLs, IADLs, incontinence and disability was correlated with an increased risk of NH 

placement and hospitalization. Lower physical functioning is significantly associated with 

increased risk of NH admissions (Muramatsu et al. 2007).

Interestingly, among older adults without dementia in Florida, Andel, Hyer, and Slack 

(2007) found that arthritis and heart disease reduced the risk of NH admission.

ADL Limitations

The strongest associations between need and service use is physical function. Research 

has found that ADL limitations and chronic disabilities are two of the strongest needbased predictors of HCBS use (Borrayo et al., 2002; Wolinsky & Johnson, 1991). 

According to Johnson & Wolinsky (1996), ADLs are significant predictors of using inhome health services among white females, but not black females.

For individuals who returned to the community from a recent hospitalization and had 

unmet ADL needs, their risks of readmission were increased. It was greater for those 

with unmet needs for new disabilities in comparison to those who were admitted with an 

existing disability (DePalma et al., 2012).

In a meta-analysis of studies on NH admission in the U.S., Gaugler et al. (2007) found 

that one of the strongest predictors of NH admission was having 3 or more ADL 

dependencies. Similarly, D’Souza et al. (2009) found that, when the ADL Index was used 

as a proxy for health status, functional impairment was positively associated with death 

and permanent NH placement. Borrayo et al. (2002) found that individuals needing ADL 

assistance were more likely to be in a NF than in an ALF or using HCBS.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 58

IADL Limitations

Some studies have found that IADL limitations are associated with both HCBS use and 

NH admissions (Chen & Thompson, 2010; Chapin et al., 2002).

Many studies have found that IADL limitations are associated with HCBS use and NH 

admissions (Andel, Hyer, & Slack, 2007; Chen & Thompson, 2010; Chapin et al., 2002). 

Cai, Salmon, and Rodgers (2009) found that worse IADLs were associated with increased 

NH admissions; however, it is important to note that higher ADLs did not show a 

significant effect on NH admission. In a 12-year epidemiological study in the U.S., IADL 

disability was found to be a significant predictor of institutionalization (Bharucha et al., 

2004).

Comorbidities

Sands et al. (2008) measured morbidities using a commonly used measure, the Charlson 

index (based on ICD-9 codes in 6 months prior to study period, in which a higher rating 

reflects greater number and severity of comorbid conditions). HCBS waiver users were 

more likely to have one or more comorbidities than NH patients.

Haller and Gessert (2007) found that having 3+ comorbidities was associated with high 

medical care utilization at end of life for NH residents with a cognitive impairment 

hospitalized within 90 days of death. 

Alzheimer’s Disease & Cognitive Impairment

Variables relating to need are strongly associated with NH admission, including 

cognitive and physical function. Many studies have found that Alzheimer’s disease and 

cognitive impairment are strongly linked to NH admissions (Bharucha et al., 2004; 

Borrayo et al., 2002; Chen & Thompson, 2010; Gaugler et al., 2007; Miller & Weissert, 

2002; Wolinsky & Johnson, 1991). Borrayo, Salmon, Polivka, and Dunlop (2002) reported 

that individuals with Alzheimer’s diagnoses or a high cognitive impairment level were 

more likely to be in an institution (nursing facility or assisted living facility) than to use 

HCBS. Between the NF or ALF, they are also more likely to be in an NF. Gaugler et al. 

(2009) found that, for persons with dementia, some predictors of NH entry include: 

severity of cognitive impairment, Alzheimer’s disease diagnosis, basic ADL 

dependencies, behavioral symptoms, and depression. 

Mental Health Status

Psychiatric issues (Cai, Salmon, & Rodgers, 2009) and being disoriented/having 

problematic behavior (Miller & Weissert, 2000) was associated with an increased 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 59

institutionalization risk. Andrews et al. (2009) found that people with schizophrenia 

enter nursing homes earlier than persons with no mental illness; however, the median 

age of NH admission for those with schizophrenia is 65. This disparity was higher for 

those at middle age (40-65).

Service Use, Mix and Spending: HCBS, Health Care, and Nursing Home Outcomes

General Health Care Experiences

Wallace et al. (1999) found that previous health care experiences predicted service use.

Miller and Weissert (2000) found that more frequent doctor visits are correlated with 

increased risk of hospital admission. Additionally, taking more medications is 

associated with hospitalization risk.

Prior Hospital Use

As mentioned previously, prior hospital use is a significant predictor of hospitalization. 

The number of hospital days an individual has is also significantly related to an 

individual’s impairment (Miller & Weissert, 2000).

Prior hospitalization is a significant predictor of nursing home admission (Miller and 

Weissert, 2000). Gaugler et al. (2007) found that this was associated with slightly greater 

odds of NH entrance. Hospital use and emergency room use has been linked to 

increased use of HCBS, increased health care utilization rates, and NH admission 

(D’Souza, 2009, Sands et al., 2008, Xu et al., 2010). One study found that NH patients, 

compared to HCBS users, were more likely to have been admitted to a hospital in the 6 

months before receiving LTC (Sands et al., 2008). Goodwin et al. (2011) found that 3⁄4 of 

new NH placements were precipitated by hospitalization. They state that being 

discharged from a hospital into a SNF was associated with increased odds of LTC 

admission.

Prior Nursing Home Use

Prior NH use was found to be a strong predictor of NH admission (Gaugler et al. 2007) 

and positively associated with future NH use (Miller & Weissert, 2000). PACE 

participants who enrolled from a NH vs. from the community were at a higher risk of 

having a 30 day NH admission (Friedman et al., 2005).
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 60

Prior Service Use

The effect of HCBS use on NH admission has resulted in mixed findings. In some cases, 

more HCBS services are associated with a greater risk of NH admission (Miller & 

Weissert, 2000; Karuza & Wu, 2011; NYSOA, 2011). Other studies indicate that HCBS use 

can delay NH admission (Brock et al., 2007; Karuza & Wu, 2011; Kosloski & Montgomery, 

1995; Chen & Thompson, 2010; Chen et al, 2011; Gaugler et al., 2005; OPPAGA, 2010; 

Shapiro & Taylor, 2002). One study looked at the use of an HMO and found that it 

appeared to help at-risk older adults postpone NH placement (Fischer et al. 2003). 

Shapiro & Taylor (2002) conducted a randomized controlled trial where individuals on 

the waiting list for Florida’s Community Care for the Elderly program who were at 

“moderate risk” received services earlier than they would have. They found that the 

intervention group had higher subjective well-being and were less likely to enter an 

institution or die during the 18-month period than the comparison group. The types of 

services these studies researched varied and included the effect of services like paid 

helpers, case management, OAA III-E/respite care, and State Medicaid Waiver. The 

studies also examined the effect of service delivery including volume of services and 

timing of care.

Volume of Services

Receiving a greater volume of attendant care, homemaking services, and home-delivered 

meals was associated with a lower risk of hospitalization (Xu et al., 2010).

Additionally, volume of services has an effect on institutionalization. In Rhode Island, 

individuals who received more than one service had increased community tenure (Brock 

et al., 2007; Karuza & Wu, 2011).

Timing of Care

Regarding service delivery, Gaugler et al. (2005) found that timing of care affected

institutionalization. Caregivers of individuals with dementia who used in-home help 

services earlier in their caregiving experiences were more likely to delay 

institutionalization (Gaugler et al., 2005). In MacMillan et al.’s (2007) study period, at 

three months after assessment, 31.4% of diverted individuals used services (the highest 

proportion in the study). This further supports that timing of care is an important 

variable to consider in NH diversion research.

IADL Help

Miller and Weissert (2000) analyzed multiple studies that examined adverse outcomes for 

older adults. They found that having paid helpers was positively associated with NH 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 61

use. However, Chen and Adams-Thompson (2010) looked at various types of service use 

on adults’ ability to remain in the community, including paid ADL/IADL personal care 

services and other HCBS. They found that the only service that supported older adults to 

remain in the community was paid IADL personal care services. The authors suggested 

that the studies should look at how IADL personal care services and other HCBS help 

older adults remain in the communities longer before their first NH visit or return home 

after institutional care. This would inform efficiency of service provision.

Supporting these findings, IADL limitations emerged in the New York POMP study as a 

significant factor predicting NH risks (New York Office for the Aging, 2010). Similarly, 

the Advanced POMP (Karuza & Wu, 2011) found that limitations in IADLs were more 

significantly associated with increased risk for NH placement than commonly recognized 

limitations in ADLs. This finding extended the knowledge base and suggested the 

importance of considering IADL limitations in the study of and delivery of OAA HCBS.

Case Management

A research recommendation emerging from the Advanced POMP (Karuza & Wu, 2011) 

was to explore the question: Is there an optimal strategy for program implementation and 

service delivery (role of case manager)? Shapiro & Loh (2009) found that cost savings for 

HCBS resulted from reduction in use of NHs. They posited that case management, 

which was used by all programs evaluated, helped to target the specific services to the 

needs of the client and increase the efficiency of HCBS programs. The author suggested 

that HCBS results in cost savings because the use of case management that targets specific 

services is tailored to the needs of the client, which is associated with a shift in client 

services from expensive care service (e.g., NH) towards less expensive services (e.g., 

prescriptions). In Rhode Island, utilization of case management services was associated 

with increased likelihood of NH admissions, but the authors suggested that this may be 

because use of case management services is an indicator of higher need (Brock, et al. 

2007).

The New York POMP project (NYSOA, 2010) found that intensity of case management 

services was a statistically significant predictor for NH placement in two counties, 

controlling for other factors. To explain this association, a case study analysis indicates 

that users of case management were already very frail and in the highest need of services, 

likely explaining their need for NH services (NYSOA, 2011). Additionally, local county 

policies and procedures affected NH placement. In one county (for which case 

management was more significant), case managers assist clients with NH placement and 

they have a cap on personal care services. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 62

OAA III-E / Respite Care

Georgia provided respite (OAA III-E) data in an analysis of their POMP data. This 

indicated that the relationship between service intensity and mean survival was flat for 

respite, but it was a significant predictor of decreased NH placement (Brock et al., 2007; 

Karuza & Wu, 2011). Kosloski and Montgomery (1995) found that there is a negative 

relationship between the amount of respite received and NH placement. 

Medicaid Waiver

Pande, Laditka, Laditka, and Davis (2007) examined the frailty of older adults enrolled in 

South Carolina’s Medicaid home care waiver program in 1995 and 2005. They found that 

older adults in 2005 were significantly more likely to have chronic conditions in 

comparison to those in 1995. Additionally, they were more likely to receive specific 

services in 2005. The authors suggest that this indicates older adults are remaining in the 

community longer and that Medicaid Waiver services may delay institutionalization.

Service Use, Mix, and Spending: Cost Savings Predictors

Cost savings is an important outcome to measure when examining the effects of HCBS on 

health care and NH use. A recent analysis of the available evidence for AARP (Mollica et 

al., 2009) concluded that “HCBS is cost-effective.” The report found that “over time, 

states that invest in HCBS experience slower Medicaid expenditure growth than states 

with low HCBS spending.” For example, Kaye, LaPlante, and Harrington (2009) 

analyzed state spending data over 10 years and found evidence that states with 

established HCBS systems controlled spending better than did states with lower levels of 

HCBS and less established HCBS systems. Several recent state-specific studies have also 

found evidence of reduced total utilization and/or cost savings from HCBS (Senecal, 

2009; Weissert et al., 1997; GAO, 1994; Alecxih, Lutzky, & Corea, 1996; Shapiro & Loh, 

2010; Shapiro et al., 2009).

One study looked at the effects of OAA III-C services. At the state-level, for every 

additional dollar spent per person over 60 in the non-institutionalized population, there is 

a decrease of approximately 6.6% in the rate of change in nursing home residency status 

for adults over 65. When overall state HCBS spending is accounted for, that effect 

disappears (Buys et al., 2011 manuscript).

Some studies examined state-specific services. Felix et al. (2011) found a net 3-year 

savings of $2.619 million for Medicaid in Arkansas as a result of their Community 

Connector Program. This program trained community health workers to identify 

individuals in disadvantaged communities who are at-risk of NH placement and connect 

them with services.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 63

Shapiro and Loh (2010) tracked services from five categories in Florida (Alzheimer’s 

Disease Initiative, Community Care for the Elderly, Home Care for the Elderly, the 

Medicaid Waiver, OAA Title III B)—those that were larger in scale and had enough 

observations for a meaningful evaluation of cost avoidance. The limitation is that this is 

a nonrandom selection of HCBS programs in Florida. Cost avoidance varied widely 

among the five HCBS programs that were analyzed. The study gave the cost avoidance 

in dollar amounts but did not look at pathways. For all service types, the greatest cost 

avoidances were for NH expenses; impacts on other health care expenditures were 

relatively small. In some cases, the HCBS programs were associated with cost increases 

in some expense categories, including prescription drug expenses, and transportation.

► OAA III-B: Based on equation by equation OLS, Older Americans Act III-B enrollees 

spent $65 less on NH care, $20 more on long-term home health services, $55 more on 

prescription drugs, and $69 more on other expenses than nonusers monthly. Results 

for inpatient expenses, outpatient expenses, physician expenses, and transportation 

expenses were not statistically significant (Shapiro & Loh, 2010).

► Alzheimer's Disease Initiative (ADI): Results of the 3 stage-least-squares equation 

showed cost avoidance associated with ADI (per member per month) was $781 less 

for NH expenses, $569 more for prescription drug expenses and $102 more for 

transportation expenses. Results for total inpatient expenses, total outpatient 

expenses, total physician expenses, total long-term home health, and total other 

expenses were not statistically significant. (Besides ADI, the study did not report 

3SLS results for any other program because of the high correlation with the mental 

health assessment variable) (Shapiro & Loh, 2010). In general, the results did not 

seem sensitive to the multiple enrollments in programs. (Shapiro & Loh, 2010)

► Community Care for the Elderly (CCE): Based on OLS equation by equation, CCE 

service users spent $12 less on inpatient care, $4 less on outpatient care, $2 less on 

physician expenses, $175 less on NH care, $3 less on transportation, $9 less on 

prescription drug, and $38 less on other expenses than nonusers monthly. Results

for total long-term home health expenses were not statistically significant (Shapiro & 

Loh, 2010).

► Home Care for the Elderly (HCE): Based on OLS equation by equation, HCE users 

spent $3 more on physician expenses, $152 less on NH care, $6 less on long-term 

home health services, $3 more on transportation services, and $65 more on 

prescription drugs than nonusers monthly. Results for total inpatient expenses, total 

outpatient expenses, and total other expenses were not statistically significant

(Shapiro & Loh, 2010).

► Medicaid Wavier (MW): Based on OLS equation by equation, MW enrollees spent 

$9 more on outpatient care, $117 less on NH care, $8 less on long-term home health 

services, $12 more on transportation services, $63 more on prescription drug, and 

$364 more on other expenses than nonusers monthly. Results for total inpatient 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 64

expenses and total physician expenses were not statistically significant (Shapiro & 

Loh, 2010). 

Shapiro & Loh’s 2010 study found consistent evidence that HCBS use in Florida 

produced cost savings compared with applicant non-users and non-user/non-applicants. 

In 2006, NH residents who received HCBS saved $373.94/month more than NH residents 

who did not apply for or receive services. This was a $4,487 average annual savings. The 

largest and most consistent benefit of enrollment in HCBS was for Medicaid NH 

expenses. 

Shapiro & Loh (2010) found that several HCBS programs, including CCE and Older 

Americans Act III-B, III-C and III-E, produced the most consistent and robust 

Medicaid cost savings relative to those in the non-user/non-applicant group. Isolating 

the effects of individual programs was difficult, because a substantial proportion of the 

study population was enrolled in multiple programs.

Several recent studies have emphasized the effectiveness of lower intensity HCBS. In a 

study by Smith and Frick (2008), high intensity HCBS use generally did not produce 

more Medicaid cost savings than moderate intensity HCBS use. Shapiro and Loh (2010) 

and Shapiro et al. (2009) found that low intensity HCBS users had the largest long-term 

care cost savings. They recommended that a re-examination of client service mixes 

within and between programs may be important to determine if some services may be 

redundant and/or less predictive of NH avoidance. Another approach would be to find 

ways to minimize the general revenue expenditures associated with higher intensity 

HCBS service mix.

Future Research Considerations

Several recent studies have recommended further research on the effects of national, 

state, and agency level policies and practices on HCBS outcomes, to help inform LTSS 

systems planning, such as eligibility and targeting policies. For example, a 

recommendation based on the POMP research (Brock et al., 2007) was to expand control 

variables in HCBS outcomes models to include service agencies' policies and structure. 

While some of these variables are outside of the scope of this study design, they are 

important to keep in mind.

Table 3 below displays variables that are especially important to consider in future 

work.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 65

Table 3 : Variables Recommended for Future Exploration

Variables Recommended for Future Exploration HCBS Use Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Federal level

National LTSS targeting policies    

National LTSS eligibility policies    

Broader Environmental Factors (e.g., 

medical care environment over time)  

State level

Cost control policies  

Distribution of HCBS resources across 

state 

State target policies    

State HCBS eligibility policies    

Financing models  

NH use rates 

Strength of NH industry 

Strength of home care labor unions 

County/Local Level 

Variance in available housing options 

Distribution of HCBS resources across 

county 

Agency Level

Agency Characteristics (policies, 

structure, etc.) 

Agency home care policies/procedures 

Individual older adult/caregiver level

Service monitoring/coordinating to meet 

persons’ needs 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 66

Variables Recommended for Future Exploration HCBS Use Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Perceptions of functional ability 

Awareness of unmet needs 

Characteristics of Older Persons not 

using OAA services    

Health/needs of Caregiver 

Quality of Life improvements 

Demographic variables 

Recent studies suggest that the following policy issues are especially important to 

consider.

Case-Control Evaluations

Karuza & Wu (2011) recommended several questions for future research, including, 

How do OAA clients compare to those who do not receive OAA services? The 

Advanced POMP studies did not include cases and controls from the general 

population of older adults. Similarly, Brock et al. (2007) recommended further 

comparisons with general populations of older persons to explore over a broader 

population the effectiveness of OAA services.

Intensity

The Kansas CARE study of participants in an Aging Network nursing home diversion 

program, (Macmillan et al., 2007) diverted customers with greater level of need (higher 

LTC threshold scores) and those customers were able to stay in the community longer. 

This was an unexpected result and more research is needed on why this may be the 

case. The LTC threshold score was measured at the beginning of the study and not over 

time. Diverted CARE participants generally received OAA and other HCBS shortly 

after diversion, but were able to stay in the community without receiving services 

continuously.

Distribution of HCBS Resources across the Aging Network

Miller et al. (1998) recommended that future studies pay attention to the issue of how 

the allocation and distribution of home care resources within a county (or a state) may 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 67

influence NH risk. Similarly, a recommendation for future research emerging from the 

POMP (Karuza & Wu, 2011) was to explore: Is there an optimal strategy for program 

implementation and service delivery (distribution of limited service resources across 

Aging Network)?

Miller et al. (1998) also recommended that future studies pay attention to the issue of 

how the allocation and distribution of home care resources within a state (or a county) 

may influence NH risk.

NH Diversion/Pre-Admission Screening

Miller et al. (1998) did not study the independent effect of preadmission screening on

time to NH admission, but suggested that this variable be included in future, related 

studies. Chapin et al. (2009), reporting on the Kansas Community Tenure Study, 

recommended: “As more studies on diversion of older adults who are applying for NH 

admission are completed, it will be possible to present a clearer picture of the capacity 

for older adults with significant long-term care needs to successfully remain in the 

community. In addition, research focused on the utilization of community-based 

services will provide policymakers a better understanding of diverted older adults' 

ability to maintain community tenure given timely access to publicly funded home- and 

community-based services.”

Cross-Program Effects and Outcomes on Multiple Domains

The literature suggests that, to provide a complete picture of HCBS impacts, evaluators 

must consider cross-program effects and outcomes on a range of domains.

D’Souza et al. (2009) found that efforts to decrease Medicaid HCBS spending appeared 

to have triggered downstream increases in permanent NH placement and greater 

utilization of acute care services, typically funded by Medicare. They recommended 

that policymakers give more consideration to the overall cross-program effects of state 

HCBS budget reductions on access to preferred care settings and health outcomes.

Shapiro, Loh, & Mitchell (2011) recommended that a full estimation of the value of these 

programs must also consider the value of quality of life improvements that can be 

attributed to these services. When combined with estimates of actual cost avoidance, 

these studies would provide a more comprehensive assessment of the value of HCBS 

programs.

Miller et al. (1998) recommended that future outcomes studies focus not only on NH 

entry, but also include other outcome such as death, entering into assisted living 

facilities, hospitalization, Medicaid spend-down, and changes in quality of life, etc., in 

order to demonstrate the impact of HCBS in general, and of Aging Network home care 

in particular.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 68

Measure Limitations

Recent studies suggest that HCBS use is affected by individual traits and circumstances 

that are difficult to measure. Some commonly used measures may not always be 

accurate.

Some evidence suggests that self-reported ADLs and IADLs often overestimate a 

person’s functional ability when compared to a performance-based test. Chen & Adams 

Thompson (2010) suggested that older adults’ perceptions of their functional ability 

may be a more important indicator of community tenure than their actual physical 

abilities because they related to their awareness of their unmet needs, which is tied to 

service use.

Similarly, Chen & Adams Thompson (2010) noted that awareness of unmet need and 

ability/desire to seek assistance are related to cultures and values that are not often 

measured but understood to drive use of services.

One study of Aging Network CARES assessment participants in Kansas found that 

living alone, residing in assisted living, and having available support did not explain 

the difference in those who remained in the community versus admitted to NHs 

(Macmillan et al., 2007). However, qualitative analysis showed that most customers 

who lived alone had daily informal contact and support from family. This suggested 

that living alone cannot be used as a measure of social isolation, particularly if informal 

support is not being accounted for. These results illustrate the complexities of 

measuring caregiver availability.

These measurement limitations are further discussed in the Study Design report.

Conclusion

AHRQ released a systematic review of the comparative effectiveness of institutional 

versus home and community-based services (Wysocki et al., 2012). The review focused 

on two questions:

1. What are the benefits and harms of long-term care provided through Home

and Community-based Services (HCBS) compared to institutions such as 

nursing homes (NH) for older adults aged 60 and over who need long-term 

care (LTC)?

2. What are the costs (at the societal and personal level) of HCBS and NH (per 

recipient and in the aggregate) for adults over age 60? (pg. ES-4).

The review protocol (AHRQ, 2011) acknowledged that meaningful comparisons must 

include services of roughly equivalent type, frequency, and intensity. In addition, the 

protocol noted that assessing LTC cost impacts requires attention to expenses associated 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 69

with several factors, including housing, specific services delivered, health care 

utilization (including acute care services such as hospitals, emergency departments), 

and costs incurred by recipients and their families. In conducting the review, the 

authors noted that there is very limited evidence, in addition to methodological 

limitations, of literature relating to the outcomes of interest. They suggest that “more 

and better research is needed to draw robust conclusions about how setting influences 

outcomes and costs of older adults using LTC” (Wysocki et al., 2012, pg. ES-14).

This review supports the need for designing a study that produces credible, solid 

evidence of results that will be of critical importance to AoA, the Aging Network, and 

service recipients and families who rely on OAA and non-OAA programs and services. 

Increasingly, stakeholders seeking continued or expanded funding for health and social 

programs must be able to “make the business case” for the investment, with rigorous 

research demonstrating success in improving peoples‘ lives and achieving more 

effective use of economic resources. Congress and the Office of Management and 

Budget (OMB) at the federal level, as well as state and local governments, use program 

evaluation information to make budget decisions. A study with inconclusive results, or 

misleadingly suggesting a lack of impact or negative impact, can be quite damaging to 

efforts to demonstrate the need and gain support for HCBS programs.

Appendix A-A: Available Data Sources and Data Elements

Numerous secondary data sources exist and several of the studies combined data from 

multiple sources. For example, Miller et al. (1998) linked individual level data from a 

national longitudinal survey with state health care system data via the state of residence 

of each person. In Shapiro & Loh (2010), MDS records were matched with CIRTS 

records, using resident IDs to obtain Social Security Numbers (SSNs) from CMS, then 

matching CIRTS records with the SSNs.

For this project, the POMP may provide the most appropriate initial data source to 

consider. However, we are encountering numerous data inconsistencies across states. 

Lewin continues to explore the POMP data with AoA as many improvements in data 

integrity have been made in the last three years. The POMP data could be matched to 

other individual data if appropriate identifiers are available and it could also be 

matched to geographic policy or supply/demand related data.

Older Americans Act Program Data

Available data sources could potentially be used to examine OAA services at the 

individual level or the state or AAA level. Table 1 below provides examples of recent 

studies using OAA program data.

Table 1: Data Sets with OAA/Aging Network Specific Data
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 70

Database Data elements Study

Florida CIRTS 

Database, a 

comprehensive 

assessment database 

of community-dwelling 

candidates for LTC 

services in Florida.

Florida CARES

Demographic and health variables from 

CIRTS screening assessment (marital 

status, race, sex, IADL, ADL, mental health 

assessment). A full assessment takes place 

every 3 months.

Although 15 programs were in the CIRTS 

file (including OAA III-B, III-C, III-D, and IIIE), only five programs that were larger in 

scale had enough observations for a 

meaningful evaluation: Older Americans Act 

III-C, Alzheimer’s Disease Initiative, 

Community Care for the Elderly, Home 

Care for the Elderly, Medicaid Waiver

Shapiro & Loh, 2010, 

2007

Shapiro, Loh, & 

Mitchell, 2011

Administrative data 

from the four New York 

State AAAs 

participating in POMP

The data included in this study was 

longitudinal data collected separately from 

each county 1/1/08 - 6/30/09. The data 

elements for the study included: older 

adults’ age, gender, ethnicity, marital 

status, income, living arrangement, ADL 

and IADL scores. Family support data were 

not included because a significant number 

of cases did not report this information. 

Health status variables, such as selfreports of incontinence, stroke, Parkinson’s 

disease, Alzheimer’s disease, and mental 

status/dementia, and health events such as 

hospitalization and ER use, were also 

included. Service use variables were also 

included.

New York State 

Office for the Aging, 

2010

AAA/Kansas 

Department on Aging 

databases 

Kansas State Publicly 

Funded Services 

(SPFS) data

The Kansas Department on Aging provided 

CARE Assessment, Senior Care Act, 

Medicaid Home and Community-Based 

Services for the Frail Elderly, and Older 

Americans Act service data which were 

extracted from their KAMIS and MMIS data 

systems

Chapin et al., 2002, 

2003, 2009

Macmillan et al., 

2007

Administrative data on 

OAA participants from 

Georgia, New York, 

North Carolina, and 

Rhode Island

Data varied by state. Georgia provided data 

on service clients from selected AAAs in the 

state. NY data consisted of all consumers 

of OAA services age 60 and older in four 

counties who received at least one service 

during a 1-year period. NC data included all 

OAA consumers age 60+ from two counties 

who received at least one registered Home 

Karuza & Wu, 2011
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 71

Database Data elements Study

and Community Block Grant Service for at 

least three months. Rhode Island data 

consisted of two data sets of OAA 

participants; the second dataset was similar 

to the first, but with some differences and 

more recent.

Administrative service 

data from Rhode 

Island, Georgia, North 

Carolina, and New 

York

Data available for analysis varied by state:

Receipt of various OAA HCBS

Demographics (age, gender, ethnicity, 

living arrangements)

Presence or absence of a caregiver 

(Georgia only)

Medicaid eligibility (NC only)

Marital status (Rhode Island only)

Measures of physical functioning (ADL, 

IADL)

Health and mental status variables: selfreports of incontinence, stroke, 

Parkinson’s disease, Alzheimer’s 

disease, and dementia (NY only)

Other health event variables, such as 

emergency room use (NY only)

Brock et al., 2007

Administration on 

Aging, AGID database

AGID is an online database created by AoA 

that provides information on clients served, 

providers of services, and expenditures for 

particular programs. Additionally, AGID 

includes population characteristics from the 

Census Bureau.

Buys et al. (draft 

manuscript currently 

under review)

Administration on 

Aging, 1995, 1996, 

2004

Administration on 

Aging Survey 1995 

State spending on Older Americans Act

State HCBS expenditures in 1992, including 

expenditures for HCBS for the functionally 

impaired elderly supported by Medicaid, 

Older Americans Act, Social Services Block 

Grant, state source programs, and other 

funding sources

Muramatsu et al., 

2007

Muramatsu et al., 

2002
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 72

State Data Sources

State datasets provide information on applicants for non-OAA services in a state, such 

as Medicaid waiver services. However, states are inconsistent in the data elements 

collected. Thus, these data may not be easily comparable across states (Table 2).

Table 2: Other State Specific Data Sets

Data Source Data Element Study

Indiana Medicaid data 

(Indiana Medicaid 

enrollment and claims and 

Insite databases)

Monthly data on people receiving 

HCBS vs. care in NHs, including:

Gender, Age, Race, Alzheimer’s 

status, Comorbidities, Medicare 

status, Medicaid status, Marital 

status, Geographic region within 

state, Total Medicaid expenditures 

(includes inpatient, outpatient, 

pharmacy, long-term care, and 

other expenditures), Inpatient and 

ER use

Sands et al. 2008

Xu et al., 2010

Data Archive maintained 

by University of Michigan 

Institute of Gerontology 

Data Archive 

Includes longitudinal assessment 

information for all individuals 

enrolled in the Michigan Choice 

waiver gathered using a 

comprehensive assessment 

instrument (Minimum Data Set for 

Home Care (MDS-HC ©, Morris et 

al., 1997). Assessments generally 

required at 90-day intervals. 

The data archive also contains 

administrative records. (Self-report 

data by patient/caregiver)

D’Souza et al., 2009

A services database 

provided by the Florida 

Department of Elder Affairs 

(DOEA) that can be used 

to track all individuals who 

received state-funded 

HCBS

Monthly charges for inpatient, 

outpatient, and physician services 

from the Medicaid claims data.

Shapiro & Loh, 2010

Shapiro, Loh, & Mitchell, 

2011

Kansas Department of 

Health and Environment

Verification of death of study 

participants Macmillan et al., 2007

Arkansas Medicaid Variables included: age, sex, race, 

Medicaid eligibility category, dually 

Felix et al., 2011
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 73

Data Source Data Element Study

eligibility and claims data eligible for Medicare, enrolled in 

Medicaid in year prior to CCP, 

years enrolled in Medicaid, 

enrolled in HCBS waiver for older 

adults or adults with physical 

disabilities, comorbidity index 

score, prior-year Medicaid 

spending. Also Medicaid spending 

by service type.

National Data Sets and Other Sources

The in-depth reviewed studies provide many examples of national datasets and other 

sources of data on HCBS policies, including state and regional characteristics. 

Table 3: National Data Sets and Other Sources

Data Source Data Elements Example HCBS 

Studies

CMS Minimum Data 

Set II (MDS) 

Data on all NH residents

MDS records were matched with CIRTS 

records using resident IDs to obtain 

Social Security Numbers (SSNs) from 

CMS, then matching CIRTS records 

with the SSNs. 

Permanent nursing home admission 

information

Shapiro & Loh, 2010

Chapin et al., 2002

Chapin et al., 2003

Macmillan et al., 2007

Buys et al., manuscript 

under review

MDS-HC

Identifies the number of times over the 

prior 90 days (or since the last 

assessment) that an individual has 

been admitted to hospital with an 

overnight stay or visited the ER without 

an overnight stay

D’Souza et al., 2009

OASIS Medicare home health assessment and 

service use data Chapin et al., 2003

CMS Nursing Home 

Data Compendium Per capita NH bed supply Miller, 2011

CMS Office of 

Research, 

Demonstrations and 

Annual data related to the number of 

certified home health agencies Miller, 2011
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 74

Data Source Data Elements Example HCBS 

Studies

Information (email 

communication with 

William D. Saunders, 

CMS, June 17, 2009).

Thomson Reuters

Form 64 data as compiled by Thomson 

Reuters contains Medicaid LTC 

expenditures data; share of LTC dollars 

devoted to HCBS for all states and DC, 

except for Arizona. Because Arizona 

operates its LTC program under a 1115 

demonstration waiver, HCBS-specific 

expenditure data was not available. 

Thomson Reuters also provides an 

annual report on “Medicaid LTC 

expenditures in FY 2007” (Burwell, 

Sredl, & Eiken, 2008).

Miller, 2011

Buys et al., manuscript 

under review

National Health and 

Nutrition Examination 

Survey (NHANES); the 

National Longitudinal 

Study of Aging

The authors suggested that the use of 

large datasets, such as these two 

surveys, might be particularly wellsuited to evaluate more 

comprehensively the effects of Title III 

Nutrition Services on NH entry as well 

as other relevant health outcomes.

Buys et al. and AoA plan to use both 

NHANES data linked with Medicare 

claims data, for individual-level 

comparisons between participants in 

OAA Title III Nutrition Services and nonparticipants on health outcomes and NH 

entry.

Buys et al., manuscript 

under review

Medstat (Burwell, 

2001, 2003)

State Medicaid expenditures for home 

health, personal care, nursing home, 

and HCBS waiver (aged or disabled) 

Muramatsu et al., 2007

HHS, 1999, 2000-

2004, 2008

State spending on Social Services 

Block Grant

Muramatsu et al.,2007

Buys et al, manuscript 

under review

Kassner & Williams, 

1997; Summer & Ihara, 

2004

State spending on state source 

programs

Muramatsu et al., 2007
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 75

Data Source Data Elements Example HCBS 

Studies

Census Bureau data 

sources, including 

annual Statistical 

Abstracts and the 

American Community 

Survey

State-level data on the share of the 

state population who are Black or 

Hispanic, Data on per capita income, 

Female labor force participation, State 

poverty rate, Housing affordability 

(household housing costs >=30% of 

income, %),Total state population

Miller, 2011

CDC Behavioral Risk 

Factor Surveillance 

Survey

Includes measures of disease 

prevalence and health status, state 

prevalence of activity limitations 

because of a physical, mental or 

emotional problem, percent of 

population reporting good or fair health

Miller, 2011

CDC annual HIV/AIDS 

Surveillance Reports State-level AIDS prevalence Miller, 2011

National Climatic Data 

Center, National 

Oceanic and 

Atmospheric 

Administration, U.S. 

Department of 

Commerce (1992)

Mean heating degree days Miller et al., 1998

Institute for Health and 

Aging, University of 

California at San 

Francisco 

Number of licensed residential care 

beds per 1,000 persons 65+ 

(Harrington, DuNah, & Bedney, 1993)

Medicaid NH per diem (Swan, Keo de 

Wit, Pickard, & Clark, 1993)

Miller et al., 1998

Health Resources 

Publishing (1992)

Adult day centers per 100,000 persons 

65+ Miller et al., 1998

Health and Retirement 

Study (HRS), a 

nationally 

representative 

longitudinal survey of 

older persons 

conducted by the 

University of Michigan 

Institute for Social 

Research 

Variables such as physical functioning, 

receipt of services (home care by a 

medical professional and additional 

service, e.g., adult day care, social 

worker, outpatient rehabilitation, 

transportation, or meals), NH 

placement, and demographics

Muramatsu et al., 2007

Muramatsu et al., 2008

Brock et al., 2007
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 76

Data Source Data Elements Example HCBS 

Studies

Assets and Health 

Dynamics among the 

Oldest Old (AHEAD) 

survey, conducted from 

Oct 1993-July 1994

The survey examined the impacts and 

interrelationships of changes and 

transitions for older Americans in three 

domains: health, financial, and family.

Muramatsu et al., 2002

Second Longitudinal 

Study of Aging (LSOA 

II), a collaborative effort 

of the National Center 

for Health Statistics 

and the National 

Institute on Aging

Nationally representative longitudinal 

study of adults age 70+. Chen & Thompson, 2010

Consortium to 

Establish a Registry for 

Alzheimer's Disease 

(CERAD) data

The CERAD project enrolled individuals 

between April 1987 and Jan 1995. Miller et al., 1998

AARP 

Annual report on “State-funded home 

and community-based services 

programs for older adults” (Mollica, 

Simms-Kaselein, Kassner, 2009), which 

provides data on state governments’ 

spending on HCBS.

Buys et al., manuscript 

under review

Other Potential Information Sources

In the absence of consistent national data collection on state LTSS policies, recent 

surveys of state Medicaid agencies and aging and disability agencies may be an 

additional source for this data. In a February, 2012 AARP Public Policy Institute report, 

a survey of state aging and disability agencies (SADAs) and Medicaid agencies from 48 

states and DC regarding LTSS policy trends reported (Cheek et al., 2012):

► Implementing or planning to implement Medicaid Managed LTSS

► Focusing on improving integration of Medicare and Medicaid services for dual 

eligibles

► Cuts made to aging and disability services programs (non-Medicaid) in FY 2011 or 

expecting to cut these programs in 2012

► Provider rate cuts

► Restrictions to some Medicaid services, such as personal care services
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 77

► Level of demand for ADRC services, information and referrals, and respite care in 

FY 2011

► Numbers of HCBS participants and Medicaid nursing facility residents in FY 2010 

and 2011, and expected trends for 2012

► Changes in staffing of state officials

► Plans for using the new health care reform law to expand HCBS

An earlier study (Kitchener, Terence, & Harrington, 2004) reported on a survey of state 

Medicaid waiver programs, that asked about state use of eligibility criteria, caps, and 

waiting lists in various Medicaid waiver programs. The survey asked about all waiver 

programs regarding cost containment strategies used in 2002. Results indicated that, of 

the 76% of waiver programs responding, 57% used some type of financial cap. A third 

(33%) used more restrictive financial eligibility criteria than for institutional services, 

and the vast majority of states limited the number of waiver slots available. Overall, the 

waiver programs reported that 157,640 persons were on waiting lists in 2002. A 

limitation of the study was that no response was received for 58 waivers, and some 

states may have underreported or not responded due to fear of negative public 

reactions or legal action by advocacy organizations. Thus, the study recommended 

policy initiatives to require the systematic recording and public reporting of cost control 

techniques (e.g., waitlists) in use in public programs. The authors recommended that 

future studies examine the implications of states’ use of waiver waiting lists and other 

cost controls. They recommended that this line of enquiry examine, for example, the 

scope, quality, and cost of services received by people enrolled in waiver programs 

compared with persons held on waiting lists.

Appendix A-B: Detailed Key Variable Table Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Predisposing

Age 9 1 12* 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Shapiro & 

Loh, 2009; 

Xu et al., 

2010

Miller, 1998 Karuza & 

Wu, 2011; NYSOA, 

2010; Friedman et al., 

2005; Goodwin et al., 

2011; Andel Hyer, & 

 

7 An asterisk (*) indicates that this variable has yielded mixed findings in the literature. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 78 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Shapiro & Loh, 

2007; Felix et al., 

2011; Alkema, 

Reyes, & Wilber, 

2006; Liu, 

McBride, & 

Coughlin, 1990; 

Mitchell & Krout, 

1998; Xu et al., 

2010

Slack, 2007; Bharucha 

et al., 2004; Cai, 

Salmon, & Rodgers, 

2009; Chen and 

Thompson, 2010; 

Miller & Weissert, 

2000; Muramatsu et 

al., 2007; Muramatsu 

et al., 2008

Education 0 2 4 0 Alkema, 

Reyes, & 

Wilber, 

2006; Li, 

2006

Muramatsu et al., 

2007; Muramatsu et 

al., 2008; Miller, 1998; 

Chen & Thompson, 

2010

Gender 9 2 13 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Shapiro & 

Loh, 2010; 

Shaprio & Loh, 

2007; Xu et al., 

2010; Sands, 

2008; Felix et al., 

2011; Alkema, 

Reyes, & Wilber, 

2005; Laditka, 

Laditka, & Drake, 

2006

Xu et al., 

2010; 

Miller & 

Weissert, 

2000

Muramatsu et al., 

2008; Muramatsu et 

al., 2007; NYSOA, 

2010; D’Souza, 2009; 

Karuza & Wu, 2011; 

Chen & Thompson, 

2010; Macmillan et 

al., 2007; Miller, 1998; 

Sands et al., 2008; 

Goodwin et al., 2004; 

Cai, Salmon, and 

Rodgers, 2009; Miller 

& Weissert, 2000; 

Ness, Ahmed, and 

Aronow, 2004; 

Gauger et al., 2007

Race/Ethnicity 13 2 14 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Shapiro & 

Loh, 2010; 

Shapiro & Loh, 

2007; Xu et al., 

2010; Felix et al., 

2011; Mitchell 

and Krout, 1998; 

Wallace et al., 

Haller & 

Gessert, 

2007; Xu 

et al., 

2010

Brock, 2007; NYSOA, 

2010; Miller, 2011; 

Sands et al, 2008; 

Muramatsu et al., 

2008; Karuza & Wu, 

2011; Stevens et al., 

2004; Muramatsu et 

al., 2007

Ness, Ahmed, & 

Aronow, 2004; Andel, 

Hyer, & Slack 2007; 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 79 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

1999; Goins, 

Tincher, & 

Spencer, 2003; 

Sands et al., 

2008; Liu, 

McBridge, & 

Coughlin, 1994; 

Laditka, Laditka, 

& Davis, 2006; 

Casado, van 

Vulpen, & Davis, 

2011

Cai, Salmon, & 

Rodgers, 2009; 

Gaugler et al., 2007; 

Miller & Weissert, 

2000; Feng et al., 

2011

Geographic 

Location

5 1 3 0 Muramatsu & 

Campbell, 2002; 

McAuley, 

Spector, Van 

Nostrand, & 

Shaffer, 2004; 

Mitchell & Krout, 

1998; Sun, 2011; 

Li, 2006.

Haller 

and 

Gesset, 

2007

Coward, Home, & 

Peek, 1995; 

Macmillan, 2007; 

Chapin et al., 2003

Living Alone 4 1 8* 0 Weddle, Wilson, 

Berkshire, & 

Heuberger, 2010; 

Xu et al., 2010; 

Alkema et al., 

2006; Muramatsu 

& Campbell, 2002

Xu et al., 

2010

Muramatsu et al., 

2008; NYSOA, 2010; 

Karuza & Wu, 2011; 

Macmillan, 2007; 

Chapin et al., 2003; 

Cai, Salmon, and 

Rodgers, 2009; 

Gaugler et al., 2007; 

Miller & Weissert, 

2000

Marital Status 5* 1 8* 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Xu et al., 

2010; D’Souza, 

2009; Wallace et 

al., 1999

Xu et al., 

2010

Miller, 1998; 

Muramatsu, 2007; 

Karuza & Wu, 2011; 

Gaugler et al., 2007; 

Miller & Weissert, 

2000; Andel, Hyer, & 

Slack, 2007; Borrayo 

et al., 2002; Ness, 

Ahmed, and Aronow, 

2004

Informal 3 2 9* 0 Xu et al., 2010; Van Chen & Thompson, 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 80 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Caregiver 

Availability

Chen & 

Thompson, 2010; 

Mitchell & Krout, 

1998

Houtven 

& Norton, 

2004; Xu 

et al., 

2010

2010; Chapin et al., 

2003; Karuza & Wu, 

2011; Miller & 

Weissert, 2000; 

Gaugler et al., 2007; 

Van Houtven & 

Norton, 2004; 

Friedman et al., 2005; 

Lo Sasso & Johnson, 

2002; Andel, Hyer & 

Slack 2007

Having children 1 0 3 0 Muramatsu & 

Campbell, 2002

Muramatsu et al, 

2007; Muramatsu et 

al., 2008; Gaugler et 

al., 2007

Social Support 0 0 1 0 Bharucha et al., 2004

Enabling

Income 6 1 6 0 Muramatsu & 

Campbell, 2002; 

Shapiro & Loh, 

2007; Chen & 

Thompson, 2010; 

Casado, van 

Vulpen, & Davis, 

2010; Wolinsky & 

Johnson, 1991; 

Borrayo et al., 

2004

Miller & 

Weissert, 

2000

Muramatsu et al., 

2008; Chen & 

Thompson, 2010; 

Macmillan et al., 

2007; Muramatsu et 

al., 2007; Karuza & 

Wu, 2011; Miller & 

Weissert 2000

Home ownership 0 0 3 0 Muramatsu et al., 

2007; Cai, Salmon, & 

Rodgers, 2009; 

Gaugler et al., 2007

Payment method 2 0 6* 0 Muramatsu & 

Campbell 2002; 

Chen & 

Thompson; 2010

Muramatsu et al., 

2008; Chen & 

Thompson, 2010; 

Macmillan et al., 

2007; Sands, 2008; 

Ness, Ahmed, & 

Aronow, 2004; 

Borrayo, Salmon, 

Polivka, & Dunlop, 

2002
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 81 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Caregiver or 

Spouse age

0 0 2 0 Muramatsu et al., 

2007; Yaffe et al., 

2002

Caregiver support 1 0 1 0 Casado, van 

Vulpen, & Davis, 

2010

Miller & Weissert, 

2000

Caregiver burden 0 0 4 0 Gaugler et al., 2009; 

Miller, Rosenheck, 

Schnieder, 2012; 

Spillman and Long, 

2009; Yaffe et al., 

2002

NH bed supply 0 0 6 0 Miller, 1998; 

Muramatsu et al., 

2008; Muramatsu et 

al., 2007; Miller, 2011; 

Miller & Weissert 

2000; Borrayo et al., 

2002

Number HHAs 0 0 3 0 Miller 1998; 

Muramatsu et al., 

2008; Miller, 2011

Medicaid HCBS 

spending

0 1 1 0 D’Souza 

et al., 

2009

D’Souza et al., 2009

Miller 1998

State HCBS 

spending

2 1 2 0 Muramatsu & 

Campbell, 2002; 

Muramatsu et al., 

2008

D’Souza 

et al., 

2009

Muramatsu et al., 

2008; D’Souza, 2009; 

Hahn et al., 2011; 

Thomas & Mor, 2013

Pre-admission 

screening

0 0 3 0 Chapin et al., 2003; 

Macmillan et al., 

2007; Miller, 1998

Need

Alzheimer’s 

Disease/Cognitive 

impairment

2 1 11 0 Xu et al., 2010; 

D’Souza; 2009

Xu, 2010 Miller, 1998; NYSOA, 

2010; D’Souza, 2009; 

Muramatsu et al., 

2008; Muramatsu et 

al., 2007; Bharucha et 

al., 2004; Borrayo et 

al., 2002; Chen and

Thompson, 2010; 

Gaugler et al., 2007; 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 82 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Miller and Weissert, 

2002; Wolinsky & 

Johnson, 1991

Comorbidities / 

Chronic Medical 

Conditions

3* 2 4 0 Xu et al., 2010; 

Sands 2008; Felix 

et al., 2011

Xu et al., 

2010; 

Haller & 

Gessert

Banaszak-Holl et al., 

2004; Borrayo et al., 

2002; Muramatsu, 

2007; Sands, 2008

Psychiatric issues 

/ Problem 

behavior

0 0 3 0 Cai, Salmon & 

Rodgers, 2009; Miller 

& Weissert, 2000; 

Andrews et al., 2009

Worse ADL/IADL 

performance in 

general

0 0 2 0 Miller & Weissert, 

2000; Muramatsu et 

al., 2007

ADL limitations 10 0 10 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Shapiro & 

Loh, 2010; 

Shapiro & Loh, 

2007; Xu et al., 

2010; Chen & 

Thompson, 2010; 

D’Souza et al., 

2009; Johnson & 

Wolinsky, 1995; 

Borrayo et al., 

2002; Wolinsky & 

Johnson, 1991

Chen et al., 2010; 

D’Souza, 2009; 

Muramatsu et al., 

2007; Muramatsu et 

al, 2008; NYSOA, 

2010; Karuza & Wu, 

2011; Miller 1998; 

Chapin et al.; 2002; 

Borrayo et al., 2002; 

Gaugler et al., 2007

IADL limitations 5 0 7 0 Muramatsu & 

Campbell, 2002; 

Shapiro et al., 

2009; Shaprio & 

Loh, 2007; Chapin 

et al. 2002; Chen 

& Thompson 

2010

Miller, 1998; Karuza, 

2011; NYSOA, 2010; 

Andel, Hyer, & Slack, 

2007; Chapin et al., 

2002; Chen & 

Thompson, 2010; Cai, 

Salmon, & Rodgers, 

2009

Self-health ratings 0 0 2 0 Muramatsu et al., 

2007; Cai, Salmon, & 

Rodgers
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 83 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Perceived unmet 

need

4 1 1 0 Kosloski & 

Montgomery, 

1994; Mitchell & 

Krout, 1998; Chen 

& Thompson, 

2010; Casado, 

van Vulpen, & 

Davis, 2010

DePalma 

et al., 

2012

Karuza & Wu, 2011

Use

Prior 

Hospitalization

2 1 6 0 D’Souza 2009; 

Sands et al, 2008

Xu et al., 

2010

Sands et al., 2008; 

Goodwin et al., 2001; 

D’Souza et al., 2009; 

Xu et al., 2010; 

Gaugler et al., 2007; 

Miller & Weissert; 

2000

Prior NH Use 0 0 4 1 Muramatsu et al. 

2007; Friedman et al., 

2005; Miller & 

Weissert; 2000; 

Gaugler et al., 2007; 

Shapiro & Taylor, 

2002; OPPAGA, 2010

Shapiro 

& Loh 

2010

HCBS Use 1 1 12 4 Chen & 

Thompson, 2010

Xu et al., 

2010

Chen & Thompson, 

2010; Macmillan et 

al., 2007; Karuza, 

2011; NYSOA, 2010; 

D’Souza, 2009; Brock 

et al. 2007; Gaugler et 

al., 2005; Kosloski et 

al., 1995; Chen & 

Thompson, 2010; 

Chen et al., 2004; 

Miller & Weissert 

2000; Fischer et al., 

2003

Shapiro 

et al 

2009; 

Shapiro 

& Loh, 

2010; 

Shapiro 

& Loh, 

2007; 

Felix et 

al., 

2011

Paid Helpers 0 0 2 0 Miller & Weissert, 

2010; Chen & 

Thompson, 2010

Case 

Management

0 0 2* Karuza 2011; NYSOA, 

2010

Medicaid waiver 0 0 1 2 Pande, Laditka, Shapiro 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 84 Variables with Significant Relationships to Outcomes of Interest

7

HCBS UseHealth Care UseCommunity Tenure/NH useCost ImpactsHCBS UseHealth Care UseCommunity Tenure/NH useCost Impacts

Laditka, & Davis, 2007 et al., 

2009; 

Shapiro 

& Loh, 

2010

Timing of Care 0 0 2 0 Macmillan et al. 2007; 

Gaugler et al. 2005

Volume of 

services

0 1 2 1 Xu et al., 

2010

Brock et al., 2007; 

Karuza & Wu 2011

Shapiro 

& Loh, 

2010

Low-intensity 

HCBS

0 0 0 0

1

Shapiro 

& Loh, 

2010

Combination of 

services

0 0 3 1 Karuza & Wu, 2011; 

Brock et al, 2007; 

Chapin et al, 2003

Shapiro 

et al, 

2009

OAA III-B 0 0 0 2 Shapiro 

et al, 

2009; 

Shapiro 

& Loh, 

2010

OAA III-C 1 0 1 2 Xu et al., 2010 NYSOA, 2010 Shapiro 

et al., 

2009

Buys et 

al. 2011

OAA III-E/Respite 0 0 3 1 Brock, 2007; Karuza,

2011

Kosloski & 

Montgomery, 1995

Shapiro 

& Loh, 

2009

Appendix A-C: Detailed Bibliography

HCBS Studies Selected for In-Depth Review, Data Extraction, and Analysis

The 19 HCBS studies selected for in-depth review are summarized below. These studies 

were selected for more in-depth review, data extraction, and analysis because they: (1) 

specifically examined OAA services (POMP and Kansas Community Tenure 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 85

Study/CARE related studies), and/or (2) provided useful information to inform the 

study design. 

N0. Study Reason for Including /Summary

1

Brock, D.B., Rabinovich, B., 

Severynse, J., Ficke, R.USDHHS, 

Administration on Aging. (2007). 

Risk factors for nursing home 

placement among OAA service 

recipients: Summary analysis of 

data from five sources. (233-02-

0087)

The Performance Outcomes Measurement Project 

(POMP) in Rhode Island, Georgia, North Carolina, 

and New York 

The authors use data from the Performance 

Outcomes Measurement Project (POMP) in Rhode 

Island, Georgia, North Carolina, and New York to 

examine the effect of the receipt of OAA services on 

the potential delay in nursing home (NH) placement 

among OAA service clients age 60 and older. The 

OAA services across the states were similar, but 

there was still some variation state-to-state. They 

also analyzed the national Health and Retirement 

Study. 

The authors used a time-to-event analysis (time to 

NH placement) using the Cox Proportional Hazard 

Model. 

Among their findings, Rhode Island POMP data 

analysis showed that risk of NH admission 

decreased for clients receiving more than one 

service, as compared with only one service. 

Utilization of case management services was 

associated with increased likelihood of NH 

admissions, but the authors suggested that this may 

be because use of case management services is an 

indicator of higher need.

A recommendation based on the results of POMP 

studies was to include personal and institutional 

health resources and characteristics, as well asarea 

characteristics in future research.

2

Chapin, R., Baca, B., Macmillan, 

K., Rachlin, R., & Zimmerman, M. 

(2009). Residential Outcomes for 

Nursing Facility Applicants Who 

Have Been Diverted: Where Are 

They 5 Years Later? The 

Gerontologist, 49(1): 46-56.

CARE Study

The authors use data from the Client Assessment 

and Referral Evaluation (CARE) program. This 

assessment is completed when individuals seek 

nursing facility (NF) admission (both Medicaid and 

private pay). In addition to CARE program data, the 

authors used data from publicly-funded community 

based services (Medicare, Medicaid, and state), NF 

admission data, and death records.

This was a five-year prospective study – they 

tracked those who applied for NF admission and 

those who were diverted. They then tracked the 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 86

N0. Study Reason for Including /Summary

diverted individuals over a period of five years. 

Those diverted received services (many types, 

including OAA-funded services) during the five-year 

period. The use of these services was intermittent.

A notable finding from this study is that many of the 

individuals diverted from NF were diverted until 

death, not just until NF admission. 

Additionally, the characteristics of those diverted 

versus those not diverted were very similar; 

suggesting that more could be done to divert 

individuals from NFs.

A limitation of this study is that they only tracked 

individuals over a period of 60 months. The authors 

recommend future studies track diverted individuals 

over the remaining life course.

3

Chapin, R., Zimmerman, M., 

Macmillan, K., Rachlin, R., 

Nakashima, M., Hayes, J., Oslund, 

P., Swaim, T., Burke, J., Shea, J., 

Reed, C. (2002) Longitudinal study 

of customers diverted through the 

care program: Technical 

report. School of Social Welfare 

Office of Aging and Long Term 

Care., University of Kansas , 

Lawrence, Kansas.

CARE Study

This study reviewed whether Medicaid-HCBS/FE 

Waiver Program services contributed more to 

community tenure than other services. The Kansas 

Department on Aging (KDOA) was interested in 

learning how long diverted older adults remain in the 

community (after the CARE assessment) and how 

effective State Publicly Funded Services (SPFS) are 

in helping older adults increase community tenure 

and avoid NF admission.

The Kansas Department on Aging provided CARE 

Assessment, Senior Care Act, and Medicaid Home 

and Community Based Services for the Frail Elderly 

and Older Americans Act service data. Data were 

extracted from the KAMIS and MMIS data systems.

The authors conducted a survival analysis along 

with a matched comparison group. They also 

analyzed qualitative data from interviews with the 

customers or their personal caregivers.

The authors found that the use of Medicaid 

HCBS/FE Waiver Program services, State General 

Fund (SGF), and/or OAA before a NF admission 

appeared to impact the length of stay – many of the 

short-term residents (64.9%) received SPFS 

compared to 42.9% of permanent Medicaid NF 

residents.

4

Chapin, R., Zimmerman, M., 

Chapin, R., Zimmerman, M., 

CARE Study

The authors presented their analysis of community
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 87

N0. Study Reason for Including /Summary

Macmillan, K., Rachlin, R., Reed, 

C., Hickey, A., Baca, B., WieboldLippisch, L., Henning, E., Oslund, 

P., Hayes, J., Katz, B., & Shea, J. 

(2003). Examination of the Use of 

Medicare Home Health Services

and Informal Caregiving and Their 

Relationship to Successful 

Community Tenure and 

Appendices. School of Social 

Welfare Office of Aging and Long 

Term Care. University of Kansas, 

Lawrence, Kansas.

tenure, risk factors for a NF admission, and cost 

benefits of SPFS compared to NF admission. 

This builds on the 2002 study and utilizes the same 

methodology. The analysis of this study for 2003 

included Medicare Home Health Service use data 

(OASIS) and an additional six months of SPFS, 

Minimum Data Set (MDS), and death data. The 

authors also conducted interviews with older adults, 

and their caregivers, who received the CARE 

assessment.

The authors found that SPFS are cost effective and 

diverted customers continued to have high rates of 

community tenure. This data confirms that State 

Publicly Funded In-Home Services, Medicare Home 

Health Services, and Informal Support contribute to 

diversion and community tenure.

Regarding OAA services, they found that the annual 

savings for diverted CARE customers who received 

OAA services was $441,524.16.

To determine factors that contribute to community 

tenure, the authors used the Cox Proportional 

Hazards Model Analysis and Logistic Regression 

Analysis. 

5

Chen, Y., & Adams Thompson, E. 

(2010). Understanding factors that 

influence success of home- and 

community-based services in 

keeping older adults in community 

settings. Journal of Aging and 

Health. 22(3): 267-291.

The authors researched factors that influence 

success of HCBS in increasing community tenure.

This study addressed three dimensions—personal 

factors, services use, and remaining in the 

community--in temporal order. The study 

constructed a theoretical framework to predict the 

likelihood of older adults residing in the community. 

They tested the framework using a complex 

structural equation model and nationally 

representative sample from a longitudinal survey 

(The Second Longitudinal Study of Aging) with data 

for six years. 

The study examined many types of non-OAA HCBS 

(discretionary services, nondiscretionary services, 

paid ADL and IADL help, unpaid ADL and IADL 

help) and different frequencies, and amounts, of 

these services.

The results revealed the importance of paid IADL 

personal care services, and of elders’ awareness of 

their unmet needs. In the complex model that was 

tested, these two factors stood out as the only 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 88

N0. Study Reason for Including /Summary

significant factors that supported older adults 

remaining in the community.

6

D'Souza, J.C., James, M.L., 

Szafara, K.L. & Fries, B.E. (2009). 

Hard Times: The Effects of 

Financial Strain on Home Care 

Services Use and Participant 

Outcomes in Michigan. The 

Gerontologist. 49(2): 154-165. 

The authors investigated the effect of program 

resources on individuals’ enrollment in the MI HCBS 

waiver program for elderly and disabled adults. 

These services include personal care, homemaker, 

meals, adult day care, private duty nursing, skilled 

therapies, volunteer, and total formal service time.

The data source for this study was a data archive 

maintained by the University of Michigan Institute of 

Gerontology – this includes participant data 

collected as a part of the Minimum Data Set for 

Home Care.

Using dates of major policy and budget changes, the 

study defined four distinct time periods over four 

years. Data for over 100,000 participants were used 

to examine temporal trends in informal care hours 

and the six outcomes (emergency room use, 

hospitalization, caregiver burden, death, NF use, 

and permanent NF placement), controlling for 

demographics, functional status, and cognitive 

status. The study obtained adjusted odds of 

outcomes using discrete-time survival analysis.

The study investigated the effect of HCBS funding 

on participant outcomes. 

They found that as resources diminished, mean 

formal care hours decreased, and three adverse 

outcomes increased: hospitalization, ER use, and 

NF placement.

7

Felix, H.C., Mays, G.P., Stewart, 

N.C., & Olson, M. (2011).Medicaid 

savings resulted when community 

health workers matched those with 

needs to home and community 

care. Health Affairs. 30(7): 1366-

1374.

The Arkansas Community Connector Program was 

the first US initiative to test a mechanism of 

identifying priority populations with unmet LTC 

needs, to help them gain access to HCBS. The 

study measured outcomes on growth in use of 

Medicaid HCBS and growth in overall Medicaid 

spending in program participants versus comparison 

group members

Using a longitudinal, quasi-experimental design, the 

study compared 919 Medicaid recipients served by 

the Community Connector program for three years, 

in three intervention counties, with a statistically 

matched group of 944 Medicaid recipients in five 

nearby counties.

This study examined participation in the Arkansas 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 89

N0. Study Reason for Including /Summary

Community Connector Program, which used 

community health workers specifically for the 

purpose of providing targeted outreach to residents 

in need of HCBS, who may be at risk of entering 

NHs, and connecting them to HCBS.

Results suggested that Medicaid realized a 

substantial return of nearly three dollars on each 

dollar invested in the program. The program also 

increased spending on HCBS and decreased 

spending on NHs. 

The findings suggest that it can pay to actively seek 

out people with unmet LTC needs and help them 

gain access to formal services.

8

Karuza, J. & I-Hsin, W. 

Administration on Aging. 

(2011). Cross validation of nursing 

home placement prediction 

models: Findings from the 

advanced performance outcome 

measurement project.

New York POMP Study

The Advanced POMP initiative’s objective is to 

examine the effects of HCBS provided by the aging 

network on the risk of nursing home placement for 

older adults.

The authors conducted a time-to-event analysis

(time to NH placement) using Cox Proportional 

Hazards Regression Models applied to client data 

from the participating states. They examined 

administrative data on OAA participants from 

Georgia, New York, North Carolina, and Rhode 

Island

The outcomes measured were: (1) remaining in the 

community; (2) permanent nursing home placement; 

(3) mortality; (4) loss to follow-up; and, (5) the end of 

the study period. 

Results showed that IADLs were more significantly 

associated with increased risk for NH placement

than commonly recognized limitations in ADLs.

The findings suggest that OAA funded services, 

when administered in combination, are more 

effective than single OAA services.

9

Macmillan, K.R., Rachlin, R., 

Chapin, R., Chandran, D., Leedahl, 

S., Baca, B., Zimmerman, M., & 

Oslund, P. (2007). The Community 

Tenure Study: Community Tenure 

Status of CARE Assessment 

Customers 60 Months after 

Diversion. School of Social Welfare 

CARE Study

This study updates the status of participants in 

Chapin et al’s studies in 2002 and 2003. The 

outcomes measured were: (1) The community 

tenure status of diverted customers in 3-month 

intervals; (2) The state publicly funded service 

utilization of diverted customers in 3-month intervals; 

and, (3) The demographic and other differences 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 90

N0. Study Reason for Including /Summary

Office of Aging and Long Term 

Care. University of Kansas. 

Lawrence, Kansas. 

between diverted customers who maintained 

community tenure and those who became 

permanent nursing facility residents.

Macmillan et al. (2007) used several data sources to 

track diverted customers, including Kansas CARE 

Assessment, Senior Care Act, Medicaid HCBS/FE, 

and OAA service data (from the KAMIS and MMIS 

data systems). They also used Kansas Department 

of Health and Environment data to verify death and 

CMS MDS data to identify permanent NF admission 

information.

The services their study examined included SPFS, 

Medicare Home Health Services, and Informal 

Support.

Results of the care study indicate that being a 

customer identified as potentially lower income, at 

the time of the study, was associated with higher 

rates of NH admissions than those not qualifying for 

Medicaid.

The Kansas CARE study of participants in an Aging 

Network Nursing Home Diversion Program reported 

that diverted customers with greater level of need 

(higher LTC threshold scores) were able to stay in 

the community longer

10

Miller, S.C., Prohaska, T.R., Furner 

S.E., Freels S., Brody J.A., & Levy 

P.S. (1998). Time to nursing home 

admission for persons with 

Alzheimer’s disease: the effect of 

health care system characteristics. 

Journal of Gerontology Series B. 

53(6):S341-53.

This study analyzed the effects of health care 

system characteristics on the time to NH admission 

for persons with Alzheimer’s disease.

Up to nine years of data from 639 individuals (the 

Consortium to Establish a Registry for Alzheimer’s 

Disease (CERAD)) was merged with longitudinal 

data from 28 states, where CERAD individuals 

resided. The study used a Cox Proportional 

Hazards Model to examine risk factors associated 

with time to NH admission, taking into account 

characteristics of each state’s LTC system.

Services examined included the percent of state 

spending on HCBS and LTC, along with other 

enabling variables. This included availability of NH 

beds and availability of HCBS. They also examined 

marital status and education.

A higher percentage of state spending on HCBS 

was associated with a longer time to NH admission 

for unmarried persons with Alzheimer’s disease in 

the first three years. The authors also found that, for 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 91

N0. Study Reason for Including /Summary

married persons, a higher number of home health 

aides per 100,000 persons age 65 and over resulted 

in a longer time to NH admission.

11

Miller, N.A. (2011). Relations 

Among Home-and CommunityBased Services Investment and 

Nursing Home Rates of Use for 

Working-Age and Older Adults: A 

State-Level Analysis. American 

Journal of Public Health, 101(9): 

1735-1741.

This study examined relations among state provision 

of Medicaid-funded HCBS and NH rates of use for 

working-age and older-adults. The study examined 

associations between state socio-demographic, 

economic, supply, and programmatic characteristics 

and rates of use.

The author combined many existing data sources 

and used multivariate fixed effects models to 

examine state-level rates of NH use over a sevenyear period (2000-2007). Data were from all states 

and DC, except Arizona. 

Miller examined states’ relative investment in HCBS 

(the share of LTC dollars devoted to HCBS). LTC 

expenditures were determined by combining NF, 

home health, personal care, and 1915(c) 

expenditures for older and working age adults, 

individuals with traumatic brain injury, and 

individuals with HIV/AIDS.

The study found that greater investment in HCBS, 

coupled with reduced NH capacity, was associated 

with reduced rates of NH care for adults age 65 and 

older, but not for working-age adults. Their use was 

associated with state socio-demographic 

characteristics, as well as chronic disease 

prevalence.

12

Muramatsu, N. & Campbell, R.T. 

(2002). State Expenditures on 

home and community based 

services and use of formal and 

informal personal assistance: a 

multilevel analysis. Journal of 

Health and Social Behavior. 43(1): 

107-124.

The authors examined the use of formal HCBS and 

informal personal assistance among adults age 70 

and older, living in the community, with functional 

limitations.

The sample included respondents to a nationally 

representative survey of older adults (N=3,051) with 

at least one ADL or IADL limitation. They combined 

data from this survey (the Assets and Health 

Dynamics among the Oldest Old Survey) with state 

HCBS expenditure data. The study conducted a 

two-level multinomial logistic regression analysis of 

data on the survey population nested in 34 states. 

This approach explored links between macro and 

micro levels of social phenomena.

The authors found that, controlling for demographic, 

socioeconomic, and care needs factors, states with 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 92

N0. Study Reason for Including /Summary

higher HCBS expenditures had individuals who were 

more likely to use formal personal assistance, but 

still not less likely to use informal personal 

assistance. The probability of using any formal 

assistance increases when an individual has more 

ADL limitations and lives in a state with higher levels 

of HCBS spending. 

This was the first known study indicateing that the 

association between ADL limitations and use of 

personal assistance depends upon the level of 

HCBS expenditures of the state where individuals 

reside. The study suggested that, to the extent that 

supportive formal services can reduce the stress on 

the caregiver, state spending on HCBS may 

strengthen the existing system of informal 

caregiving.

13

Muramatsu N., Hoyem R.L., Yin H, 

& Campbell R.T. (2008). Place of 

death among older Americans: 

does state spending on home- and 

community based services 

promote home death? Med Care. 

46(8):829- 38. 

This study examined the impacts of state spending 

on HCBS and on dying at home versus dying in a 

NH among older Americans.

The sample for this study included 3,320 

respondents to the Health and Retirement Study, 

born 1923 or earlier, and who died between 1993 

and 2002 (age 70 and older). 

Discrete time survival analysis of risk of end-of-life 

NH relocation was performed to examine the 

association with states’ HCBS spending.

The authors examined total HCBS expenditures for 

the 65 and older population, and the percentage of 

LTC expenditures going to HCBS rather than NHs.

The study found that living in a state with higher 

HCBS spending was associated with lower risk of 

end-of-life NH relocation. This was found especially 

among individuals on Medicaid. State support for 

HCBS increased the chance of dying at home by 

lowering the risk of end-of-life NH relocation.

14

Muramatsu N., Yin, H., Campbell 

R.T., Hoyem, R.L., Jacob, M.A., & 

Ross, C.O. (2007). Risk of nursing 

home admission among older 

Americans: Does states’ spending 

on home- and community-based 

services matter? Journal of 

Gerontology Series B. 62(3):169-

78.

The authors examined the effect of state HCBS 

spending on NH admission among older Americans.

They performed a discrete time survival analysis of 

first LTC admissions (90 days or more) over seven 

years, using data from the Health and Retirement 

Study.

The authors examined total HCBS expenditures for 

thsoe 65 and older, and the percent of LTC 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 93

N0. Study Reason for Including /Summary

expenditures going to HCBS rather than NHs.

Living in a state with higher HCBS expenditures was 

associated with lower risk of NH admission among 

childless seniors, but was not found to be 

statistically significant among seniors with living 

children. Lack of information on existing targeting 

policies and practices prevented the study from 

exploring whether the lack of impact of HCBS 

among childless seniors reflected targeting of 

seniors who lack family, or the ineffectiveness of 

state HCBS support among that population.

15

New York State Office for the 

Aging (2010). Performance 

outcomes measures project 

(POMP): From home care to 

nursing home in New York: 

demonstrating the benefits of aging 

network services. Albany. 

New York POMP

The authors calculated the outcome variable 

“survival time” for each individual consumer as the 

difference (in months) between the date of the event 

(nursing home placement) or censoring (by death, 

movement out of the study area, end of the study 

period, or other loss to follow-up) and the date the 

consumer entered the aging network service 

system.

The authors used administrative data from January 

2008 - June 2009 from the four New York AAAs 

participating in POMP. They examined whether the 

receipt of OAA Cluster 1 services affected NH 

placement. They also used the data to identify risk 

factors that contributed to NH entry. The authors 

used the Cox Proportional Hazards Model as the 

method for survival analysis to study an individual’s 

survival time to a particular outcome.

They examined OAA Cluster 1 Services, which 

include: personal care services; homemaker/chores 

services; home-delivered meals; adult day care 

services; and, case management services.

The authors stated that the modeling of “time to 

nursing home placement‟ shows that aging network 

consumers’ relative nursing home risk can be 

reduced by total services and personal care. They 

found that intensity of case management services in 

two counties was a statistically significant predictor 

for NH placement, controlling for other factors.

16

Shapiro, A. & Loh, C.P. (2007). 

Establishing Algorithms for the 

Cost-Benefit Analysis of In-Home 

Services for Elderly in Florida.

Florida POMP

The purpose of this study was to examine the 

estimated cost savings of Medicaid HCBS in Florida. 

The authors studied five HCBS programs in Florida 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 94

N0. Study Reason for Including /Summary

(Florida Department of Elder 

Affairs and Administration on Aging 

No. XQ648). University of North 

Florida: Report for Florida 

Department of Elder Affairs. 

which were: 1) Alzheimer’s Disease Initiative (ADI); 

2) Community Care for the Elderly (CCE); 3) Home 

Care for the Elderly (HCE); 4) Medicaid Waiver 

(MW); and, 5) Older Americans Act 3b (OA3B).

The authors utilized Medicaid claim data (SFY 1999-

2005) to use a propensity score matching 

procedure, which simulated random assignment of 

Florida Department of Elder Affairs (FDOEA) clients 

into matched treatment (HCBS users) and 

comparison (waitlist) groups. They produced an 

algorithm that determined differences in Medicaid 

expenditures between the HCBS and waitlist 

groups, which produced an estimated cost savings. 

They also performed telephone interviews with over 

500 FDOEA clients to employ a “Willingness to Pay” 

procedure.

Their results suggest evidence of Medicaid cost 

avoidance through the use of HCBS programs in 

Florida. In the five HCBS programs analyzed, the 

range was a cost overage of $885/month and a cost 

savings of $1558/month.

They also found from their contingent valuation 

survey that older adults’ median estimates of the 

hedonic value of their quality of life is, on average, 

$900/month across all service programs.

17

Shapiro, A., & Loh, C.P. (2010). 

Advanced performance outcome 

measures project (POMP): 

Estimates of Medicaid and general 

revenue cost-avoidance from 

HCBS utilization (Contract No. 

XQ867)

Florida POMP

The authors examined the cost effectiveness of 

HCBS programs for Medicaid and General Revenue 

in Florida. They examined the same services from 

their 2007/2009 studies, which include: the 

Alzheimer’s Disease Initiative (ADI); Community 

Care for the Elderly (CCE); Home Care for the 

Elderly (HCE); Medicaid Waiver (MW); and, Older 

Americans Act 3b (OA3B).

They used data from their 2009 project to estimate 

models of cost-effectiveness simultaneously to 

reduce the error of measurement. They then 

estimated Medicaid and General Revenue costsavings of HCBS programs through the use of a 

sample of individuals who would be placed into NHs.

The researchers found evidence that HCBS 

utilization produces cost savings when compared 

with costs from those on waitlists for services and 

non-applicants. The largest and most consistent 

HCBS enrollment benefit was found for Medicaid NH 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 95

N0. Study Reason for Including /Summary

expenses. These NH cost savings ranged from 

$1000-$1400/month compared to the MDS group 

(depending on service use intensity). The low 

intensity HCBS users utilized the fewest general 

revenue expenses and generated the greatest 

overall cost savings, in comparison to the non-users 

and non-applicants.

18

Shapiro, A., Loh, C.P. & Mitchell, 

G. (2009). Medicaid Cost-Savings 

of Home-and Community-Based 

Service Programs for Older 

Persons in Florida. Journal of 

Applied Gerontology, 30(1): 3-21.

This study examined potential Medicaid cost savings 

that result from HCBS use by older Floridians. They 

examined the same services from their 2007study, 

which include: the Alzheimer’s Disease Initiative 

(ADI); Community Care for the Elderly (CCE); Home 

Care for the Elderly (HCE); Medicaid Waiver (MW); 

and, Older Americans Act 3b (OA3B).

The authors used Medicaid claim data (SFY 2000-

2005) to use a propensity score matching procedure 

to simulate random assignments of seniors into a 

matched treatment group (HCBS users), and a 

comparison (waitlist) group. They developed an 

algorithm that determined differences between the 

two groups in Medicaid expenditures – this 

produced an estimate of cost savings.

The authors found that some Florida HCBS 

programs showed evidence of Medicaid costsavings. The median savings varied, ranging from a 

cost overage of $277 per member per month to a 

cost-savings of $229 per member per month. This 

was especially apparent when examining nursing 

home costs.

19

Xu H., Weiner M., Paul S., Thomas 

J., Craig, B., Roseman M., 

Doebbeling, C.C., & Sands LP. 

(2010). Volume of home- and 

community-based Medicaid waiver 

services and risk of hospital 

admissions. Journal of the 

American Geriatrics Society, 58(1): 

109-115.

This study examined the effect of type and volume 

of HCBS Medicaid Waiver services on the risk of 

hospital admissions. They used Indiana Medicaid 

claims data (June 2001-December 2004). The 

Medicaid HCBS services examined included 

attendant care, home making, and home-delivered 

meals.

The authors used a Cox Survival Analysis to 

examine time to hospital admission since enrollment 

in the HCBS waiver. They adjusted for 

demographics, comorbidities, prior use of health 

services, and volume of HCBS received. The 

sample included 1,354 Indiana Medicaid recipients, 

who were enrolled over two years.

They found that a greater volume of the Medicaid 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 96

N0. Study Reason for Including /Summary

Waiver HCBS that they examined was associated 

with a lower risk of hospital admissions. The 

authors note that this association diminished, and 

was no longer significant, around one year after 

program enrollment. The lowest risk of hospital 

admissions was seen among individuals receiving 

more hours of attendant care and homemaking 

services.

In addition to the above studies selected for in-depth review, many additional studies 

were reviewed that addressed some topics of interest to this project but did not meet 

the criteria for an in-depth review and extraction of information.

Articles Providing Support for Key Variables 

Alkema, G.E., Reyes, J.Y., & Wilber, K.H. (2006). Characteristics associated with homeand community-based service utilization for Medicare managed care consumers. 

The Gerontologist, 46(2), 173-182.

Andel, R., Hyer, K, & Slack, A. (2007). Risk factors for nursing home placement in older 

adults with and without dementia. Journal of Aging and Health, 19, 213-228.

Andrews, A.O., Bartels, S.J., Xie, H., & Peacock, W.J. (2009). Increased risk of nursing 

home admission among middle aged and older adults with schizophrenia. Am J 

Geriatr Psychiatry, 17(8), 697-705.

Banaszak-Holl, J., Fendrick, A.M., Foster, N.L, Herzog, A.R., Kabeto, M.U., Kent, D.M., 

Straus, W.L., & Langa, K.M. (2004). Predicting Nursing Home Admission: 

Estimates from a 7-year follow-up of a nationally representative sample of older 

Americans. Alzheimer Dis Assoc Disord, 18(2), 83-89.

Bharucha, A.J., Pandav, R., Shen, C., Dodge, H.H., & Ganguli, M. (2004). Predictors of 

nursing facility admission: a 12-year epidemiological study in the United States. 

JAMA 52, 434-439.

Borrayo, E.A., Salmon, J.R., Polivka, L., & Dunlop, B.D. (2002). Utilization Across the 

Continuum of Long-Term Care Services. The Gerontologist, 42(5), 603-612.

Borrayo, E.A., Salmon, J.R., Polivka, L., & Dunlop, B.D. (2004). Who is Being Served? 

Program Eligibility and Home- and Community-Based Services Use. Journal of 

Applied Gerontology, 23, 120-140.

Cai, Q., Salmon, J.W., & Rodgers, M.E. (2009). Factors associated with long-stay nursing 

home admissions among the U.S. Elderly population: comparison of logistic 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 97

regression and the cox proportional hazards model with policy implications for 

social work. Social Work in Health Care, 48(2), 154-168.

Casado, B.L., van Vulpen, K.S., Davis, S.L. (2011). Unmet Needs for Home and 

Community-Based Services Among Frail Older Americans and Their Caregivers. 

Journal of Aging and Health, 23(3), 529-553.

Chen, Y., Thompson, E.A., Berkowitz, B., Young, H.M. & Ward, D. (2011). Factors and 

home-and community-based services (HCBS) that predict older adults' residential 

transitions. Journal of Service Science and Management, 4, 368-379.

Coward, R.T., Home, C., & Peek, C.W. (1995). Predicting nursing home admissions 

among incontinent older adults: a comparison of residential differences across six 

years. Gerontologist, 35(6), 732-743.

DePalma, G., Xu, H., Covinsky, K.E., Craig, B.A., Stallard, E., Thomas, J., & Sands, L.P. 

(2012). Hospital Readmission Among Older Adults Who Return Home With 

Unmet Need for ADL Disability. Gerontologist, 52(4), 1-8. 

Feng, Z., Fennell, M.L., Tyler, D.A., Clark, M, & Mor, V. (2011). Growth of racial and 

ethnic minorities in the US nursing homes driven by demographics and possible 

disparities in options. Health Affairs, 30(7), 1358-1365.

Fischer, L.R., Green, C.A., Goodman, M.J., Brody, K.K., Aickin, M, Wei, F., Phelps, L.W., 

& Leutz, W. (2003). Community-Based Care and Risk of Nursing Home 

Placement. Medical Care, 41(12), 1407-1416.

Friedman, S.M., Steinwachs, D.M., Rathouz, P.J., Burton, L.C., & Mukamel, D.B. (2005). 

Characteristics predicting nursing home admission in the program of all-inclusive 

care for elderly people. The Gerontologist, 45(2), 157-166.

Friedman, S.M., Steinwachs, D.M., Temkin-Greener, H., & Mukamel, D.B. (2006). 

Informal caregivers and the risk of nursing home admission among individuals 

enrolled in the program of all-inclusive care for the elderly. The Gerontologist, 46(4), 

456-463.

Gaugler, J.E., Duval, S., Anderson, K.A. & Kane, R.L. (2007). Predicting nursing home 

admission in the U.S.: a meta-analysis. BMC Geriatrics, 19(7), 13-27.

Gaugler, J.E., Kane, R.L., Kane, R.A., Clay, T. & Newcomer, R.C. (2005). Early 

community-based service utilzation and its effects on institutionalization in 

dementia caregiving. The Gerontologist, 45(2), 177-185.

Gaugler, J.E., Yu, F., Krichbaum, K., & Wyman, J.F. (2009). Predictors of nursing home 

admission for persons with dementia. Medical Care, 47(2), 191-198.

Goins, R.T., Tincher, A., & Spencer, S.M. (2003). Awareness and Use of Home and 

Community-Based Long-Term Care by Rural American Indian and White Elderly 

with Co-Morbid Diabetes. Home Health Care Services Quarterly, 22(3), 65-81.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 98

Goodwin, J.S., Howrey, B., Zhang, D.D., & Kuo, Y-F. (2011). Risk of continued 

institutionalization after hospitalization in older adults. Journal of Gerontology: 

Medical Sciences, 66A(12), 1321-1327.

Hahn, E.A., Thomas, K.S., Hyer, K., Andel, R., & Meng, H. (2011). Predictors of LowCare Prevalence in Florida Nursing Homes: The Role of Medicaid Waiver 

Programs. The Gerontologist, 51(4): 495-503. 

Haller, I.V. & Gessert, C.E. (2007). Utilization of Medical Services at the End of Life in 

Older Adults with Cognitive Impairment: Focus on Outliers. Journal of Palliative 

Medicine, 10(2), 400-407.

Johnson, R.J. & Wolinsky, F.D. (1996). Use of community-based long-term care services 

by older adults. Journal of Aging and Health, 8, 512-537.

Kaye S., LaPlante M., & Harrington C. (2009). Do noninstitutional long-term care 

services reduce Medicaid spending? Health Affairs, 28(1):262-272. Abstract at: 

http://content.healthaffairs.org/cgi/content/abstract/28/1/262?.

Kosloski, K. & Montgomery, R.J. (1995). The impact of respite use on nursing home 

placement. Gerontologist, 35(1), 67-74.

Krout, J.A., Oggins, J., & Holmes, H.H. (2000). Patterns of service use in a continuing 

care retirement community. Gerontologist, 40(6), 698-705.

Laditka, S.B., Laditka, J.N., & Drake, B.F. (2006). Home- and Community-Based Service 

Use by Older African American, Hispanic, and Non-Hispanic White Women and 

Men. Home Health Care Services Quarterly, 25 (3-4), 129-153.

Li, H. (2006). Rural older adults' access barriers to in-home and community-based 

services. Social Work Research, 30(2), 109-118.

Liu, K., McBride, T.D., & Coughlin, T.A.. (1990). Costs of community care for disabled 

elderly persons: the policy implications. Inquiry, 27(1): 61-72.

Lo Sasso, A.T., & Johnson, R.W. (2002). Does informal care from adult children reduce 

nursing home admissions for the elderly? Inquiry, 39(3), 279-297.

McAuley, W.J., Spector, W., & Van Nostrand, J. (2009). Formal Home Care Utilization 

Patterns by Rural-Urban Community Residence. Journal of Gerontology: Social 

Sciences, 64B(2), 258-268.

McAuley, W.J., Spector, W., Van Nostrand, J., & Shaffer, T. (2004). The influence of 

rural location on utlization of formal home care: the role of Medicaid. Gerontologist, 

44(5), 655-664.

Miller, E.A. & Weissert, W.G. (2000). Predicting elderly people's risk for nursing home 

placement, hospitalization, functional impairment, and mortality: a synthesis. 

Medical Care Research and Review, 57, 259-297.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 99

Miller, E.A., Rosenheck, R.A., & Schneider, L.S. (2012). Caregiver burden, health 

utilities, and institutional service use in Alzheimer's disease. Int J Geriatr 

Psychiatry, 27, 382-393. 

Mitchell, J. & Krout, J.A. (1998). Discretion and service use among older adults: the 

behavioral model revisited. Gerontologist, 38(2):159-168.

Ness, J., Ahmed, A., & Aronow, W.S. (2004). Demographics and payment characteristics 

of nursing home residents in the United States: a 23-year trend. Journal of 

Gerontology: Medical Sciences, 59A(11): 1213-1217.

Office of Program Policy Analysis & Government Accountability (OPPAGA). (2010). 

The State Could Consider Several Options to Maximize Its Use of Funds for 

Medicaid Home and Community-Based Services. Report No. 10-33. 

Pande, A., Laditka, S.B., Laditka, J.N., & Davis, D.R. (2007). Aging in Place? Evidence 

That a State Medicaid Waiver Program Helps Frail Older Persons Avoid 

Institutionalization. Home Health Care Services Quarterly, 26(3), 39-60.

Sands, L., Xu, H., Weiner, M., Rosenman, M., Craig, B., & Thomas, J. (2008). Comparison 

of Resource Utilization for Medicaid Dementia Patients Using Nursing Homes 

Versus Home and Community Based Waivers for LTC. Med Care. 46 (4):449-453. 

Shapiro, A. & Taylor, M. (2002). Effects of a Community-Based Early Intervention 

Program on the Subjective Well-Being, Institutionalization, and Mortality of LowIncome Elders. The Gerontologist, 42(3): 334-341. 

Spillman, B.C. & Long, S.K. (2009). Does High Caregiver Stress Predict Nursing Home 

Entry?. Inquiry, 46(2), 140-161.

Stevens, A., Owen, J., Roth, D., Clay, O., Bartolucci, A., & Haley, W. (2004). Predictors of 

time to nursing home placement in White and African American individuals with 

dementia. Journal of Aging and Health, 16, 375-397.

Sun, F. 2011. Community Service Use by Older Adults: The Roles of Sociocultural 

Factors in Rural-Urban Differences. Journal of Social Service Research, 37(2): 124-135.

Thomas, K.S. & Mor, V. (2013 manuscript). The Relationship between Older Americans 

Act Title III State Expenditures and Prevalence of Low-Care Nursing Home 

Residents. Manuscript currently under review at Health Research and Educational 

Trust.

Van Houtven, C.H. & Norton, E.C. (2004). Informal care and health care use of older 

adults. Journal of Health Economics, 23(6), 1159-1180.

Wallace, D.C., Fields, B.L, Witucki, J., Boland, C., & Tuck, I. (1999). Use of Home and 

Community-Based Services by Elderly Black and White Females. Journal of Women 

& Aging, 11(4), 5-20.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 100

Weddle, D., Wilson, F.L., Berkshire, S.d., & Heuberger, R. (2012). Evaluating Nutrition 

Risk Factors and Other Determinants of Use of an Urban Congregate Meal 

Program by Older African Americans. Journal of Nutrition in Gerontology and 

Geriatrics, 31(1), 38-58.

Weissert W., Lesnick T., Musliner M., & Foley K.A. (1997). Cost savings from home and 

community-based services: Arizona's capitated Medicaid LTC program. The 

Journal of Health Politics, Policy, and Law, 22(6): 1329-57.

Wolinsky, F.D. & Johnson, R.J. (1991). The Use of Health Services by Older Adults. 

J Gerontol, 46(6): S345-S357.

Yaffe, K., Fox, P., Newcomer, R., Sands, L., Lindquist, K., Dane, K., & Covinsky, K.E. 

(2002). Patient and caregiver characteristics and nursing home placement in 

patients with dementia. JAMA, 287(16), 2090-2097.

Articles Providing Background Information

Accius , J. AARP Public Policy Institute, (2008). The role of the Older Americans Act in 

providing long-term care.

Administration on Aging. (2010). Older Americans Act. Washington, DC. Retrieved 

from: http://www.aoa.gov/AoA_programs/OAA/index.aspx

Agency for Healthcare Research and Quality. (2011). Long-Term Care for Older Adults: 

A Comparative Effectiveness Review of Institutional versus Home and 

Community-based Care. Retrieved from: 

http://www.effectivehealthcare.ahrq.gov/ehc/products/369/906/Long-termCare_Protocol_20111220.pdf

Beebe, D. (2011, April 18). Medi-Cal Coverage of Adult Day Health Care Slated for 

Elimination. [Paraprofessional Healthcare Institute web log comment]. Retrieved 

from: http://phinational.org/archives/medi-cal-coverage-of-adult-day-healthcare-slated-for-elimination/

Branch, L.G. & Jette, A.M. (1982). A prospective study on long-term care 

institutionalization among the aged. American Journal of Public Health, 72(12): 1373-

1379.

Brown, R., C. Irvin, D. Lipson, S. Simon, and A. Wenzlow. “Research Design Report for 

the Evaluation of the Money Follows the Person (MFP) Grant Program.” 

Washington, DC: Mathematica Policy Research, Inc., October 2008. Available at: 

[http://www.mathematicampr.com/publications/PDFs/MFP_designrpt.pdf]

Buteau, E. (2010, February 17). [Web log message]. Retrieved from 

http://www.effectivephilanthropy.org/blog/2010/02/experimental-design-canbe-a-powerful-evaluation-tool/
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 101

Buys, D.R., Borch, C., Kilgore, M., Zizza, C., & Locher, G.L. OAA Nutrition 

Expenditures and Nursing Home Residency: A State Level Examination of the 

Effects of Expenditures on Home Delivered and Congregate Meal Programs on 

Rates of Nursing Home Residency. Manuscript Currently Under Review at 

Journal of Aging and Social Policy.

Cool, T.D., Shadish, W.R., & Wong, V.C. (2008). Three conditions under which 

experiments and observational studies often produce comparable causal estimates: 

New Findings from Within-Study Comparisons. J. Policy Anal. Manag., 27(4): 724-

750.

Disability Rights California. (2012). Court approves settlement of Adult Day Health 

Care case: new program for people at risk of institutionalization. [Press release]. 

Retrieved from: http://www.disabilityrightsca.org/news/2012-01-

24%20Press%20Release.htm

Duignan, P. (2008a). What added value can evaluators bring to governance, development and 

progress through policy-making? The role of large visualized outcomes models in policy 

making. Paper presented to the 2008 European Evaluation Society Biennial 

Conference: Building for the Future: Evaluation in Governance, Development and 

Progress, Lisbon, 103 October 2008.

Duignan, P. (2008b). Encouraging better evaluation design and use through a standardized 

approach to evaluation planning and implementation—Easy Outcomes. Paper presented 

to the 2008 European Evaluation Society Biennial Conference: Building for the 

Future: Evaluation in Governance, Development and Progress, Lisbon, 1-3 October 

2008.

Duignan, P. (2005-2009). Seven possible impact/outcome evaluation design types. 

Outcomes Theory Knowledge Base article No. 209. Retrieved from: 

http://knol.google.com/k/paul-duignan-phd/seven-possible-outcomeimpactevaluation/2m7zd68aaz774/10.

Duignan, P. (2009). Techniques for improving constructed matched comparison group 

impact/outcome evaluation designs. Outcomes Theory Knowledge Base Article 

No. 253. Retrieved from: http://knol.google.com/k/paul-duignan-phd/-

/2m7zd68aaz774/113

Ellis, L. & Roe, D.A. (1993). Home-delivered meals programs for the elderly: 

distribution of services in New York State. American Journal of Public Health, 83(7): 

1034-1036.

GAO (Feb, 2011). Older Americans Act: More Should Be Done to Measure the Extent of 

Unmet Need for Services. GAO-11-237. 

http://www.gao.gov/assets/320/316099.pdf

Kane, R.L. (1988). Beyond caring: the challenge to geriatrics. J Am Geriatr Soc, 36(5): 467-

472.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 102

The Lewin Group. May 2010. Projected Economic Impact of Eliminating California’s 

Medi-Cal Adult Day Health Care Program. Prepared for: Congress of California 

Seniors.

The Lewin Group. December 2011. Accelerating Adoption of Assistive Technology to 

Reduce Physical Strain among Family Caregivers of the Chronically Disabled 

Elderly Living at Home. (Draft Final Report). Prepared for: Assistant Secretary for 

Planning and Evaluation.

Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, 

MD: Agency for Health Care Research and Quality; 2011. Available at: 

http://www.effectivehealthcare.ahrq.gov.

Mollica, R. Taking the Long View: Investing in Medicaid Home and Community-Based 

Services Is Cost-Effective. AARP Public Policy Institute, March, 2009. 

http://www.aarp.org/health/medicare-insurance/info-03-2009/i26_hcbs.html.

Wysocki, A., Butler, M., Kane, R.L, Kane, R.A., Shippee, T., & Sainfort, F. (2012). LongTerm Care for Older Adults: A Review of Home and Community-Based Services 

Versus Institutional Care. AHRQ Publication No. 12(13)-EHC134-EF. Preapred for 

the Agency for Healthcare Research and Quality, U.S. Department of Health and 

Human Services.

Studies providing data on state HCBS policies:

Cheek, M., Roherty, M., Finnan, L, Cho, E., Walls, J., Gifford, K., Fox-Grage, W., & 

Ujvari, K. (2012). On the Verge: The Transformation of Long-Term Services and 

Supports. Washington, DC: AARP Public Policy Institute.

Kitchener, M., Ng, Terence, & Harrington, C. (2004). Medicaid 1915(c) Home and 

Community-Based Services Waivers: A National Survey of Eligibility Criteria, 

Caps, and Waiting Lists, Home Health Care Services Quarterly, 23(2): 55-69.

Previous Literature Reviews on Impacts of HCBS:

Doty, P. (2000). Cost-effectiveness of HCB LTC services. Washington, DC: 

USHHS/ASPE Office of Disability, Aging and LTC Policy. Retrieved from: 

http://aspe.hhs.gov/daltcp/reports/2000/costeff.htm

Grabowski, D. (2006). The Cost-Effectiveness of Noninstitutional Long-Term Care 

Services: Review and Synthesis of the Most Recent Evidence. Medicaid Care 

Research and Review, 73(1):3-28. Abstract at: 

http://www.ncbi.nlm.nih.gov/sites/entrez

Hedrick, S.C., & Inui, T.S. (1986). The effectiveness and cost of home care: an 

information synthesis. Health Serv Res, 20(6 Pt 2): 851–880.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 103

Hughes, S.L. (1985). Apples and oranges? A review of evaluations of community-based 

long-term care. Health Serv Res. 20(4):461-88.

Kemper, P., Applebaum, R., & Harrigan, M. (1987). Community care demonstrations: 

What have we learned? Health Care Financing Review, 8(4): 87-100.

The Lewin Group. (2010). UHC Gaps in Care Literature Review: Impacts of Informal 

Caregiving, Select Social Service Interventions, and Home and Community-Based 

Services. Prepared for: UnitedHealth Center for Health Reform and 

Modernization, UnitedHealth Group.

Weissert, W., & S.C. Hedrick. (1994). Lessons learned from research on the effects of 

community-based LTC. Journal of the American Geriatrics Society, 42(3):348-353.

Channeling demonstration studies:

Brown, R., & Phillips, B. (1986). The effects of case management and community services on 

the impaired elderly. Channeling Evaluation Technical Report TR-86-B-02. Princeton, 

NJ: Mathematica Policy Research, Inc. (Executive summary only).

Greene, V.L., Lovely, M.E., & Ondrich, J.I. (1993). The cost effectiveness of community 

services in a frail elderly population.The Gerontologist, 33: 177-189.

Greene, V.L., Ondrich, J.I., & Laditka, S. (1998).Can home care services achieve cost 

savings in LTC for older people? Journal of Gerontology, Series B, 53(4): S228-238.

Kemper, P. (1988). The Evaluation of the National LTC Demonstration: 10. Overview of the 

Findings. Health Services Research, 23(1): 161-174. Available at: 

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1065495/pdf/hsresearch00089-

0167.pdf

Rabiner, D.J., Stearns, S.C., & Mutran, E. (1994). The Effect of Channeling on In-Home 

Utilization and Subsequent Nursing Home Care: A Simultaneous Equation 

Perspective. Health Services Research, 29(5): 605-622.

Thornton, C., Dustan, S.M., & Kemper, P. (1988). The Evaluation of the National Long 

Term Care Demonstration: 8. The effect of Channeling on health and LTC costs. 

Health Services Research, 23(1): 129-142. 

Woolridge, J. & Schore, J. (1988). The effect of channeling on the use of nursing homes, 

hospitals, and other medical services. The Evaluation of the National LTC 

Demonstration: 7. Health Services Research, 23(1): 119-128.

Cash and Counseling demonstration:

Brown R., Carlson B., Dale S., Foster L., Phillips B., & Schore J. (2007). Cash and 

counseling: improving the lives of Medicaid beneficiaries who need personal care 

or home- and community-based services: final report. Mathematica Policy 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 104

Research, Inc. Retrieved from: http://www.mathematicampr.com/Health/cashcounselingpubs.asp.

Dale, S., & Brown, R. (2006). Reducing Nursing Home Use Through Consumer-Directed 

Personal Care Services. Medical Care, 44(8): 760-767. 

Other randomized experiments

Applebaum, R., Seidl, F.W., & Austin, C.D. (1980). The Wisconsin Community Care 

Organization: Preliminary Findings from the Milwaukee Experiment. The 

Gerontologist, 20(3): 350-355.

Nocks, B.C., Learner, R.M., Blackman, D. & Brown, T.E. (1986) .The effects of a 

community-based long term care project on nursing home utilization. The 

Gerontologist. 26(2): 150-157.

Skellie, F.A., Mobley, G.M. & Coan, R.E. (1982). Cost-effectiveness of community-based 

long-term care: current findings of Georgia's alternative health services project.

Am J Public Health. 72(4): 353–358.

Comparison group studies

Edelman, P. & Hughes, S.. (1990). The impact of community care on provision of 

informal care to homebound elderly persons. Journal of Gerontology, 45: S74-S84.

Hughes, S., Manheim, L.M., Edelman, P.L., & Conrad, K.J. (1987). Impact of long term 

home care on hospital and nursing home use and cost. Health Services Research,

22(1): 19-47. 

Kaye S., LaPlante M., & Harrington C. (2009). Do noninstitutional long-term care 

services reduce Medicaid spending? Health Affairs, 28(1):262-272. Abstract at: 

http://content.healthaffairs.org/cgi/content/abstract/28/1/262?.

Sands, L., Xu, H., Weiner, M., Rosenman, M., Craig, B., & Thomas, J. (2008). Comparison 

of Resource Utilization for Medicaid Dementia Patients Using Nursing Homes 

Versus Home and Community Based Waivers for LTC. Med Care. 46 (4):449-453. 

Smith, C. A. & Frick, K. D. (2008). Cost-Utility Analysis of High- vs. Low-Intensity 

Home- and Community-Based Service Interventions. Social Work in Public Health,

23(6): 75-98.

Vertrees, J.C., Manton, K.G., & Adler, G.S.. (1989). Cost effectiveness of home and 

community-based care. Health Care Financing Review, 10(4): 65-77.

Weissert W., Lesnick T., Musliner M., & Foley K.A. (1997). Cost savings from home and 

community-based services: Arizona's capitated Medicaid LTC program. The 

Journal of Health Politics, Policy, and Law, 22(6): 1329-57.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 105

Studies using longitudinal model

Mitchell, G., Salmon J.R., Polivka L., & Soberon-Ferrer H. (2006). The Relative Benefits 

and Cost of Medicaid Home- and Community- Based Services in Florida. The 

Gerontologist. 46(4): 483–494. 

Van Houtven C.H., & Domino M.E. (2005). Home and community-based waivers for 

disabled adults: program versus selection effects. Inquiry. 42(1):43-59. 

Studies using simulation models

Alecxih, Lutzky, & Corea. (1996). Estimated cost savings from the use of HCB 

alternatives to nursing facility care in three states. Estimated cost savings from the 

use of HCB alternatives to nursing facility care in three states. Washington, DC: 

AARP Public Policy Institute/The Lewin Group.

Coughlin, T.A., McBride, T.D., Perozek, M., & Liu, K. (1992). Home Care for the 

Disabled Elderly: Predictors and Expected Costs. Health Services Research, 27(4): 

453-479

Cross-sectional studies

Burr, J.A., Mutchler, J.E. & Warren, J.P. (2005). Effects of state spending on HCBS on 

living arrangement of older unmarried women. Aging Soc Policy, 17(1):1-18.

Reviewed Materials on Program Evaluation Design

Achen, C. (1986). The statistical analysis of quasi-experiments. Los Angeles: University 

of California Press.

Coalition for Evidence-Based Policy. (2009). Which Comparison-Group (“QuasiExperimental”) Study Designs Are most Likely to Produce Valid Estimates of a 

Program’s Impact? A Brief Overview and Sample Review Form. Retrieved from: 

http://coalition4evidence.org/wordpress/?page_id=468

Dennis, M.L. (1990). Assessing the Validity of Randomized Field Experiments: An 

Example from Drug Abuse Treatment Research. Eval Rev 14(4): 347.

Department of Education, Institute of Education Sciences, National Center for 

Education Evaluation and Regional Assistance. (2003). Identifying and Implementing 

Educational Practices Supported by Rigorous Evidence: A User Friendly Guide.

Retrieved from: 

http://ies.ed.gov/ncee/pubs/evidence_based/evidence_based.asp

Dijkers, M. P. J. M. (2009). When the best is the enemy of the good: The nature of research 

evidence used in systematic reviews and guidelines. Prepared for the NCDDR Task 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 106

Force on Systematic Review and Guidelines. Austin, TX: SEDL. 

http://www.ncddr.org/kt/products/tfpapers/

Duignan, P. (2008). Methods and analysis techniques for information collection. Outcomes 

Theory Knowledge Base Article No. 219. http://knol.google.com/k/methodsand-analysis-techniques-for-information-collection#

Duignan, P. (2009). Using a generic outcomes model to implement similar programs in 

a number of countries, districts, organizational or sector units. Outcomes Theory 

Knowledge Base Article No. 265. http://knol.google.com/k/using-a-genericoutcomes-model-to-implement-similar-programs-in-a-number-of#

Duignan, P. (2010). M& E Systems—How to Build and affordable simple monitoring and 

evaluation system using a visual approach. Outcomes Theory Knowledge Base Article No. 

267. http://tinyurl.com/otheory267. 

GAO, Program Evaluation and Methodology Division. (1991). Designing Evaluations.

GAO-PEMD-10.1.4. www.gao.gov/special.pubs/10_1_4.pdf

GAO. (2009). Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective 

Interventions. GAO-10-30. http://www.gao.gov/products/GAO-10-30

GAO. (2012). Designing Evaluations. 2012 Revision. GAO-12-208G. 

http://www.gao.gov/products/GAO-12-208GGAO, 2012

GAO. (1989). Content Analysis: A Methodology for Structuring and Analyzing Written 

Material. Transfer Paper 10.1.3. http://archive.gao.gov/f0102/157490.pdf

Johnston, M.V., Vanderheiden, G.C., Farkas, M.D., Rogers, E.S., Summers, JA.A., & 

Westbrook, J.D. (2009). The challenge of evidence in disability and rehabilitation research 

and practice: A position paper. Prepared for the NCDDR Task Force on Standards of 

Evidence and Methods. Austin, TX: SEDL. 

www.ncddr.org/kt/products/tfpapers/tfse_challenge/

Krause, D. (1996). Effective Program Evaluation: An Introduction. Chicago: 

Nelson-Hall Inc.

Moffitt, R. (1991). Program Evaluation with Nonexperimental Data. Eval Rev 15:291. 

Rossi, P.H., Freeman, H.E., & Lipsey, M.W. (1999). Evaluation: A Systematic Approach. 

Sixth Edition. Sage Publications. 

Shadish, W.R. & Cook, T.D. (2009). The Renaissance of Field Experimentation in 

Evaluating Interventions. Annu. Rev. Psychol. 60:607-29. 

Shadish, W.R. Galindo, R., Wong, V.C., Steiner, P.M. & Cook, T.D. (2011). A 

Randomized Experiment Comparing Random and Cutoff-Based Assignment. 

Psychological Methods, (2):179-191.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix A: Literature Review

Appendix A: Literature Review 107

Singer, J.D. & Willett, J.B. (1991). Modeling the Days of Our Lives: Using Survival 

Analysis When Designing and Analyzing Longitudinal Studies of Duration and 

the Timing of Events. Psychological Bulletin.110(2): 268-290. 

Treasury Board of Canada. (1998). Program Evaluation Methods: Measurement and 

Attribution of Program Results. Retrieved from: http://www.tbssct.gc.ca/cee/pubs/meth/pem-mep00-eng.asp

Trochim, W. M. (2006) The Research Methods Knowledge Base, 2nd Edition. Retrieved 

from: http://www.socialresearchmethods.net/kb/.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

i

Table of Contents

Overview .................................................................................................................... 108

Logic Model ............................................................................................................... 109

Research Questions.................................................................................................. 111

Defining the Scope of the Study .............................................................................. 111

Key Variables Supported by the Literature............................................................. 112

Variable Measure Considerations............................................................................ 114

Proposed Data Sources and Data Elements........................................................... 124

Tables and Figures

Figure 1: Logic Model.............................................................................................................110

Table 2: Key Variables ...........................................................................................................112

Figure 3: Proposed Study Design Image................................................................................122

Table 4: Suggested Data Sets ...............................................................................................125
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 108

Appendix B: Study Framework and Design

Overview

Older American Act (OAA) programs and services represent a significant federal 

investment in developing a comprehensive, coordinated, and cost-effective system of 

home and community-based services (HCBS) that enables adults to live independent 

and healthy lives in their homes and communities. Under this project, the 

Administration on Aging (AoA) seeks to design a rigorous study of the impact of OAA 

programs and services on key outcomes including HCBS use, health care use, 

community tenure, and long term services and supports (LTSS) expenditures. OAA 

services and programs are diverse, often integrated and/or provided in combination 

with other services, funded through multiple funding streams, and administered and 

delivered by different state and local-level agencies with varying data collection 

capacity. Therefore, a study of OAA funded programs and services is inherently 

challenging.

This Study Design describes The Lewin Group‘s proposed approach for understanding 

the impact of OAA programs and services on the aforementioned outcomes of interest. 

Building off of the literature reviewviii and study framework,ix this document includes a 

discussion of the challenges and trade-offs that must be considered in the selection of 

variables, research questions, study design options and secondary data sources, in order 

for AoA to endorse a rigorous impact study of OAA programs and services.

Purpose of the Study

Developing and designing a study that produces credible, solid evidence of results is of 

critical importance to AoA, the Aging Network, service recipients and families who rely 

on OAA services. Key policymakers, such as Congress and the Office of Management 

and Budget (OMB) at the federal level, as well as state and local governments, use 

program evaluation information to make budget decisions. Increasingly, funding for 

health and social programs is dependent on stakeholders’ ability to make the business 

case for the investment, with rigorous research demonstrating success in improving 

peoples‘ lives and achieving more effective use of resources. A study with inconclusive 

results or a study that suggests a lack of impact when in fact the design prevented the 

impact from being detected can be detrimental. 

 

viii The Lewin Group (2012). Study of the Global Outcomes of the Older Americans Act Programs and 

Services- Review of the Literature Department of Health and Human Services, Administration on 

Aging. 

ix The Lewin Group (2012). Study of the Global Outcomes of the Older Americans Act Programs and 

Services – Study Framework. Department of Health and Human Services. Administration on Aging. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 109

This study focuses on the impact of services provided through OAA funded HCBS 

programs authorized under Titles III-B, C, D, and E of the OAA. These programs 

include a range of supportive services, nutrition services, health promotion and disease 

prevention programs, as well as services for family caregivers. While the mix and type 

of services offered differ by state and locality, the vast majority of OAA funding is used 

for the provision of nutritional services.

Process for Developing the Study Design

To begin, the team conducted a thorough review of the literature to examine the 

methodology, findings, etc. of related studies. These studies informed the research 

questions, logic model, and initially proposed OAA global outcomes study design. 

Subsequently, AoA and Lewin held multiple meetings to discuss these documents. 

Following the development of a study design, AoA and Lewin engaged various 

stakeholders through:

► Convening an expert group to provide feedback on the study design – with 

representatives from federal and state government, local agencies, and academia.

► Conducting key informant interviews with state representatives to better 

understand the available state administrative data for OAA, data collection and 

reporting processes. 

► Conducting key informant interviews with selected expert group members to 

receive additional feedback and additional experts as needed. 

Next steps based on feedback received are reflected in the Limitations and 

Recommendations document. In the process of designing this study, the research team 

identified a number of strengths and limitations to conducting a global outcomes study 

of the OAA. 

Logic Model

The logic model below depicts the inputs, activities, outputs, and outcomes of OAA 

funded HCBS programs and services. We included “non-OAA funded activities” (e.g., 

informal services, state funded HCBS, private pay) as these activities often occur in 

combination with OAA-funded activities and impact the key outcomes of interest for 

this project. Assumptions and external factors are presented.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 110

Figure 4: Logic Model
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 111

Research Questions

Based on the expected outcomes depicted in the logic model, the following research 

questions for an OAA global outcomes study were proposed. What is the impact of 

OAA funded HCBS programs and services on:

► Community tenure

► Health care utilization

► Costs of care for older adults (e.g., LTSS, health care costs)

► Physical, mental, and emotional health and wellness (i.e., preventive measures) of 

care recipients and caregivers

► Unmet needs among older adults

► Caregivers (e.g., strain, burden, depression, health, etc)

► Coordination of services (e.g., care management)

In addition to answering questions about the impact of service use on desired 

outcomes, it is recommended that AoA consider a design that measures the strength of 

association of covariates/intervening variables on the key outcomes including, but not 

limited to, mix, type, and intensity of services. This step would answer the following 

sample questions: 

► What is the impact of OAA services alone or in combination with services paid for 

by other sources? 

► What is the impact of service mix and intensity on outcomes of interest? [if possible, 

we will isolate OAA services]

► What subgroups had the most favorable outcomes? (e.g., health conditions, 

demographics, functional status)

Defining the Scope of the Study

A critical methodological decision hinges on how non-OAA funded services are 

factored into the study design. For example, does the design consider the OAA service 

package alone or in combination with a similar service package that is funded by 

alternative sources? This decision will impact the comparison group, sample size, 

research questions and generalizability of the findings.

Another important issue discussed frequently in the literature, and one that we are 

confronted with in this design, is the level of exposure to HCBS services (i.e., mix, 

duration, intensity and timing). Consideration of this step will ensure that individual 

variation (e.g., a caregiver that receives minimal exposure to respite care every week 

will likely respond differently than a caregiver who receives 24 hours of respite care 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 112

every week) are factored into the final models. Several studies that were reviewed 

developed exposure algorithms that may be applicable for use in the final models to 

address this issue (NYSOA, 2010). 

Furthermore, another important issue raised in the literature is the impact of certain 

policy or other contextual factors on individual outcomes (e.g., nursing home admission 

or HCBS use). These measures include the amount of state spending on Medicaid 

HCBS, the number of nursing home beds in a geographic area, availability of Waiver 

slots, the number of home health agencies in a particular area, and the use of preadmission screening prior to NH use.

Key Variables Supported by the Literature

Upon a review of the literature, articles were identified that support the association 

between the variables listed below and the key outcome variables of interest (LTSS 

expenditures, community tenure/NH use, health care use, and HCBS use). The list 

below also includes independent variables at the policy/funding level, as well as those 

related to characteristics of older adults and their service use. The variables are 

organized into the Andersen Health Behavioral Model’s predisposing, enabling, need, 

and use categories and shown in Table 2 below:

Table 5: Key Variables

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Predisposing Variables

Age √ √ √

Education √ √ √

Gender √ √ √

Race/Ethnicity √ √ √

Geographic Location √ √ √

Living Alone √ √ √

Marital Status √ √ √

Informal Caregiver Availability √ √ √

Having Children √

Social Support √

Enabling Variables 

Income √ √ √
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 113

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Home Ownership √

Payment Method √ √ √

Caregiver or Spouse Age √

Support for Caregiver √ √

Caregiver Burden √

NH Bed Supply √

Number of Home Health Agencies √

Medicaid HCBS Spending √ √

State HCBS Spending √

Pre-admission screenings √

Physician Access √

Need Variables 

Alzheimer's/Cognitive 

Impairment/Dementia

√ √ √

Comorbidities/Chronic Medical 

Conditions

√ √ √

Psychiatric Issues/Problematic 

Behavior

√ √

Worse ADL/IADL performance in 

general

√ √

ADL Limitations √ √ √

IADL Limitations √ √

Self-health ratings √ √

Unmet need √ √ √

Service Use Variables 

Hospital Admissions/Use √ √ √

HCBS Use √ √ √ √

NH Use √ √

ADL/IADL Help √ √

Case Management √

Medicaid Waiver √

Timing of Care √

Volume of Services √ √ √
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 114

Independent Variables Included in Outcome 

Analysis

HCBS 

Use

Health 

Care Use

Community 

Tenure/NH 

Use

Cost 

Impacts

Low-intensity HCBS √ √

Combination of Services √ √

Use of OAA III-B √

Use of OAA III-C √ √ √

Use of OAA III-E/Respite √ √

Variable Measure Considerations

Several of the key variables proposed for the OAA study are well established measures 

(e.g., sex as male/female/unknown). However, the literature points to several 

limitations of many measures, which must be considered prior to including them in the 

study. The considerations that were most frequently discussed in the literature are 

described below. Additionally, discussions with states also revealed related variable 

considerations, which are included in this section.

Predisposing Variables

Measures for Informal/Social Support

Studies identified measurement concerns regarding the amount or level of informal 

support, as well as concerns regarding proxy measures (living alone, marital status, and 

having children) used to represent this support. Results from these studies illustrate the 

complexities of measuring informal support.

Amount/Level of Informal Support: A study conducted by Xu and colleagues (2010) was 

limited by an inability to assess the amount of informal care provided. The researchers 

measured the existence of “informal help” with a yes/no response, but did not further 

examine the level of informal support. Similarly, a study by Sands and colleagues (2008) 

was not able to quantify the amount of formal or informal caregiving an individual 

received nor the influence the amount of that support may have on the inpatient use 

and/or expenditure for using HCBS waiver services. In state interviews, state 

representatives reported being able to measure the intensity of caregiving support (e.g., 

the number of hours a caregiver helps an older adult each week). However, other states 

did not require that their AAAs record and report this information. The existence of and 

availability of a caregiver may be recorded; however, the amount of hours they provide 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 115

informal support is not recorded or it is not reported consistently to the state. In some 

states, caregiver availability may not be recorded at all.

Living Alone: Living alone has been used as a proxy measure for social support (Krout et 

al., 2000). One study of the Aging Network CARES assessment of participants in Kansas 

found that living alone and having available support did not explain the difference in 

those who remained in the community versus those admitted to a nursing facility 

(Macmillan et al., 2007). However, additional qualitative analysis showed that most 

consumers who lived alone had daily informal support and contact from family 

members. This suggests that living alone may not be an appropriate proxy measure of 

social isolation, particularly if informal support has not been taken into account. 

Marital Status: Similar to findings in the literature regarding living alone, marital status 

may not always serve as a proxy measure for informal caregiving and/or social support. 

In some cases, due to missing data, researchers have used marital status as a proxy 

measure for informal support (Miller et al. 1998). However, while two systematic reviews 

found an association between being married and lower odds or reduced risk of nursing 

facility placement (Gaugler et al. 2007; Miller and Weissert, 2000), findings from more 

recent research did not draw the same conclusion. For older adults with dementia, 

Andel, Hyer and Slack (2007) found that married individuals were more likely to be 

placed in a nursing home. The same study found that widowed status reduced risk of 

nursing home placement among older adults without dementia. The authors suggest 

that this may be due to the unique nature of this population. Having limited financial 

resources, a spouse may actually be a burden or an additional strain on resources that 

may lead to a nursing home placement. The highlighted studies demonstrate the 

importance of including marital status as a measure, but that it may not be an appropriate 

proxy measure for social support.

Having Children: A study by Gaugler and colleagues (2007) found that having children 

lowered the odds of nursing home admission. Muramatsu and colleagues (2007) also

found that living close to or living with children was significantly associated with lower 

nursing home admissions. However, other studies found that the existence of children 

may not be a suitable proxy measure for social or informal support. One such study by 

Karuza & Wu (2011) argues that using proxy measures to quantify informal care is 

problematic if the geographic distance between parent and child(ren) and the child’s level 

of involvement in providing care is not determined. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 116

Enabling Variables

Income / Payment Method

Alkema and colleagues (2006) point out that education may be a proxy measure for 

socioeconomic status and has been used as such in sociological research. This method 

was used by Miller et al. (1998) in their study, where education level was used as a proxy 

for income. Another proxy measure for income is Medicaid status (Chapin et al. 2003) or 

welfare status (Casado et al. 2010) , which includes receiving Medicaid, Social Security 

Supplemental Income, Temporary Assistance to Needy Families, and/or Supplemental 

Nutrition Assistance Program. However, these proxy measures do not capture everyone 

with low income because of the less than 100 percent (generally 50-70 percent) 

participation rates among those eligible for these programs (GAO, 2005). 

Policy Variables

The literature supports the need to include policy variables in examining the impact of 

services on individuals. However, understanding how these variables are measured and 

their effect on/how they are affected by policy is important. They include: 

► State Spending on Medicaid HCBS: Muramatsu et al. (2007) used two variables to look 

at state HCBS spending and other state level funding: One was per capita HCBS 

spending (total HCBS spending/population of 65 or older), which measures the 

absolute amount of money going towards HCBS; and (2) Percentage of LTC 

spending going to HCBS rather than nursing homes, which measures the extent of 

the state's LTC system expenditure for HCBS as compared to institutional settings. 

Some studies examined general Medicaid HCBS spending, but not per person 

spending (D’Souza et al., 2009; Miller, 1998) and found a potential association 

between the reduction in HCBS spending and an increase nursing home admissions.

► Number of Home Health Agencies and Nursing Home Bed Supply: Previous HCBS 

studies have identified a number of factors related to the availability of resources in 

the community that are important in HCBS outcomes. The New York Office for the 

Aging (2010) recommended, based on the lessons learned from the study, that 

future studies include community contextual factors, including the availability and 

accessibility of long-term care services. Contextual factors discussed in the literature 

may include the following:

 Number of home health agencies. A study by Miller and colleagues (1998) 

found that for individuals who are married, a greater number of home health 

agencies is associated with a longer stay in the community. 

 Right supply of nursing home beds. Muramatsu and colleagues (2008) used 

county-level NH bed availability per person 65 or older to measure the 

variability of nursing home bed supply within a state as a proxy for long-term 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 117

care availability. While the nursing home bed supply may be a suitable proxy 

of nursing home use in some instances, the Maramatsu study also points out 

that the use of this variable as a proxy for long-term care availability should be 

done with care. Variations across states may exist because states with a smaller 

nursing home bed supply may be more likely to have a weaker nursing home 

industry and therefore be better positioned to lower nursing home utilization 

and HCBS funding than states with larger, more powerful nursing home 

industries.

► Pre-Admission Screening: A study conducted by Chapin and associates (2009) 

suggests that providing HCBS information during a pre-admission screening may 

have the potential to permanently prevent a nursing home admission. Findings 

from this study would suggest that examining the use and timing of a preadmission screening is a variable to consider in the study of OAA services.

► State Policies on Uniform Assessment Tools: Some states may not require that their 

AAAs or local service providers use the same assessment tools, or even assess 

individuals at the same time intervals. This may be an issue in obtaining uniform 

data on OAA recipients across the state as well as tracking programs related to 

physical, emotional, and mental health. 

Caregiver Characteristics, Assistance, & Burden

Connecting caregiver measures to individual outcomes may be difficult, specifically, 

measuring the amount of assistance the caregiver receives and the caregiver’s burden and 

how these measures impact a consumer’s outcomes. This relates to the discussion above 

regarding the measurement of informal and social support. As previously discussed 

marital status and having children may not be appropriate proxy measures for social 

support. Additionally, these measures do not capture the amount of informal and social 

support an older adult may receive and, therefore, possibly the burden placed on a 

caregiver providing support.

Casado, van Vulpen, and Davis (2010) reported that if a caregiver has substitute help, 

they report having less unmet HCBS services. A study conducted by Miller and Weissert 

(2000) found an association between the support a caregiver receives and the risk of 

nursing home entry. The study found that the greater the support a caregiver received, 

the risk of nursing home admission declined. More specifically, greater caregiver burden 

has been associated with increased nursing home placement in a number of studies 

(Gaugler et al., 2009; Miller, Rosenheck, Schnieder, 2012; Spillman and Long, 2009; Yaffe 

et al., 2002). However, when utilizing the caregiver burden variable, it is essential to 

examine data sets for consistent use of the measure. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 118

An additional concern from state discussions is data collected related to the caregiver and 

the care recipient for those who are receiving Family Caregiver Support Program 

services. With only minimal data being collected on the care recipient, information might 

be lacking on the intensity of caregiving needed (e.g., the number of I/ADLs the 

caregiver has to assist with). Incomplete data may also prove to be problematic when 

tracking the same care-recipient over various OAA services.

Need Variables

ADL and IADL Limitations

Chen & Adams (2010) noted that awareness of unmet need and ability/desire to seek 

assistance are related to cultures and values that are not often measured but understood 

to drive use of services. Evidence suggests that self-reported ADLs and IADLs often 

overestimate a person’s functional ability when compared to performance-based tests 

and therefore may not be omitted from research. However, the same study by Chen & 

Adams (2010) suggests that older adults’ perceptions of their functional ability may be a 

more important indicator of community tenure than their actual physical abilities because 

an individual’s perception of the functional ability is tied to their awareness of their 

unmet needs and their service use. 

Additionally, for studies examining an individual over time, it is important to track 

whether ADL/IADL status changed over time. Xu and colleagues (2010) cited this as a 

limitation to their study because functional status was only obtained from the preadmission screening used to determine functional eligibility for the waiver program and 

not once services had been received. This did not allow for an examination of the impact 

of waiver program services on functional status. As mentioned previously, this may be a 

concern, as various states have differing policies on which tool to use to conduct 

assessments and how often an individual is reassessed. 

A study by Karuza & Wu (2011) examined the impact of OAA services on individual 

outcomes across four states. One state in the study used unmet need score as a proxy for 

ADL/IADL scores. However, Karuza and Wu point out that this is a limitation of the 

study and state, “the use of proxy measures to quantify client-based characteristics, e.g., 

limitations on Activities of Daily Living or Instrumental Activities of Daily Living, raises 

concerns with the validity and reliability of these measures, especially if the analyses use 

them as potential covariates or control variables”. 

While awareness of unmet need and self-perceived functional status may not be deemed 

appropriate measures for all studies, for the purposes of examining service use and the 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 119

impact of services on functional status over time, these measures should be considered 

without substitution by a proxy measure, if possible, in the OAA study. 

Prior Use Variables

Prior Nursing Home Use / Defining Community Tenure

Prior nursing home stays has been a significant predictor of future nursing home 

admissions. However, prior nursing home stay was not measured consistently. For 

example, Muramatsu and colleagues (2008) defined “permanent” nursing home 

relocation as a stay at a nursing home or other health care facility at the time of death. 

However, for living study subjects, a long-stay nursing home admission was not defined 

in the Muramatsu study. Cai and associates (2009) identified a long-stay nursing home 

admission as “the stays after admission that exceed three months from the date of 

admission.” This measurement is consistent with prior research. 

Chapin and colleagues (2003) interviewed nine state aging program representatives and 

found only two (Missouri and Connecticut) that routinely tracked community tenure and 

defined the measure. MO defined this as “Anyone who hasn’t chosen nursing facility” 

and CT as “to begin services and remain on the program without long-term placement.” 

Karuza & Wu’s (2011) analysis defined “survival” as any outcome that was not a 

permanent nursing facility placement. An additional state data concern is whether the 

state tracks the reasons for an OAA service recipient ending their service use. Some 

states may record the beginning and end date of service without tracking the reasons 

(e.g., death, NH admission, etc) for termination of services.

Measurement of Prior Service Use

Volume of Services: According to the literature, volume of services has an impact on 

institutionalization. In Rhode Island, individuals who receive more than one service have 

an increase in community tenure (Brock et al., 2007; Karuza & Wu, 2011). These studies 

did not have a control variable; therefore, they had to use receipt of one service as the 

control. Xu and associates (2010) posit that researchers need to track “the mechanism by 

which greater volume of formal services is associated with lower risk for hospitalization. 

It is possible that greater availability of services reduced risk for potentially preventable 

events”.

Service Intensity and Combination: A study by Shapiro and colleagues (2009) recommends 

that the intensity of services be examined, as that measure may impact outcomes, 

specifically cost-savings. They found that low intensity HCBS users utilize fewer general 

revenue expenses and generate the greatest overall cost savings, compared to non-users 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 120

(Shapiro & Loh, 2010). However, other studies show that costs savings related to service 

intensity may also be due to the combination of services received. For example, in the 

Shapiro (2009) study, when multiple services users were removed from the analysis, the 

Alzheimer’s Disease Initiative (ADI) cost savings disappeared. Savings originally 

observed in the study from ADI may be a result of savings from the combination of ADI 

and other services.

Timing of Care: Regarding service delivery, Gaugler et al. (2005) found that timing of care 

impacted institutionalization. Caregivers of individuals with dementia who used inhome help services earlier in their caregiving experiences were more likely to delay 

institutionalization (Gaugler et al., 2005). In the MacMillan and colleagues’ (2007) study, 

at three months after assessment, 31.4% of diverted individuals used services (the highest 

proportion in the study). This further supports that timing of care is an important 

variable to consider in nursing home diversion research. However, without data on 

when the receipt of services was first needed it may be difficult to measure the impact on 

institutionalization. 

Case Management: One series of studies highlights the importance of understanding both 

state and local policies toward receipt of services. For example, Karuza & Wu (2011) 

found that intensity of case management services was a statistically significant predictor 

for nursing home placement in two counties, after controlling for select factors. 

Additional investigation into why this is the case (NYSOA 2010) revealed that local 

county policies and procedures impacted nursing home placement. In one county (for 

which case management was more significant), case managers who assist clients with 

nursing home placement and there is a cap on personal care services.

Cost of Service Use

Another data concern is whether a state can connect expenditures to the individual. 

Some states may only have the ability to report expenditures data in aggregate on the 

service level. This prevents researchers from accurately assessing the amount of money 

spent to provide OAA services at the individual level.

Proposed Study Design

We suggest that the study approach with the greatest potential for delivering conclusive 

and actionable results would be a quasi-experimental design that includes: 1) a 

retrospective study component drawing on existing individual data from federal 

sources; and 2) a prospective study component using data to be collected during the 

study period. Evaluators of HCBS and other health and social service programs usually 

rely on quasi-experimental techniques as the best impact study design and most 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 121

powerful alternative to random experiments. We know from the literature that 

demonstrating impacts on HCBS expenditures can take several years, with a lag 

between increased spending on HCBS expansion and savings from this investment

(Kaye, LaPlante, & Harrington, 2009). Thus, we recommend a study period of seven

years; a plausible option given the retrospective/prospective design option. This study 

period time can be modified depending upon time, data, or funding constraints. 

To reduce bias and error in assessing program outcomes, we recommend incorporating 

multiple methods, measures, and data sources. In using a quasi-experimental design 

with a matched comparison group, we will explore different potential methods 

including propensity score matching for constructing a matched comparison group in 

the final design. This will allow us to compare older adults who had an outcome of 

interest with older adults who are matched on certain key characteristics relevant to the 

outcome, to determine whether OAA services contributed to the difference in outcome. 

The matching process will allow us to control for factors that are believed to contribute 

to different outcomes (e.g., gender, IADL status, etc.).

The key to unbiased inference in this approach, as in any quasi-experimental design, is 

proper specification: choosing a set of key factors to be controlled, so that uncontrolled 

variables are approximately random, i.e., not correlated with the controlled variables or 

the treatment (Achen, 1986). Knowledge about the factors that affect receipt of services 

and factors correlated with outcomes (i.e., community living tenure and healthcare 

utilization) identified in literature review are essential to the success of the design.

The research team proposes using this design in a study of 3 states. A primary purpose 

of the study will be to identify a core set of variables that all states should collect. States 

have been granted great flexibility regarding program administration and data 

collection. As a result, states lack uniform and comprehensive data. Conducting this 

study, as discussed in the Limitations and Recommendations report (Appendix C), can 

address that issue.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 122

Figure 6: Proposed Study Design Image
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 123

Proposed Study Design Description

The above graphic depicts the proposed design for the Study of the Global Outcomes of 

Older American Act Programs and Services. The goal is to measure the 

impact/association of OAA service use by older adults on four key outcomes (health 

care utilization, HCBS expenditures, NH admissions and community tenure). Further, 

the design allows for the examination of various service use trajectories over time (e.g., 

HCBS user in a NH, NH user exits NH, etc.) and the extent to which OAA programs 

and services are associated with these trajectories. The details of the study design are 

discussed below.

Intervention: Receipt of OAA services, alone or in combination with other HCBS.

Target Population: The target population for this study is older adults age 60 and above 

(those eligible for OAA services). In the selection of the cohorts in stage 1, the sample 

will be selected and then connected back to those who received OAA services in stage 2 

of the design. Those who have received OAA services are the treatment, or intervention, 

group and those who are not matched to OAA service use are the comparison group.

Pre-Stage: Three states will be selected through an RFP process to participate in this 

study design. The states will have the data capacity and availability to participate in the 

stages outlined below.

Stage 1: The proposed design takes place over a six-year study period and consists of 

three stages. In Stage 1, (depicted in the blue box), the study sample is selected using 

MDS, Medicaid/Medicare claims data, and OAA or state-funded services data, that 

cover a one year period of time (2009-2010) and an entire state (e.g., Georgia). Three 

cohorts of older adults will be identified from this dataset: Cohort 1 will be a sample of 

older adults who have been admitted to nursing homes; Cohort 2 will be a sample of 

older adults enrolled in any HCBS (e.g., Medicaid, state-funded, OAA); and, Cohort 3

will be a group of individuals who were not admitted to a nursing home and did not

receive HCBS. This group will be matched to older adults in Cohorts 1 and 2 on certain 

key variables to ensure comparability (propensity score matching). Cohort 3 will serve 

as the comparison group. 

Stage 2: During Stage 2 of the study (represented in the purple box), the HCBS service 

use by Cohort will be measured. This will be accomplished through the analysis of 

Medicare/Medicaid Claims, POMP, State OAA administrative records, HRS and any 

other available sources. The study period covers 2006-2009. Data will be analyzed by 

service type, mix, and intensity of service use. A group of individuals will be non-users.

Stage 3: In the final stage of the study, Stage 3, (illustrated in the green box), using the 

same combination of data sources, two additional years of service use and client 

disposition—status post intervention will be examined. In sum, the study will have a 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 124

longitudinal data base (2006-2012 or the most current year available to us) that includes 

three unique cohorts of individuals: nursing home users at baseline, HCBS users at 

baseline, and non-users at baseline. This combination will allow for both within-group 

and between-group data modeling. Results will demonstrate if receipt of OAA services, 

whether alone or in combination with other services, will directly impact: health care 

utilization, HCBS expenditures, nursing home (NH) admissions and community tenure. 

Proposed Data Sources and Data Elements

A high-quality study hinges on valid and reliable data. Regardless of how well a study 

is designed, the study is only as good as the data utilized. While available data sources 

provide a cost-effective and efficient way of collecting data, all sources collected from 

the literature review were found to have limitations in terms of the measures 

/indicators collected, the quality of the data, the frequency of the collection, or

comparability of the measures /indicators over localities and states. Many studies have 

used supplemental surveys, questionnaires or interviews to collect additional data that 

complement the data collected through regular reporting systems and allow for more 

robust study results. While supplemental data collection methods can be costly, they 

are likely the only way to ensure that high quality data is collected on certain key 

variables identified in the literature as critical for inclusion (e.g., availability of informal 

caregivers, level of unmet need, functional status, etc.).

Based on our understanding of the data sources available for use in a study of OAA 

programs and services, as well as the efficiencies needed by AoA in executing this 

study, the design will primarily utilize secondary data sources (which are discussed in 

more depth below). While it is feasible to use a study design that is solely based on 

secondary data sources, AoA must be keenly aware of and thoughtfully consider the 

trade-offs that are inherent in using a design that is solely based on secondary data. 

In Table 4, the data sources proposed in the study design and how they are applicable 

and useful to the study design are identified. The purpose for using these secondary 

data sources in the OAA HBCS study is to compile a cadre of multiple independent, 

intervening, and outcome variables that are critical to building a reliable study model.

Numerous secondary data sources exist and several of the studies from the literature 

review combined data from multiple sources. For example, Miller et al. (1998) linked 

individual level data from a national longitudinal survey with state health care system 

data via the state of residence of each person. In Shapiro & Loh (2010), MDS records 

were matched with Florida CIRTS records, using resident IDs to obtain Social Security 

Numbers (SSNs) from CMS, then matching CIRTS records with the SSNs. Based on our 

proposed study design, we suggest using some of the data sets described below. We 

are proposing the use of Medicaid and Medicare claims data and exploring datasets 

similar to HRS, like NLTCS and NHATS. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 125

Table 7: Suggested Data Sets

Data 

Source Data Elements

CMS 

Medicare 

Claims Data

The Medicare Claims Files contain information collected by Medicare to pay 

for health care services provided to a Medicare beneficiary. Data are 

available for each institutional and non-institutional claim type, which each 

record being a claim. Some of the information contained within these 

utilization files includes:

► procedure and diagnosis information

► dates of service

► revenue center detail

► payment and charge amounts

► beneficiary demographic information

► limited professional provider and facility data (http://www.resdac.org/cmsdata/file-family/Medicare-Claims) 

CMS 

Medicaid 

Claims Data

MAX Personal Summary File

The Personal Summary File contains one record for every individual enrolled 

for at least one day during the year. The file contains demographic data (e.g. 

date of birth, gender, race), basis of eligibility, maintenance assistance 

status, monthly enrollment status, and a utilization summary. A small 

number of the records in the file are limited records for individuals who were 

assumed to be eligible, had a paid claim and then were ultimately determined 

to be ineligible. (http://www.resdac.org/cms-data/files/max-ps) 

CMS 

Minimum 

Data Set 

(MDS) 

“The CMS MDS is The Long Term Care Minimum Data Set (MDS) is a 

standardized, primary screening and assessment tool of health status which 

forms the foundation of the comprehensive assessment for all residents 

(regardless of payer) of long-term care facilities certified to participate in 

Medicare or Medicaid. The MDS contains items that measure:

► physical, 

► clinical,

► psychological,

► psycho-social functioning, and

► life care wishes” (http://www.resdac.org/cms-data/files/mds-3.0) 

This dataset was used in prior HCBS studies (Buys et al manuscript; Chapin 

et al., 2002; 2003; Macmillan et al., 2007; Shapiro & Loh, 2010)

MDS-Home 

Care

This dataset identifies the number of times over the prior 90 days (or since 

the last assessment) that an individual has been admitted to hospital with an 

overnight stay or visited the ER without an overnight stay. It was used in a 

prior HCBS study (D’Souza et al., 2009). It also includes functional status, 

being used by 9 states for their Nursing Facility Level of Care Assessment. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 126

Data 

Source Data Elements

Home Health 

Outcome and 

Assessment 

Information 

Set (OASIS)

“The Home Health Outcome and Assessment Information Set (OASIS) 

contains data items that were developed for measuring patient outcomes for 

the purpose of performance improvement in home health care. OASIS 

assessments are collected at specified time points for adult (age 18 or over) 

Medicare and Medicaid patients. OASIS assessment instrument includes:

► socio-demographic, 

► environmental, 

► support system, 

► health status, 

► functional status, and 

► health service utilization characteristics”.( http://www.resdac.org/cmsdata/files/oasis) 

One study used this for Medicare home health assessment and service use 

data (Chapin et al., 2003)

POMP – the 

Performance 

Outcomes 

Measures 

Project

Original Performance Outcome Measure (POM) demonstration projects 

provided a core set of performance measurement tools to quantify the impact 

of OAA Title III services for consumers. The objectives of POM were to: 

develop performance measure across several program domains under Older 

American Act (OAA); to conduct sample surveys using POM instruments; 

and, to use data from the surveys to improve program management and to 

justify future program expenditures. 

This then grew into POMP and included Arizona; Florida; Georgia; Iowa; New 

York; North Carolina; Ohio; Rhode Island; and, South Carolina. Next 

Generation POMP and the Sentinel Project included more states. We 

examined multiple studies that used POMP data (Brock et al., 2007; Karuza 

& Wu, 2007; Shapiro & Loh, 2010).

State OAA 

Administration 

Data

We recommend the use of state OAA data from a state that uses a statewide 

database. Prior studies have used state data sources, like Florida’s CIRTS 

records (Shapiro & Loh, 2010). To determine the feasibility of using a state’s 

administrative data, we interviewed select states on their data collection and 

warehousing procedures. 

National 

Surveys 

related to 

Long Term 

Care

The Health and Retirement Study

The Health and Retirement Study (HRS) is an ongoing longitudinal panel 

study that surveys a representative sample of more than 26,000 Americans 

over the age of 50 every two years. (http://hrsonline.isr.umich.edu/) 

It collections variables such as physical functioning, receipt of services (home 

care by a medical professional and additional service, e.g., adult day care, 

social worker, outpatient rehabilitation, transportation, or meals), NH 

placement, and demographics. This survey has been used by prior HCBS 

studies (Brock et al., 2007; Muramatsu et al., 2007; 2008)
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 127

Data 

Source Data Elements

The National Long Term Care Survey (NLTCS)

The NLTCS is a longitudinal survey designed to study changes in the health 

and functional status of older Americans (aged 65+) that was last fielded in 

2004. It also tracks health expenditures, Medicare service use, and the 

availability of personal, family, and community resources for caregiving. 

(http://www.nltcs.aas.duke.edu/) 

The National Health and Aging Trends Studies (NHATS)

The NHATS, first fielded in 2011, will gather information on a nationally 

representative sample of Medicare beneficiaries ages 65 and older. Inperson interviews will be used to collect detailed information on activities of 

daily life, living arrangements, economic status and well-being, aspects of 

early life, and quality of life. (http://www.nhats.org/) 

Bibliography

Achen, C.H. (1986). The Statistical Analysis of Quasi-Experiments. University of California 

Press: Berkley and Los Angeles, CA.

Alkema, G.E., Reyes, J.Y., & Wilber, K.H. (2006). Characteristics associated with homeand community-based service utilization for Medicare managed care consumers. 

The Gerontologist, 46(2), 173-182.

Andel, R., Hyer, K, & Slack, A. (2007). Risk factors for nursing home placement in older 

adults with and without dementia. Journal of Aging and Health, 19, 213-228.

Brock, D.B., Rabinovich, B., Severynse, J., Ficke, R.USDHHS, Administration on Aging. 

(2007). Risk factors for nursing home placement among OAA service recipients: 

Summary analysis of data from five sources. (233-02-0087) 

Cai, Q., Salmon, J.W., & Rodgers, M.E. (2009). Factors associated with long-stay nursing 

home admissions among the U.S. Elderly population: comparison of logistic 

regression and the cox proportional hazards model with policy implications for 

social work. Social Work in Health Care, 48(2), 154-168.

Casado, B.L., van Vulpen, K.S., Davis, S.L. (2011). Unmet Needs for Home and 

Community-Based Services Among Frail Older Americans and Their Caregivers. 

Journal of Aging and Health, 23(3), 529-553.

Chapin, R., Zimmerman, M., Macmillan, K., Rachlin, R., Nakashima, M., Hayes, J., 

Oslund, P., Swaim, T., Burke, J., Shea, J., Reed, C. (2002) Longitudinal study of 

customers diverted through the care program: Technical report. School of Social 

Welfare Office of Aging and Long Term Care, University of Kansas , Lawrence, 

Kansas.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 128

Chapin, R., Zimmerman, M., Chapin, R., Zimmerman, M., Macmillan, K., Rachlin, R., 

Reed, C., Hickey, A., Baca, B., Wiebold-Lippisch, L., Henning, E., Oslund, P., 

Hayes, J., Katz, B., & Shea, J. (2003). Examination of the Use of Medicare Home Health 

Services and Informal Caregiving and Their Relationship to Successful Community 

Tenure and Appendices. School of Social Welfare Office of Aging and Long Term 

Care. University of Kansas, Lawrence, Kansas.

Chen, Y., & Adams Thompson, E. (2010). Understanding factors that influence success 

of home- and community-based services in keeping older adults in community 

settings. Journal of Aging and Health, 22(3): 267-291.

D'Souza, J.C., James, M.L., Szafara, K.L. & Fries, B.E. (2009). Hard Times: The Effects of 

Financial Strain on Home Care Services Use and Participant Outcomes in 

Michigan. The Gerontologist. 49(2): 154-165. 

Gaugler, J.E., Kane, R.L., Kane, R.A., Clay, T. & Newcomer, R.C. (2005). Early 

community-based service utilzation and its effects on institutionalization in 

dementia caregiving. The Gerontologist, 45(2), 177-185.

Gaugler, J.E., Duval, S., Anderson, K.A. & Kane, R.L. (2007). Predicting nursing 

home admission in the U.S.: a meta-analysis. BMC Geriatrics, 19(7), 13-27.

Gaugler, J.E., Yu, F., Krichbaum, K., & Wyman, J.F. (2009). Predictors of nursing home 

admission for persons with dementia. Medical Care, 47(2), 191-198.

General Accounting Office (2005). Means-Tested Programs: Information on Program 

Access Can Be an Important Management Tool. GAO-05-221.

Kaye SH, LaPlante MP, Harrington C. (2009). Do Noninstitutional Long-Term Care 

Services Reduce Medicaid Spending? Health Affairs, 28(1):262- 272.

Karuza, J. & Wu, I-Hsin. Administration on Aging. (2011). Cross validation of nursing 

home placement prediction models: Findings from the advanced performance 

outcome measurement project.

Krout, J.A., Oggins, J., & Holmes, H.H. (2000). Patterns of service use in a continuing 

care retirement community. Gerontologist, 40(6), 698-705.

Macmillan, K.R., Rachlin, R., Chapin, R., Chandran, D., Leedahl, S., Baca, B., 

Zimmerman, M., & Oslund, P. (2007). The Community Tenure Study: Community 

Tenure Status of CARE Assessment Customers 60 Months after Diversion. School 

of Social Welfare Office of Aging and Long Term Care. University of Kansas. 

Lawrence, Kansas. 

Miller, S.C., Prohaska, T.R., Furner S.E., Freels S., Brody J.A., & Levy P.S. (1998). Time to 

nursing home admission for persons with Alzheimer’s disease: the effect of health 

care system characteristics. Journal of Gerontology Series B. 53(6):S341-53.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix B: Study Framework and Design

Appendix B: Study Framework and Design 129

Miller, E.A. & Weissert, W.G. (2000). Predicting elderly people's risk for nursing home 

placement, hospitalization, functional impairment, and mortality: a synthesis. 

Medical Care Research and Review, 57, 259-297.

Miller, E.A., Rosenheck, R.A., & Schneider, L.S. (2012). Caregiver burden, health 

utilities, and institutional service use in Alzheimer's disease. Int J Geriatr 

Psychiatry, 27, 382-393. 

Muramatsu N., Yin, H., Campbell R.T., Hoyem, R.L., Jacob, M.A., & Ross, C.O. (2007). 

Risk of nursing home admission among older Americans: Does states’ spending on 

home- and community-based services matter? Journal of Gerontology Series B. 

62(3),169-78.

Muramatsu N., Hoyem R.L., Yin H, & Campbell R.T. (2008). Place of death among older 

Americans: does state spending on home- and community based services promote 

home death? Med Care, 46(8):829- 38. 

New York State Office for the Aging (2010). Performance outcomes measures project 

(POMP): From home care to nursing home in New York: demonstrating the 

benefits of aging network services. Albany. 

Sands, L., Xu, H., Weiner, M., Rosenman, M., Craig, B., & Thomas, J. (2008). Comparison 

of Resource Utilization for Medicaid Dementia Patients Using Nursing Homes 

Versus Home and Community Based Waivers for LTC. Med Care. 46 (4):449-453. 

Shapiro, A., & Loh, C.P. (2010). Advanced performance outcome measures project 

(pomp): Estimates of Medicaid and general revenue cost-avoidance from HCBS 

utilization (Contract No. XQ867)

Shapiro, A., Loh, C.P. & Mitchell, G. (2009). Medicaid Cost-Savings of Home-and 

Community-Based Service Programs for Older Persons in Florida. Journal of 

Applied Gerontology, 30(1): 3-21.

Spillman, B.C. & Long, S.K. (2009). Does High Caregiver Stress Predict Nursing Home 

Entry?. Inquiry, 46(2), 140-161.

Xu H., Weiner M., Paul S., Thomas J., Craig, B., Roseman M., Doebbeling, C.C., & Sands 

LP. (2010). Volume of home- and community-based Medicaid waiver services and 

risk of hospital admissions. Journal of the American Geriatrics Society, 58(1): 109-115.

Yaffe, K., Fox, P., Newcomer, R., Sands, L., Lindquist, K., Dane, K., & Covinsky, K.E. 

(2002). Patient and caregiver characteristics and nursing home placement in 

patients with dementia. JAMA, 287(16), 2090-2097.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

i

Table of Contents

Introduction ............................................................................................................... 130

Expert Feedback........................................................................................................ 130

State Discussions ..................................................................................................... 132

Limitations of the Study Design............................................................................... 133

Research Questions.................................................................................................. 135

Recommendations .................................................................................................... 138

Tables and Figures

Table 1: State OAA Data – Risks and Mitigation Strategies ...................................................132

Table 2: Modified Research Questions ..................................................................................135
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 130

Appendix C: Limitations and Recommendations

Introduction

The Administration on Aging (AoA) and The Lewin Group (Lewin) hosted an Expert 

Group meeting on December 18, 2012. After this meeting, the team held several key 

informant interviews with additional stakeholders and calls with states regarding their 

Older Americans Act (OAA) data systems. This report addresses study strengths and 

limitations identified from the following:

► Expert Feedback—Suggestions of the expert group and key informants are 

included. AoA and Lewin considered ways to address these suggestions, while also 

recognizing the limitations of secondary data available.

► State Feedback—Discussions of the implications of state feedback on the 

exploratory study design are included. While some states, or AAAs within states, 

could be prepared to participate in the design proposed, there are still many 

limitations to consider based upon the site or state data collection policies and 

procedures.

Identified limitations are followed by a multi-step recommendation to prepare AoA and 

the states to conduct this OAA study.

Expert Feedback

To determine the feasibility of conducting this study using the proposed methodology, 

the research team discussed the design with a group of experts from various fields. The 

experts identified strengths and limitations of the study design, and some suggestions 

for moving forward. Below are the major action items that resulted from the meeting:

Research Questions & Measuring Impact 

► Narrow the research questions to those answerable by this design, specifically, those 

that can be addressed using secondary data. 

► Consider alternative variables to measure program impact. For example:

 avoidable hospitalization instead of any hospitalization, 

 unnecessary ER visits,

 improved access to primary care services. 

► Be mindful in the findings that addressing unmet need may be correlated with 

increased service use; implying the program actually costs money as opposed to 

savings.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 131

► Review additional literature including work done on Tailored Care (TCARE), which 

examines delayed institutionalization due to the provision of resources to 

caregiversx. 

Define Target Population

► Clearly define the target population and study sample. For example, the study 

design should note that residents in nursing homes in both the retrospective AND 

prospective design will be omitted from the analysis.

Defining a control group

► Attempt to include total non-users of services for the control group. A potential 

data source would be those on OAA waiting lists.

OAA Data limitations

► A potential data limitation is the inability to link claims data to OAA data (many 

states do not collect personal identifier information). 

► In selecting a few individual states for this project, mindfulness must be given to the 

potential lack of minority representation.

Increased Focus on Post-Acute Services

► Consider including post-acute services as well as individuals living in assisted living 

in the study design to broaden the study sample. 

► Consider examining the relationship between OAA expenditures to Medicare 

expenditures (e.g. does an increase in OAA expenditures lead to a decrease in 

Medicare expenditures by reducing the number of hospital readmissions?). 

Concern over Private Pay & Informal Support

► Explore a means to capture the private pay HCBS user population.

► Explore a means to capture informal supports. 

► Explore linking the Medicare users under Cohort 3 to national surveys (NLTCS, 

NHATS, and HRS). However, this may not be feasible at a state level. 

► Explore linking housing data to service data. This would have capture residential 

care and assisted living consumers.

 

x http://www4.uwm.edu/tcare/
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 132

Additional Data Concerns

► Use the CMS Chronic Condition Warehouse to capture a longitudinal trajectory. 

This would minimize some of the problems associated with annual claims data. 

Items to Consider

► Reclassify this project as an exploratory study as opposed to an “evaluation design”.

State Discussions

To determine the feasibility of conducting this study across different states, the research 

team discussed specific questions and concerns about data with states that have statewide databases for their OAA programs. The question template may be found in 

Appendix C-A. These questions were identified as key in determining a state’s 

suitability for being included in an exploratory study. The types of questions asked 

focused on whether:

► The state could provide the overall unduplicated service count;

► The state uses consistent data collection methods across the state;

► The state enacted policies relatively consistently across the state;

► The state collected the appropriate personal identifying information to connect state 

OAA administrative data to Medicare/Medicaid claims data; and

► The state tracked details about informal caregiving use among OAA clients.

The research team conducted interviews with Georgia, Florida, Massachusetts, 

Minnesota, and Ohio. Table 1 captures data challenges identified, as well as suggested 

mitigation strategies. 

Table 8: State OAA Data – Risks and Mitigation Strategies

Risk Mitigation Strategy

The state does not collect Social 

Security numbers for OAA service 

recipients, making linking state OAA 

datasets with CMS data difficult

Ensure that the state, if chosen for piloting this design, 

has other identifiers that can be used to link individuals 

to Medicare/Medicaid claims data (e.g., name, gender, 

DOB, address).

Longitudinal data tracking isn’t 

available (uniformly) or reliable

We recommend excluding states lacking reliable 

longitudinal data. Some states may have migrated to 

a new system recently, in which case additional time 

would need to be spent cleaning and merging 

databases. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 133

Risk Mitigation Strategy

There is a lack of information on 

informal caregiving intensity for the 

intervention group

Ideally, this state would not be selected for the 

exploratory study; however, if the state has strong 

data otherwise, this will partially offset the limitation.

The state does not require use of a 

uniform assessment and reassessment conducted over 

standard time intervals (e.g., reassessed every six months)

Identify the necessary components from an 

assessment and ensure that these are collected. 

Determine the re-assessment time periods, if 

applicable.

The state cannot track expenditures 

data at the individual level

Ideally, this state would not be selected for the 

exploratory study; however, if the state has strong 

data otherwise, this will partially offset the limitation. 

The state has some AAAs and 

providers collecting data at a more 

advanced level than other AAAs

If an exploratory study were conducted, these 

advanced AAAs could be chosen rather than 

conducting a study among all the state’s AAAs. 

Limitations of the Study Design

General Limitations

The use of secondary data can be economical, both in terms of cost and time. It can also 

allow evaluators to look at a much larger sample size than would often be feasible if 

primary data were collected. However, evaluators have less control over the quality, 

completeness, and consistency of secondary data. More specifically, below are a few of 

the challenges and questions that should be considered when utilizing and selecting 

secondary data sources for the OAA global outcomes study:

► Validity—Is the data collected, measured and documented accurately? Is it 

measuring what it was intended to measure?

► Reliability—Do all people collecting and entering data define, collect and document 

the data in the same way?

► Change to Measures over Time—Does the data get collected in the same way over 

time or do measure/indicators change over time?

► Comparability/Consistency— Are data collected in the same way across sources 

and across agencies/localities/states (e.g., does Indiana measure health status using 

the same scale as New York and do two counties within New York measure it the 

same way)?

► Missing Data—Are data sources complete? How often are data missing? Are data 

for individuals or agencies with certain characteristics more likely to be missing, 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 134

creating a bias? Are certain people not included at all in the data (e.g., caregivers 

under 50)? What methods can be used to account for missing or incomplete data?

► Ability to Link Sources—Are there ways to accurately link data in different 

sources? Is data in the different sources compatible (e.g., collected at the same 

frequencies, measured in the same way, etc.)?

► Completeness of Data—Do the various sources available include all or the majority 

of the key variables that need to be included in the model? Can the existing data 

serve as a proxy for what is being measured in the study?

► Frequency/Length of Data Collection—Is the data collected frequently enough to 

allow for measuring the outcomes of interest? If using retrospective data, has the 

data been collected long enough to evaluate long-term outcomes?

► Sample—Does the secondary data source collect information from a large enough 

sample of the population of interest? What are the implications of the sampling 

design on usability of the data for the study?

This list highlights some of the challenges that must be considered upfront when 

determining the use of secondary data sources and which sources to prioritize. 

Additionally, all of these issues must be considered when collecting primary data. 

However, an important difference between primary data collection and secondary data 

use is that the parties involved in developing the data collection tools and processes are 

able to control many of these factors upfront. This ensures that the data collected is 

suitable for the study design.

Study Design Limitations

The proposed study design recommends a method for examining the effect of OAA 

programs on service recipients, while also using a comparison group. The lack of an 

adequate comparison group in previous studies is a limitation of the HCBS impact 

literature (Grabowski, 2006; Shapiro & Loh, 2010). In the current design, the research 

team attempted to mitigate this limitation by identifying a control group using 

Medicare claims data; however, this design is limited in examining key variables for the 

treatment and control group. Specifically, this designs limitations include: 

► State Older Americans Act (OAA) Data: A select group of states (FL, GA, MA, 

MN, and OH) were interviewed regarding their OAA Data Collection policies and 

procedures. These interviews highlighted some of the limitations of state OAA data, 

along with identifying states with more comprehensive data systems from our small 

sample. Specific limitations included that impact the study design included:

 The absence of Social Security Numbers (SSNs), which provide the most 

reliable method to match to Medicare and Medicaid claims.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 135

 Lack of information regarding the reasons for termination of services, which 

limits the ability to measure community tenure as an outcome.

► Informal Caregiver Data: As discussed in the expert group meeting and key 

informant interviews, difficulties will be encountered obtaining caregiver, due 

largely to the non-existence of such data. 

► Private Pay Data: This is a limitation for both the treatment and comparison 

groups. States do not uniformly collect information on private pay services received 

by OAA users. Also, this information would not be readily available for the 

proposed treatment or comparison groups. 

Research Questions

The following research questions for an OAA global outcomes study were proposed 

and further examined for feasibility to address the limitations identified. Currently, not 

every question can be answered by every state or organization. These questions may 

require modification depending upon the states and/or AAAs chosen by AoA to 

participate in an exploratory study (discussed in Table 2). 

Table 9: Modified Research Questions

Original 

Question

Components to be 

Measured Suggestion Explanation

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

community 

tenure?

► OAA services by 

type, intensity, 

volume, and 

duration

► Community tenure 

(months remaining 

in the community; 

nursing home 

residency)

This question may 

remain the same.

Most states track the 

beginning and end date of 

service use; however, not all 

states track the reasons for 

OAA service termination 

(e.g., NH admission). 

These reasons can be 

identified through Medicare 

and Medicaid claims data.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 136

Original 

Question

Components to be 

Measured Suggestion Explanation

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

health care 

utilization?

► OAA services by 

type, intensity, 

volume, and 

duration

► Health care 

utilization (times 

hospitalized for 

potentially 

avoidable

hospitalizations, 

visits to ER, nights 

spent in hospital, 

number of 

physician visits)

This question may 

remain the same.

Measures of health care 

utilization would be identified 

through Medicare and 

Medicaid claims data. 

Identifying outcomes would 

not be affected by OAA state 

data; however, it is important 

to choose a state with 

comprehensive data 

systems that may also keep 

detailed information on 

these outcomes for 

additional analysis. 

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

costs of care for 

older adults (e.g., 

LTSS, health care 

costs)?

► OAA services by 

type, intensity, 

volume, and 

duration

► LTC costs for both 

Medicaid and 

Medicare covered 

services

This question may 

remain the same; 

however, the state 

must already 

collect 

comprehensive 

individual 

expenditures data 

to best answer it or 

collect primary 

data as a part of 

this study design.

Some states can connect 

OAA expenditure data 

directly to the consumer 

(e.g., Mrs. Smith’s service 

use cost $20,000 over 3 

years). Others only report

service expenditures in 

aggregate. Those states 

collecting individual-level 

expenditure data may be 

best able to answer this 

question. 

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

physical, mental, 

and emotional 

health and 

wellness (i.e., 

preventive 

measures) of care 

recipients and 

caregivers?

► OAA services by 

type, intensity, 

volume, and 

duration

► Physical 

health/wellness

► Emotional 

health/wellness

► Mental 

health/wellness

This question may 

remain the same; 

however, in order 

to be answered, 

the state must 

have a more 

comprehensive 

data system that 

tracks these 

measures 

uniformly, or the 

state must collect 

primary data as a 

part of this study 

design.

Not every state assess OAA 

clients consistently across 

the state (some states allow 

AAAs and providers to use 

their own assessment 

forms). Some of these 

assessments may be selfreport, while others may be 

actual client-level 

evaluations. Additionally, 

time intervals for collecting 

this information vary 

considerably. No physical, 

emotional and mental health 

information will be available 

for the comparison group 

without primary data 

collection.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 137

Original 

Question

Components to be 

Measured Suggestion Explanation

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

unmet needs 

among older 

adults?

► OAA services by 

type, intensity, 

volume, and 

duration

► Measures in 

changes of 

perceived unmet 

need (self-report)

► Assessed unmet 

need

This question may 

not be feasible or 

may be difficult 

given available 

data. Only one 

state (GA) tracked 

this type of 

information. This 

could be answered 

if states collect 

primary data as 

part of this study 

design. 

The suggested measure for 

this question is self-reported 

and assessed unmet need. 

Some states may assess 

unmet need (including when 

developing a service plan) 

but they do not compare the 

plan to the actual service 

delivered. Or, they do not 

re-assess the consumer if 

unmet need is a selfreported measure. 

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

caregivers (e.g., 

strain, burden, 

depression, 

health, etc.)?

► OAA services by 

type, intensity, 

volume, and 

duration

► Caregiver strain

► Caregiver burden

This is feasible if 

the state conducts 

more detailed 

caregiver 

assessments or if 

they collect primary 

data as a part of 

the study.

The research team must 

ensure the state assesses 

caregivers for these factors 

(strain and burden). 

Potential concerns are:

Similar use of valid and 

reliable instruments.

Frequency of administration.

What is the impact 

of OAA-funded 

HCBS programs 

and services on 

coordination of 

services (e.g., 

care 

management)?

► OAA services by 

type, intensity, 

volume, and 

duration

► Case 

management: 

effect of case 

management, 

review services 

received before 

and after case 

management

► Potentially 

preventable 

hospital 

readmission

This question may 

remain the same; 

however, the state 

must have a 

comprehensive 

data system that 

includes more 

detailed case 

management 

information that 

includes OAA 

services for the 

individual.

Some states may collect 

detailed case management 

information in their systems. 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 138

Recommendations

While it is feasible to conduct this study, the rigorousness and generalizability of the 

study will depend on the data gathered, cleaned, merged and analyzed to overcome 

many of the data limitations described above. It will also be heavily dependent on the

strength of the chosen state’s OAA data system and timeline when the state migrated to 

a statewide data system. In addition, the time and funds allocated to conducting this 

study will also determine the study’s validity and reliability. 

Based upon the limitations of data sources available, and feedback from expert group 

participants and key informants, the research team recommends a multi-step approach 

outlined below.

Prepare for and Conduct an Exploratory Study

This recommendation will help AoA identify additional limitations with state data, test 

the feasibility of the current design, and recommend a set of core variables required to 

be collected uniformly across states for future program administration and evaluation. 

The steps below outline recommendations for approaching this study. 

Step 1: Issue RFP to conduct study

AoA issues an RFP to oversee the proposed study. The awarded research team would 

work with AoA to prepare for and conduct the study.

Step 2: Convene Technical Working Group

Within three months of the contract begin date, AoA in collaboration with the awarded 

research team establishes a Technical Working Group (TWG). A TWG can assist the 

research team to identify the minimum key variables necessary for a 

retrospective/prospective design, as proposed in the Study Framework and Design

document (Appendix B). From work conducted to date, these key variables may 

include those listed below. In many cases, states are collecting some of this information 

on service recipients already; however, the collection of this information across OAA 

service categories and AAAs is inconsistent.

► Social Security Number

► Date of Birth

► Gender

► Race/Ethnicity

► Address / Geographic Location

► Living Alone status

► Marital Status

► Informal Caregiver Availability

► Having Children

► Social Support

► Income

► Home Ownership
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 139

► Payment Method

► Caregiver or Spouse Age

► Support for Caregiver

► Caregiver Burden

► Physician Access

► Alzheimer's/Cognitive 

Impairment/Dementia

► Comorbidities/Chronic Medical 

Conditions

► Psychiatric Issues/Problematic 

Behavior

► ADL Limitations

► IADL Limitations

► Self-health ratings

► Unmet need measures

► Hospital Admissions/Use

► Prior / Current HCBS Use (OAA 

and non-OAA)

► Nursing Home Use 

Service use should be tracked by a combination of services, timing of care, volume, and 

intensity. Additionally, collection of expenditures data on the individual level is an 

important variable for identifying cost savings. 

It is also important to identify the effect of policy variables. The research team would 

be tasked with identifying and collecting this data, which should include: 

► NH Bed Supply

► Number of Home Health Agencies

► Medicaid HCBS Spending

► State HCBS Spending

► Pre-admission screenings

Step 3: Issue RFP to states

Prior to the completion of step 2, AoA issues an RFP to identify up to three states 

willing to participate in organizing existing data at the state level. Criteria for this RFP 

should include that the state:

► use a statewide data collection system or has a statewide data collection protocol in 

place used consistently across AAAs. Ideally, the data collection system or protocol 

should be in place across the state for more than 5 years;

► collect Social Security Numbers (SSNs) across the states (personal identifying 

information). If not, state must collect name, DOB, and zip code to link data to 

Medicare/Medicaid claims;

► preferably collect personal identifying information for all services, including Title 

III-C and D services. 

► collect detailed information on waitlisted individuals (if a waitlist is maintained), 

including demographic information, services needed, etc.; 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 140

► collect data beyond the standard National Aging Program Information Systems 

(NAPIS) State Program Reports (SPR) elements, such as:

 Informal Caregiver Availability

 Informal Caregiving Intensity

 Consumer Private Pay Resources

 Detailed data on an older adult in a caregiver dyad (where caregiver receives 

services)

 Reason for service termination (community tenure)

 Requiring case management notes be made available in an electronic format

► has standardized assessment protocols (e.g. tool and re-assessment intervals); 

► report on volume of services received;

► differentiates services by program and funding source;

► report expenditures data at the individual-level;

► track difference between planned and actualized service delivery to measure unmet 

need; and

► is able to demonstrate data sharing relationships with partner agencies in their state 

(e.g. State Medicaid agency)

In this RFP, it should be noted that a future step/activity, if funded, would include 

statewide collection of primary data. 

Step 4: Assist states / Collect available secondary data

Step 4 has two concurrent components that will begin upon awarding three states 

grants to conduct the study:

1) AoA and the awarded research team will work with the three selected states in 

gathering and preparing their OAA administrative data. As described in step 3, 

the selected states will have the data capacity to answer many of the study’s 

posed research questions and will be able to quickly begin implementing this 

step. A vital component of data preparation is for the states to collaborate with 

partner agencies in their state (e.g., State Medicaid agency) to consider data 

systems integration and/or data use agreements for better use of state data. 

2) While the selected States are preparing and assisting in the collection of data, the 

research team should assist the states as necessary. In addition, the research team 

will begin preparations for linking the state-level data with other national datasets. 

This will require that AoA facilitate a multi-agency collaboration, specifically, a 

strategic partnership with the Centers for Medicare and Medicaid Services (CMS) 

and other HHS agencies, Chronic Disease Warehouse (CDW), Housing and Urban 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 141

Development (HUD), and the Department of Transportation (DoT) to conduct a 

comprehensive evaluation of HCBS programs and services. Many older adults 

receive a combination of services, or experience a trajectory of service use that 

includes both AoA and CMS HCBS. They may also benefit from services provided 

through HUD, such as elderly and disabled-designated subsidized housing 

(Section 202 and Section 811). This option will better assess OAA services in 

combination with other service and the consumer’s HCBS experience overall.

Step 5: Expand work to include primary data collection

Step 5 includes expanding the work described in steps 2-4 in the selected states to 

include primary data collection, as funds and time allow. Primary data collection will 

strengthen the data by gathering detailed information on such key aspect of caregiving 

as informal supports. This step could be accomplished through either a state-wide 

random selection of OAA participants, or targeted data collection that occurs with select 

AAAs in the chosen states.

As states begin collecting the primary data, AoA and the research team should facilitate 

additional interviews and/or focus groups with representatives (state and/or local 

level) in all 50 states and territories to identify deficits in data collection and access the 

landscape for implementing national requirements for data collection. In addition to 

feedback received from the TWG in step 2, step 5 will allow for the identification of 

potential variables classified as “mandatory” for ongoing, systematic data collection 

and better understand the challenges states may face in updating current data collection 

efforts to meet new national requirements.

Step 6: Design and field the study

The research team selected in Step 1 should work with AoA and the three participating 

states in finalizing the study design and fielding the study with data collected at the 

state and federal level. The proposed design in the Study Framework and Design

should be the foundation for this design, and modified as appropriate given available 

state secondary data and any additional primary data. 

Step 7: Reconvene TWG and present findings

Once the exploratory study is conducted in the three selected states reconvene the TWG 

and present results for feedback and discussion. TWG members, in collaboration with 

AoA and the research team, will define a core set of variables that AAAs will employ to 

ensure consistent data collection across agencies. It is recommended that results from 

the study, in addition to information collected in Step 5, be incorporated into core 

variable recommendations. Once the recommendations are finalized, AoA and the 

research team prepares an OMB package for approval of new SPR data requirements 

going forward.
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 142

Step 8: Training & technical assistance for data collection

With changes to SPR data requirements, many states will require intensive T&TA as 

they make updates to their current data systems. However, some states will be more 

prepared than others, or already do, collect the core variable requirements, while others 

will require a complete overhaul. Those states with little updates required should begin 

collecting and reporting the new SPR requirements as soon as possible. Other states that 

require more effort to update data collection systems should begin collecting and 

reporting the new SPR requirements as soon as they are able. 

Suggestions for Future Research

Conducting this study can position AoA to design and implement a more 

comprehensive evaluation of OAA services in the future. Suggestions for future 

research are discussed below. 

Comparison group data collection

Depending on funding availability, facilitating surveys among comparison groups of 

interest (e.g., private pay, Assisted Living) should be considered. These surveys would 

collect information on a representative sample of non-OAA service recipients to get a 

better understanding of their informal caregiving and private pay services and would 

address the limitation of only using claims data. AoA should work with the research 

team to prepare for this data collection, completing such tasks as:

► Developing survey tools

► Pre-testing survey tools

► Developing sampling procedure

► Identifying surveying method (i.e., phone, mail, etc.)

Additionally, to address the private pay limitation, one option involves partnering with 

a managed care provider company that offers a continuum of care (e.g., health care, 

medical home, personal care, long-term care insurance). This option gives the team the 

opportunity to link private insurer data with those identified for a comparison group.

Conduct an evaluation

As all states begin collecting newly required data elements, AoA might consider using 

this newly acquired data in a national evaluation. This evaluation could result in 

definitive conclusions providing a full picture of the impacts of OAA Title III services to 

date. Specifically, AoA will better understand the impact of OAA services across the 

board in promoting positive outcomes for service recipients, including increased 

community tenure and decreased health care use. This evaluation will also examine the 
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 143

potential cost savings of OAA service use. Additionally, it will enhance capacity for 

ongoing research and provide information about the programs that AoA and the Aging 

Network can use in future program planning.

Appendix C-A: State Data Questions Guide

Interview Protocol

Interview Questions

1. Describe how your agency/organization collects data, i.e., is this handled by a 

contractor or the State itself? Do you assess data validity? How?

2. What are the State’s current policies or criteria for services? (e.g. service caps; 

state vs. local service caps; eligibility requirements; waiting list policy)

a. What information do you collect on individuals on the wait list? Is this 

the same information for those receiving services?

3. What are the units of measure used for services in each Title III Category? Does 

your state use the SPR definitions or is there deviation from those units of 

measure?

a. III-B: Personal care, Home maker, Chore, Adult day care/Adult day 

health, case management, assisted transportation, transportation, legal 

assistance, I&A, Outreach

i. Please be very clear on the I&A measures.

ii. Do you report adult day as respite?

iii. Is case management information collected electronically (i.e., case 

notes)?

b. III-C: Home-delivered meal, congregate meals, nutrition education, 

nutrition counseling, high nutritional risk, NSIP meals

i. Is there any potential confusion for how these may be measured? 

(e.g., meals served out of a senior center connected to an Assisted 

Living Facility – do the ALF residents get counted?)

ii. What meals count?

c. III-D: Health promotion [OAA – how much money spent / how many 

people are served]

i. What do you collect beyond OAA required collection?

d. III-E: Counseling/Support Group/Training, Case Management, Respite 

Care, Supplemental Services

i. Do you identify what the supplemental service is?

4. Do you track the volume of services received by each client? [be specific for 

which services and programs]

5. What data elements does your state collect? Included OAA, Waiver, etc... 

services. Of this data, what can you report?
Exploratory Study of the Global Outcomes of the

Older Americans Act Programs and Services Appendix C: Limitations and Recommendations

Appendix C: Limitations and Recommendations 144

a. Do you have a data dictionary or a data list for these elements?

b. What tools do you use for data collection? Is this consistent across the 

state?

c. Are there consistent answer categories for variables across the state? 

(services, health, demographic)

d. How integrated are your systems? Are state-funded and Waiver 

programs captured in the same system? 

6. Do all AAAs collect the same or some of the same OAA data elements? Is the 

data collection process state-wide?

a. How does the state receive data from the local level? Is it at the individual 

level or the aggregate level?

b. Do you use a statewide database? 

c. What is the timeline for local to state roll up of data, i.e., daily, monthly, 

annual?

d. Can the system produce an unduplicated count across services?

e. Does the system contain PII? Specifically, SSN or Medicare number?

f. Does the system track community tenure? What is the beginning point 

(referral, assessment, etc.) to termination (reason, nursing home, death, 

etc)?

g. Is state/local working with Medicare to track services utilization – care 

transitions, etc...?

7. Can your data be tracked longitudinally? [be specific for which services and 

programs]

8. What is the level of specificity for your OAA data? Do you separate OAA 

services out from similar non-OAA funded services (e.g., respite)?

a. How do you differentiate OAA services from non-OAA funded services 

(e.g., Medicaid)? Do you use consistent measures?

9. How do you track expenditures data?

a. Be clear on difference between expenditures and cost

10. How often do you collect data on the functional, physical, behavioral status of 

your OAA clients? Is that consistent across the state?

11. What demographic variables do you collect? Is this consistent across the state?

12. Do you track informal caregiving use?

13. Do you track unmet need?

a. Do you track utilization?

b. Do you track the difference between planned services delivery and 

actualized service delivery?

c. Do you have a system in place for ensuring that you are serving people in 

the underserved categories (rural, minority, etc)? Do you use targeting 

criteria?