Living Well Cross-Site Evaluation

Executive Summary: Evaluation Year Three

Access the full annual report and other evaluation materials here.
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 2

Executive Summary: Evaluation Year Three

Introduction

Grant Background

In the last two decades, the number of individuals with intellectual and developmental 

disabilities (I/DD) living in community settings and receiving home and community-based 

services (HCBS) has increased dramatically. While this has led to better outcomes, including 

people with I/DD making more of their own choices and being a part of the community, there 

are great variations in the accessibility of quality HCBS across the country. To address these 

variations and the associated risks, the Administration for Community Living (ACL) awarded 

eight five-year grants across two cohorts in 2017 and 2018. The goal of the grants is to build 

and implement model approaches focusing on two core components and incorporating eight 

key features to promote independence, integration, and inclusion in community life. Additional 

information about the Living Well grants is available in Volume I of the annual report.

Core Components and Key Features

Core Components

Community 

Monitoring

Community 

Capacity

Building

Core components for enhancing and assuring the independence, integration, 

safety, health, and wellbeing of individuals living in the community
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 3

Key Features

Partnerships

Initiation and coordination of partnerships 

or coalitions with local and state-level 

organizations, agencies, and other 

relevant stakeholders, including at least 

one self-advocacy organization in the 

design, implementation, and replication 

of grantee activities;

Meaningful & active engagement 

with self-advocates and families

Continuous, meaningful, and active 

engagement of self-advocates and family 

members throughout the life cycle and in 

all stages of the project; 

Evidence based practices 

for service improvements

Use of evidence based and innovative 

strategies to: (1) improve access to and 

quality of community services, (2) reduce 

and mitigate abuse and neglect, and (3) 

support empowerment, selfdetermination, and self-advocacy; 

Building capacity of direct support professionals (DSP) and HCBS providers: 

Prevention-based tools and technical 

assistance to address common needs, 

such as changing the ‘culture of abuse 

and neglect’ in HCBS settings and transferring knowledge of positive behavior;

Reducing abuse and neglect through 

community monitoring: 

Collection, analysis, and dissemination 

of data to develop and implement 

coordinated community monitoring 

that builds on existing local or state 

infrastructure and partnerships;

Addressing healthy and safety 

with data tools:

Data tools and evidence based practices 

for monitoring high-risk individuals and 

addressing reoccurring issues of health 

and safety concerns;

Program and outcome evaluation:

Process and outcome evaluation to 

analyze delivery and impact of project 

activities; and,

Sustainability

Assurance of organizational, financial, 

and/or community stability to continue 

and refine grantee work.
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 4

Grantees

2017: COHORT 1

University of Georgia Institute on 

Human Development

University of New Hampshire 

Institute on Disability

Virginia Commonwealth University 

Partnership for People with 

Disabilities

2018: COHORT 2

Alaska Governor’s Council on 

Disabilities and Special Education

University of Idaho Center 

on Disabilities and Human 

Development

Indiana Family and Social Services 

Administration

University of Missouri-Kansas City 

Institute for Human Development

Wisconsin Board for People with 

Developmental Disabilities

Findings and Analysis

ACL contracted with the Lewin Group (Lewin), as a subcontractor to New Editions Consulting, 

to conduct a cross-site evaluation of Living Well grants. Lewin collected data through 

interviews, virtual stakeholder meetings, and materials review (i.e., semi-annual reports). 

Findings and analyses from the third year of data collection are presented here.

Findings

Living Well grantees are tasked with “developing and testing one or more model approaches 

of a coordinated and comprehensive system that includes two interrelated core components 

for enhancing and assuring the independence, integration, safety, health, and well-being of 

individuals living in the community.”1 The two core components are: (1) Community Monitoring 

and (2) Community Capacity Building. Each grantee is using a detailed work plan to guide 

progress toward specific goals and objectives. As grantees achieve milestones indicated in 

their work plans, their Living Well models are taking shape in varied ways in response to the 

context in which they are being designed and implemented. 

A series of profiles summarizing grantee-specific background and experience, relevant state contextual factors, model overview, and key grant-funded activities comprise Volume II of the annual report. 

1 Living W ell-Model Approaches for Enhancing the Quality, Effectiveness and Monitoring of Home and Community-Based Services for 

Individuals with Developmental Disabilities; HHS-2017-ACL-AOD-DNIQ-0221. https://acl.gov/grants/living-well-model-approachesenhancing-qualityeffectiveness-and-monitoring-home-and-1
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 5

Cross-Site Analysis 

Lewin analyzed data to determine how grantees are implementing their models, whether they 

are meeting the goals of the grant, and whether their models impact the quality of life for 

individuals with I/DD. The analysis examines model design, activities, outputs and outcomes, 

and sustainability using the core components and key features as framework. The cross-site 

analysis is presented in Volume III of the annual report and summarized here.

Model Design

Grantees designed their models to address specific needs within the context of their state, 

and their interpretation of the term model approaches, as used in the Living Well Funding 

Opportunity Announcement (FOA), varied. Lewin analyzed data on grantees’ model approaches 

using four domains (process, structure, roles, context) to understand how grantees are 

implementing their models. Within each domain, Lewin identified emergent themes.

• Process: Implementation of the Living Well grants is an iterative process that began prior 

to the grant award and is ongoing. Grantees worked collaboratively with stakeholders, 

utilizing existing stakeholder groups, to plan and execute their grants from the pre-award 

stage. Grantees also incorporated ongoing quality assurance processes into their model 

design and worked to anticipate and adapt to barriers to implementation.

• Structure: Structure refers to the components and techniques used to implement the 

grant. This domain includes elements such as the evidence based interventions employed 

by grantees, communication, measurement, and evaluation. Grantees are continually 

adapting their project structure to meet changing needs and are strategically engaging 

stakeholders to effect progress toward their grant goals.

• Roles: This domain examines the individuals and groups responsible for implementing the 

grant activities and how their roles are organized. Grantees are utilizing a collaborative 

approach to leadership to facilitate active participation from all stakeholders. Additionally, 

while grantees are formalizing stakeholder roles though Memorandums of Understanding 

(MOUs) and setting clear expectations, grantees are also allowing roles to evolve as state 

contextual factors change.

• Context: Each Living Well grant is designed and implemented in a unique environment. The 

context domain considers internal factors, such as staffing and organizational capacity, 

and external factors, such as state policy and demographics. Grantees are planning 

for anticipated and unknown factors and are working to leverage the unique internal 

capabilities of their teams and stakeholders.

Community Monitoring

Community Monitoring refers to the development and implementation of a coordinated system 

to monitor the health and safety of individuals with I/DD living in community settings. To develop and support comprehensive community monitoring systems in their states, grantees are 

developing and implementing a wide range of activities by utilizing the eight Key Features of the 

grant. The Key Features most closely associated with this core component are:
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 6

Reducing Abuse and Neglect through Community Monitoring: Grantees are incorporating 

this key feature by assessing current community monitoring systems in their states, 

advocating for improvements to reporting systems, and working to enhance the 

quality of critical incident data reporting through training for provider staff.

Addressing Health and Safety with Data Tools: Grantees are using data tools to 

integrate and analyze data from multiple sources. While several grantees are 

utilizing existing data tools, others are developing new data tools to better address 

the health and safety of individuals with I/DD in their communities.

Partnerships: While grantees are required to partner with at least one developmental 

disability (DD) network agency, at least one state agency, and at least one 

additional partner, all grantees have robust stakeholder networks. Grantees are 

convening cross-agency partners as well as collaborating with peers and external 

experts to improve the community monitoring systems in their states.

Community Capacity Building

Community Capacity Building refers to efforts to support, develop, and build knowledge 

among individuals with I/DD and the people and systems that support them in the community. 

Grantees are developing and implementing many different activities to build capacity in their 

communities by using the eight Key Features of the grant. The Key Features most closely 

associated with this core component are:

Meaningful and Active Engagement with Self-Advocates and Families: Grantees are 

working to build knowledge and advocacy skills among individuals with I/DD and 

their families. Grantees are also engaging self-advocates and families as leaders, 

trainers, and advocates as well as incorporating their feedback in all aspects of 

their grant work.

Building Capacity of DSPs and HCBS Providers: Grantees’ efforts to build provider 

capacity focused on developing and implementing trainings (e.g. Person Centered 

Thinking © (PCT) trainings in Georgia and Dream Big training in New Hampshire) as 

well as working to expand career pathways and growth opportunities for DSPs.

Partnerships: Partnerships were also central to grantees’ efforts to build community 

capacity. Grantees using partnerships to align their grant work with existing 

stakeholder goals and strategies and foster stakeholder engagement.

Outputs and Outcomes

Grantees utilized process and summative evaluation techniques to assess their progress 

towards outputs and outcomes as well as sustainability across their grant activities. Grantees 

used different evaluation strategies to measure their progress and developed several 

approaches to ensure that their efforts are sustainable.
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 7

EVALUATIONS

• Process Evaluation: Grantees gather data regarding their processes to determine whether 

grant activities are being implemented and resulting in outputs aligned with their 

workplans. In order to measure and assess these processes and outputs, grantees are 

debriefing stakeholder meetings, conducting focus groups, and evaluating their outputs 

such as toolkits, questionnaires, and trainings to assess their impacts including reach of 

materials and number of participants at a training. Data from these evaluation activities are 

used to monitor progress as well as support improvements in ongoing grant initiatives.

• Outcome Evaluation: Outcomes are tied closely to the effectiveness of an output. Grantees 

are implementing tools to measure short-term outcomes as well as gather baseline data to 

be able to demonstrate progress toward long-term outcomes.

SUSTAINABILITY STRATEGIES

• Develop durable products: Durable products include published resources and recorded 

trainings. They are typically available online and will not require future funding to be 

accessible by their intended audiences.

• Secure funding: Efforts to secure future funding for Living Well grant activities ensures that 

the goals of the Living Well grant are incorporated into future work and that progress 

being made towards those goals will continue beyond the grant term.

• Align partner goals: Grantees are working with their stakeholders to align current and future 

initiatives with Living Well grant goals. Some grantees are also working towards a shared 

understanding of ownership of future responsibilities and continuation of specific grant 

activities beyond the grant term.

Scalability and Replicability

Through this grant, ACL seeks models that that can be scaled or replicated within and across 

states. Grantees are addressing scalability and replicability through all aspects of their models 

and all grantees reported that at least part of their model could be scaled or replicated in some 

form. Strategies varied and while some aligned closely with sustainability strategies, grantees 

also noted several challenges such as the need to ensure fidelity and accessibility. Grantees 

are using nationally available tools and frameworks and developing shared understanding 

across stakeholders to overcome these challenges.

Methodology

Evaluation Overview and Purpose

Each grantee designed one or more models integrating community monitoring and capacity 

building with the goal of enhancing health, safety, integration, and independence of individuals 

with I/DD living in the community. These models work toward:
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 8

• Enhancing the health and safety of individuals with I/DD;

• Strengthening the direct support professional workforce;

• Strengthening the role of self-advocacy in improving and assuring the quality of home and 

community-based services; and

• Increasing the coordination and capacity of states to effectively implement comprehensive 

high quality HCBS.

A set of questions focusing on quality and effectiveness, scalability and replicability, and 

sustainability guide the cross-site evaluation. These questions and a more detailed discussion 

of the evaluation methodology are included in Volume IV of the annual report.

Data Collection and Analysis Timeline 

November – December 2020: Conduct initial planning interviews

January - May 2021: Attend virtual stakeholder meetings

May – June 2021: Analyze Spring 2021 Semi-Annual Reports

May 2021: Conduct Key Informant Interviews

June 2021: Conduct Grant Leadership Interviews

July – August 2021: Analyze data and develop final report

September 2021: Poll grantees on topics for additional products. Complete report and products

October 2020

Create workplan, 

finalize OY2 products, 

and present findings

October-December 2020

Revise evaluation tools 

& conduct virtual initial 

planning interviews

January 2021

Submit final 

evaluation tools

January-May 2021

Conduct virtual 

site visits

March 2021

Modify data collection 

tools - grant leadership and key infomant 

interview guides

April 2021

Conduct key 

informant interviews 

and analyze data

June 2021

Conduct grant leadership 

interviews, analyze data, and 

attend stakeholder meetings

June-September 2021 

Report data

Conclusion

Lewin examined grantees’ model design, activities, outputs, and outcomes to understand how 

they are implementing their grant models, the extent to which they are meeting grant goals, 

and how their models impact the quality of life for individuals with I/DD. While grants are still in 

progress, clear cross-site themes are emerging and suggest the several key takeaways. 
LIVING WELL CROSS-SITE EVALUATION: EXECUTIVE SUMMARY 9

• Grantees are leveraging existing models and finding that ongoing stakeholder relationships 

are critical to project success.

• Meaningful engagement of self-advocates and family members guides project activities 

and builds the capacity of individuals and families as advocates and leaders.

• Building a stable and competent DSP workforce is critical to the health, safety, and 

inclusion of individuals with I/DD.

• Data tools help to integrate existing data sources and interpret data in order to guide 

action and evaluation progress.

• Results from internal process evaluations are useful for grantees to refine their strategies 

and determine how models might be scaled or replicated in the future. Grant outcomes 

realized within the grant period are primarily short-term and more heavily focused on the 

core component of Community Capacity Building.

While contextual factors, most significantly the COVID-19 pandemic, have changed the environment in which the Living Well grants are being implemented, the importance of addressing 

Community Monitoring and Community Capacity building remains vital to the health, safety, 

and well-being of individuals with I/DD. As grantees approach the final one to two years on 

their Living Well projects, they are increasingly focused on sustaining, scaling, and replicating 

the valuable work completed to date.