Living Well Cross-Site Evaluation

Volume III: Analysis

Access the full annual report and other evaluation materials here.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 2

Volume III: Analysis

Introduction

The Administration for Community Living (ACL) requires each Living Well grantee to implement 

one or more model approaches for enhancing and assuring the independence, integration, 

safety, health, and well-being of individuals with intellectual and developmental disabilities (I/

DD) living in the community. Model approaches include the interrelated core components of 

Community Monitoring and Community Capacity Building and address eight Key Features.1 

The Lewin cross-site evaluation team (Lewin) collected data from the grantees through a series 

of virtual interviews, site visits, and a review of written reports submitted by the grant teams 

to ACL. Lewin then analyzed these data to determine how grantees are implementing their 

models, whether they are meeting the goals of the grant, and whether their models impact the 

quality of life for individuals with I/DD. The analysis examines model design, activities, outputs 

and outcomes, and sustainability.2 A detailed discussion of the evaluation methodology is found 

in Volume IV of this report.

Model Design

Grantees designed their models to address specific needs within the context of their state, and 

their interpretation of the term model approaches as used in the Living Well Funding Opportunity Announcement (FOA) varied (Exhibit 1). Lewin analyzed data on grantees’ model approaches 

using four domains (process, structure, roles, context)3 to understand how grantees are implementing their models. Within each domain, Lewin identified emergent themes.

Exhibit 1. Grantee Interpretation of Model Design

Grantees’ understanding of the term model approaches when they designed their projects varied 

and encompassed one or more of the following interpretations:

• An intervention that is scalable and replicable 

at the national level with high fidelity requiring 

minimal adjustments

• A framework that resembles other Projects of 

National Significance (PNS) grants

• A logic model or blueprint with defined steps 

for how to create systems change

• A set of best or promising practices

• A set of intervention strategies to build capacity and improve community monitoring

• A policy framework that can be replicated in 

other localities or disability communities

• An approach that can be replicated both as a 

whole and in part (e.g., specific interventions)

1 Each Living Well grant must address eight Key Features, as defined in the Funding Opportunity Announcement (FOA): 1) Partnerships, 2) 

Meaningful and active engagement with self-advocates and families, 3) Evidence based practices for service improvements, 4) Building 

capacity of direct support professionals and home and community-based services providers, 5) Reducing abuse and neglect through 

community monitoring, 6) Addressing health and safety through data tools, 7) Program and outcome evaluation, and 8) Sustainability.

2 This volume focuses on cross-site themes emerging from the eight Living Well grantees and features examples from their work. For 

more information about each grantee’s project, please refer to Volume II. Grantees’ projects are in various stages of implementation, but 

products and programs are hyperlinked, where possible.

3 The domains are adapted from the Change Implementation Framework by Wendy Hirsch.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 3

Process

Implementation of the Living Well grants is an iterative process that began prior to the grant award 

and is ongoing. Lewin identified the following themes from analyses of grantees’ processes.

Establish buy-in from stakeholders through collaborative planning pre- and post-award. Nearly all 

grantees convened stakeholders prior to receiving their award to review FOA requirements, 

assess needs within the state, identify key staff or stakeholders to engage in the project, and 

evaluate state contextual factors. This process generated substantial buy-in early on from key 

stakeholders on priority issues and strategies. 

Capitalize on existing stakeholder groups or project structures. All grantees capitalized on existing partnerships with key stakeholder groups or project structures 

in some capacity. Grantees that could model their grant on an existing structure 

(e.g., adapting the project structure from a recent PNS grant such as Partnerships in Employment) and those with established stakeholder groups with a 

clear vision and purpose were able to launch their grant more efficiently.

Incorporate ongoing improvement and quality assurance processes into model design. 

Grantees use continuous stakeholder feedback as the primary source of quality 

improvement data. While grantees use both formal (e.g., surveys, structured 

conversations, data) and informal (e.g., open discussion during meetings) 

feedback, they rely more heavily on natural processes for quality improvement 

over defined, concrete strategies. Grantees also cited use of the Living Well 

grant technical assistance as a strategy for improvement.

Anticipate and adapt to barriers to implementation. While some challenges were known in advance 

(e.g., state budget limitations) and others could not be expected (e.g., COVID-19 pandemic), 

grantees recognized the need to anticipate barriers and remain agile to meet changing needs 

and circumstances. 

Nearly all grantees 

convened stakeholders 

prior to receiving 

their award to review 

FOA requirements, 

assess needs within 

the state, identify key 

staff or stakeholders to 

engage in the project, 

and evaluate state 

contextual factors. 

Structure

Structure refers to the components and techniques used to implement the grant. This 

domain includes elements such as the evidence based interventions employed by grantees, 

communication, measurement, and evaluation.

Adapt the project structure to meet evolving context and need. Grantees universally identified 

the need to modify their project structure as circumstances change. Most notably, grantees 

adapted to both emerging needs and public health measures (e.g., stay-at-home orders, 

physical distancing) as a result of the COVID-19 pandemic. Grantees transitioned activities to 

virtual formats and developed materials and programs to support the health and well-being of 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 4

people with I/DD during the pandemic. Grantees also cited other structural changes, such as 

changes in their communication strategies (e.g., increased use of social media, videos, virtual 

meeting platforms), new evaluation approaches (e.g., adapting evaluation approach to virtual 

activities), and increased focus on community monitoring as other structural changes.

Strategically engage stakeholders. Grantees engaged stakeholders frequently and through 

multiple methods, such as meetings, regular email updates, and online chat. Several grantees 

engaged professional facilitators who, as neutral parties, were instrumental in keeping meetings 

on track, collecting participant feedback, and disseminating resources. Grantees found that 

soliciting verbal commitments from stakeholders on specific action items led to greater follow 

through than open-ended discussions. The COVID-19 pandemic exacerbated home and 

community-based services (HCBS) provider staffing shortages and diverted the attention of 

state agency staff, which created challenges to stakeholder engagement. 

Promising Practice: The Idaho Living 

Well team structured their grant to align 

closely with Community NOW! (CNOW!), 

a self-advocate-led workgroup designed 

to improve the developmental disability 

service system to best meet the needs of 

adults with I/DD using Medicaid HCBS. In 

2017, CNOW! issued a series of 17 recommendations to the Idaho Department of 

Health and Welfare Division of Medicaid, 

towards which the group continues to 

work. Prior to applying for the grant, the 

Idaho Living Well team met with CNOW! to 

align grant objectives with these recommendations. This collaboration allows 

the Living Well grant to leverage existing 

stakeholder relationships and structures 

(e.g., workgroup charter, committee structure) to work towards common goals.

Roles

This domain examines the individuals and groups responsible for implementing the grant 

activities and how their roles are organized. Themes emerged in this domain related to both 

leadership and stakeholder roles.

Employ a collaborative approach to leadership. Many grantees described their leadership style 

as neither fully top-down nor bottom-up. Grantees want their projects to evolve organically 

and collaboratively in response to stakeholder feedback, while recognizing the need for clear 

guidance and a distinct point of contact. Additionally, many of the grantees position themselves 

as conveners or facilitators of expert stakeholders. 

Formalize stakeholder roles. Grantees find value in having clearly defined, consistent roles for 

stakeholders. While grantees were required to establish memoranda of understanding (MOU) 

with certain partners, grantees noted the benefit of formalizing stakeholder relationships and 

setting clear expectations in this way.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 5

Allow stakeholder roles to evolve. Grantees shift stakeholder roles as their projects evolve and 

engage new stakeholders to support specific grant activities. Some stakeholders transition 

off the project (e.g., due to retirement or a new job), and some stakeholders become more 

engaged (e.g., pilot sites becoming more connected with project leadership).

Context

Each Living Well grant is designed and implemented in a unique environment. The context 

domain considers internal factors, such as staffing and organizational capacity, and external 

factors, such as state policy and demographics. Refer to Exhibit 2 for a summary of contextual 

factors experienced by each grantee.

Exhibit 2. Contextual Factors by Grantee

Contextual Factors AK GA ID IN MO NH VA WI

Existing stakeholder relationships • • • • • • • •

Understanding of state priorities and needs • • • • • • • •

Budget concerns • • • •

Existing lawsuits and settlements • •

Political climate • • •

Large state geography • • •

Unanticipated change(s) in state leadership • • •

COVID-19 pandemic • • • • • • • •

Plan for known contextual factors. All grantees accounted for known or expected 

factors in their grant design and implementation. In particular, all grantees 

considered relationships with their current stakeholders (e.g., capacity to 

engage, shared goals) and current state needs and priorities. Grantees also 

planned for state budget shortages, legislative priorities, legal action (e.g., 

lawsuit, settlement agreement), and geography.

Anticipate and capitalize on unknown factors. Grantees also recognized the importance of planning for and responding to factors that may emerge during the course 

of the grant. Most significantly, the COVID-19 pandemic changed the environment 

in which grants are implemented (Exhibit 3). While the pandemic posed many challenges, grantees also took advantage of opportunities created by increased virtual 

engagement, a heightened awareness of health and well-being, and flexibilities in 

the service system allowed as a result of relaxed regulations.

Leverage unique internal capabilities. Grantees designed model approaches that 

leverage their strengths and qualifications. Many grantees had strong relationships with state agencies prior to the grant award, which facilitates their efforts. Others noted 

their role as neutral parties within their state systems as critical to convening stakeholders and 

moving forward. Grantees’ current and prior experience with other PNS grants and national 

initiatives are also significant facilitators of success with their Living Well grants.

While the pandemic 

posed many challenges, grantees also took 

advantage of opportunities created by 

increased virtual engagement, a heightened awareness of 

health and well-being, 

and flexibilities in 

the service system 

allowed as a result of 

relaxed regulations.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 6

Exhibit 3. Impact of the COVID-19 Pandemic on Living Well Grants

From its widespread recognition in the United their stakeholders and communities. Grantees 

States in the spring of 2020, the COVID- plan to use their experience responding to the 

19 pandemic exacerbated many of the chal- COVID-19 pandemic to continue their advocacy 

lenges faced by individuals with I/DD living of improved community monitoring systems, 

in HCBS settings and the systems that sup- enact policies, increase the health and safety 

port them, elevating concerns about health and of those within the community, build the capacsafety, abuse and neglect, and social isola- ity of the direct support workforce, and support 

tion. The pandemic also highlighted the impor- community living.

tance of community living and the value of 

the direct support workforce, raising aware- Product Highlight: The COVID-19 pandemic 

ness and creating opportunities for positive raised concerns surrounding the health, safety, 

change. All Living Well grantees adapted to the and social connectedness of individuals with I/

physical distancing and safety measures put DD. The Wisconsin Living Well team created the 

in place by COVID-19, and they continued to COVID-19 Toolkit to provide guidance on how 

respond to the needs of their community in a individuals can stay healthy and safe during 

variety of ways (e.g., creating health and safety the COVID-19 pandemic as well as strategies 

resources, advocating for improvement in com- for staying connected. The toolkit also includes 

munity monitoring systems, increasing access a number of products targeted at increasto vaccinations, transitioning meetings to a vir- ing awareness of individuals’ rights and how 

tual format). The grantees reported that, by to report abuse since the COVID-19 pandemic 

transitioning their meetings to a virtual format, changed how many people with I/DD live and 

they are able to engage individuals that live in work. All resources are written in plain language 

rural or hard-to-reach areas, involve new stake- and developed by self-advocates. The toolkit 

holders, and increase participation by alleviating was piloted by more than 150 individuals across 

transportation barriers. There were challenges the Wisconsin Living Well stakeholders includexperienced while transitioning meetings to a ing self-advocates, provider agencies, and 

virtual setting, including adapting trainings and managed care organizations. The pilot included 

materials, connectivity issues, and reduced rap- an evaluation component, which was used to 

port and human connection. Moving forward, compile detailed recommendations to improve 

many grantees plan to use a hybrid approach, the toolkit and prepare it to be expanded into a 

using in-person interaction and virtual for- broader toolkit addressing abuse and neglect.

mats as most appropriate to meet the needs of 

4 Administration for Community Living (2020). ACL’s Logic Model Guidance. Retrieved from https://acl.gov/sites/default/files/

programs/2020-10/ACL%20Logic%20Model%20Guidance.pdf

Activities

Grantees engaged in activities related to the two core components of Community Monitoring 

and Community Capacity Building by addressing the eight Key Features (Exhibit 4) . Lewin 

analyzed grantees’ activities to determine the Key Features most closely associated with each 

core component and identify emergent themes. Activities are grantees’ interventions or actions 

that produce direct products (e.g., documents, number of people reached, number of services 

provided) in support of program outcomes and impact.4
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 7

Exhibit 4: Eight Key Features

Grantees incorporate eight Key Features into their Living Well models.

Partnerships

Initiation and coordination of partnerships 

or coalitions with local and state-level 

organizations, agencies, and other 

relevant stakeholders, including at least 

one self-advocacy organization in the 

design, implementation, and replication 

of grantee activities

Meaningful & active engagement 

with self-advocates and families

Continuous, meaningful, and active 

engagement of self-advocates and family 

members throughout the life cycle and in 

all stages of the project 

Evidence based practices 

for service improvements

Use of evidence based and innovative 

strategies to: (1) improve access to and 

quality of community services, (2) reduce 

and mitigate abuse and neglect, and (3) 

support empowerment, selfdetermination, and self-advocacy 

Building capacity of direct support professionals (DSP) and HCBS providers: 

Prevention-based tools and technical 

assistance to address common needs, 

such as changing the ‘culture of abuse 

and neglect’ in HCBS settings and transferring knowledge of positive behavior

Reducing abuse and neglect through 

community monitoring: 

Collection, analysis, and dissemination 

of data to develop and implement 

coordinated community monitoring 

that builds on existing local or state 

infrastructure and partnerships

Addressing healthy and safety 

with data tools:

Data tools and evidence based practices 

for monitoring high-risk individuals and 

addressing reoccurring issues of health 

and safety concerns

Program and outcome evaluation:

Process and outcome evaluation to 

analyze delivery and impact of project 

activities and,

Sustainability

Assurance of organizational, financial, 

and/or community stability to continue and 

refine grantee work.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 8

Community Monitoring

Community Monitoring refers to the development and implementation of a coordinated system 

to monitor the health and safety of individuals with I/DD living in community settings. In 2018, 

the U.S. Department of Health and Human Services Office of Inspector General (OIG) issued a 

report documenting findings from a review of federal and state requirements for reporting and 

monitoring critical incidents involving Medicaid beneficiaries with developmental disabilities 

residing in group homes. OIG’s review occurred in four states (Connecticut, Massachusetts, 

New York, and Maine) and revealed significant gaps in compliance for reporting and monitoring 

critical incidents. For example, state agencies did not ensure that critical incidents were appropriately reported, recorded, and addressed.5 Responding to these gaps is challenging; states 

often have complex and fragmented systems for community monitoring that include multiple 

agencies responsible for receiving and responding to reports of suspected abuse. States may 

also face vulnerabilities or weaknesses in their systems related to clearly defining incidents, 

balancing risk with health and safety, designing community monitoring systems that reflect the 

cultural diversity of the state, and implementing effective incident management systems.6

Grantees engaged in cross-system collaboration, data collection and analysis, and developing 

policy recommendations to improve systems of community monitoring. The three Key Features 

most closely associated with this core component are:

Reducing abuse and neglect through community monitoring

Addressing health and safety with data tools

Partnerships

Reducing Abuse and Neglect through Community Monitoring

This Key Feature focuses on the use of existing infrastructure and partnerships to 

collect, analyze, and share relevant data for a coordinated system of community monitoring. 

Lewin examined grantee activities associated with this Key Feature to surface themes, which 

are as follows:

Assess current systems of community monitoring. In collaboration with key stakeholders, grantees 

are assessing their complaint and critical incident reporting processes. Grantees are identifying 

gaps in their statewide community monitoring systems and developing recommendations 

for improvement. Grantees are also exploring circumstances leading to abuse and neglect, 

barriers to reporting and monitoring, strategies for balancing individual choice and safety, and 

the crosswalk between critical incident and Adult Protective Services (APS) data. Through 

5 U.S. Department of Health and Human Services Office of Inspector General, Administration for Community Living, and Office for Civil 

Rights. (2018). Ensuring Beneficiary Health and Safety in Group Homes Through State Implementation of Comprehensive Compliance 

Oversight. Retrieved from: https://oig.hhs.gov/reports-and-publications/featured-topics/group-homes/group-homes-joint-report.pdf

6 Bourne, M.L, Sowers, M., & Vegas, L. (2017). Health and Welfare Review: Report and Self-Assessment. National Association of State 

Directors of Developmental Disability Services. Retrieved from https://www.nasddds.org/uploads/documents/NASDDDS_Health_and_

Welfare_Review_Report_10-2017_FINAL.pdf
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 9

these processes, grantees generate policy recommendations, improve reporting practices, and 

enhance partnerships across fragmented systems.

Example 1: Wisconsin

The Wisconsin Living Well team reviewed the Wisconsin incident management system with stakeholders 

and prepared a cross-department 

recommendation report. The report 

includes comprehensive recommendations around funding, system improvement, and regulatory requirements for multiple state agencies 

including APS, Division of Quality Assurance, Office of Caregiver Quality, 

Department of Justice, and the Division of Long-Term Care.

Example 2: Idaho

The Idaho Living Well team established a Quality Assurance workgroup 

as part of their grant. This workgroup 

is identifying gaps in the state’s community monitoring system. The Idaho 

Council on Developmental Disabilities (ICDD) funded a contract with the 

Criminal Justice Department at Boise 

State University to conduct a threepart series of studies on the circumstances of abuse, neglect, exploitation, and the barriers to reporting and 

monitoring.

Advocate for improved tracking and reporting systems. Addressing gaps in systems of community 

monitoring may include tracking employees with substantiated cases of abuse and neglect, 

incident reporting among at-risk adults, and incident management systems that facilitate timely 

action in response to incidents. As a result, grantees are advocating for registries and reporting 

systems to prevent and respond to incidents of abuse and neglect.

Example 1: Idaho

The Idaho Living Well team is working 

towards implementation of a statewide registry of abuse, neglect, and 

exploitation. This registry will monitor 

individuals who, after full investigation, are found more likely than not to 

have committed an incident of abuse 

or neglect but may not be formally 

charged and convicted of a crime. 

Example 2: Wisconsin

The Wisconsin Living Well team is 

advocating to use federal recovery 

funding to invest in technology and 

infrastructure for a statewide incident 

management system to better facilitate communication between all abuse 

and neglect reporting systems in Wisconsin to allow state agencies, managed care organizations, independent 

consultant agencies, and service providers greater access to information. 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 10

Enhance the service of critical incident data through training. Effective systems of community 

monitoring rely on quality data. Grantees implemented trainings, primarily targeted to staff 

responsible for inputting and analyzing incident reporting data, which aim to standardize how 

critical incident data are used. Grantees used several strategies to implement effective training 

programs, such as soliciting input from stakeholders and standardizing training statewide. 

The Idaho Living Well team’s Quality 

Assurance workgroup met with one 

of Idaho’s Guardianship and Conservatorship Monitoring Coordinators 

to learn about the state guardianship 

system and the current process for 

reporting critical incidents. From this 

experience, the workgroup identified 

gaps within the system as well as necessary training needs for guardians 

and individuals under guardianship.

Example 1: Idaho Example 2: New Hampshire

The New Hampshire Living Well team 

is planning statewide training for those 

responsible for data input on critical 

incident reports through their Health 

Risk Screening (HRS) platform. The 

training is intended to increase accuracy and consistency of data reporting.

Addressing Health and Safety with Data Tools

Data tools monitor and address the health and safety of individuals with I/DD at increased risk (e.g., due to age or severity of disability) or issues of specific concern within a system 

(e.g., corrective action tracking system, emergency response process for unexpected deaths). 

Early in their grants, many grantees identified available data sources and explored how data could 

be analyzed, presented, and used. During this evaluation cycle, grantees reported using tools to 

collect primary data and to integrate and interpret data from multiple existing sources.

Integrate and analyze multiple sources of data. States collect a great deal of data related to the 

health and safety of individuals with I/DD; however, the data are often inaccessible, residing in 

systems that are incompatible, or collected without a clear plan for analysis or use. Grantees 

evaluated these sources of data during the initial years of the grant, and identified tools to help 

them integrate and interpret fragmented data. Grantees are using these tools to identify health 

and safety risks of individuals with I/DD and monitor quality in their states. 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 11

Example 1: New Hampshire

The New Hampshire Living Well team conducted a crosswalk of data used by their state 

for oversight and monitoring (e.g., OIG’s key components of health and safety compliance 

oversight, sub-assurance monitoring requirements to assure quality and effectiveness for 

Medicaid HCBS waivers, file review audits, provider re-designation process). The team 

used this crosswalk to establish a critical incident management reporting system through 

the Health Risk Screening (HRS) platform and in alignment with sentinel events and mortality reporting. This platform will gather data to run monthly analysis reports and identify 

trends. The system will also be used to report performance measures, for example, by the 

state Bureau of Developmental Services (BDS) to the Center for Medicare & Medicaid Services (CMS) for HCBS waiver performance.

Example 2: Virginia

The Virginia Living Well team is examining predictors and buffers of risk in 

the state by merging new datasets on 

abuse and exploitation crises with existing datasets, including the National 

Core Indicators (NCI). NCI is a voluntary effort by public developmental 

disability (DD) agencies to track performance by examining key indicators 

through annual surveys. The team 

worked with state agency partners to 

embed risk questions on eight key indicators of an early death or hospitalization in the NCI surveys, and they are 

in the second year of collecting data.

Example 3: Missouri

The Missouri Living Well team continues to develop its data dashboard 

by integrating data from numerous 

sources (e.g., NCI, Centers for Disease Control and Prevention, and 

U.S. Census Bureau) to promote 

data-driven decision-making. The 

dashboard allows users to compare 

experiences and outcomes for people with disabilities to the population 

as a whole. The Missouri Living Well 

team continues to explore new data 

sources that may be integrated into 

the dashboard in the future.

Leverage existing data tools. A common challenge facing grantees is not a lack of data, but rather 

how the data are used for monitoring and action. Grantees reported capitalizing on existing 

tools to use data in new ways or expanding the data collected to address health and safety. For 

example, grantees are linking data tools, which allow them to take action, such as assigning 

training to DSPs based on specific incidents that occur, identifying and addressing common 

medication errors, and monitoring conditions associated with preventable injury and death. 

Grantees also established processes for stakeholder groups or project teams to review and 

interpret data to identify gaps or areas of concern.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 12

Example 1: Alaska

The Alaska Living Well team analyzes 

data from its DD Shared Vision Alignment Survey to refine their interventions. The survey, now in its third year of 

data collection, is completed by family 

members, self-advocates, providers, 

and community members and captures data on whether self-advocates 

are living meaningful lives. The project 

team noted that this survey could be 

replicable in other states and has been 

beta-tested with self-advocates. 

Example 2: Indiana

The Indiana Living Well team triangulates data between their Medicaid 

HCBS waivers and the Person-Centered Individualized Support Plans

(PCISPs) to audit quality and better 

inform their Living Well and Partners 

in Transformation steering group activities.

Partnerships

Grantees are addressing systems of community monitoring that are complex, 

fragmented, and often function in silos. While grantees are required to partner with at least one 

DD network agency, at least one state agency, and at least one additional partner, all grantees 

have robust stakeholder networks. Grantees facilitate collaboration across state agencies and 

engage with peers and experts outside of their state to improve their systems of community 

monitoring.

Convene cross-agency partners. As grantees’ understanding of their state community monitoring 

systems develop, the need for cross-agency collaboration becomes more apparent. The 

responsibilities for reporting, defining, investigating, tracking, and preventing abuse, neglect, 

and exploitation are spread across multiple agencies. For example, a state’s I/DD agency may 

define critical incidents and set policy for reporting and responding, while that same state’s 

APS agency defines abuse, neglect, and exploitation differently and uses different criteria for 

responding and substantiating reports. Additionally, providers in a state may report critical 

incidents using one or more data systems, while a state’s APS system tracks reports in a 

separate system. To address this fragmentation, grantees formed workgroups and developed 

recommendations for systems improvements.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 13

Example 1: Idaho

The Idaho Living Well Quality Assurance workgroup conducted a review of the state monitoring system for abuse and neglect reports, which included the court system and established 

subcommittees to develop training recommendations related to the monitoring of abuse and 

neglect. Representatives from ICDD, the Idaho Department of Health, Welfare Division of 

Medicaid, the Domestic Violence Council, the Idaho Coalition Against Sexual Assault and Domestic Violence, the Crime Victims Compensation Fund, Adult Protection, DisAbility Rights 

Idaho, and the Center on Disabilities and Human Development collaborate through the workgroup to develop quality assurance recommendations on abuse and neglect. 

Example 2: New Hampshire

The New Hampshire Living Well team 

maintains strong partnerships with 

the state Bureau of Developmental Services and Bureau of Quality. 

These relationships proved essential 

in the development of the critical incident management platform to meet 

multiple agencies’ needs.

Example 3: Wisconsin

The Wisconsin Living Well team reviewed the Wisconsin incident management system with stakeholders (including APS, the Wisconsin Division of 

Quality Assurance, Office of Caregiver 

Quality, the state Department of Justice, and Division of Long-Term Care) 

and shared a cross department recommendation report with the Wisconsin 

Department of Health Services in May.

Collaborate with external experts and peers. Grantees also sought help from partners outside of their 

state, including peer agencies, national organizations, and the Living Well grant technical assistance 

contractor. Living Well grantees receive one-on-one technical assistance and participate in quarterly 

meetings to share innovative practices and learn from grantees and other experts. Grantees value 

this diffusion of knowledge and resources as they hone their individual models.

The Alaska Living Well team continues to support the Alaska Mental Health Trust Authority’s efforts around adapting the Crisis Now Arizona model for Alaska. The Alaska 

Mental Health Trust Authority pledged money to support its implementation. The Crisis 

Now model is a roadmap to safe, effective crisis care that diverts people in distress from 

the emergency department and jail by developing a continuum of crisis care services that 

match an individual’s clinical needs. The Alaska team provided stakeholder input regarding this model and Alaskans with developmental disabilities.7

Example 1: Alaska

7 Crisis Now (2017). Exceptional Practices Lead to System’s Success in AZ. Retrieved from http://bhltest2.com/wp-content/

uploads/2018/05/CrisisNow-LeadToSystemSuccessInAZ.pdf
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 14

The Virginia Living Well team, in collaboration with the Living Well grant technical assistance 

contractor and the Department of Behavioral Health and Developmental Services, developed the Modules for Group Best Practices. These modules guide best practices for decision-making in systems for people with disabilities and their families participating in groups 

such as committees or councils. The modules will remain as a durable product, hosted on 

Virginia’s Living Well website, for Regional Quality Councils and other group members to 

better understand quality frameworks and data-driven decision-making.

Example 2: Virginia

Community Capacity Building

Community Capacity Building refers to efforts to support, develop, and build knowledge 

among individuals with I/DD and the people and systems that support them in the community. 

Nationally, most people with I/DD live in community settings. Among people with I/DD who 

use Medicaid-funded services, a majority (92%) receive HCBS.8 While living in HCBS settings 

(e.g. individual or family homes, small group homes) is associated with positive outcomes 

related to choice, relationships, and employment, barriers to full inclusion remain; people with I/

DD experience low rates of integrated employment and often do not fully exercise choice and 

control in their lives.9 The workforce of direct support professionals (DSP) who play a key role in 

facilitating health, well-being, and community inclusion of individuals with I/DD experience high 

turnover (51.3% weighted average), due in part to low wages, few benefits, and limited training, 

which leads to poorer outcomes for individuals receiving support.10,11 Grantee efforts focused 

on knowledge and capacity building among individuals with I/DD, their families, and HCBS 

providers through training, partnerships, and empowerment.

The three Key Features most closely associated with grantees’ progress towards the core 

component of Community Capacity Building are:

Meaningful and Active Engagement with Self-Advocates and Families

Building Capacity of DSPs and HCBS Providers

Partnerships

8 Larson, S.A., Eschenbacher, H.J., Taylor, B., Pettingell, S., Sowers, M., & Bourne, M.L. (2020). In-home and residential long-term 

supports and services for persons with intellectual or developmental disabilities: Status and trends through 2017. University of 

Minnesota, Research and Training Center on Community Living, Institute on Community Integration. Retrieved from https://ici-s.umn.

edu/files/aCHyYaFjMi/risp_2017.pdf.

9 National Core Indicators (2019). In-Person Survey: 2017-2018 Final Report. Retrieved from https://www.nationalcoreindicators.org/

upload/core-indicators/17-18_IPS_National_Report_PART_I_update_CA_entitlement.pdf.

10 National Core Indicators. (2019). National Core Indicators 2018 Staff Stability Survey Report. Retrieved from: https://www.

nationalcoreindicators.org/resources/staffstability-survey/

11 Friedman, C. (2018). Direct Support Professionals and Quality of Life of People with Intellectual and Developmental Disabilities. 

Intellectual and developmental disabilities, 56(4), 234-250. Retrieved from: https://doi.org/10.1352/1934-9556-56.5.23.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 15

Meaningful and Active Engagement with Self-Advocates and Families

Build knowledge and advocacy skills among self-advocates and families. One strategy for 

reducing incidents of abuse and neglect is increasing awareness of risk factors, indicators, and 

methods for reporting among people with I/DD. Grantees implemented trainings designed to 

build knowledge and capacity among people with I/DD on topics such as identifying safe and 

healthy relationships and guiding one’s own choices.

The Alaska Living Well team, along with the Alaska DD Collaborative Advisory Group (a 

statewide stakeholder group, which includes individuals with DD, family members, DD 

service providers, Division of Vocational Rehabilitation, Tribal Vocational Rehabilitation, 

state DD agency staff, and other community members dedicated to advancing the shared 

DD vision in Alaska), organized a March 2021 campaign called Living the Vision as part 

of DD Awareness month. The Alaska Living Well team also partnered with the Key Coalition, one of the grant’s statewide advocacy organization partners, to host its Key Campaign. The Key Campaign continued outreach and training of self-advocates, culminating 

in virtual legislative visits with Alaska’s representatives and senators over two weeks in 

February 2021, where 140 individuals (including self-advocates, families, providers, and 

advocates) spoke on living with I/DD and receiving HCBS. 

Example 1: Alaska

The Idaho Living Well team worked with CNOW! to review, modify, and approve recommendations moved forward from each of the Idaho Living Well workgroups associated 

with their grant objectives. This partnership expanded to include the Culturally Responsive Advisory Group (CRAG), which focuses on policy recommendations that are culturally and linguistically relevant to the Latino population. CRAG provides feedback on 

CNOW! meeting materials, the Quality Assurance workgroup scope of work, proposed 

research studies, DSP competencies, and other topics. 

Example 2: Idaho

The Virginia Living Well team organized three virtual advocacy events with 666 total attendees. They planned and hosted two of the events in partnership with a group called 

A Life Like Yours Self-Advocacy Alliance. The Alliance, created by the Arc of Virginia, 

is comprised of people with disabilities and advocates across twenty self-advocacy 

groups across the state. The team also developed durable materials, such as the 2021 

Advocacy Toolkit, One Page Guide to Advocacy, and Budget Hearing Guide Video for 

self-advocates.

Example 3: Virginia
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 16

Engage self-advocates and families as leaders, trainers, and advocates. In addition to educating 

individuals with I/DD about abuse and neglect, grantees are building capacity by supporting 

and engaging self-advocates as leaders. Grantees are empowering self-advocates through 

leadership roles in their projects, as co-trainers delivering trainings to providers, and as peeradvocates, working with other individuals with I/DD to build peer leadership networks.

All trainings sponsored by the Alaska Living Well team had co-instruction, including plan of care 

trainings, health safety, avoiding abuse and neglect, person-centered emergency planning, employment, the I Have Rights campaign, COVID-19 Alaska Toolkit, supported decision-making 

agreement trainings, and the Self-Advocacy Summit. Co-instruction requires a person with I/DD 

to partner with a person without I/DD in the delivery of any training activities.

Self-advocates serve as co-trainers for two trainings offered to HCBS providers through 

the Georgia Living Well grant: Supporting Informed Decision-Making training and Supporting Social Roles. The Georgia Living Well team pivoted during the COVID-19 pandemic to ask co-trainers to pre-record their segments for a virtual training, including an 

introduction video, sharing slides, and telling stories. 

Example 2: Georgia

The Virginia Living Well team continues to host Leadership for Empowerment and Abuse 

Prevention (LEAP) trainings by and for individuals with I/DD, which they adapted to a 

single 90-minute virtual training session. LEAP provides trainings to adults with I/DD on 

healthy relationships and information on how to prevent abuse to adults with disabilities. 

Additionally, four young adults from Florida participated in a pilot LEAP training to help 

tailor the curriculum to a teenage audience, which is slated to start in fall 2021. 

Example 3: Virginia

The Wisconsin Living Well team continues to deliver the Safe and Free Curriculum at 

pilot sites, with self-advocates serving as trainers about two times per year. The curriculum, developed in collaboration with People First Wisconsin, covers topics important 

for self-advocates to learn about their rights, how to avoid abuse, and how to have 

healthy relationships. The team is working towards an entirely peer-to-peer version of 

the Safe and Free Curriculum with People First Wisconsin.

Example 4: Wisconsin

Example 1: Alaska
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 17

Solicit and act on feedback from self-advocates and families. Grantees engage self-advocates and 

families as stakeholders in grant design and implementation, as trainers and advocates, and as 

participants in training and skill-building activities. This year, grantees also sought input from 

self-advocates and families on the impact of these activities, the quality of their experiences, 

and strategic direction for the future.

Example 1: Indiana

The Indiana Living Well team conducted focus groups with self-advocates and families to plan for future 

grant activities. The team also surveyed its stakeholder steering group, 

Partnerships in Transformation, using a validated tool called Factors of 

Effective Partnerships and found that 

the factors the Indiana Living Well 

team most excelled at were group 

representation (all appropriate stakeholders at the table) and communicating effectively.

Example 2: Virginia

The Virginia Living Well team was 

highlighted in VCU news for results 

of their study that showed the impacts of the LEAP training for and by 

self-advocates. After completing the 

training, the project team asked people living with disabilities to watch video vignettes and respond to vignettes 

in their own words, identifying healthy 

and unhealthy relationships. Participants became better at explaining 

why scenarios were abusive or exploitative and improved their planning 

on how to act to resolve the situation.

Building Capacity of DSPs and HCBS Providers

This Key Feature includes tools and technical assistance that help with the prevention 

of abuse and neglect by improving access to and quality of HCBS, enhancing the competency 

of DSPs, and addressing common needs within agencies (e.g., training, orientation, workplace 

culture). Grantees’ efforts to build capacity focused on training and expanded career pathways 

for DSPs.

Develop and implement training to build HCBS provider capacity. Training requirements for DSPs are 

largely determined at the state level and typically require around 40 hours of training on basic topics 

or skills (e.g., first aid, documentation); competency-based training is rarely required. Yet, DSPs play 

a critical role supporting individuals to reach their desired outcomes, and require a high degree 

of knowledge and skill to do so successfully.12 Grantees seek to build workforce capacity through 

trainings that address these vital skills, such as social roles, choice, and relationships.

12 Hewitt, A., Macbeth, J., Merrill, B., & Kleist, B. (2018). The Direct Support Workforce Crisis: A Systemic Failure. Impact, 31 (1). 

Retrieved from https://ici.umn.edu/products/impact/311/Systemic-Failure/#Systemic-Failure
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 18

Example 1: Alaska

The Alaska Living Well team launched a Community Relationship Building Program in 

partnership with the DD service provider agency. The program encouraged individuals 

with I/DD to build natural support connections that could be maintained during the 

COVID-19 pandemic social distancing mandates. Cohort members focused on maintaining connections with friends and family through virtual platforms. The program virtually convened the initial training cohort monthly to discuss adaptations for maintaining 

social connections during the pandemic.

Example 2: Georgia

The Georgia Living Well team offers a Supporting Social Roles training that explains 

social roles and offers strategies for DSPs to support individuals with I/DD to make 

and sustain connections to foster community inclusion. The trainings transitioned to a 

virtual platform during the COVID-19 pandemic and will continue for provider partners 

for the remainder of the grant. The Georgia team also offers professional development 

for DSPs through the College of Direct Support. In early 2021, there was a significant 

increase in the number of lessons assigned compared to early 2020 showing the importance of online learning throughout the COVID-19 pandemic.

Example 3: New Hampshire

The New Hampshire Living Well team partnered with the New Hampshire Council on 

Developmental Disabilities, ABLE New Hampshire, and People First of New Hampshire 

to provide seven virtual trainings for DSPs called Dream Big: Tools for Supporting Individual Choice and a Robust Community Life. Survey results from the 117 participants 

indicate a very high level of satisfaction with the trainings. To sustain the Dream Big 

training efforts, the New Hampshire Living Well team plans to record the trainings and 

host them on RELIAS (New Hampshire’s online training platform), create a step-by-step 

manual of how to implement the training that corresponds with the accompanying slide 

deck, and identify personnel within area agencies and private provider networks that 

are willing to participate in a train-the-trainer workshop.

Example 4: Virginia

The Virginia Living Well team conducted Person Centered Thinking (PCT) trainings between October 2020 and March 2021 for 295 people, primarily HCBS provider staff, 

to build competency on person-centered values and principles. The team evaluated 

participants’ knowledge using pre- and post-tests in eight core PCT skills and found a 

statistically significant improvement in knowledge for each skill.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 19

Final Product Highlight: New Hampshire Medication Administration Training

Early in their grant design, the New Hampshire 

Living Well team identified medication administration as an important factor for promoting health and safety among people with I/DD 

using HCBS. Additionally, the team learned that 

the training was only offered in English; many 

DSPs who did not speak English as a first language struggled to meet this foundational 

requirement for their jobs, and were more likely 

to make medication errors that resulted in the 

revocation of medication privileges and eventual termination from their positions. The team 

updated the state’s medication administration 

training, reduced the amount of technical and 

clinical language, and translated it into Spanish 

and Nepalese, the two most commonly spoken 

languages within the workforce after English. 

When the COVID-19 pandemic prompted closures and physical distancing requirements, 

the state DD agency sought help converting the 

training to a virtual format. Within four days, the 

New Hampshire Living Well team responded to 

the request and produced the training virtually, 

including a series of videos created by self-advocates, a state Developmental Disability nurse, 

and home provider. The New Hampshire Living 

Well team intends to share the training with 

other states. The goal is that the new training 

with improved accessibility will help attract and 

retain providers in the DSP workforce.

Establish growth opportunities for DSPs. The national average wage for DSPs is under $12.00 

per hour, and about half of DSPs use publicly funded benefits (e.g., food, housing, or medical 

assistance).13 DSP turnover rates vary widely, with an average annual turnover of 51%.14

Providing career growth opportunities is important to both the personal well-being of DSPs, 

as well as organizational stability. Grantees are seeking innovative strategies to enhance DSP 

wages and construct pathways for career growth and development.14

Example 1: Idaho

The Idaho Living Well team conducted a statewide survey to collect information from 

DSPs, provider agencies and individuals with I/DD on issues related to wages, training, 

recruitment, and retention of DSPs in the state. The team is also looking how wage 

increases implemented by increased federal funds may impact these issues and influence the DSP wage advocacy work of the bFair 2DirectCare workgroup. 

Example 2: New Hampshire

The New Hampshire Living Well team worked with Community Support Network, Inc., the 

association of New Hampshire’s ten area agencies that provide services to people with DD, 

to develop the Care and Support program. The program provided 11 high school students an 

introduction into the role of a DSP over the course of an 18-week multi-modal program. Upon 

completion of the program, the students have all of the necessary required training to apply 

for and become DSPs, helping to build capacity of the workforce. Of the 11 participants, three 

were hired by agencies as DSPs upon the completion of the program.

13 Hewitt, A., Macbeth, J., Merrill, B., & Kleist, B. (2018). The Direct Support Workforce Crisis: A Systemic Failure. Impact, 31 (1). 

Retrieved from https://ici.umn.edu/products/impact/311/Systemic-Failure/#Systemic-Failure

14 National Core Indicators. (2019). National Core Indicators 2018 Staff Stability Survey Report. Retrieved from: https://www.

nationalcoreindicators.org/resources/staffstability-survey/
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 20

Partnerships

The Key Feature15 of partnerships emerged as significant to both core components of 

the Living Well grant. Specific to Community Capacity Building, grantees explored opportunities 

to align their work with partners to enhance the sustainability of their grants. Grantees also 

sought concrete roles for partners and solicited commitments to their work.

Align partnerships with existing goals and strategies. One strategy for sustaining grant initiatives 

after the award ends is to align with existing goals or efforts in the state. Regular communication with partners emerged as another important strategy to building relationships and aligning 

goals. Grantees achieved this through regular meetings, as well as co-planning for the future.

Example 1: Alaska

The Alaska Living Well Project Director co-chairs the Alaska Work Matters Task Force, 

which brings together representatives from the state DD agency, transportation, corrections, juvenile justice, disability groups, individuals with lived experience, and employers. The task force is working to understand employment barriers to make policy 

recommendations to the legislature in February 2022.

Example 2: Virginia

Staff from the Virginia Living Well team joined the Integrated Housing Advisory Committee to highlight inequities in supported living services to Virginia’s Department of 

Justice, including recommendations from A Life Like Yours Self-Advocacy Alliance. The 

recommendations spanned topics from the right to hold a lease, the right to control who 

owns a key to the home, and the ability to change HCBS providers.

Example 3: Missouri

Five county-based pilot sites participating in Missouri’s Living Well grant engaged in 

monthly strategic planning sessions. The planning informed material for Missouri’s Living Well product library to promote scaling and replication of emerging and promising 

practices across the state.

Foster stakeholder engagement and ownership through concrete roles. Grantees formed crossagency workgroups, subcommittees, and steering bodies to generate and implement 

recommendations. Fundamental to this strategy is a clear purpose and charge for each group’s 

members, as well as the authority to take action.

15 Administration for Community Living (2020). ACL’s Logic Model Guidance. Retrieved from https://acl.gov/sites/default/files/

programs/2020-10/ACL%20Logic%20Model%20Guidance.pdf
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 21

Example 1: Idaho

The Idaho Living Well team organizes grant activities using a workgroup 

structure. This year, the Quality Assurance workgroup broke into subcommittees to address identified 

gaps in knowledge related to reporting and monitoring in the state. Each 

subcommittee has a clear purpose 

and roles. Subcommittees met with 

the Self-Advocacy Training workgroup to learn from self-advocates 

about topics they think are important 

to cover in trainings related to each 

topic area. Members of the Idaho’s 

CRAG also joined the Quality Assurance subcommittees.

Example 2: Indiana

The Indiana Living Well team is the 

sole state-agency grantee among 

all Living Well grantees in both cohorts. They established a stakeholder steering committee, Partners in 

Transformation, to provide collaboration and guidance. While the Indiana team sought a collaborative approach, stakeholders looked to the 

state agency to lead and direct the 

work. The Indiana Living Well team 

organized subgroups to increase 

ownership by stakeholder and expedite the work.

Outputs and Outcomes

Outputs are measurable, direct products or activities that contribute to outcomes. Outputs may 

include documents and materials developed, number of people reached, or number of services 

provided, but do not address the value or impact the products or services. Short-term (or 

immediate) outcomes include changes in attitudes, knowledge, or skill. They lead to changes 

in areas such as behavior, decision making, and policies (i.e., intermediate outcomes). These 

outcomes are a precursor to achieving the desired long term outcome or impact of the grant, 

which is community- and system-level change.16 Grantees assessed their progress towards 

outputs and outcomes by addressing the Key Feature of program and outcome evaluation.

Program and Outcome Evaluation

Each grantee is required to use process and summative evaluation techniques to 

analyze how the project is being delivered, the extent to which the project is being 

implemented as planned, and the effectiveness of their model in achieving project outcomes. 

Grantees’ evaluation methodologies vary, and evaluation activities are ongoing. 

Process evaluation. Process evaluation determines whether grant activities are implemented as 

planned and result in desired outputs. Grant outputs include, but are not limited to, stakeholder 

engagement activities (e.g., meetings, surveys), resources (e.g., toolkits, websites), trainings 

and curricula, data tools, and assessments (e.g., mapping current reporting systems, assessing 

systems for gaps). Results of grantees’ process evaluations enable them to track and report on 

progress, identify and address potential barriers, and improve future grant activities. 

16 Administration for Community Living (2020). ACL’s Logic Model Guidance. Retrieved from https://acl.gov/sites/default/files/

programs/2020-10/ACL%20Logic%20Model%20Guidance.pdf
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 22

Grantees’ internal evaluators use methods such as surveys, focus groups, questionnaires, 

debrief sessions, and tracking forms (e.g., training or meeting frequency and attendance) 

for their process evaluations. All grantees engage their internal evaluators in regular (e.g., 

quarterly) project meetings to share data on how and to what extent project activities are being 

implemented as planned. The data from these evaluation activities are used to monitor progress 

and support improvements in grantees’ initiatives.

Example 1: Wisconsin

The Wisconsin Living Well team evaluates whether their stakeholder meetings are inclusive and accessible to self-advocates through debrief sessions with their leadership 

team after each meeting. One debrief session revealed that self-advocates experienced 

difficulty understanding dense material presented at the meeting. The Wisconsin Living 

Well team responded to this finding by working with speakers prior to each meeting to 

ensure presentations are delivered in plain language and by assigning a team member 

to monitor chat and respond to any questions from self-advocates.

Example 2: Indiana

Indiana Living Well Leadership team also meets with key partners before and after each 

Partners in Transformation stakeholder meeting to plan for the session, and debrief 

whether it met its intended purpose and objectives. The team assesses elements such 

as the overall impact of the meeting, questions asked, and stakeholder engagement.

Example 3: Georgia

The Georgia Living Well team administers annual surveys to provider staff at their pilot 

sites. The survey collects data on the type and nature of services provided, participation in staff training, staff retention and turnover rates, and staff pay, bonuses, and incentives. As they collect the surveys each year, their evaluation team can analyze trends 

in staff capacity and share the data with providers to review current status and growth 

on key staff indicators.

Outcome evaluation. Outcome evaluation measures the extent to which grants are achieving the 

desired effect and meeting their stated objectives. Long-term project outcomes for Living Well 

grants include increased independence, self-determination, community integration, health, and 

well-being of individuals with I/DD. Short- and intermediate-term outcomes include improved 

community monitoring and a reduction in risk and prevalence of critical incidents (e.g., abuse, 

neglect) among individuals with I/DD using HCBS. Additional short- and intermediate-term outcomes include enhanced engagement and leadership of self-advocates and family members, 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 23

increased capacity of states to provide HCBS in integrated settings and respond to critical 

incidents, and enhanced knowledge and skills of DSPs leading to improved workforce stability.

To date, grantees report more outcomes relating to the core component of Community 

Capacity Building than the core component of Community Monitoring. This aligns with 

grantees’ activities (refer to Appendix A for a summary of activities by core component), is 

consistent with the type of data that grantees are using to measure outcomes, and reflects the 

unique nature of each of the two core components. Most internal evaluation findings during this 

evaluation period focus on short-term outcomes, such as increased knowledge and awareness 

of rights or enhanced DSP skills, which are expected to lead to longer term impacts. 

Grantee outcomes related to the core component of Community Monitoring focus on enhanced 

health and safety for individuals with I/DD. Two grantees reported short-term outcomes related 

to improving access to the COVID-19 vaccine through vaccination events and collaboration 

with grant partners. Many grantees are also working towards improvements in their respective 

states’ incident reporting platforms to improve community monitoring and improve health and 

safety. While grantees report progress on these initiatives, the outcomes they are working 

towards are long-term and require ongoing effort and evaluation.

Grantee outcomes related to the core component of Community Capacity Building focus 

on increased capacity of self-advocates and their family members, as well as the capacity 

of provider agencies and DSPs. Grantees measured and reported outcomes in this area by 

showing growth in leadership capacity and increased knowledge through post-activity or posttraining surveys. This increased capacity is expected to contribute long-term outcomes, such 

as increases in community living rates, which are still in progress. 

Example 1: Wisconsin

The Wisconsin Living Well team and 

their pilot sites are using the Council 

on Quality and Leadership Personal 

Outcome Measures (POM) survey to 

assess progress among people using HCBS towards personally identified outcomes. This survey was completed as a baseline measure and will 

be repeated in 2023 to show growth 

and challenges in supports provided 

and personal outcomes.

Example 2: Missouri

The Missouri Living Well team assisted 

their state with a rapid implementation 

of StationMD during the COVID-19 

pandemic. StationMD is a telemedicine platform for individuals with I/DD 

aimed at reducing emergency department visits. The Missouri Living Well 

team is evaluating the return on investment to determine the effectiveness 

of the platform and whether the state 

should continue to provide access to 

StationMD for individuals enrolled in 

Medicaid HCBS I/DD waivers in the 

state. For example, the Missouri team 

is examining outcomes, such as reduced emergency department visits.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 24

Sustainability

Sustainability is a Key Feature required of Living Well grantees and integral to the 

success of their projects. All grantees worked to address sustainability of their projects and 

activities in at least one way. Key strategies include developing durable products, securing 

future funding, and aligning partner goals and future responsibilities. 

Develop durable products. Durable products, including published resources and recorded 

trainings, are typically available online and will not require future funding to be accessible 

by their intended audiences. These products and recorded trainings are accessible to their 

intended audiences via online platforms and will continue to be accessible beyond the term of 

the grant. Examples include:

• The New Hampshire Living Well team’s medication administration training;

• The Virginia Living Well team’s Regional Quality Council recorded training modules;

• The Georgia Living Well team’s trainings and career tracks on the College of Direct Support 

platform; and,

• The Wisconsin Living Well team’s Healthy, Safe, and Connected COVID-19 toolkit. 

Secure funding. Securing future funding is also a strategy for sustainability beyond the term of 

the Living Well grants. Efforts to secure future funding for Living Well grant activities ensure 

that the goals of the Living Well grant are incorporated into future work and that progress being 

made towards those goals will continue beyond the grant term.

Example 1: Alaska

The Alaska Living Well team is working to secure future funding for specific activities such as their Supported Decision-Making Agreement pilot 

as well as embedding key aspects 

of their grant work into the state DD 

Council’s next five-year plan. 

Example 2: Georgia

The Georgia Living Well team is developing a proposal for short-, mid-, and longterm interventions to grow and sustain 

the direct support workforce given the 

federal funding from the American Rescue Plan through the state Department 

of Community Health.

Align partner goals. Finally, multiple grantees are working with their partners to align Living Well 

grant goals with existing state initiatives. Some grantees are also working towards a shared 

understanding of ownership of future responsibilities and continuation of specific grant activities 

beyond the grant term. 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 25

Example 1: Alaska

The Alaska Living Well team continues to ensure that the DD Shared Vision, a common 

vision statement passed into state statute, is at the center of their grant activities. This 

ensures that their grant work aligns with all other stakeholders in the state and will support future efforts. 

Example 2: Indiana

The Indiana Living Well team has spent considerable time forming a stakeholder group 

and working through the Charting the LifeCourse Ambassador series together. This 

framework helped the team establish their Partners in Transformation workgroup with 

a shared understanding of their goals as they move forward with their grant work and 

look beyond the term of the grant. 

Example 3: Idaho

The Idaho Living Well team leveraged and built on strong partnerships with stakeholders that are committed to the long-term vision of the CNOW! initiative. The team 

plans to implement sustainable self-advocate train-the-trainer models to conduct statewide trainings on specific topics including: self-advocacy, self-determination, individual 

rights, supported decision-making, principles of person-centered thinking, how to report abuse, neglect, and exploitation, and post-secondary employment and education 

opportunities.

As grantees continue into the final years of their grants, sustainability strategies will become 

more important and are expected to continue to evolve. 

Scalability and Replicability 

Through this grant, ACL seeks models that can be scaled or replicated within and across states 

to improve the lives, health, well-being, and self-determination of individuals with I/DD. Grantees 

are addressing scalability and replicability through all aspects of their models, including their 

overall model design and specific activities and interventions. All grantees reported that at 

least part of their model could be scaled or replicated in some form, but their approach and 

interpretation of scalability and replicability varies. One grantee noted that, while parts of their 

model could be replicated in other settings, each part of their model complements and builds 

off of the other parts and the entire model may be necessary to achieve progress towards 

the desired outcomes. Another grantee noted that, while parts of their model could be scaled 

or replicated elsewhere, many of their activities are dependent on and tailored to the unique 

context present in their state. 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 26

Grantees also noted several specific strategies towards achieving scalability and replicability. 

More than half of the grantees are considering the importance of ensuring understanding 

among stakeholders before attempting to replicate or scale specific initiatives or activities 

in order to ensure the fidelity of the activity once it is scaled. Another strategy to overcome 

challenges related to scalability and replicability is use of a nationally available tool or platform. 

Examples include Therap, Charting the LifeCourse, and the NCI survey. Using these tools 

and frameworks ensures that other organizations or stakeholders have access to the platform 

or tool and can then follow the grantee’s model for implementing the tool while also making 

adjustments for their own specific state context. 

Conclusion

Lewin examined grantees’ model design, activities, outputs, and outcomes to understand how 

they are implementing their grant models, the extent to which they are meeting grant goals, 

and how their models impact the quality of life for individuals with I/DD. While grants are still in 

progress, clear cross-site themes are emerging and suggest the following key takeaways.

Model Design

• Leveraging existing stakeholder relationships and project models contributes to efficient 

implementation of project activities. 

• Ongoing stakeholder relationships are critical to project success, and stakeholder roles 

should be clear and defined, strategic, and adaptive. 

• Grantees must plan for known and unknown contextual factors and remain flexible to 

changing circumstances or opportunities.

Activities

• Robust partnerships, both within and across states, facilitate innovative solutions and 

effective implementation of project activities. In particular, meaningful engagement of 

self-advocates and family members guides project activities and builds the capacity of 

individuals and families as advocates and leaders.

• Building a stable and competent DSP workforce is critical to the health, safety, and 

inclusion of individuals with I/DD. 

• States typically have access to numerous data sources related to abuse, neglect, and 

well-being of individuals with I/DD. Data tools help to integrate and interpret data in order to 

guide action and evaluation progress. 

Outputs and Outcomes

• Results from internal process evaluations are useful for grantees to refine their strategies 

and determine how models might be scaled or replicated in the future. 
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 27

• Grant outcomes realized after three and four years of implementation (for cohorts two and 

one, respectively) are primarily short-term and more heavily focused on the core component of Community Capacity Building. Achieving systems change outcomes generally takes 

longer than the five-year grant period and requires long-term sustainability strategies.

While contextual factors, most significantly the COVID-19 pandemic, have changed the environment in which the Living Well grants are being implemented, the importance of addressing 

Community Monitoring and Community Capacity building remains vital to the health, safety, 

and well-being of individuals with I/DD. As grantees approach the final one to two years on their 

Living Well projects, they are increasingly focused on sustaining, scaling, and replicating the 

valuable work completed to date.
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 28

Appendix A: Grantee Activities by Core Component

Grantee Sample Activities Community 

Monitoring

Capacity 

Building

Alaska

Key Campaign •

Community Relationship Program Pilot (continues but largely paused 

due to COVID-19)

•

SDMA Pilot and Symposium •

Person-Directed Plan of Care Trainings •

I Have Rights Campaign •

Person-Centered Emergency Training •

Alaska COVID-19 Toolkit •

NCI Survey • •

DD Alignment/Shared Vision Survey •

Georgia

Therap – Collect and Analyze Data •

College of Direct Support Training for DSPs •

Annual DSP Staff Survey (data submission was delayed due to 

COVID-19)

•

Supporting Informed Decision-Making Training •

Supporting Social Roles Training •

Idaho

QA Training Subcommittees •

Recommendations to Retain a Quality DSW •

Analyze Abuse/Neglect Reporting System •

Caregiver Registry of Abuse •

Three Studies – Abuse and Neglect •

CRAG – Diversity & Inclusion/Cultural Competence •

Community Now! •

Indiana

CtLC •

Align Efforts Across Systems •

Conduct Trainings and Events •

Explore Data Collection Systems and Strategies •

Quality Indicators and Surveys •

Missouri

CtLC for Systems Transformation Pilot •

Virtual Data Dashboard •

Therap •

Quillo •

Missouri Quality Outcome Video series •

StationMD •

Innovation Collaboratives (will not move forward with second cohort) •

New 

Hampshire

Critical Incident Management through HRS •

HRS Platform Implementation •

Medication Administration Training •

Care and Support DSWs Pilot with Technical High School •

Dream Big: Tools for Supporting Individual Choice & Robust Community 

Based Life for DSPs

•
LIVING WELL CROSS-SITE EVALUATION: ANALYSIS 29

Grantee Sample Activities Community 

Monitoring

Capacity 

Building

Virginia

Coalition for Community Safety •

Trend Reports •

NCI and abuse data analysis •

RQC Training Modules •

LEAP Trainings •

Person-Centered Thinking Trainings •

Discovering Me! •

CHAT •

F2F Trainings •

Wisconsin

Incident Management System Review and Policy recommendations on 

reporting systems

•

Safe and Free Curriculum •

COVID-19 Toolkit •

Living Well Wednesdays

Pilot Sites and Intervention Plans •