Measuring Systems 

Change: A Brief Guide

Administration for Community Living

Office of Performance and Evaluation

ACL Data Council
TABLE OF CONTENTS

Purpose 1

Systems Change: A Brief 2

What Do We Mean by System? 2

Examples of system change efforts at ACL 3

What Do We Mean by Systems Change? 5

How Can We Measure Systems Change Incrementally? 6

Elements of Systems Change 7

Sample Evaluation Questions 7

What Evaluation Methods Are Effective to Use for Assessing 

Systems Change? 10

Systems Change Evaluations: Examples From ACL 12

National Community of Practice for Supporting Families Evaluation 12

Model Approaches for Living Well Cross-Site Evaluation 13

References 16
Purpose

Many programs funded by the Administration for Community Living 

(ACL) focus on improving the systems that serve older adults and 

individuals with disabilities. This brief is intended as a resource for 

ACL staff who oversee such programs. It provides an introduction 

to measuring systems change, including how systems and systems 

change are defined and how they can be operationalized, as 

well as how to approach their evaluation. The brief also includes 

selected examples from ACL evaluations that have focused on 

systems change and that can provide additional guidance to ACL 

staff.
Systems Change: A Brief

ACL plays a critical role in helping create systems that “maximize the 

independence, well-being, and health of older adults, people with disabilities 

across the lifespan, and their families and caregivers” (ACL, n.d.). Thus, its funding 

and programs are intended to affect networks, programs, and services, as well 

as individuals. As part of this effort, ACL collects data to measure the impact and 

effectiveness of the initiatives and programs it funds.

In addition to measuring the impact of ACL’s services on individuals, it is important 

that ACL determine the extent to which its efforts lead to improvements at 

the system level. Since systems and systems change can be difficult to define 

and operationalize, measuring systems change may seem daunting. However, 

understanding and considering some basic concepts of systems will enable 

evaluators to operationalize systems change and measure changes in incremental 

steps.

What Do We Mean by System?

A system is a group of subsystems or entities that are interconnected or interrelated 

and function for a common purpose to form a complex and unified whole (Coffman, 

2007). In the context of ACL, systems consist of multiple actors, programs, policies, 

agencies, or institutions at the national, state, and local levels with the common 

goal of achieving better outcomes for older adults and adults with disabilities.

2
EXAMPLES OF SYSTEM CHANGE EFFORTS AT ACL

3

• University Centers for Excellence in Developmental Disabilities 

Education, Research, and Service are a nationwide network of 

independent but interlinked centers, representing an expansive 

national resource for addressing issues, finding solutions, and 

advancing research related to the needs of individuals with 

developmental disabilities and their families.

• The No Wrong Door System initiative is a collaborative effort of ACL, 

the Centers for Medicare & Medicaid Services, and the Veterans Health 

Administration that promotes and assists state efforts to streamline 

access to long-term services and support options for older adults and 

individuals with disabilities.

• State Grants to Enhance Adult Protective Services are demonstration 

grants to enable states to enhance Adult Protective Services systems 

statewide and include innovations and improvements in practice, 

services, data collection, and reporting.

• The Alzheimer’s Disease Programs Initiative is a competitive grants 

program open to states and community-based organization. It assists 

states to develop/expand the dementia capability of their home- and 

community-based services (HCBS) system, and it helps communitybased organizations operating within an existing dementia-capable 

system to expand the capability of the system within which they 

operate. 
continued

4

• State Councils on Developmental Disabilities work to address identified 

needs by conducting advocacy, systems change, and capacity building 

efforts that promote self-determination, integration, and inclusion. Key 

activities include conducting outreach, providing training and technical 

assistance, removing barriers, developing coalitions, encouraging 

citizen participation, and keeping policymakers informed about 

disability issues.
What Do We Mean by Systems Change? 

Systems change occurs when there are fundamental and sustained changes in 

policies, processes, relationships, and power structures, as well as deeply held 

values and norms. Positive changes in these elements can be thought of in terms 

of two key concepts: more effective pathways and more conducive institutional 

structures. 

The capacity of pathways increases with improvements in the scale, quality, and 

comprehensiveness of programs and services. Effectiveness in these areas is 

defined as follows: 

• Scale: Sufficient program supply and effective accessibility and outreach/

recruitment to meet the needs of the target population 

• Quality: Programs, services, staffing, and supports in the system that meet 

quality standards and are adequately tailored for the purpose of meeting 

client needs

• Responsiveness: Systems providing the right mix of programs, services, and 

supports to meet the diverse needs of clients

The connections (or relationships) of pathways improve with positive changes in 

linkage, alignment, and cross-system coordination. Effectiveness in these areas is 

defined as follows:

• Linkage: Clients able to access and successfully transition from one program 

or service to another

5
• Alignment: Steps or services building on one another and/or having 

complementary purposes

• Cross-system coordination: Stakeholders within a system able to pursue 

linkages and alignment that support the system goals

Institutional structures represent the structural context in which pathways are 

constructed or maintained. They become more conducive as new sets of incentives, 

constraints, and opportunities are created (e.g., through new policies, knowledge, 

or additional resource allocations) that allow and encourage stakeholders to build 

and maintain more effective pathways. 

• Incentives: Potential benefits for taking a particular action

• Constraints: Rules or limits that prevent a particular action or make an 

action difficult

• Opportunities: Conditions that enable a particular action, such as pursuing 

a goal in a new way (i.e., innovation)

How Can We Measure Systems Change Incrementally?

Conceptualizing systems as pathways and structures and understanding the 

elements that define them, then, provide the opportunity to define and measure 

change along those elements. Ideally, assessments should focus on determining 

whether systems have changed for the positive—specifically, whether pathways 

have moved from being less effective to being more effective, and whether 

structures have moved from potentially posing barriers to providing conducive 

structures. Table 1 below presents some of the questions that may be posed to 

determine whether positive systems change has occurred. 

6
Table 1. Sample Evaluation Questions for Assessing Positive Systems Change 

Capacity of Pathways

Elements of 

Systems Change Sample Evaluation Questions

Increased scale

• Has the number of program slots increased?

• Have providers been added to the system?

• Does the program/service supply match the 

needs of clients?

• Has the accessibility to programs/services 

improved?

• Has the ability to provide transportation to 

access programs/services improved?

• Are providers doing a better job of spreading 

awareness about their programs and attracting 

clients to them?

Improved quality

• Are more providers within the system 

systematically adhering to program quality 

standards?

• Are more providers within the system engaging in 

continuous quality improvement efforts?

• Are more providers within the system 

implementing and improving their fidelity to 

evidence-based programs and practices?

• Are more providers working to tailor their 

services and supports to the needs of their 

clients?

7
Elements of 

Systems Change Sample Evaluation Questions

Improved 

responsiveness

• Have service gaps been identified?

• Have changes been made to address service 

gaps?

• Is there a sufficient supply of programs/services 

tailored to the different needs of the clients?

Connections of Pathways

Elements of 

Systems Change Sample Evaluation Questions

Improved linkage

• Has the level of effort needed to enroll in a 

program/service decreased?

• Can clients easily transition from one program/

service to another?

• Has coordination among providers improved?

• Have new linkages been created, or have existing 

linkages been improved (e.g., institutionalized 

warm handoffs, data sharing among providers)?

• Are fewer clients “falling through the cracks”?

Improved alignment

• Are outcomes between programs/providers 

aligned to ensure they contribute to the same 

ultimate goals?

• Are programs/providers doing a better job of 

coordinating?

8
Elements of 

Systems Change Sample Evaluation Questions

Improved crosssystem coordination

• Have system leaders identified shared goals as 

well as ways that programs and services can link 

and align to contribute to shared goals?

• Is there better integration across the system 

components or entities?

• Has duplication of effort been reduced?

Institutional Structures 

Elements of 

Systems Change Sample Evaluation Questions

Improved 

institutional 

structures

• Is there access to additional or new sources 

of funding for evidence-based practices that 

have not been used in the past or for innovative 

programming?

• Are there less restrictive policies or more 

supporting policies to encourage programs/

providers to engage in joint planning or 

cross-training?

• Have effective services and processes been 

institutionalized?

9
What Evaluation Methods Are Effective to Use for Assessing 

Systems Change?

10

Developmental evaluation facilitates assessments of where 

things are and reveals how things are unfolding; helps to 

discern which directions hold promise and which ought to 

be abandoned; and suggests what new experiments should 

be tried. 

Gamble, 2017, p. 18

Systems tend to be complex and in a state of continuous development and 

adaptation (Preskill, Gopal, Mack, & Cook, 2014). As a result, change within 

systems may be unpredictable and nonlinear. Thus, traditional evaluation 

approaches such as summative evaluations that are based on a defined sequence 

of steps and linear chains of cause and effect in static environments may not be 

very effective for assessing systems change. Instead, evaluation approaches or 

frameworks that can account for dynamic and unpredictable environments and that 

allow for flexibility may be more effective. 

Developmental evaluation is one approach that can be effective for complex 

dynamic environments, such as systems, as it is intended to support innovation 

within a context of uncertainty. In this approach, the evaluator is not a third-party, 

objective observer but rather becomes part of the team that is working to bring 

about systems change (Hargreaves, 2010). The developmental evaluator’s primary 
role is to bring evaluative thinking into the process of development and intentional 

change and to work with strategic decision makers to interpret data and identify 

implications.

The main goal in developmental evaluation is to create intentional learning and 

iterative and adaptive processes. This will facilitate simultaneous questioning and 

learning. Developmental evaluators can use an array of data collection methods, 

such as interviews, focus groups, surveys, and observation. These methods can be 

used to describe how pathways and structures differ across time points (e.g., from 

baseline to follow-up) and to explore how specific strategies have contributed to 

identified changes. 

For additional details and tools for applying a developmental evaluation approach, 

see the resources listed in the reference list. Developmental evaluation represents 

but one strategy, and other evaluation approaches or frameworks that can be 

adopted for assessing system change also exist. For other approaches, see the 

examples listed below.

11
Systems Change Evaluations: 

Examples From ACL

NATIONAL COMMUNIT Y OF PRACTICE FOR SUPPORTING 

FAMILIES EVALUATION

12

The Community of Practice (CoP) for Supporting Families of Individuals 

with Intellectual & Developmental Disabilities was designed to help states 

build capacity to create policies, practices, and systems to better assist and 

support families that include a member with intellectual or developmental 

disabilities across the lifespan. Outcomes reported in the annual project 

reports (National Community of Practice, n.d.) and a national evaluation 

(ACL, 2019) show that, since the inception of the program, CoP participating 

states have succeeded in building capacity and making changes to their 

systems. As part of the national evaluation, the evaluation team adopted a 

framework developed by Wenger et al. (2011) to examine the link between 

system activities and outcomes through the creation of value in communities 

and networks. Applying the framework, the evaluation team examined 

activities and outcomes for five cycles of value creation: immediate 

value, potential value, applied value, realized value, and reframing value. 

Qualitative data were collected through a reporting tool, telephone 

interviews, and site visits.
13

Figure 1. Cycles of Value Creation Adopted for the CoP Evaluation.

Cycle 1

Inherent value 

of activities 

and 

interactions

Immediate Value

Cycle 2

Changes in 

knowledge 

capital

Potential Value

Cycle 3

Changes in 

practice

Applied Value

Cycle 4

Performance 

Improvement

Realized Value

Cycle 5

Redening 

success

Reframing Value

MODEL APPROACHES FOR LIVING WELL CROSS-SITE EVALUATION

Living Well grants are intended to develop and test model approaches for 

enhancing the quality, effectiveness, and monitoring of HCBS for people 

with developmental disabilities. To achieve this goal, Model Approaches 

for Living Well grants focused primarily on building the capacity of HCBS 

systems and enhancing community monitoring to prevent abuse, neglect, 

and exploitation. The cross-site evaluation examined whether grantees 

were successful in increasing coordination and capacity in their states to 

effectively implement comprehensive, high-quality HCBS. For this effort, 

the evaluation team focused on three key features of systems change: 
MODEL APPROACHES FOR LIVING WELL CROSS-SITE EVALUATION, continued

14

partnerships, engagement with self-advocates and families, and building 

competencies of direct service professionals (DSPs) and HCBS providers. 

Qualitative and quantitative data were collected through an annual 

reporting tool, interviews, and site visits. To access the multiyear evaluation 

products, see https://acl.gov/programs/program-evaluations-and-reports

(ACL, 2020).

Figure 2. Capacity Evaluation Framework for the 

Living Well Cross-Site Evaluation.

Key Feature Activity

Capacity Building

Partnership Engage new state partners

Engage specialized partners

Engagement 

with selfadvocates and 

families

Reach marginalized populations

Improve supports for individuals with dual 

diagnoses and other disability types

Develop and implement trainings for selfadvocates and families

Building 

competencies of 

DSPs and HCBS 

providers

Design and implement DSP trainings

Use technology and social media

Improve DSP recruitment and retention

Make improvement based on DSP and provider 

feedback
As noted, developmental evaluation and the frameworks applied for the CoP and 

Living Well program evaluations present some examples for how systems change 

evaluations may be approached. Regardless of which strategy is used, evaluators 

may want to consider examining changes in the elements that make up the 

pathways and institutional structures of a system to allow an opportunity to assess 

changes in incremental steps before long-term outcomes can be realized. 

15
References

Administration for Community Living. (n.d.). About ACL. https://acl.gov/about-acl

Administration for Community Living. (2019). National Community of Practice for Supporting Families 

evaluation. Annual report 2018–2019. https://acl.gov/sites/default/files/programs/2019-10/ACL_

CoP_Annual_Report_Final_SF_508.pdf

Administration for Community Living. (2020, December 28). Program evaluations and reports. 

https://acl.gov/programs/program-evaluations-and-reports

Coffman, J. (2007). A framework for evaluating systems initiatives. BUILD Initiative. 

https://www.buildinitiative.org/Portals/0/Uploads/Documents/Framework%20for%20

Evaluating%20Systems%20Initiatives.pdf

Gamble, J. A. A. (2017). A developmental evaluation primer. The J. W. McConnell Family Foundation. 

https://mcconnellfoundation.ca/wp-content/uploads/2017/07/A-Developmental-EvaluationPrimer-EN.pdf

Hargreaves, M. B. (2010). Evaluating system change: A planning guide. Mathematica Policy Research, 

Inc. https://www.mathematica.org/download-media?MediaItemId=%7BF80106C2-3849-4619-

95DB-7EBBC90ED045%7D

Latham, N. (2014). A practical guide to evaluating systems change in a human services system context. 

Center for Evaluation Innovation. https://stepup.ucsf.edu/sites/stepup.ucsf.edu/files/SystemsChange-Evaluation-Toolkit_FINAL_10-12-14_0.pdf

National Community of Practice. (n.d.). Supporting families of individuals with intellectual & 

developmental disabilities. http://supportstofamilies.org/innovations-strategies/national-copresources/

Preskill, H., Gopal, S., Mack, K., Cook, J. (2014). Evaluation complexity: Propositions for improving 

practice. FSG. https://www.fsg.org/publications/evaluating-complexity

Wenger, E., Trayner, B., & de Laat, M. (2011). Promoting and assessing value creation in communities 

and networks: A conceptual framework. Rapport 18, Ruud de Moor Centrum, Open University of the 

Netherlands. https://www.asmhub.mn/uploads/files/11-04-wenger-trayner-delaat-value-creation.pdf

16
This brief was prepared by the Office of Performance and Evaluation (OPE), Center for Policy and 

Evaluation, Administration for Community Living (ACL), U.S. Department of Health and Human Services (HHS), in partnership with New Editions Consulting, Inc., under contract number HHSP2332015

00113I/75P00119F37003

For additional information contact:

Office of Performance and Evaluation

Center for Policy and Evaluation

Administration for Community Living

330 C St SW

Washington, DC 20201

www.acl.gov/contact
Administration for Community Living

Office of Performance and Evaluation

ACL Data Council