1

National Community of Practice for Supporting 

Families Evaluation

Annual Report 2018-2019

Prepared for: Administration for Community Living 

Office for Performance and Evaluation

Submitted by: New Editions

The Lewin Group

September 24, 2019
2

Contents

Executive Summary .........................................................................................................................3

Introduction and Background ..................................................................................................3

Evaluation Design ....................................................................................................................3

Methodology .............................................................................................................................4

Data Collection Tools...............................................................................................................5

Findings ....................................................................................................................................5

Conclusion and Recommendations..........................................................................................7

Introduction and Background ..........................................................................................................8

National Community of Practice for Supporting Families......................................................8

Evolution of the Charting the LifeCourse Framework............................................................9

Evaluation Purpose and Goals ..............................................................................................10

Methodology ..................................................................................................................................10

Evaluation Design ..................................................................................................................10

Data Collection.......................................................................................................................13

Communication.......................................................................................................................14

Data Analysis..........................................................................................................................14

Findings..........................................................................................................................................15

Cycle 1.....................................................................................................................................15

Cycle 2.....................................................................................................................................17

Cycle 3.....................................................................................................................................19

Cycle 4.....................................................................................................................................21

Cycle 5.....................................................................................................................................23

Conclusion and Recommendations ...............................................................................................24

Evaluation Limitations ...........................................................................................................24

Recommendations...................................................................................................................25

Next Steps................................................................................................................................26

Appendix ........................................................................................................................................27

Appendix A: Data Collection Activities Timeline..................................................................27

Appendix B: Reporting Tool Survey for Groups 1-3 .............................................................28

Appendix C: Telephone Interview Guide...............................................................................38

Appendix D: Site Visit Guide..................................................................................................40
3

Executive Summary

Introduction and Background

In 2012, the Administration for Community Living (ACL) funded a National Community of 

Practice for Supporting Families (CoP) to build capacity across and within states to create policies, 

practices, and systems to better assist and support families that include a member with intellectual 

or developmental disabilities (I/DD) across the lifespan. ACL awarded a five-year grant to the 

National Association of State Directors of Developmental Disabilities Services (NASDDDS) in 

collaboration with the University of Missouri-Kansas City Institute on Human Development 

(UMKC-IHD) and the Human Services Research Institute (HRSI). NASDDDS selected five states 

to participate in the community of practice through a competitive application process: Connecticut, 

District of Columbia, Oklahoma, Tennessee, and Washington. Additionally, Missouri served as a 

demonstration state, due to its role in developing the Charting the LifeCourse (CtLC) framework.

As the grant period ended, ACL observed not only the success of the CoP participants, but growth 

of the CoP to include an additional ten states not affiliated with the grant. In 2017, ACL awarded 

an evaluation contract to New Editions Consulting and its partner, The Lewin Group, to evaluate 

the CoP.

Evaluation Design

Within its grant proposal, NASDDDS identified three project outcomes for the CoP.

Project Outcomes

Ø State and national consensus on a national framework and agenda for improving supports for 

families with children and adults with I/DD

Ø Enhanced state policies, practices, and sustainable systems that result in improved supports to 

families

Ø Enhanced capacity of states to replicate and sustain exemplary practices to support families and 

systems

These outcomes continue to drive the work of the CoP, and this evaluation examines state and 

national progress towards meeting these outcomes. Results of the evaluation will:

§ Inform the Administration on Disabilities (AoD)1by providing outcomes data and 

recommendations to improve supports to families of individuals with I/DD, including the 

future application of the CtLC framework and CoP platform.

§ Provide AoD with an updated systematic description of the range of CoP organizational 

structures, practices, and development across CoP states, which include consideration of 

the degree to which CoP states apply the principles and tools of the CtLC framework.

 

1 The Administration on Intellectual and Developmental Disabilities (AIDD) initiated the evaluation in 2017. The 

Administration on Disabilities (AoD) now oversees the evaluation.
4

Methodology

The evaluation team adapted a framework developed by Wegner, Trayner, and de Laat2for 

promoting and assessing value creation in communities and networks (Exhibit 1). This framework 

assesses value creation across members, leaders, and partners of communities and networks. The 

framework allows the evaluation team to link specific activities of communities and networks with 

desired outcomes, while considering the value of the learning that involvement in those social 

networks facilitates.

Exhibit 1. Value Cycle Framework

All states that participated in the CoP for at least two years between 2012 and 2018 are included in 

the evaluation. The evaluation stratified states into three groups based on their stage of 

development (Exhibit 2).

Exhibit 2. Stratification of States by Group

Group States

Group 1

Six original CoP states

§ Connecticut

§ District of Columbia

§ Missouri (mentor state)

§ Oklahoma

§ Tennessee

§ Washington

 

2 Wenger, E., Trayner, B., and de Laat, M. (2011). Promoting and Assessing Value Creation in Communities and 

Networks: A Conceptual Framework. Rapport 18, Ruud de Moor Centrum, Open University of the Netherlands. 

Retrieved from http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-

Wenger_Trayner_DeLaat_Value_creation.pdf.
5

Group States

Group 2

Four expansion states with exposure to the CoP or 

CtLC framework prior to joining the CoP

§ Maryland

§ Ohio

§ Pennsylvania

§ South Dakota

Group 3

Six expansion states with little or no prior 

exposure to the CoP or CtLC framework prior to 

joining the CoP

§ Alabama

§ Delaware

§ Hawaii

§ Indiana

§ Kansas

§ Oregon

Data Collection Tools

The evaluation team developed a series of tools to collect data from states and the national project 

team (Exhibit 3). The evaluation team pilot-tested the reporting tools with two states and 

conducted modified cognitive testing to determine whether the tools performed as expected. The 

team revised the tools following the testing.

Exhibit 3. Data Collection

Data 

Collection 

Method

Description Timeline

Reporting 

Tools

The team designed a reporting tool for each of the three 

groups of participants and loaded the tools into an online 

platform for states to complete. Each tool combined closedand open-ended questions to document progress during the 

past year.

February 2019

Telephone 

Interviews 

The evaluation team developed interview guides specific to 

each state based on the data submitted through the reporting 

tool. Guides included a core set of questions for each 

evaluation group.

March – April 2019

Site Visits

The evaluation team developed a site visit guide, which was 

adapted for each visit. The guides specified protocols for 

outreach, communication, planning, and documenting 

findings during the visit.

May – July 2019

Findings

After thorough analysis of the data collected through the reporting tools, telephone interviews, and 

site visits, several themes emerged related to each value cycle.

Cycle 1: Immediate Value

Value Cycle 1 evaluates the immediate value of state CoP activities to both members and the state. 

Activities that fall into this cycle have inherent value that may include information sharing, 

problem solving, peer-to-peer support, partnerships and collaboration, and listening to family 

voices. These activities may facilitate policy and practice changes in later cycles, which can

contribute to improved supports and individual outcomes.
6

Cycle 1: Key Themes

Ø Stakeholders share information and support

Ø States derive benefit from their CoP structures and processes including increased connections 

with one another and peer-to-peer supports

Ø Family members and self-advocates inform needed services and supports, system processes, 

and policy

Ø Diverse partners engage around a shared purpose, generating buy-in and promoting

collaboration across different systems

Cycle 2: Potential Value

Value Cycle 2 examines CoP activities and outputs expected to produce value in later cycles. This 

may include using the CtLC framework and tools, trainings or informational sessions, creating and 

disseminating products, identifying innovation areas, advancing current initiatives, changing

thinking or knowledge, and more. Such changes in knowledge capital may lead to changes in 

practice.

Cycle 2: Key Themes

Ø Members’ use and diffusion of the CtLC framework advances consensus

Ø States deploy diverse strategies to build knowledge

Ø States target new and existing initiatives for change

Ø Knowledge building leads to new ways of thinking

Cycle 3: Applied Value

Applied value examines the ways in which knowledge leads to innovation and change. In 

particular, Value Cycle 3 examines how state CoPs changed systems or practices to improve 

performance. This may include changes to policy (e.g., new waivers, new waiver services, and new 

legislation), changes to programs or services, revision to ISPs or other processes, developing new 

initiatives, creating new positions to meet a particular need, and measuring or monitoring services.

Cycle 3: Key Themes 

Ø States change policies and services to advance supports to families

Ø States revise forms and processes to align with the CtLC framework

Cycle 4: Realized Value

Realized value refers to the effects of new practices that occur during Value Cycle 3. Impacts may 

include decreased reliance on formal services, reduced waiting lists and process times, changed 

behavior, increased access and efficiency in navigating systems, greater satisfaction, and better 

lives.

Cycle 4: Key Themes 

Ø Stakeholders including CoP members, state service systems staff, families, and self-advocates

behave in new and different ways

Ø State service systems operate more efficiently
7

Cycle 5: Reframing Value

In Value Cycle 5, prior changes in knowledge capital and practice lead to redefining success. 

Examples include measuring outcomes in new ways, and changing goals, strategies, or system 

operations because of the CoP. This cycle includes strategies for sustainability and how states

redefined or revised their use of the CtLC framework.

Cycle 5: Key Themes

Ø State CoPs advance new goals, strategies, and operations

Ø State CoPs measure outcomes in new ways

Conclusion and Recommendations

The evaluation team identified a series of recommendations to inform ACL on future grants.

¡ Leverage the CoP platform in other systems change projects.

¡ Require strategic partnerships that can influence change at systems and grassroots levels.

¡ Identify clear performance measures for grantees.

¡ Support strategies for virtual engagement of grantees and states.

¡ Identify and model sustainability practices.

¡ Explore strategies to promote and incentivize evaluation participation after grant funding ends.
8

Introduction and Background

In 2012, the Administration on Disabilities (AoD) within the Administration for Community 

Living (ACL) funded a National Community of Practice for Supporting Families (CoP) to build 

capacity across and within states to create policies, practices, and systems to better assist and 

support families that include a member with intellectual and developmental disabilities (I/DD) 

across the lifespan. As the grant period ended, ACL observed not only the success of the CoP 

participants, but growth of the CoP to include an additional ten states not affiliated with the grant. 

In 2017, ACL awarded an evaluation contract to New Editions Consulting (New Editions) and its 

partner, The Lewin Group (Lewin) to identify how participating states are achieving project 

outcomes. In the 2017-2018 evaluation year, the evaluation team, comprised of New Editions and 

Lewin, developed the evaluation design based on a comprehensive environmental scan including 

stakeholder input. The Lewin Group led the implementation of the evaluation during the 2018-

2019 evaluation year. This report documents findings from the first year of data collection.

National Community of Practice for Supporting Families

Wingspread Report

In 2011, leaders in the disability field came together at the Wingspread Summit to develop policy 

recommendations to improve supports to families with a member with I/DD. The Wingspread 

Summit produced a report that outlines recommendations, ensuring that people with I/DD are fully 

integrated and included in society, and recognizing the critical role of families in their lives. The 

U.S. Administration on Disabilities (AoD)3used these recommendations to fund projects that year, 

including a request for proposal for what became the CoP.

4

Wingspread Summit Report Recommendations

Ø Design the structure and functions of state service systems to include a focus on supporting 

families reflective of the fact that most people with I/DD are living with their families in the 

community.

Ø Develop and fund the National Supporting the Family Initiatives (now the CoP) which explores 

principles, practices, and data indicators used to inform practice and policy related to supporting 

families across the lifespan.

Ø Develop the National Data Collection Initiative (now the Family Information Systems Project) 

which contains consistent and uniform data, identifying the impact on families and people with 

I/DD and cost-effectiveness of supporting families across the lifespan.

Ø Increase recognition of the family’s role in developing key federal policies and national 

programs.

Community of Practice Development

AoD awarded a five-year grant (2012-2017) to the National Association of State Directors of 

Developmental Disabilities Services (NASDDDS). NASDDDS collaborated with the University of 

Missouri-Kansas City Institute on Human Development (UMKC-IHD) and the Human Services 

Research Institute (HRSI) to form a national project team. The objective of this team, established 

 

3 The Administration on Intellectual and Developmental Disabilities (AIDD) awarded funding to the CoP. The 

Administration on Disabilities (AoD) now oversees the evaluation of the CoP.

4 National Association of State Directors of Developmental Disabilities Services. (2017). Setting a Fresh Course: 

Assessing the Impact of the National Community of Practice for Supporting Families of Individuals with Intellectual 

and Developmental Disabilities. Final Report.
9

in October 2012, was to “build capacity across and within states to create policies, practices, and 

systems to better assist and support families that include a member with I/DD across the lifespan.” 

The goal of supporting families, as defined by the CoP, is to “best support, nurture, love, and 

facilitate opportunities for the achievement of self-determination, interdependence, productivity, 

integration, and inclusion in all facets of community life for their family members.”5

NASDDDS sought states to participate in the CoP through a competitive application process. 

Fifteen states applied and the national project team selected five states to participate: Connecticut, 

District of Columbia, Oklahoma, Tennessee, and Washington. Additionally, Missouri served as a 

demonstration state due to its role in developing the Charting the LifeCourse (CtLC) framework.

Evolution of the Charting the LifeCourse Framework

The CtLC framework helps individuals and families develop a vision for a good life, think about 

their options, identity and find supports, and discover how they can live their best life. The CtLC 

framework is comprised of a set of principles that support the core belief that “all people have the 

right to live, love, work, and play and pursue their dreams in their community.”6 This belief drives 

the work of CoP states on supporting families.

CtLC Principle Description

ALL People ALL individuals and families are considered in the CoP’s vision, values, policies 

and practices for supporting people with I/DD

People within the 

Context of Family 

and Community

Individuals and families need supports that address all life stages as family 

member roles and needs change over time.

Strategies to Support 

Families

Supports address all facets of life and include discovery and navigation 

(information, education, skill building), connecting and networking (peer 

support), and goods and services (daily living and financial supports).

Quality of Life 

Domains and

Outcomes

Support structures that focus on self-determination, community living, social 

capital and economic sufficiency help plan for life outcomes, not just services.

Life Stages and 

Trajectory

Individuals and families can focus on a specific life stage, understanding how 

past, current, and future life stages will impact their life trajectory.

Integrated Delivery 

of Supports

Individuals and families can utilize a variety of integrated supports, including 

those that are publicly or privately funded and based on eligibility, available to 

anyone in the community, and relationship-based.

Initially called the LifeCourse Framework, the CtLC framework is rooted in various theories and 

concepts from the disability field, which include the notion of self-determination, family systems 

theory, person-and-family centered care, and other various models of disability. The CtLC 

 

5 This definition is adapted from the Wingspread Final Report: Hecht, E., Reynolds, M., Agosta, J., & McGinley, K. 

(2011). Building an agenda for supporting families with a member with intellectual and developmental disabilities. 

Report of the Wingspread Conference on building a family support agenda, March 6-8, 2011. Racine, Wisconsin: 

Johnson Foundation.

6 https://www.lifecoursetools.com/
10

framework evolved throughout the course of the five-year CoP grant, and the model continues to 

develop based on the experiences of its users.

Evaluation Purpose and Goals

Within its grant proposal, NASDDDS identified three project outcomes for the CoP.

Project Outcomes

Ø State and national consensus on a national framework and agenda for improving supports for 

families with children and adults with I/DD

Ø Enhanced state policies, practices, and sustainable systems that result in improved supports to 

families

Ø Enhanced capacity of states to replicate and sustain exemplary practices to support families and 

systems

These outcomes continue to drive the work of the CoP, and this evaluation examines state and 

national progress towards meeting these outcomes. Results of the evaluation will:

¡ Inform AoD by providing outcomes data and recommendations to improve supports to families 

of individuals with I/DD, including potential future application of the CtLC framework and 

CoP platform.

¡ Provide AoD with an updated systematic description of the range of CoP structures, practices, 

and development across CoP states, which include consideration of the degree to which CoP 

states apply the principles and tools of the CtLC framework.

Methodology

Evaluation Design

Evaluation Activities

The evaluation team engaged in a series of activities during the 2017-2018 evaluation cycle to 

inform the design of the evaluation.

¡ Environmental scan and white papers. The evaluation team conducted an environmental 

scan to provide ACL with a comprehensive review of the types of supports available to

families of individuals with I/DD. This environmental scan informed two white papers. The 

first white paper summarized best practices in supports to families, components of the CtLC 

framework, and CoP state implementation of the framework. The second white paper included 

CoP state findings from telephone interviews, site visits, and attendance at the Annual CoP 

Meeting and CtLC Showcase, as well as materials obtained from these events.

¡ Telephone interviews. The evaluation team conducted individual and group telephone 

interviews with all CoP states to understand their organizational structure, goals, challenges, 

activities, and innovations.

¡ Technical expert panels. The evaluation team convened two technical expert panels to inform 

the design, implementation, and interpretation of findings.
11

· Subject matter experts. New Editions engaged industry leaders with professional and 

lived expertise, and experience in programs serving individuals with I/DD and their 

families in home and community-based settings.

· NASDDDS and UMKC-IHD. Experts from NASDDDS and UMKC-IHD guided the 

development of the CoP evaluation and its subsequent expansion.

¡ Site Visits. The evaluation team conducted site visits to selected states to gain an 

understanding of the activities and outcomes reported by states, in addition to interviewing 

state leadership teams, providers and case managers, intake and outreach staff, and families and 

self-advocates.

¡ Annual CoP Meeting and CtLC Showcase. The evaluation team attended the Annual CoP

Meeting and CtLC Showcase in Kansas City, Missouri from April 18-20, 2018.

Evaluation Questions

There are several evaluation questions that directly correspond with the three project outcomes 

established by the CoP (Exhibit 1).

Exhibit 1. Project Objectives and Corresponding Evaluation Questions

Project Outcomes Evaluation Questions

State and national consensus on a 

national framework and agenda for 

improving support for families with 

children with I/DD

§ How, and to what extent, did the national and state CoPs 

achieve consensus on a national framework for supporting 

families?

§ How, and to what extent, are state CoPs using the CtLC 

framework to enhance supports to families of individuals 

with I/DD?

Enhanced state policies, practices, 

and sustainable systems that result 

in improved supports to families

§ What components of the national and state CoPs contribute 

to enhanced state policies, practices, and systems, and to 

what extent?

§ How, and to what extent, have the enhanced state policies, 

practices, and systems improved the supports available to 

families?

§ How, and to what extent, have the lives of families of 

children with I/DD improved or changed as a result of the 

work of the CoP?

Enhanced capacity of states to 

replicate and sustain exemplary 

practices to support families and 

systems

§ How are exemplary practices defined, identified, and 

verified, and which were most likely to be replicated and 

sustained, and why?

§ How, and to what extent, do state CoPs build capacity to 

sustain their work after formal affiliation with the CoP 

concludes?

§ What secondary outcomes did state CoPs experience beyond 

the stated project outcomes (if any)?
12

Evaluation Framework

The evaluation team adapted a framework developed by Wegner, Trayner, and de Laat7for 

promoting and assessing value creation in communities and networks. This framework assesses 

value creation across members, leaders, and partners of communities and networks. The 

framework allows the evaluation team to link specific activities of communities and networks with 

desired outcomes, while considering the value of the learning that involvement in those social 

networks facilitates. This conceptual framework considers five different cycles of value creation, 

which provide a context for organizing, understanding, and analyzing data. These five value 

creation cycles, adapted to evaluate the CoP, include:

¡ Cycle 1: Immediate Value – Value of activities and interactions themselves. The activities 

and interactions between members of the CoP have inherent value. The value to members may 

include problem solving, support, information sharing, collaboration, or mutual understanding. 

Activities in Cycle 1 are expected to strengthen relationships, encourage information and 

knowledge exchange, and improve communications among stakeholders.

¡ Cycle 2: Potential Value – Changes in knowledge capital. Activities that increase 

knowledge, such as developing products and conducting trainings, generate value that is not 

realized immediately. The evaluation team expects changes in knowledge capital to lead to 

changes in practice by establishing a consistent knowledge base and producing clarity on 

needed systems changes.

¡ Cycle 3: Applied Value – Changes in practice. Applied value examines the ways in which 

knowledge leads to innovation, new approaches, or changes in practice. In particular, Cycle 3 

examines how state CoPs changed systems or practices with a particular focus on how and why 

those changes are expected to improve outcomes for people with I/DD and their families.

¡ Cycle 4: Realized Value – Performance improvement. Realized value refers to the effect of 

new practices that occurred during Cycle 3, such as improvements in performance that are 

meaningful to stakeholders. To capture Cycle 4 value, the team evaluated the impact of efforts 

made during earlier cycles that increased self-determination, interdependence, productivity, 

integration, and inclusion in all facets of community life for people with I/DD, as well as 

supports to families.

¡ Cycle 5: Reframing Value – Redefining success. In Cycle 5, learning leads to redefining 

success, such as new goals, strategies, and performance metrics. To capture Cycle 5 value, the 

evaluation team examined how states realigned their existing goals and strategies to better 

support families and people with I/DD, including strategies for sustainability.

Evaluation Participants

All states that participated in the CoP for at least two years between 2012 and 2018 are included in 

the evaluation. The evaluation stratified states into three groups based on their stage of 

development (Exhibit 2).

 

7 Wenger, E., Trayner, B., and de Laat, M. (2011). Promoting and Assessing Value Creation in Communities and 

Networks: A Conceptual Framework. Rapport 18, Ruud de Moor Centrum, Open University of the Netherlands. 

Retrieved from http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-

Wenger_Trayner_DeLaat_Value_creation.pdf.
13

Exhibit 2. Stratification of States by Group

Group Description States

Group 1 Six original CoP states

§ Connecticut

§ District of Columbia

§ Missouri (mentor state)

§ Oklahoma

§ Tennessee

§ Washington

Group 2 Four expansion states with exposure to the CoP or CtLC 

framework prior to joining the CoP

§ Maryland

§ Ohio

§ Pennsylvania

§ South Dakota

Group 3 Six expansion states with little or no prior exposure to 

the CoP or CtLC framework prior to joining the CoP

§ Alabama

§ Delaware

§ Hawaii

§ Indiana

§ Kansas

§ Oregon

Data Collection

The evaluation team designed a series of tools to collect data from states and the national project 

team (Exhibit 3). The evaluation team pilot-tested the reporting tools with two states and 

conducted modified cognitive testing to determine whether the tool performed as expected. The 

team revised the tools following the testing.

Exhibit 3. Data Collection Tools

Data 

Collection 

Tool

Description Timeline

Reporting 

Tools

The team designed a reporting tool for each of the three groups of 

participants and loaded the tools into an online platform for states to 

complete. Each tool combined closed- and open-ended questions to 

learn about progress during the past year. States submitted data in the 

reporting tool for their respective group using an online platform. The 

evaluation team provided the group-specific link and a document 

containing the questions from the tool to each state.

February 

2019

Telephone 

Interview 

Guides

The evaluation team developed interview guides specific to each state 

based on the data submitted in the reporting tool. Guides included a 

core set of questions for each evaluation group. Using these guides, the 

evaluation team conducted one-hour interviews with each state CoP and

the national project team.

March –

May 2019

Site Visit 

Guides

The evaluation team developed a site visit guide, which was adapted to 

each visit. The guides specified protocols for outreach, communication, 

planning, and documenting findings during the visit. The evaluation team 

conducted seven in-person and virtual site visits. In consultation with the 

national project team and ACL, Lewin identified a sample ofstates that 

represented each of the three evaluation groups’ diverse structures and 

approaches. To ease burden on states, the evaluation team aligned visits 

with existing stakeholder, technical assistance (TA), and planning 

meetings.

May – July 

2019
14

Note: The evaluation team assessed the value of conducting materials review of documents from 

states (e.g., CoP meeting notes, products, and presentations) and determined to exclude it as a data 

collection method.

Lewin conducted visits with a representative sample of states participating in the CoP (Exhibit 4).

Exhibit 4. Lewin Conducted Seven Site Visits

State Date Meeting Title InPerson Virtual

Indiana May 16 – 17, 2019 Annual TA Visit from National Project 

Team ✔

District of Columbia June 7, 2019 State CoP Meeting ✔

Missouri June 12, 2019 Family-to-Family / Stakeholder Meeting ✔

South Dakota June 24, 2019 State CoP Meeting ✔

Maryland June 25 – 26, 2019 Annual TA Visit from National Project 

Team ✔

Kansas July 17, 2019 Monthly Statewide Planning Meeting ✔

Ohio July 31, 2019 State CoP Meeting ✔

Communication

The national project team (NASDDDS and UMKC-IHD) served in an advisory role to Lewin 

during the evaluation year through regular meetings on evaluation activities and their review of 

evaluation tools and products. Lewin communicated key evaluation activities to states through 

email, participation in TA calls hosted by the national project team, and a presentation at the 

Annual CoP Meeting in Kansas City, Missouri in April 2019. Lewin also engaged with states 

during attendance at the Annual CtLC Showcase after the Annual CoP Meeting in Kansas City, 

Missouri in May 2019.

Data Analysis

The evaluation team analyzed data collected through the reporting tools, telephone interviews, and 

site visits using qualitative methods (Exhibit 5).

Exhibit 5. Data Analysis Methods

Data Collection 

Tool Description

Reporting Tools

After receiving submissions from all states, Lewin reviewed reporting tool data to 

ensure all submissions were complete. Lewin followed up with project teams that 

submitted incomplete responses to determine whether blank fields were 

intentional or oversights. Lewin provided the opportunity to submit additional 

data. Using ATLAS.ti, the evaluation team coded data by value creation cycle, then 

by descriptive code.

Telephone 

Interview Guides

With each team’s permission, Lewin recorded the telephone interviews and 

documented each call with notes. The evaluation team coded data by value cycle 

and descriptive code using ATLAS.ti.
15

Data Collection 

Tool Description

Site Visit Guides

Lewin produced a summary for each site visit, documenting activities and 

discussions. The team manually coded these notes to identify themes by value 

cycle.

Findings

After thorough analysis of the data collected through the reporting tools, telephone interviews, and 

site visits, several themes emerged from each value cycle.

Cycle 1

Value Cycle 1 captures the immediate value of state CoP activities that may facilitate policy and 

practice changes in later cycles and contribute to improved supports and individual outcomes.

¡ Stakeholders share information and support. The CoP serves as a conduit for information 

exchange. States commonly use the CoP to share information with stakeholders regarding 

progress towards goals, new activities and initiatives, community events, CtLC tools and 

resources, and strategies for supporting families. States emphasized the importance of 

streamlining information dissemination efforts to ensure that accurate information is reaching 

the public from trusted sources (e.g., select state agencies or LifeCourse Ambassadors). CoP 

members also use the CoP to connect with one another and provide peer-to-peer supports. 

Some states developed web-based platforms to facilitate networking and communications. 

Nationally, CoP states are also sharing information with other CoP states regarding promising 

practices and lessons learned.

¡ Family members and self-advocates guide the agenda. Families and self-advocates are often 

a part of CoP leadership. Their voices and perspectives inform needed services and supports, 

system processes, and policy. Many family members are also LifeCourse Ambassadors who 

participate in CoP meetings, trainings, and quality improvement efforts.

¡ States derive benefit from their organizational structure and processes. States have 

varying CoP structures. Some states maintain formal, top-down structures in which project 

leaders clearly define roles and responsibilities, goals, and activities for the CoP; other states 

use an informal, bottom-up structure in which change happens more organically in response to 

visionary leaders or grassroots support. In some states, executive state teams maintain a 

feedback loop to share information with regional or local CoPs, whereas other states have one 

state CoP team. States reported that commitment and engagement from the state developmental 

disabilities (DD) agency or the DD Council are contributing factors towards achieving CoP 

goals. On a national level, select states (including those the evaluation team visited) reported 

that meetings and technical assistance (TA) provided by the national project team greatly 

benefitted their state’s efforts, in addition to relationships developed with other CoP states.

¡ Diverse partners engage around a shared purpose. States are developing strong partnerships 

with various organizations to generate buy-in and promote collaboration across different 

systems. Partners include family support organizations, advocacy organizations, health and 

human service providers, provider agencies, education systems, employment initiatives, aging 

departments, Vocational Rehabilitation, University Centers for Excellence in Developmental 
16

Disabilities (UCEDDs), Protection and Advocacy Systems (P&As), and more. Many of the 

representatives from these organizations attend CoP meetings.

The Ohio CoP consistently shares information and resources on available statewide 

programs and different family support models. This encourages communication, 

learning, and collaboration among existing efforts across the state. Lead partner 

organizations include the Department of Developmental Disabilities (DODD), DD 

Council, and both University Centers for Excellence in Developmental Disabilities 

(UCEDDs). All partners use the CtLC framework in their policy and systems change 

work.

CoP leadership and CoP members in Delaware are engaged in all CoP activities. They 

participate in outreach events, meet with stakeholders to understand their needs, 

and disseminate information and resources. The Division Of Developmental 

Disabilities Services (DDDS) collaborates with the Division of Medicaid & Medical 

Assistance (DMMA) and Division of Services for Aging and Adults with Physical 

Disabilities (DSAAPD) to develop supports for family caregivers of individuals with 

Dementia and Alzheimer’s.

States engaged in a variety of activities that generated immediate value (Exhibit 6).

Exhibit 6. Value Cycle 1 Activities Reported by States

State

Sharing information

Peer-to-peer support

Partnership/

collaboration between 

organizations

Listening to family/

self-advocate 

perspectives

Connection to national 

CoP

State CoP structure

State agency leadership 

participation

CoP member 

engagement

Group 1

Connecticut ✔ ✔ ✔ ✔ t ✔ t

District of Columbia ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Missouri ✔ ✔ ✔ ✔ t ✔ ✔ t

Oklahoma ✔

Tennessee ✔ ✔ ✔ ✔ ✔

Washington ✔ ✔ ✔ ✔ ✔

Group 2

Maryland ✔ ✔ ✔ ✔ ✔ ✔

Ohio ✔ ✔ ✔ t ✔ ✔ ✔ t

Pennsylvania ✔ ✔ ✔ ✔ ✔ t

South Dakota ✔ ✔ ✔ ✔ ✔ ✔

Group 3

Alabama ✔ ✔ ✔

Delaware ✔ ✔ ✔ ✔ ✔

Hawaii ✔ ✔ ✔ ✔

Indiana ✔ t ✔ ✔ t t

Kansas ✔ ✔ t

Oregon ✔ ✔ ✔
17

State

Sharing information

Peer-to-peer support

Partnership/

collaboration between 

organizations

Listening to family/

self-advocate 

perspectives

Connection to national 

CoP

State CoP structure

State agency leadership 

participation

CoP member 

engagement

Total number of states reporting 

each activity (out of 16) 13 8 16 10 5 10 7 4

✔ State reported this activity this evaluation year.

t State did not report this activity this evaluation year due to past CoP advancements in this area.

Cycle 2

Value Cycle 2 examines CoP activities and outputs, such as use of the CtLC framework and tools, 

trainings or information sessions, creating and disseminating products, identifying innovation 

areas, advancing current initiatives, and changes in thinking or knowledge that are expected to 

produce value in later cycles.

§ Members’ use and diffusion of the CtLC framework advances consensus. The CtLC 

framework is used both as an intervention to help individuals and families develop a vision for 

a good life, think about options, and identify supports, and to inform strategic reframing by 

helping systems and organizations prioritize strategies and evaluate activities. States are 

integrating the CtLC framework into front door interactions, trainings, transitions across the 

lifespan, individual service or support plans (ISPs) and other service planning processes, and 

strategic plans. Many states reported an existing Person Centered Thinking (PCT) foundation 

in their state, which the CtLC framework enhanced or supported. States emphasized the value 

of diffusing the CtLC framework and tools across sectors including health care financing, 

vocational rehabilitation, education, early childhood, mental health, behavioral health, 

employment, aging, housing, managed care, and case management.

§ States deploy diverse strategies to build knowledge. All CoP states conducted trainings on 

the CtLC framework. On a national level, CoP members completed the Ambassador Series and 

participate in the Annual CoP Meeting and CtLC Showcase. States commonly referenced the 

Ambassador Series as a key national training that helps support the spread and understanding 

of the CtLC framework. On the state level, trainings address utilization of the CtLC framework 

in areas such as technology and assistive technology, employment, self-determination, selfadvocacy, case management, education, long-term support services, and planning services. 

State initiatives and trainings that incorporate the CtLC framework include Employment First, 

Partners in Policymaking, Youth Leadership Forum, and Person Centered Thinking (PCT) 

training. Audiences include individuals, families, support coordinators, case managers, intake 

and referral teams, personal support workers, providers, and staff in other state departments. 

Many CoPs also sponsor informal trainings or information sessions. To supplement ongoing 

training and education, many states have created and disseminated educational materials on the 

CtLC principles and tools. These materials apply CtLC to person centered practices, 
18

employment, rights, planning, self-advocacy, self-direction, education, community supports, 

and more.

§ States target new and existing initiatives for change. Every state CoP reported advancing 

current initiatives or goals to better support families in the past year. Primary goals and focus 

areas for state CoPs include:

· Enhancing front door supports;

· Including individuals and families in planning and implementation of services and 

supports;

· Increasing the use of technology to support individuals with I/DD and families;

· Advancing employment, post-secondary education, transition support, peer-to-peer support, 

capacity building, and supported decision making;

· Implementing the Centers for Medicare & Medicaid Services (CMS) Home and 

Community Based Services (HCBS) Settings Final Rule;

· Embedding CtLC in current practices; and,

· Developing systems based recommendations.

· Several states announced participation in new National Innovation Workgroups including 

Family and Self-Advocate Engagement, Cultural and Linguistic Competency, 

Implementation and Practice, and System Design and Quality.

¡ Knowledge building leads to new ways of thinking. States reported various ways in which 

the CoP or CtLC framework led to changes in thinking and knowledge. For example, the CtLC 

tools support families and self-advocates to consider alternative supports within the community 

before accessing state-funded services. The CtLC framework also helps professionals frame 

conversations using a common language, facilitates transitions throughout the lifespan, and 

contributes to an overall cultural shift among states.

South Dakota uses the CtLC framework in infrastructure and systems planning to 

provide a different lens towards progress. The CtLC framework is now working its 

way through several sectors within the state including Education, Department of 

Corrections, LTSS Systems, etc. The state CoP engages in informal CtLC training and 

education sessions with case managers, service coordinators, families, and selfadvocates. South Dakota also created and disseminated the "Charting a PersonCentered Path to Employment" resource, which outlines services and provides 

anticipatory guidance.

Connecticut deploys diverse strategies to build knowledge. The Department of 

Developmental Services (DDS) and Council on Developmental Disabilities (CDD) 

provide the Ambassador Series and other training opportunities to families, selfadvocates, and professionals (e.g., case managers, Department of Education, and 

Department of Rehabilitation Services). DDS also creates and disseminates a 

newsletter to families with CoP and CtLC information and resources.

States engaged in various activities to increase knowledge (Exhibit 7).
19

Exhibit 7. Value Cycle 2 Activities Reported by States

State

Using CtLC tools/principles

National training

State training

Informal training/

education

Family outreach

Creating/

disseminating products

Diffusion across sectors

Identifying innovation/

focus areas

Advancing current 

initiatives/goals

Workgroup/learning 

community

Website/technology

Changes in thinking/

knowledge

Group 1

Connecticut ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

District of Columbia ✔ ✔ ✔ ✔ t ✔ ✔ ✔ ✔ ✔ ✔ ✔

Missouri ✔ ✔ t ✔ ✔ ✔ ✔ ✔ ✔ ✔

Oklahoma ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Tennessee ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Washington t ✔ ✔ t ✔ ✔ ✔ ✔ ✔

Group 2

Maryland ✔ ✔ ✔ ✔ t ✔ t ✔ ✔ ✔ ✔

Ohio ✔ ✔ ✔ ✔ t ✔ ✔ ✔ ✔ ✔

Pennsylvania t t ✔ ✔ t ✔ ✔ ✔ ✔ ✔

South Dakota ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Group 3

Alabama ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Delaware ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Hawaii ✔ ✔ ✔ ✔ t ✔ ✔ ✔

Indiana ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Kansas ✔ t ✔ ✔ ✔ ✔ ✔

Oregon ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Total number of states reporting 

each activity (out of 16) 14 12 15 14 7 9 10 13 13 8 7 15

✔ State reported this activity this evaluation year.

t State did not report this activity this evaluation year due to past CoP advancements in this area.

Cycle 3

Applied value examines the ways in which knowledge leads to innovation, new approaches, or 

changes in practice, such as changes to policy (e.g., new waivers, new waiver services, and new 

legislation), changes to programs or services, revision to ISPs or other processes, new initiatives, 

creating new positions to meet a particular need, and measuring or monitoring services.

¡ States change policies and services to advance supports to families. Many states reported 

policy changes within the past year. Examples include the development and implementation of 

new waivers that embed the CtLC principles and language, new waiver services, passing the 

Technology and Employment First Act or other legislation, convening a Legislative Caucus on 

waiver waiting lists, and revising guidelines on eligibility or services offered. States noted the 

impact of the CMS HCBS Settings Final Rule in guiding other policy changes. States also use 
20

the CtLC framework to change various programs and services, such as employment programs 

and grants, self-directed services, initiatives for preventing abuse, neglect, and exploitation, and 

assistive technology programs.

¡ States revise forms and processes to align with the CtLC framework. States used the CtLC 

framework to revise ISPs, administrative policies and procedures, hiring and recruitment 

processes, and to inform intake and referral processes. Several states created new staffing 

positions to support collaboration within and between systems (e.g., service coordinators, 

family engagement director, community services director, employment director, and other 

liaisons). States reported increased skills of those who facilitate planning, transition, intake and 

referral, and case management to be more self-directed.

Oklahoma convened a Legislative Caucus that assured $2 million in new money went 

towards Medicaid services for people on the waiting list.

Indiana created a Person Centered Individualized Support Plan (PCISP) that reflects 

the CtLC life stages, life domains, and strength-based supports. The state is now 

measuring the completion of PCISPs for all individuals.

The District of Columbia enacted the Disability Services Reform Amendment Act of 

2018 to improve the quality of life for people with I/DD by improving the complaint 

process, assuring no new people are committed, and enhancing supported decision 

making.

States engaged in a variety of specific activities to change policy and practice (Exhibit 8).

Exhibit 8. Value Cycle 3 Activities Reported by States

State

Change to policy

Change to program/services

Change in use of services

Revision to ISP or other 

forms

/processes

Embedded CtLC principles/tools 

into operations

New initiatives

Aligning with other initiatives

Changes to recruitment/

hiring practices

New staffing position created

Improved skills/

capabilities of staff

Measuring/

monitoring services

Group 1

Connecticut ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

District of Columbia ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Missouri ✔ ✔ ✔ ✔ ✔ ✔

Oklahoma ✔ ✔ ✔ ✔

Tennessee ✔ ✔ ✔

Washington ✔ ✔ ✔ ✔
21

State

Change to policy

Change to program/services

Change in use of services

Revision to ISP or other 

forms

/processes

Embedded CtLC principles/tools 

into operations

New initiatives

Aligning with other initiatives

Changes to recruitment/

hiring practices

New staffing position created

Improved skills/

capabilities of staff

Measuring/

monitoring services

Group 2

Maryland ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Ohio ✔ ✔ ✔ ✔ ✔

Pennsylvania ✔ ✔ ✔ ✔ ✔ ✔

South Dakota ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Group 3

Alabama ✔ ✔ ✔ ✔ ✔

Delaware ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Hawaii ✔ ✔ ✔

Indiana ✔ ✔ t t ✔

Kansas ✔ ✔

Oregon ✔ ✔ ✔ ✔ ✔

Total number of states reporting 

each activity (out of 16) 12 10 6 11 13 7 5 1 7 7 12

✔ State reported this activity this evaluation year.

t State did not report this activity this evaluation year due to past CoP advancements in this area.

Cycle 4

Realized value refers to the effects of new practices that occur during Value Cycle 3. Impacts may 

include decreased reliance on formal services, reduced waiting lists and process times, changed 

behavior, increased access and efficiency navigating systems, increased satisfaction, and better 

lives.

¡ People behave in new and different ways. States reported increased involvement by families 

in leadership and planning processes. States also identified a deeper understanding by families 

that the CtLC framework can produce real change, both individually and within systems. 

During site visits, many stakeholders shared how the CtLC framework influenced changes in 

their behavior, both in their professional and personal lives. For example, the CtLC framework

helped stakeholders to focus more holistically. They no longer focus only on formal services, 

but what they can do to support people’s lives.

¡ State service systems operate more efficiently. More people are receiving community-based 

services and are having increased efficiency in accessing them. Additionally, states reported 

reductions in waiting list and processing times due to changes in policy.
Pennsylvania’s Employment First Act of 2018 expanded employment opportunities 

for individuals with disabilities. Over the last two years, Pennsylvania increased the 

proportion of the population of people with I/DD with competitive employment from 

12% to 16%. Additionally, Pennsylvania modified eligibility criteria for three waivers, 

allowing individuals with autism spectrum disorder to transfer to self-directed care 

and other services. Individuals with autism spectrum disorder can now access 

services they previously could not. Pennsylvania identified these outcomes as a direct

result of the ALL principle within the CtLC framework.

In Washington State, the Individual and Family Services Waiver allowed the state to 

bring 5,000 more people into services, an outcome that Washington State identified 

as directly influenced by CoP work. Additionally, Washington State held trainings on 

the community engagement service offered by the wavier, after which the number of 

people using that service rose from 20 to 900.

22

The evaluation team identified a series of activities related to realized value (Exhibit 9).

Exhibit 9. Value Cycle 4 Activities Reported by States

State

Less reliance on formal 

services

Reduction in waiting 

list/process times

New skills for 

using services/

supports

Changed behavior

Better experience 

with services/

systems navigation

Increased access/

efficiency

Increased satisfaction/

better lives

Group 1

Connecticut ✔ ✔

District of Columbia ✔ ✔ ✔

Missouri t t

Oklahoma ✔ ✔ ✔

Tennessee ✔

Washington ✔ ✔ ✔ ✔

Group 2

Maryland ✔ ✔ ✔

Ohio ✔

Pennsylvania ✔ ✔

South Dakota ✔ ✔ ✔

Group 3

Alabama

Delaware ✔ ✔ ✔ ✔

Hawaii

Indiana t t t

Kansas ✔ ✔

Oregon ✔ ✔

Total number of states reporting 

each activity (out of 16) 3 4 1 6 3 7 6

✔ State reported this activity this evaluation year.

t State did not report this activity this evaluation year due to past CoP advancements in this area.
23

Cycle 5

In Value Cycle 5, changes in knowledge capital and practice lead to redefining success, such as 

measuring outcomes in new ways, and changing goals, strategies, or system operations because of 

the CoP. This cycle includes strategies for sustainability and how states redefined or revised their 

use of the CtLC framework.

¡ State CoPs advance new goals, strategies, and operations. States have aligned their state 

DD quality outcomes with the CtLC framework, and embedded it into scopes of work, 

strategic plans, and evaluation methods. States have significantly changed how their state DD 

agency approaches families and business practices, and always steer people to consider 

community resources first. Other states that initially focused on serving adults now target 

individuals at a younger age and now look at both the family and individual instead of the 

individual only.

¡ States measure outcomes in new ways. Several states changed goals, strategies, or systems 

operations as a result of the CoP or CtLC framework and some are also measuring and 

monitoring outcomes in new ways. For example, Oregon embedded the CtLC framework into 

the scope of work and evaluation methods required of Family/Peer Support Contractors 

(OCFN) in order to better articulate the intent of those contracts and expected outcomes.

In Maryland, local CoP leaders voiced that organizational decisions were often made 

by the national or state teams and trickled down without consulting others. State CoP 

leadership realized the need for a change in strategy and brainstormed ways to 

better include local CoP input, such as new circular pipelines of information between 

the state and local groups, a regular TA call, and the instatement of sharing and 

strategizing meetings. The emergence of these concerns created the potential for 

CoP reorganization to further benefit and include stakeholders.

Fewer states have reframed value as a result of the CoP because they are still progressing towards 

sustainable change. Exhibit 10 documents key state activities.
24

Exhibit 10. Value Cycle 5 Activities Reported by States

State

Measuring outcomes 

in new ways as a result 

of the CoP

Changes in 

goals/strategies/

system operations as a 

result of the CoP

Group 1

Connecticut

District of Columbia ✔

Missouri ✔ ✔

Oklahoma ✔

Tennessee ✔

Washington ✔ ✔

Group 2

Maryland ✔

Ohio ✔

Pennsylvania ✔

South Dakota ✔

Group 3

Alabama ✔

Delaware ✔

Hawaii ✔

Indiana ✔

Kansas ✔

Oregon ✔ ✔

Total number of states reporting 

each activity (out of 16) 6 12

✔ State reported this activity this evaluation year.

t State did not report this activity this evaluation year due to past CoP advancements in this area.

Conclusion and Recommendations

Evaluation Limitations

The evaluation team acknowledges several limitations of this study.

¡ Retrospective data collection. Data collected during this evaluation year required participants 

to recall earlier activities and events. While the reporting tools focused on activities within the 

prior year, interview questions sometimes required participants to recall prior activities.

Therefore, there may be some limitations in the accuracy and/or completeness of data.

¡ Post-grant evaluation. ACL initiated this evaluation after the grant to NASDDDS ended. 

States’ continued participation in the CoP is voluntary and self-funded. Given this context, the 

evaluation team designed the evaluation to minimize burden and collaborated with the national 
25

project team to engage states and communicate value. The goal of being parsimonious may 

result in omitting important data points.

¡ Evolution of national framework. Achieving state and national consensus on a framework for 

supporting families is an outcome of the CoP. The CtLC framework is widely used and 

accepted as this national framework. CtLC evolved during the course of the grant period and 

continues to grow. States used the framework at various stages of its implementation, and 

measures of fidelity for the framework have not yet been established. A snapshot evaluation 

fails to capture the nuances of an evolving program.

Recommendations

The evaluation team identified a series of recommendations to inform ACL on future grants.

¡ Leverage the CoP platform in other systems change projects. The evaluation team 

observed both immediate value in the use of the CoP platform (e.g., access to information and 

support, engagement of diverse partners) and long-term value in states’ ability to leverage the 

platform to achieve systems change (e.g., changing policies and services to advance supports to 

families). These findings suggest that the CoP platform could be effective in convening 

stakeholders to effect systems change on other topics of interest.

¡ Require strategic partnerships. Stakeholder partnerships, including engagement of selfadvocates, are a frequent requirement of projects of national significance (PNS), such as the 

CoP. Unique to the CoP is the requirement that state teams include both the state DD agency 

and DD council. This reflects the national project team’s emphasis on addressing supports to 

families through both personal and system perspectives. The evaluation team recommends 

incorporating specific expectations for strategic partnerships into future grant opportunities.

¡ Identify clear performance measures for grantees. The CoP provided states with flexibility 

to address the issues most pressing to their unique context. States consistently noted challenges 

related to quantifying and measuring their progress toward desired outcomes. The evaluation 

team recommends developing clear performance measures that are broad enough to account for 

variations among states, but specific enough for states to measure and report on progress both 

individually and collectively.

¡ Support strategies for virtual engagement of grantees and states. While face-to-face 

stakeholder engagement is usually ideal, the time and cost of attending regular in-person 

meetings can be prohibitive. This is particularly salient when stakeholders also have family 

caregiving responsibilities. Many CoP states explored virtual options for communicating with 

and convening stakeholders including conference calls, virtual meeting platforms (e.g., Zoom, 

Skype), and social media (e.g., Facebook). The evaluation team recommends additional TA on 

future projects to support and enhance states’ ability to engage with stakeholders virtually.

¡ Identify and model sustainability practices. At the end of the grant, NASDDDS not only 

sustained the CoP with participating states, but expanded to include additional states. During 

this evaluation cycle, NASDDDS invited a third cohort of state to join the CoP. The evaluation 

team recommends documenting best practices related to the sustainability of this initiative for 

use on future projects.
26

¡ Explore strategies to promote and incentivize evaluation participation after grant 

funding ends. States’ continued participation in the CoP is voluntary and self-funded. Their 

participation in the evaluation is also voluntary and may be burdensome in terms of time and 

effort spent on data collection activities. The evaluation team recommends the exploration of 

various incentive strategies to promote states’ willingness to continue participating in the 

evaluation.

Next Steps

The evaluation team will continue to collect and analyze additional data during the 2019-2020 

evaluation cycle.
27

Appendix

Appendix A: Data Collection Activities Timeline

The evaluation team collected data between February and July 2019.

Evaluation Tool Dates for Data 

Collection

Feb 

2019

Mar 

2019

Apr 

2019

May 

2019

Jun 

2019

Jul 

2019

Reporting Tools 2/4/2019-3/1/2019 ✔ ✔

Telephone 

Interviews 3/25/2019-5/15/2019 ✔ ✔ ✔

Site Visits 5/16/2019-7/31/2019 ✔ ✔ ✔
28

Appendix B: Reporting Tool Survey for Groups 1-3

Purpose

This reporting tool is designed to collect information from each state on CoP structure and 

members, use of the CtLC framework, and improvements made to family supports. Information 

regarding changes made to policies, programs and services, and initiatives, as well as the impact or 

anticipated impact of these changes will also be captured. The information collected will be used to 

inform specific questions asked during telephone interviews. This version of the reporting tool is 

tailored to Group 1 states, which include the six original CoP states (CT, DC, MO, OK, TN, and 

WA). All states are asked to complete the reporting tool one time this evaluation year with a time 

commitment of less than two hours.

Instructions

1. Please complete each question to the best of your abilities and based on your understanding of 

the question in context of your state. Additionally, please complete each question in the order 

that they appear (Note: The online tool contains skip logic, which may change the numbering 

of the questions from what is seen in this document). Questions may be in the format of 

multiple choice, check boxes, fill-in-the-blank, or descriptive narrative.

2. Please submit your responses using the following Research.Net link. This Word document 

contains all questions asked through Research.Net and can be shared among respondents to 

prepare their responses.

3. These questions are to be completed by CoP facilitators and/or members of the CoP leadership 

team in your state that oversee day-to-day CoP activities. If you do not know the answer to a 

certain question, please inquire with a colleague who knows the answer. If this is not possible 

or your state does not have any activity to report in a particular area, please leave that field 

blank. Since states are focusing their efforts in different areas, it is expected that you will not 

have a response to every field.

Note: When we refer to the word “state” within this document, we are referring to all of the 

activities that occur within your state. We are not referring specifically to state-funded activities or 

activities for which a state agency is responsible.

If you have any questions, please email the evaluation team at Natalie.Boonchaisri@Lewin.com.

Reporting Tool Questions

1. CoP Structure and Members

a. List the name, title, and email address of individual(s) responsible as the CoP point(s) of 

contact in your state. These are the individuals with whom the evaluation team will 

communicate. You may enter up to three individuals.

· Contact 1 Name: ___________________________________

· Contact 1 Title: ____________________________________

· Contact 1 Email Address: ____________________________

· Contact 2 Name: ___________________________________
29

· Contact 2 Title: ____________________________________

· Contact 2 Email Address: ____________________________

· Contact 3 Name: ___________________________________

· Contact 3 Title: ____________________________________

· Contact 3 Email Address: ____________________________

b. Do the individual(s) responsible as the CoP point(s) of contact in your state also rely on others 

for leadership, administration, and/or oversight of the CoP? We refer to these individuals as 

CoP leadership. Please select yes or no and complete the accompanying questions, if any.

☐ Yes [move forward to question 1c]

☐ No [skip question 1c and begin at question 1d]

c. Identify the various roles of the individuals who participated in CoP leadership within the last 

year. As a reminder, CoP leadership refers to the individuals that the CoP point(s) of contact in 

your state also rely on for leadership, administration, and/or oversight of the CoP. Select all 

that apply.

☐ Self-Advocates

☐ Family Members

☐ Professional – Advocacy Organization

☐ Professional – State Agency

☐ Professional – Service Provider/Case Management

☐ Professional – Developmental Disabilities (DD) Council

☐ Professional – University Center for Excellence in Developmental Disabilities (UCEDD)

☐ Professional – State Protection & Advocacy System (P&A)

☐ Other, please specify: _____________________

d. How many individuals comprise the CoP leadership in your state? As a reminder, CoP 

leadership refers to the individuals that the CoP point(s) of contact in your state also rely on 

for leadership, administration, and/or oversight of the CoP. Select one.

☐ 1-2 individuals

☐ 3 individuals or more

☐ Not applicable

e. Did CoP leadership perform any of the following activities in the last year? As a reminder, 

CoP leadership refers to the individuals that the CoP point(s) of contact in your state also rely 

on for leadership, administration, and/or oversight of the CoP. Select all that apply.

☐ Develop/facilitate training

☐ Identify/review CoP outcomes

☐ Review consumer satisfaction information

☐ Facilitate cross-agency/organizational meetings

☐ Build/maintain relationships

☐ Engage community resources

☐ Facilitate networking

☐ Support communication
30

☐ Identify/secure needed resources

☐ Initiate partnerships

☐ Other, please specify: _____________________

☐ Not applicable

f. How are individuals with I/DD currently engaged in CoP leadership? As a reminder, CoP 

leadership refers to the individuals that the CoP point(s) of contact in your state also rely on 

for leadership, administration, and/or oversight of the CoP. Select all that apply.

☐ Active CoP facilitator or leader

☐ Active CoP member

☐ Active CoP trainer

☐ Advisor on policy development

☐ Advisor on service and support development or implementation

☐ Other, please specify: _____________________

☐ Not applicable

g. How are family members currently engaged in CoP leadership? As a reminder, CoP 

leadership refers to the individuals that the CoP point(s) of contact in your state also rely on 

for leadership, administration, and/or oversight of the CoP. Select all that apply.

☐ Active CoP facilitator or leader

☐ Active CoP member

☐ Active CoP trainer

☐ Advisor on policy development

☐ Advisor on service and support development or implementation

☐ Other, please specify: _____________________

☐ Not applicable

2. Improvements to Family Supports

a. Did your CoP contribute to any of the following improvements or changes related to family 

supports in your state within the last year? Select all that apply. If no intended improvement or 

change was observed, but your state is actively working towards it, please select that field. If 

no intended improvement or change was observed and your state is not actively working 

towards it, please leave that field unchecked.

☐ Increased family member knowledge on how to better support their loved one(s)

☐ Increased family member skills to navigate and access services

☐ Enhanced family member ability to advocate for services and policy change

☐ Increased family-to-family support

☐ Increased self-advocacy

☐ Increased use of non-disability community support

☐ Increased access to self/family-directed services

☐ Increased access to transportation

☐ Increased access to respite/childcare

☐ Increased access to/use of transition services
31

☐ Increased access to adaptive equipment

☐ Increased access to/use of technology by families or people with I/DD to promote 

independence and well-being

☐ Other, please specify: _____________________

☐ Not applicable

3. Impact of Partnerships

a. As a result of your CoP activities, did partnerships (within or outside of the DD system) 

contribute to any of the following improvements or changes within the last year? Select all that 

apply. If no intended improvement or change was observed, but your state is actively working 

towards it, please select that field. If no intended improvement or change was observed and 

your state is not actively working towards it, please leave that field unchecked.

☐ Reduced fragmentation of services for individuals with I/DD

☐ Improved system efficiency

☐ Encouraged innovation

☐ Increased collaboration

☐ Maximized use of existing community assets

☐ Other, please specify: _____________________

☐ Not applicable [skip question 3b and begin at question 4a]

b. Describe the impact or anticipated impact of the selected improvement(s) or change(s) 

resulting from partnerships on families and people with I/DD.

[The responses you selected from question 3a will carry over to this question]

4. Impact of Changes to Policies

a. Did your CoP activities influence any of the following improvements or changes to STATE 

POLICY (e.g., law, regulation, executive order, or policy) within the last year? Select all that 

apply. If no intended improvement or change was observed, but your state is actively working 

towards it, please select that field. If no intended improvement or change was observed and 

your state is not actively working towards it, please leave that field unchecked.

☐ Reduced fragmentation of services for individuals with I/DD

☐ Improved system efficiency

☐ Encouraged innovation

☐ Increased collaboration

☐ Maximized use of existing community assets

☐ Other, please specify: _____________________

☐ Not applicable [skip question 4b and question 4c and begin at question 5a]

b. Describe the STATE POLICY (e.g., law, regulation, executive order, or policy) change that 

influenced or resulted in the selected improvement(s) or change(s) in your state within the last 

year.

[The responses you selected from question 4a will carry over to this question]

c. Describe the impact or anticipated impact of the selected improvement(s) or change(s) 

resulting from STATE POLICY (e.g., law, regulation, executive order, or policy) change on 
32

families and people with I/DD.

[The responses you selected from question 4a will carry over to this question]

5. Impact of Changes to Programs and Services

a. Did your CoP activities influence any of the following improvements or changes to 

PROGRAMS and SERVICES within the last year? Select all that apply. If no intended 

improvement or change was observed, but your state is actively working towards it, please 

select that field. If no intended improvement or change was observed and your state is not 

actively working towards it, please leave that field unchecked.

☐ Reduced fragmentation of services for individuals with I/DD

☐ Improved system efficiency

☐ Encouraged innovation

☐ Increased collaboration

☐ Maximized use of existing community assets

☐ Other, please specify: _____________________

☐ Not applicable [skip question 5b and question 5c and begin at question 6a]

b. Describe the PROGRAM and/or SERVICE change that influenced or resulted in the selected 

improvement(s) or change(s) in your state within the last year.

[The responses you selected from question 5a will carry over to this question]

c. Describe the impact or anticipated impact of the selected improvement(s) or change(s) 

resulting from PROGRAM and/or SERVICE change on families and people with I/DD. [The 

responses you selected from question 5a will carry over to this question]

6. Impact of Changes to Initiatives

a. Did your CoP activities influence any of the following improvements or changes to EXISTING 

INITIATIVES or NEW INITIATIVES within the last year? Select all that apply. If no intended 

improvement or change was observed, but your state is actively working towards it, please 

select that field. If no intended improvement or change was observed and your state is not 

actively working towards it, please leave that field unchecked.

☐ Reduced fragmentation of services for individuals with I/DD

☐ Improved system efficiency

☐ Encouraged innovation

☐ Increased collaboration

☐ Maximized use of existing community assets

☐ Other, please specify: _____________________

☐ Not applicable [skip question 6b and question 6c and begin at question 7a]

b. Describe the EXISTING INITIATIVE change(s) or NEW INITIATIVE(S) that influenced or 

resulted in the selected improvement(s) or change(s) in your state within the last year.

[The responses you selected from question 6a will carry over to this question]

c. Describe the impact or anticipated impact of the selected improvement(s) or change(s) 

resulting from EXISTING INITIATIVE change(s) or NEW INITIATIVE(S) on families and 

people with I/DD.

[The responses you selected from question 6a will carry over to this question]
33

7. Progress towards AIDD Priority Areas

a. Select the following Administration on Intellectual and Developmental Disabilities (AIDD) 

priority areas that your state progressed towards within the last year. Select all that apply. If 

your state did not progress towards a particular priority area, please leave that field unchecked. 

If your state focused elsewhere, please select not applicable.

☐ Protection of the rights of individuals with I/DD and preventing their abuse, neglect, and 

exploitation

☐ Individuals with I/DD and their families’ ability to access home and community-based 

services and supports that are self-directed and ensure opportunity for full and meaningful 

community participation

☐ Participation of individuals with I/DD in the competitive, integrated workforce

☐ Participation of individuals with I/DD in system and service delivery design

☐ Better supporting families of individuals with I/DD

☐ Increasing systems’ focus on community integration at all levels and stages of service from 

information and referral, needs assessments, service matching, and service provision

☐ Not applicable [skip question 7b, question 7c, and question 7d and begin at question 8a]

b. Describe the specific goal(s) your CoP identified within the last year that align with 

the selected Administration on Intellectual and Developmental Disabilities (AIDD) priority 

areas.

[The responses you selected from question 7a will carry over to this question]

c. For the goal(s) identified for each selected AIDD priority area, identify your state's current 

state of progress. [The responses you selected from question 7a will carry over to this 

question]

· Not started

· In progress

· Achieved

d. For the goal(s) identified for each selected AIDD priority area, briefly describe how progress 

towards this goal is being monitored in your state and what is being measured. If your state is 

not monitoring and/or measuring progress towards this goal, please leave that field blank.

[The responses you selected from question 7a will carry over to this question]

8. Use and Effect of CtLC Framework

a. Indicate whether use of the Charting the LifeCourse (CtLC) framework in your state 

INFLUENCED any of the following improvements or changes within the last year. Select all 

that apply. If no improvement or change was observed, please leave that field unchecked.

☐ Increased number of people receiving services and supports from state-funded services

☐ Increased number of people receiving services and supports from non-state-funded services

☐ Expanded reach of services and supports geographically

☐ Expanded reach of services and supports by population (e.g., new age groups)

☐ New services available to support families

☐ Improvements in existing services available to support families
34

☐ Improved process of accessing services (supported consumer navigation)

☐ Less time spent at front door/seeking services

☐ Shortened wait lists to receive services

☐ Improved connections between consumers and their communities

☐ Improved connections between agencies serving individuals with I/DD and their 

caregivers/families

☐ Increased access to goods and technologies to support life goals of individuals with I/DD

☐ Professional development of those who use the CtLC framework in their work

☐ Other, please specify: _____________________

☐ Not applicable [skip question 8b and begin at question 9a]

b. For each selected improvement or change that was influenced by the Charting the LifeCourse

(CtLC) framework, identify the degree to which the CtLC framework influenced the observed 

improvement or change. Please use the following definitions for high, medium, and low to 

identify the degree to which the CtLC framework influenced the observed improvement or 

change: [The responses you selected from question 8a will carry over to this question]

· High – The improvement or change likely would not occur without the influence of the 

CtLC framework.

· Medium – The improvement or change likely would occur differently or to a lesser extent 

without the influence of the CtLC framework.

· Low – The improvement or change likely would occur without the influence of the CtLC 

framework.

c. If there is more you’d like to tell us about your state’s activities, please use the box below to 

provide us with any additional information.

9. Group-Specific Questions

Questions Specific to Group 1 States (CT, DC, MO, OK, TN, and WA)

a. What actions have you taken in the last year to support sustainability of your CoP? Select all 

that apply.
35

☐ Formalized structures or processes associated with the CoP

☐ Enhanced or formalized leadership roles

☐ Strengthened or formed new partnerships

☐ Increased use or application of the CtLC framework or tools

☐ Increased involvement of families and self-advocates

☐ Increased community support

☐ Identified or secured sustainable funding

☐ Other, please specify: _____________________

Questions Specific to Group 2 States (MD, OH, PA, and SD)

a. Have you adopted practices in your state from other participants in the national CoP? Please 

select yes or no and complete the accompanying questions, if any.

☐ Yes [move forward to next question]

☐ No [skip next question]

b. Identify the various practices your state adopted from other participants in the national CoP. 

Select all that apply.

☐ Practices related to CoP structure (e.g., organizational structure, meeting schedule)

☐ Practices related to knowledge building and dissemination (e.g., products, tools, training)

☐ Practices related to policy change

☐ Practices related to program or service changes

☐ Practices related to engagement with families and/or self-advocates

☐ Other, please specify: _____________________

c. To the best of your knowledge, have other participants in the national CoP adopted practices 

from your state? Please select yes or no and complete the accompanying questions, if any.

☐ Yes [move forward to next question]

☐ No [skip next question]

d. Briefly describe the practice(s) that other participants in the national CoP adopted from your 

state.

e. Which of the following objectives did you intend to achieve in your state by joining the 

National Supporting Families Community of Practice? Select all that apply.

☐ Enhance supports to families

☐ Improve formal service and support programs for people with I/DD

☐ Improve informal supports for people with I/DD

☐ Increase engagement of family members and self-advocates in service delivery and/or policy 

development

☐ Increase partnerships and collaboration
36

☐ Increase use of the CtLC tools and framework

☐ Other, please specify: _____________________

f. Identify the degree to which having previous exposure to the CtLC framework (before joining 

the expansion state cohort) in your state influenced your state’s ability to achieve outcomes.

g. High – If my state did not have previous exposure to the CtLC framework (before joining the 

expansion state cohort), my state would not have achieved the outcomes we’ve achieved today.

h. Medium – Having previous exposure to the CtLC framework (before joining the expansion 

state cohort) influenced my state’s ability to achieve outcomes, but to a lesser extent.

i. Low – Having previous exposure to the CtLC framework (before joining the expansion state 

cohort) did not influence my state’s ability to achieve outcomes.

Questions Specific to Group 3 States (AL, DE, HI, IN, KS, and OR)

a. To what degree did the opportunity to engage with other states through the national CoP help 

your state CoP to achieve or progress toward its goals within the last year?

b. High – Engaging with other states was essential in achieving or progressing toward my state 

CoP’s goals within the last year.

c. Medium – Engaging with other states was helpful in achieving or progressing toward some of 

my state CoP’s goals within the last year.

d. Low – Engaging with other states did not affect my state CoP’s ability to achieve or progress 

toward goals within the last year.

e. N/A – My state did not engage with other states within the last year.

f. To what extent does your state CoP align with the CtLC framework as a national framework 

for supporting families?

g. Fully – All of my state CoP activities are fully aligned with the CtLC framework.

h. Partially – My state CoP activities are partially aligned with the CtLC framework.

i. Little or not at all – My state CoP activities are minimally or not yet aligned with the CtLC

framework.

j. What contextual factors within your state have influenced your state CoP’s success? Select all 

that apply.

☐ Existing experience with person-centered approaches

☐ Existing partnerships

☐ Meaningful engagement with families and self-advocates

☐ Dedicated CoP leadership

☐ Availability of funding for CoP activities and administration

☐ Other, please specify: _____________________

Variations between groups 1-3

The states involved in the CoP were divided up into three groups for the purpose of the evaluation. 

This appendix refers to the reporting tools used with Group 1 states, which includes CT, DC, MO, 

OK, TN and WA. Group 2 includes states with exposure to the CtLC framework prior to joining 
37

the expansion state cohort. Group 2 states include MD, OH, PA, and SD. Group 3 includes states 

with little or no exposure to the CtLC framework prior to joining the expansion state cohort. Group 

3 states include AL, DE, HI, IN, KS, and OR. The reporting tool questions were kept consistent 

across all three state groupings.
38

Appendix C: Telephone Interview Guide

Purpose

This document will guide the telephone interview conducted with the state CoP team from each 

participating state, and should take one hour or less of time. Interview guides will be tailored to 

each state using data collected from the reporting tool. Facilitators will use this document as a 

guide to cover all necessary areas. Facilitators and note-takers should review each state’s 

respective reporting tool prior to the interview.

Interview Script

Introduction

Introduce yourself and the note taker briefly with your names and role on the project. Ask for or 

state their name and job title/role in the CoP.

Thank you for your time today. In this conversation, we’ll ask questions to get more narrative 

context for the data you shared through the reporting tool. We’re interested to hear about your CoP 

and its activities directly from you. Today, we’ll focus on connecting the dots between activities 

common across states, and progress towards outcomes you reported in the reporting tool, 

especially those affecting the AIDD priority areas.

For note taking purposes, we’d like to record this interview to be able to refer back to. Is your team 

comfortable with us doing so? Wait for them to confirm. Thank you. Do you have any questions 

before we get started?

1. First and foremost, we reviewed your reporting tool responses. Are there any recent changes, 

successes, or challenges within your CoP that you’d like to highlight for us before we get 

started?

2. There are several accomplishments noted in your reporting tool that we’d like to ask more 

about:

· The section includes state specific prompts that are tailored to the data collected from the 

reporting tool data.

· Example Prompt: Your state noted the launch of a new program that has embedded the 

CtLC framework into its structure/goals. Can you provide more details on this 

accomplishment?

Improvements/changes and the CtLC framework

We’d like to ask about the improvements/changes occurring in your state’s CoP and your use of 

the CtLC framework.

1. You reported a number of improvements/changes in the reporting tool as a result of the CoP 

and/or the CtLC framework. [Reference a couple of examples from the reporting tool. List the 

specific performance improvements/changes noted by the state in the reporting tool.]

· Ask the state to expand upon one specific performance improvement referenced above.
39

i. Do you believe the reported improvements/changes would have occurred without 

influence of the CoP and/or CtLC framework?

ii. Without the CoP and/or CtLC framework, would it have taken longer or been more 

difficult to achieve these improvements/changes?

2. In the reporting tool, you indicated progress towards specific AIDD priority areas within the 

last year. I’d like to ask more about the activities that are contributing to your progress. [Ask 

only about priority areas reported in the reporting tool]

List several areas where progress has been made toward specific AIDD priority areas 

referenced by the state. Include whether the state has achieved this goal, or if they are still 

working towards completing it.

Ex. Individuals with I/DD and their families’ ability to access home and community-based 

services and supports that are self-directed and ensure opportunity for full and meaningful 

community participation – in progress (Connecticut)

· Include a state specific prompt that asks the state CoP team to identify the specific 

activities that led to the progress toward AIDD priority areas referenced above.

· One of the initial goals of the project was to achieve state and national consensus on a 

national framework for support families. To what extent have you achieved consensus as a 

state and with the national CoP?

Sustainability

Let’s move on to sustainability.

1. How would you describe your state CoP’s strategy and ability to build capacity and sustain 

your efforts?

2. Are there any lessons learned or promising practices you would share with other states or 

organizations that led to the improvements/changes seen within your state?

Changes in Goals/Strategies

For our last topic, I’m going to ask about changes in goals and expectations.

1. Have you changed goals or strategies because of participation in the CoP? If yes, describe.

2. Have you changed the way you measure success because of participation in the CoP? If yes, 

how?

3. Have you seen any secondary or unexpected outcomes?

Wrap Up

1. Is there anything else you’d like to share today?

2. Do you have any questions before we end?

Thank you so much for participating today! This is incredibly valuable information that we will 

use to inform our evaluation findings. In the next few days, if you think of anything you wish 

you’d shared or asked, please reach us by email. As for next steps, we will soon conduct site visits 

to select states. If your state is selected, you will soon receive an email from a member of the 

evaluation team to determine if this is feasible. Thank you again, and have a great day!
40

Appendix D: Site Visit Guide

Purpose

This guide contains protocols for planning and implementing site visits with CoP states, including 

note-taking templates and questions to guide interviews with CoP leadership.

Lewin will conduct up to six site visits between May and mid-August on an annual basis. Selected 

site visit states will represent various levels of CoP participation, implementation, success, and 

geography across the U.S. During site visits in year one, Lewin will attend prescheduled CoP 

meetings, stakeholder meetings, TA visits conducted by NASDDDS, and/or trainings. During 

these meetings, Lewin will maintain the role of observers. If possible, Lewin will also hold a brief 

meeting with CoP leadership after the prescheduled meeting to discuss any remaining questions. 

This revised approach is intended to reduce evaluation activity burden on states.

Timeline

Exhibit 1 outlines the timeline for activities associated with each site visit counting from the first 

day of the site visit.

Exhibit 1. Site Visit Timeline

Timeframe Action Item Completion 

Date

1- 2 

Months 

Prior

Contact the state point(s) of contact by email and request dates of their 

upcoming CoP meetings, stakeholder meetings, trainings, and/or TA 

visits

60 days before

§ Request meeting with CoP leadership, time permitting

§ Confirm site visit date(s), times, and locations

§ Request agenda and list of attendees

45 days before

Month of 

Site Visit

Develop site visit materials, including an agenda and adapted 

evaluation questions, and send to ACL one week prior to site visit 2 weeks before

If meeting with CoP leadership is permitted, share adapted evaluation 

questions with the state point(s) of contact prior to the site visit 1 week before

Conduct site visit

Develop site visit summary and share with ACL 1 week after

Scheduling

Initial Outreach

The evaluation team sent an email to all states on April 24, 2019 to inform them of the revised 

approach to site visits and that communication will be sent to select site visit states in the coming 

weeks. Email text is included in Figures 2 and 3.
41

Figure 2. Outreach Email Text to All States

From: Natalie Boonchaisri

To: Point(s) of Contact in All CoP States

Subject: ACL Supporting Families CoP Evaluation - Site Visits and Materials Review

Message:

Thank you for your time and participation in ACL’s evaluation of the Supporting Families 

Community of Practice. We’ve enjoyed learning about your activities and achievements through

completion of the reporting tool and telephone interviews to date, and we look forward to 

conducting the remaining telephone interviews. We hope to see many of you at the upcoming 

Annual CoP Meeting and Charting the LifeCourse Showcase next week.

The next step in the evaluation is to conduct site visits with up to six states. After hearing 

feedback from states, NASDDDS, and UMKC, our team is revising our approach to site visits. 

Instead of visiting the six original CoP states, we will select six states from the entire cohort of 

Supporting Families Community of Practice states. This will allow us to gather data from states 

representing various levels of CoP participation, approaches to implementation, and geography 

across the U.S. Additionally, in efforts to reduce state burden, we propose aligning our visits 

with existing activities, such as CoP meetings, stakeholder meetings, TA visits conducted by 

NASDDDS, and/or trainings. Site visits will take place over 1-2 days. If time permits, the 

evaluation team will request to schedule a brief meeting with CoP leadership to discuss any 

remaining questions.

We will be conducting the materials review on an ad hoc basis. Several of you have already 

supplied materials to enhance our understanding of state activities. We will reach out 

individually to request any additional materials based on your reporting tool and telephone 

interview responses.

We will communicate with states proposed for site visits in the coming weeks. If you have any 

questions, please email the evaluation team at Natalie.Boonchaisri@Lewin.com. We thank you 

in advance for your support in this evaluation.
42

Figure 3. Outreach Email Text to Select Site Visit States

From: Natalie Boonchaisri

To: Indiana, Missouri, Maryland, Ohio, Pennsylvania, and South Dakota

Subject: ACL Supporting Families CoP Evaluation - Site Visit Request

Message:

Thank you for your time and participation in ACL’s evaluation of the Supporting Families 

Community of Practice. It was great to see you at last week’s Annual CoP Meeting and 

Charting the LifeCourse Showcase in Kansas City, Missouri. The next step in the evaluation is 

to conduct site visits with up to six states. In efforts to reduce state burden, we propose to attend 

existing CoP meetings, stakeholder meetings, TA visits conducted by NASDDDS, and/or 

trainings. During these meetings, the evaluation team will maintain the role of observers.

Due to the unique activities occurring in [INSERT STATE NAME], the evaluation team 

concluded that a site visit to your state would help inform the evaluation. Are there any CoP 

meetings, stakeholder meetings, TA visits conducted by NASDDDS, and/or trainings 

happening in your state between now and mid-August that your team would welcome us to 

observe? If time permits, we would also like to have a brief 30-minute discussion with CoP 

leadership after the meeting to ask a few remaining questions. Please let us know if this is 

feasible and if you have any questions.

Follow Up

The evaluation team will follow up individually with select states to confirm upcoming meetings, 

request a brief meeting with CoP leadership during the visit, and identify relevant details, including 

time, location, attendees, and agenda.

Sample Agenda

Activity Attendees Location

CoP Meeting,

Stakeholder Meeting,

TA Visit, and/or

Training

§ CoP Leadership

§ Project Partners

§ Stakeholders

§ Lewin Project Manager (or designee)

§ Lewin Task Coordinator

TBD

Meeting with CoP Leadership

§ CoP Leadership

§ Lewin Project Manager (or designee)

§ Lewin Task Coordinator

TBD
43

Site Visit Protocols

Evaluation Team Attendance and Roles

Two members of the evaluation team will conduct each site visit: (1) the project manager or 

designee; and (2) the task coordinator. Both members of the evaluation team will observe the 

identified meeting and take notes using the template in Appendix A. During meetings with CoP 

leadership, both evaluation team members will participate in active discussion and take notes using 

the template in Appendix B.

If requested by the state, team members will introduce themselves and briefly summarize their 

purpose for attending the identified meeting. Team members may respond to questions about the 

evaluation, if requested by the state. The evaluation team will not request time on the agenda to 

address the group, but will gladly provide updates if requested by the state.

Potential CoP Leadership Meeting Topics

The following prompts will guide discussion with CoP leadership, based on available time and 

relevancy to the specific state. Specific questions will be informed by telephone interview 

transcripts and responses to the reporting tool.

¡ Please share your reactions to the identified meeting. What are you pleased with? Do you have 

any areas of concern or challenges you anticipate? Did anything surprise you?

¡ What environmental or contextual factors (e.g., leadership, complying with HCBS final rule, 

waiver renewals, new director/staff) contributed to the achievements/successes seen in your 

state? How are you managing those factors?

¡ Please discuss any other challenges or barriers experienced in your state and strategies used to 

overcome those barriers. Additionally, please discuss any promising practices.

¡ Optional: Do you feel self-advocates and families have access to the supports needed to 

meaningfully engage in your CoP? Have you faced any challenges in your engagement of selfadvocates and families, and have you identified any strategies for overcoming those 

challenges?

¡ Optional: Please discuss any progress toward project sustainability and/or any practices that 

are leading toward scalability or replicability of your model.

¡ Optional: Are there any other updates related to your CoP that we haven’t discussed that you 

would like to share with the evaluation team?

Follow Up

Within a week of each site visit, the evaluation team will send an email to each state thanking them 

for the opportunity to visit and will submit a site visit summary to ACL.