Living Well 

Cross Site Evaluation

Volume III 

Methodology
Living Well Cross Site Evaluation Volume III: Methodology 2

I. Evaluation Overview

The Administration on Disabilities (AoD), through the Administration for Community Living’s (ACL) Office of

Performance and Evaluation (OPE), awarded an evaluation contract to New Editions Consulting, which subcontracted

with The Lewin Group (Lewin). In 2017-2018, New Editions gathered data from states in Cohort One (Georgia, New

Hampshire, and Virginia) and documented their Living Well models to inform the evaluation design. Lewin initiated the

evaluation in 2018. The purpose of this cross-site evaluation is to determine:

• How the Living Well grants are being implemented across grantee sites;

• Whether Living Well grantees are meeting the goals of the project; and

• Whether the models implemented across the sites impact the quality of life of individuals with intellectual and

Exhibit 1. Eight Key Features of Living Well Models 

Partnerships

Initiation and coordination of 

partnerships or coalitions with 

local and state-level organizations, 

agencies, and other relevant 

stakeholders, including at least 

one-self advocacy organization, in 

the design, implementation, and 

replication of grantee activities

Meaningful and active 

engagement with selfadvocates and families

Continuous, meaningful, and active 

engagement of self-advocates and 

family members throughout the life 

cycle and in all stages of the project

Evidence based practices 

for service improvements

Use of evidence based and 

innovative strategies to (1) improve 

access to and quality of community 

services, (2) reduce and mitigate 

abuse and neglect, and (3) support 

empowerment, self-determination, 

and self-advocacy

Building capacity of DSPs 

and HCBS providers

Prevention-based tools and 

technical assistance to address 

common needs, such as changing 

the ‘culture of abuse and neglect’ 

in HCBS settings and transferring 

knowledge of positive behavior

Reducing abuse 

and neglect through 

community monitoring

Collection, analysis, and 

dissemination of data to develop 

and implement coordinated 

community monitoring that 

builds on existing local or state 

infrastructure and partnerships

Addressing health and 

safety with data tools

Data tools and evidence based 

practices for monitoring highrisk individuals and addressing 

reoccurring issues of health and 

safety concerns

Program and 

outcome evaluation

Process and outcome evaluation 

to analyze delivery and impact of 

project activities

Sustainability

Assurance of organizational, 

financial, and/or community 

stability to continue and refine 

grantee work

II. Project Objectives

Grantees are implementing model approaches of a coordinated and comprehensive system that include two interrelated

core components for enhancing and assuring the independence, integration, safety, health, and well-being of individuals

living in the community: (1) community monitoring, and (2) community capacity building. Each model will work toward:

• Enhancing the health and safety of individuals by increasing their independence, social capital, self-determination,

community integration, productivity and participation.

• Strengthening the knowledge, skills, specialization, economic/career prospects and work force stability, retention and

advancement of direct support professionals.

• Incentivizing the effective adoption and use of evidence based and/or promising practices related to supporteddecision making, person-centered thinking, competitive integrated employment and similar strategies
Living Well Cross Site Evaluation Volume III: Methodology 3

• Strengthening and elevating the leadership roles that self-advocacy organization, Centers for Independent Living and

other peer support networks, working together with families and others, must play in improving and assuring the

quality of home and community based services.

• Enhancing the effectiveness and coordination of efforts by the state, the Developmental Disabilities Assistance and

Bill of Rights Act (DD Act) network and others to anticipate, avoid, investigate, reduce and remediate abuse, neglect

and other rights violations in the home and community-based services (HCBS) service delivery system for people

with developmental disabilities.

• Increasing the capacity of States to effectively implement the HCBS settings rules by providing such services in the

most integrated setting appropriate to the needs of individuals with developmental disabilities of all ages.

III. Evaluation Questions

A set of questions guide the cross-site evaluation (Exhibit 2).

Exhibit 2. Living Well Cross-Site Evaluation Questions

Focus Evaluation Questions

Quality and 

Effectiveness

What are the performance measures for determining how the models contribute to improving 

the quality of life of individuals with intellectual and developmental disabilities (I/DD)?

How effective are each of the approaches to Living Well?

What is the best approach for AIDD’s future work in this area?

Scalability and 

Replicability

Are the models scalable nationally and/or replicable in other states?

Is the grantee model at the right level of scale to test the model? Are there elements missing?

What demographic factors impact or are a factor in the ability to scale the Living Well models?

What is core to the model, or can parts of the model be scaled? Is the whole model needed to 

be effective?

What changes to policy and practice in the service delivery systems are needed to replicate the 

Living Well model?

Sustainability

Are these models sustainable?

What features are necessary to sustain the projects?

How do systems need to change and/or what systems must be in place in order to implement 

one of the models being tested under the Living Well Grant Program?

IV. Data Collection and Analysis

During the 2018-2019 evaluation year, Lewin collected qualitative, quantitative, and observational data through three

methods: (1) interviews, (2) annual reporting tool, and (3) site visits. Prior to collecting data, Lewin reviewed the narrative

each grantee submitted with its grant application and semi-annual reports submitted by grantees in Cohort One to ACL

during the first year of their grant. Using an iterative approach to developing data collection tools, Lewin designed an

interview guide based on these background materials. The annual reporting tool and site visit protocol also built on

previous data collection activities. Data collection tools are available in Appendices A, B, and C.
Interviews. Lewin held one-hour, semi-structured interviews with each grantee’s project team. Interview questions aligned 

with the eight key features and focused on grantees’ project design and progress to date. Lewin recorded and transcribed 

each interview and used the data to inform evaluation activities for the remainder of the year.

Annual Reporting Tool. Following the interviews, each grantee submitted responses to an annual reporting tool. The 

reporting tool gathered a combination of qualitative and quantitative data documenting grantee’s progress toward project 

goals by key feature. Grantees submitted their data using an online tool. The evaluation team reviewed each response for 

completeness and uploaded data to ATLAS.ti for analysis. Lewin coded data by key feature, then applied descriptive codes 

and identified themes.

Site Visits. Lewin conducted a site visit with each grantee, which included observation of a Living Well stakeholder 

meeting and a meeting with the grant leadership team. Lewin developed a site visit protocol, which guided data collection 

during each visit. The content and structure of each grantee’s stakeholder meeting varied based on the focus and needs of 

the particular grant. Following each visit, the Lewin team documented their observations in a site visit summary and 

identified themes that emerged from the visits.

V. Timeline

Building on the preliminary evaluation activities that occurred in 2017-2018, Lewin initiated year one of the evaluation in

November 2018. A timeline of activities indicates when each activity occurred (Exhibit 2).

Exhibit 2. Timeline of Year One Evaluation Activities

Living Well Cross Site Evaluation Volume III: Methodology 4
Living Well Cross Site Evaluation Volume III: Methodology 5

Appendix A

Interview Guide
Living Well Cross Site Evaluation Volume III: Methodology 6

Purpose

This document will guide interviews with Living Well grantees and should take no longer than one hour to complete. 

Facilitators will use this document as a guide to encourage a conversation with grantees about their program, rather than 

as an exact script. Sub-questions are additional prompts that the facilitator should consider using to guide the dialogue 

away from answers we can collect from grantee applications and reports, and toward areas of new information or greater 

specificity. The facilitator and note taker should be familiar with the grantee’s application and status reports and should 

know in advance which bullets will be useful in prompting conversation.

Interview Script

Introduction

Introduce yourself and the note taker briefly with your names and role on the project. Ask for or state the name(s) and job 

title(s)/role(s) of the interviewee(s) in the Living Well demonstration for the recording.

Thank you for your meeting with us today. Through these annual conversations, we’re looking to gather contextual 

information to deepen our understanding of how you developed your intervention and areas of focus, and to identify 

your state’s capacity across the eight Key Features that underpin the Living Well demonstration. We’re interested to hear 

in your own words about the work you are doing increase community integration and independence of individuals with 

developmental disabilities and to improve the quality of home and community-based services (HCBS)

We know that your activities include multi-faceted approaches to achieving system-wide change, and recognize that 

our time is limited, so we’ll focus today on activities common across grantees, particularly those efforts affecting the 

Administration on Intellectual and Developmental Disabilities’ (AIDD) areas of focus: 

• The development of a coordinated system for Community Monitoring 

• Improvements to community services that support individuals with I/DD through Community Capacity Building

As a reminder, we’ve asked to record this interview to ensure fidelity in the event that we need to clarify sections of our 

notes. Do we have your permission to record this interview? (wait for them to confirm). Thank you.

Do you have any questions before we get started?

Context for the Intervention

Let the interviewees know you’re going to start with questions about the broader context of their intervention. Also let 

grantees know we have reviewed their application and other materials, including call notes, and would like to ask some 

follow up or clarifying questions.

1. Before we begin, are there any recent updates, success stories, or challenges that you would like to share? Overall, 

how are things going so far?

2. How did you select your interventions and area(s) of focus?

Partnerships and Engagement of Advocates

Share that we will ask about stakeholder partnerships and engagement of self-advocates and families.

1. How did you identify stakeholders to engage as partners in the design and implementation of your Living Well 

model? 

a. Do these relationships pre-date the Living Well project? How do you anticipate these partnerships will support 

your efforts under this grant?
Living Well Cross Site Evaluation Volume III: Methodology 7

2. How did you identify self-advocates and families of individuals with I/DD to engage in the development and 

implementation of your project?

a. How does the engagement of advocates fit into the broader partnerships you have built to implement this 

model?

Service Improvement and Direct Support Professional (DSP) Training

Share that we will now ask about strategies for improving I/DD services in the community.

1. How did you identify opportunities for service improvements in your state?

a. What is the evidence base you drew from in designing service improvements in your state?

b. Are there any barriers or challenges to service improvement that you have encountered or anticipate?

2. What trainings or technologies are you employing in your intervention to build DSP and provider competencies?

a. State-specific prompt (Alaska): We see from your grant application that one of your key strategies to strengthen 

the network of DSPs in Alaska is to develop distance-delivered trainings on person-centered thinking and 

supportive decision-making. How are you approaching development of these trainings? Did you draw on 

existing trainings in the development of these programs?

b. What is your approach to implementation? Are these trainings and/or technologies being launched statewide, 

or with staggered rollout?

c. What infrastructure or capacity currently exists in your state to support this implementation?

3. Have you seen any service improvements to date? If so, can you share what they are?

Community Monitoring

The next section addresses plans for community monitoring and enhanced support for individuals with increased risk.

1. What agencies and community partners constitute your state’s system for community monitoring?

a. How do these partners coordinate to close gaps in community monitoring? What is your process for 

collaboration?

2. Are there tools or technologies that your state employees use to facilitate the tracking, redressing, and prevention 

of abuse and neglect?

3. How are individuals who are at increased risk identified?

a. What tools or strategies do you employ to monitor and address the health and safety of these individuals, while 

encouraging community integration?

b. State-specific prompt (Alaska): We see from your grant application that a key strategy of your efforts to 

increase independence, integration, and safety of individuals with I/DD includes reducing incarceration and 

institutionalization. What is Alaska’s current capacity for Crisis Intervention and Jail Diversion, and does your 

approach include these elements?

4. Have you seen any changes in the quality of the monitoring efforts or the results of these efforts on consumer 

safety/protection so far? If so, can you share what these are?

Evaluation

Share that we will briefly discuss the program evaluation plan.

1. How are you measuring outcomes from the intervention?

Sustainability

Share that we will now discuss sustainability.

1. Have you given any early consideration to sustaining the project long-term?
Living Well Cross Site Evaluation Volume III: Methodology 8

Wrap Up

Share that we will now conclude the interview.

1. Is there anything else you’d like to share today?

2. Do you have any questions before we end?

Thank you so much for participating today! We appreciate the opportunity to hear from you directly about the valuable 

work that you and your partners are doing to improve supports for individuals with I/DD. In the next few days, if you think 

of anything you wish you’d shared or asked, you can reach me via email at living_well_evaluation@lewin.com. Thank you 

again, and have a great day!
Living Well Cross Site Evaluation Volume III: Methodology 9

Appendix B

Annual Reporting 

Tool Questions
Living Well Cross Site Evaluation Volume III: Methodology 10

Purpose

Annually, this reporting tool will collect standardized information from each grantee on their Living Well project structure 

and members, capacity across the Eight Key Features of the grant, and improvements made to integration and community 

monitoring. Information regarding changes made to policies, programs and services, and initiatives, as well as the impact 

or anticipated impact of these changes will also be captured. This tool is designed to capture changes over the lifespan of 

the grant across all grantees, and it is expected that some questions will not be applicable to all grantees due to their areas 

of focus or timeline of activities. Along with telephone interviews, the evaluation team will use information from this tool 

to inform specific questions asked during site visits. All grantees are asked to complete the reporting tool one time this 

evaluation year. Completion of this web-based survey tool is self-directed, with an expected time commitment of less than 

four hours. 

Instructions

1. Please complete each question to the best of your abilities and based on your understanding of the question in 

context of your state. Additionally, please try to complete questions in the order that they appear, as the online 

tool contains skip logic, which may change the numbering of the questions from what is seen in this document. 

However, if you do not have the answer to a question in front of you, and need to return to that question later 

within the online tool, you will be able to do so. Questions may be in the format of multiple choice, check boxes, 

or descriptive narrative.

2. Please submit your responses using the following Research.Net link: https://www.research.net/r/QX2GWL7. This 

Word document contains all questions asked through Research.Net and can be shared among respondents to 

prepare their responses.

3. These questions are to be completed by members of the Living Well leadership team for your grant that oversee 

day-to-day Living Well activities. If you do not know the answer to a certain question, please inquire with a 

colleague who knows the answer. If this is not possible or you do not have any activity to report in a particular 

area, please leave that field blank.

4. If a question is not applicable to your project design or timeline of activities, please select “Not Applicable” or leave 

the field blank.

5. Note: When we refer to the word “state” within this document, we are referring to all of the activities that occur 

within your state. We are not referring specifically to state-funded activities or activities for which a state agency 

is responsible.

If you have any questions, please email the evaluation team at Living_Well_Evaluation@Lewin.com

Reporting Tool Questions

Living Well Project Partnerships

1. Identify the stakeholders that are currently represented on the formal body of stakeholders for your Living Well 

grant. In this tool, formal body of stakeholders refers to the partnerships with local and state level organizations 

established through MOUs and/or interagency agreements for the purpose of the Living Well grant.

(In subsequent years, grantees will be asked only to explain any changes to this group, i.e. Question #2.)

Select all that apply.

☐ Self-Advocacy Organization

☐ Family Network or Organization

☐ Advocacy Organization

☐ State DD Agency

☐ Service Provider
Living Well Cross Site Evaluation Volume III: Methodology 11

☐ Case Management

☐ Developmental Disabilities (DD) Council

☐ University Center for Excellence in Developmental Disabilities (UCEDD)

☐ State Protection & Advocacy Agency (P&A)

☐ Other, please specify: _____________________

2. Identify any changes in stakeholder participation over the last year, including the addition of any new partners 

not reflected in the MOU, or the withdrawal of any partners previously engaged on the grant. If any changes in 

stakeholder participation occurred, please also indicate why the changes occurred.

3. In what ways are the formal body of stakeholders convened for the purpose of this grant advancing your project 

goals?

Select all that apply.

☐ Aligning Living Well with other projects or initiatives in the state

☐ Facilitating access to data or information

☐ Identifying or securing needed resources

☐ Initiating partnerships with new or different stakeholders

☐ Strengthening the voice of self-advocates and families in statewide systems or policy change

☐ Providing technical expertise 

☐ Providing cultural expertise or brokerage

☐ Other, please specify: _____________________

4. Please describe your activities associated with each response selected for the previous question since the beginning 

of your grant.

Engagement of Self-Advocates and Family Members

5. How have self-advocates engaged with your Living Well project since the project began? In this question, Living 

Well models refers to all aspects of your Living Well project related to community monitoring and community 

capacity building.

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

Select all that apply.

☐ Developing Living Well models

☐ Testing Living Well models

☐ Implementing Living Well models

☐ Scaling Living Well models

☐ Other, please specify: _____________________

☐ Not Applicable

6. Please describe activities associated with each response selected for the previous question.

7. How have family members of individuals with I/DD engaged with your Living Well project since the beginning 

of your grant? In this question, Living Well models refers to all aspects of your Living Well project related to 

community monitoring and community capacity building.

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)
Living Well Cross Site Evaluation Volume III: Methodology 12

Select all that apply.

☐ Developing Living Well models

☐ Testing Living Well models

☐ Implementing Living Well models

☐ Scaling Living Well models

☐ Other, please specify: _____________________

☐ Not Applicable

8. Please describe activities associated with each response selected for the previous question.

9. What actions have you taken since beginning of your grant to advance or improve full and meaningful 

participation of self-advocates and families in your Living Well initiative?

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

Select all that apply.

☐ Provided support outside of meetings to understand materials and meeting objectives

☐ Provided transportation, childcare, respite services, or other supports necessary to attend meetings

☐ Ensured self-advocates or families lead project activities

☐ Changed project activities based on self-advocate or family feedback

☐ Other, please specify: ____________________

☐ Other, please specify: ____________________

☐ Other, please specify: ____________________

☐ Not Applicable

10. For each of the following Living Well grant outcomes, please indicate the extent to which you have progressed 

toward this outcome since the beginning of your grant.

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

☐ Enhanced the health and safety of individuals by increasing their independence, social capital, selfdetermination, community integration, productivity, and/or participation

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Strengthened the knowledge, skills, specialization, economic/career prospects, workforce stability, retention, 

and/or advancement of direct support professionals

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Incentivized the effective adoption and use of evidence-based and/or promising practices related to supported 

decision-making, person-centered thinking, competitive integrated employment, and similar strategies.

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Elevated the leadership roles that self-advocacy organizations, centers for independent living, and other peer 

support networks play in improving and assuring the quality of home- and community-based services

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)
Living Well Cross Site Evaluation Volume III: Methodology 13

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Not Applicable/Other (please specify): _____________________

11. For each item that you responded “To a Moderate extent” or “To a Great extent,” please describe your activities 

since the beginning of the grant.

(If you responded “Not Applicable” to the previous question, you may skip this question).

Building the Competencies and Capacity of DSPs and HCBS Providers

12. Since the beginning of your grant, have you provided any training to DSPs and HCBS service providers to 

support improvements to service delivery and/or quality?

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

☐ Yes [If selected, proceed to question #13]

☐ No [If selected, skip questions #13-19 and begin at question #20]

13. Please list the training activities in which you engaged since the beginning of your grant.

14. Were these existing trainings, or were they specially developed for the Living Well project?

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

☐ Existing trainings previously utilized in the state

☐ Existing trainings not previously utilized in the state

☐ Newly developed trainings created for this grant

☐ Other, please specify: ____________________

15. How are the trainings employed for your Living Well project delivered?

(In subsequent years, grantees will only be asked to report on activity since the previous evaluation cycle.)

Please select all that apply.

☐ In-Person

☐ Live Webinar

☐ Web-Based Recording

☐ Web-Based Interactive

☐ Other, please specify ______________

16. How are learner outcomes being tracked?

Please select all that apply.

☐ Pre/Post-Tests

☐ Retrospective Pre-Tests

☐ Feedback Surveys

☐ Group Discussions

☐ On-the-Job Review or Monitoring

☐ Other, please specify ______________

☐ Not Applicable
Living Well Cross Site Evaluation Volume III: Methodology 14

17. Since the beginning of your grant, have you scaled or modified tools, training, or technical assistance to DSPs and 

HCBS providers to reach larger audiences?

☐ Yes [If selected, proceed to question #18]

☐ No, but in planning [If selected, proceed to question #18]

☐ No [If selected, skip question #18 and proceed to question #19]

18. Please describe the tools, trainings, and/or technical assistance that you have scaled or modified since the 

beginning of your grant to reach larger audiences.

19. Please describe how your tools, training, or technical assistance for DSPs and HCBS providers are leading 

to evidence-based practices to reduce the incidence of and risk factors for abuse, neglect, and exploitation of 

individuals with I/DD living in the community.

Community Monitoring and Supporting Individuals at Increased Risk

20. Please identify the entities responsible for collecting, analyzing, or sharing relevant data related to monitoring 

abuse and neglect of people with I/DD in your state at this time.

(In subsequent years, grantees will be asked to explain any changes to this structure or capacity.)

☐ State DD agency

☐ Adult Protective Services

☐ Child Protective Services

☐ Protection and Advocacy Agency

☐ Providers

☐ Case management/support coordination entities

☐ Other state agency (please specify): ____________________

☐ Other (please specify): ____________________

☐ Other (please specify): ____________________

21. Please describe how you are measuring progress toward increased health and safety of individuals with I/DD.

22. For each of the following strategies to support community monitoring, please indicate the extent to which you 

have used the strategy since the beginning of your grant.

☐ Completing on-site safety reviews

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Documenting implementation to confirm corrections have been made to deficient practices

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Establishing corrective action tracking systems

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Monitoring health and safety against Appendix G in the state’s HCBS waiver application or an emergency 

response process to review unexpected deaths
Living Well Cross Site Evaluation Volume III: Methodology 15

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Identifying risk and/or safety concerns

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Using findings to develop, implement, and improve the tools, technical assistance, and other strategies 

employed under the grant to reduce abuse and neglect within HCBS settings

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Not Applicable/Other (please specify) ______________

23. For each item that you responded “To a Moderate extent” or “To a Great extent,” please describe the impact or 

anticipated impact of the selected strategies in state progress toward advanced health and safety of individuals 

with I/DD.

[If you responded “Not Applicable” to the previous question, you may skip this question].

Evaluation

24. As a result of evaluating Living Well activities to date, have you made any changes to your project design or 

approach?

☐ Yes [If selected, proceed to question #25]

☐ No [If selected, skip question #25 and begin at question #26]

25. Please describe how you have made changes to your project design or approach as a result of your evaluation 

activities to date.

Sustainability, Scalability, and Replicability

26. For each of the following strategies to sustain the Living Well project after the grant expires, please indicate the 

extent to which you have used the strategy since the beginning of your grant.

☐ Identifying sources of funding to sustain project work after the grant expires

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ State-level policy change

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Creation of permanent training resources

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)
Living Well Cross Site Evaluation Volume III: Methodology 16

☐ Creation of implementation toolkits to facilitate replicability

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Building training capacity through train-the-trainer models

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Building local (e.g., provider, county boards) capacity to sustain change

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Empowering self-advocates and families

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Systems change through integration with other grants and projects 

ƈ Not at all (No activity to report since the beginning of the grant, but there may be activity in the future)

ƈ To a moderate extent (Some activity to report since the beginning of the grant, with more planned in the 

future)

ƈ To a great extent (This has been a major area of focus)

☐ Not Applicable/Other (please specify) __________

27. Please list the other grants and projects across your state and network of partners with which you are integrating 

Living Well.

28. What contextual factors within your state have influenced your approach to Living Well in your state?

Select all that apply.

☐ Existing experience with person-centered approaches

☐ Existing partnerships with stakeholders

☐ Meaningful engagement with families and self-advocates

☐ Availability of funding for Living Well activities and administration

☐ Continuity of partners and leadership across efforts to achieve system change

☐ Alignment of Living Well activities with other initiatives in the state

☐ Public awareness of incidents or gaps in the state’s monitoring of abuse and neglect

☐ Other, please specify: _____________________

Additional Information

29. If there is more you’d like to tell us about your state’s activities, please use the box below to provide us with any 

additional information.

Please insert this information into the online tool.
Living Well Cross Site Evaluation Volume III: Methodology 17

Appendix C

Site Visit Protocol
Living Well Cross Site Evaluation Volume III: Methodology 18

Purpose

This guide contains protocols for planning and implementing site visits with the eight Living Well grantees, including notetaking templates and questions to guide interviews with grant leadership. 

Lewin will conduct the site visits annually between April and July. During site visits in year one, Lewin will attend a Living 

Well stakeholder meeting and hold a meeting with the Grant Leadership Team for each Living Well grantee. The evaluation 

team may schedule additional activities based on the grantees level of implementation at the time of the site visit.

Timeline

Figure 1 outlines the timeline for activities associated with each site visit counting from the first day of the site visit.

Figure 1. Site Visit Timeline

Timeframe Action Item Completion Date

1-2 Months Prior

Contact the state point of contact by email and request dates of their 

upcoming stakeholder meetings. 60 days before

Request one-hour grantee meeting with leadership after stakeholder 

meeting

Confirm site visit date(s), times, and locations

Request agenda and list of attendees

45 days before

Month of Site Visit

Develop site visit materials, including an agenda and adapted evaluation 

questions, and send to ACL one week prior to site visit 2 weeks before

Share adapted evaluation questions with the state point of contact prior to 

the site visit 1 week before

Conduct site visit

Develop site visit summary and share with ACL 1 week after

Scheduling

Initial Outreach

The evaluation team sent an email to all grantees on March 13, 2019 to request their availability for an upcoming site 

visit. Email text is included in Figure 2.
Living Well Cross Site Evaluation Volume III: Methodology 19

Figure 2. Outreach Email Text

From: The Evaluation Team

To: Site Visit Leads

Subject: Living Well Evaluation Activities

Message:

Good Afternoon,

Thank you again for taking the time to speak with us last month during the Living Well evaluation interview. We are writing to request your 

action on scheduling the Annual Living Well Evaluation Site Visit.

Please confirm your site visit availability by Wednesday, March 20, 2019 by email.

Site visits will take place over 1-2 days, during which time we aim to engage with a broader audience of stakeholders to learn more about 

your Living Well project activities. These site visits will take place across a window running from mid-April through the end of July, and we 

are looking to coordinate the timing so that we can overlap with any planned stakeholder meetings or activities within your state for your 

Living Well grant. If you have any planned Living Well stakeholder meetings between April and July that the evaluation team could attend, 

please provide the date(s) or an approximate timeframe if dates are still pending.

If you do not have planned Living Well stakeholder meetings between April and July or prefer to schedule your site visit at another time, 

please provide suggested dates or timeframes that would work well for a site visit with your team.

If you have any questions, please email the evaluation team at Living_Well_Evaluation@Lewin.com. We thank you in advance for your 

support in this evaluation.

Sincerely,

The Living Well Evaluation Team

Follow Up

The evaluation team will follow up individually with each grantee to confirm availability during the identified 

stakeholder meeting, request a meeting grant leadership during the visit, and identify relevant details, including time, 

location, attendees, and agenda.

Sample Agenda

Time Activity Attendees Location

Half-Day/Full-Day Meeting

Grant Leadership Team 

Meeting with Stakeholder 

Groups

Grant Leadership

Project Partners as indicated by MOU

Lewin Project Director (or designee)

Lewin Task Coordinator (or designee)

TBD

One-Hour Meeting Meeting with Grant 

Leadership Team

Grant Leadership

Lewin Project Director (or designee)

Lewin Task Coordinator (or designee)

TBD

Other Activities, TBD TBD TBD TBD
Living Well Cross Site Evaluation Volume III: Methodology 20

Site Visit Protocols

Evaluation Team Attendance and Roles

Two members of the evaluation team will conduct each site visit: (1) the project director or manager or designee; and 

(2) the task coordinator or designee. One member of the evaluation team will conduct the interview with each grantee,

with the second member taking notes using the template in Appendix A. During stakeholder meetings, both evaluation

team members will take notes using the template in Appendix B.

Living Well Stakeholder Meeting

Members of the evaluation team will participate in each grantee’s stakeholder meeting as observers. As requested by the 

grantee, team members will introduce themselves and briefly summarize their purpose for attending the stakeholder 

meeting. Team members may respond to questions about the cross-site evaluation, if requested by the grantee. During 

year one, the evaluation team will not request time on the agenda to address the group.

Living Well Grant Leadership Interview 

Participants

• Grant Leadership (e.g., Project Director/PI, Project Coordinator, Evaluator)

• Lewin Project Director (or designee)

• Lewin Task Coordinator (or designee)

Room Set Up

• If possible, arrange everyone around one big table so the group can all see each other.

• Use a circle, semi-circle, or square set up.

• Avoid classroom or lecture style seating.

• The note taker can sit at or just outside the circle.

Introductory Script

Members of the evaluation team will introduce themselves to each person as they arrive and start once all expected 

participants arrive or briefly after the scheduled start time if participants are still gathering. 

Thank you for the information your state submitted through data reporting and everything you shared in phone interviews. 

Today we’re going to touch on some of the same topics to get more information and clarity. We want to learn more 

about activities in your state. We’re especially interested in what leads to positive outcomes. I’m going to ask a series of 

conversation prompts. We’ll also leave time at the end so everyone can share additional thoughts they have.

Before we jump in, could we first go around the room for introductions? Please share your name, any organization 

affiliation, and your role on the Living Well grant.

Discussion Prompts

The following prompts will guide discussion with grant leadership. The evaluation team will discuss all items in the 

first section and select from the other prompts below based on available time and relevant topics to the specific grantee. 

Specific questions will be informed by telephone interview transcripts and responses to the reporting tool.

The following questions are required of all sites:

• Are there any recent developments or milestones for your Living Well project that you would like to share?

• [If leadership meeting occurs after stakeholder meeting] Please share your reactions to the stakeholder

meeting. What are you pleased with? Do you have any areas of concern or challenges you anticipate? Did
Living Well Cross Site Evaluation Volume III: Methodology 21

anything surprise you?

• [If leadership meeting occurs before the stakeholder meeting] Please share any expectations you have about

the stakeholder meeting. What are you excited about? Do you have any areas of concern or challenges you

anticipate?

• [The following prompts are structured around the 8 Key Features of Living Well models.]

• Organizational partnerships – Please discuss the level of buy-in to your Living Well project by partners,

including any variations in the level of engagement or commitment by partners. Do you feel partners have

clarity on their role and the vision for the project? Are there any partners that you anticipate needing to engage

in the future? Have you faced any challenges in your engagement of partners, and have you identified any

strategies for overcoming those challenges?

• Active engagement with self-advocates and families – Please discuss the level of buy-in to your Living

well project by self-advocates and family members, including any variations in the level of engagement or

commitment. Do you feel self-advocates and families have clarity on their role(s) in the project? Do you feel

self-advocates and families have access to the supports needed, if any, to meaningfully engage in your Living

Well project? Have you faced any challenges in your engagement of self-advocates and families, and have you

identified any strategies for overcoming those challenges?

The following prompts are optional:

• Evidence based practices and innovative strategies – Please describe any progress toward implementing

your evidence-based practices since we last spoke. Are there any changes to your project design or highlights

(successes or challenges) to share?

• Direct support professionals – Please describe your progress toward building the competency and capacity of

DSPs since we last spoke. Are there any changes to your project design or highlights (successes or challenges)

to share?

• Systems for community monitoring – Please describe your progress toward enhanced systems for

community monitoring since we last spoke. Are there any changes to your project design or highlights

(successes or challenges) to share?

• Data tools and evidence based practices to monitor and address health and safety of individuals with

developmental disabilities – Please describe your progress toward monitoring and addressing health and

safety since we last spoke. Are there any changes to your project design or highlights (successes or challenges)

to share? Are you able to access needed data sets to produce your desired measurements?

• Evaluation plan – Please discuss how your internal evaluation is guiding and informing the implementation

of your Living Well project.

• Sustainability/Scalability/Replicability plan – Please discuss any progress toward project sustainability and/

or any practices that are leading toward scalability or replicability of your model.

• Are there any other updates related to your Living Well grant that we haven’t discussed that you would like to

share with the evaluation team?

Follow Up

Within a week of each site visit, the evaluation team will send an email to each grantee thanking them for the opportunity 

to visit and will submit a site visit summary to ACL.