Living Well Cross-Site Evaluation

Volume IV: Methodology

Access the full annual report and other evaluation materials here.
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 2

Volume IV: Methodology

Evaluation Overview

The Administration on Disabilities (AoD), 

through the Administration for Community 

Living’s (ACL) Office of Performance and 

Evaluation (OPE), awarded an evaluation 

contract to New Editions Consulting, which 

subcontracted with The Lewin Group (Lewin). 

In 2017-2018, New Editions gathered data 

from states in Cohort One (Georgia, New 

Hampshire, and Virginia) and documented 

their Living Well models to inform the 

evaluation design. Lewin initiated the 

evaluation in the fall of 2018 and completed 

its third year of data collection in the fall 

of 2021. The purpose of this cross-site 

evaluation is to determine:

• How the Living Well grants are being

implemented across grantee sites;

• Whether Living Well grantees are meeting the goals of the project; and

• Whether the models implemented across the sites impact the quality of life of individuals with

intellectual and developmental disabilities (I/DD).

Each grantee designed at least one integrated model of Community Monitoring and Community

Capacity Building to enhance the independence, integration, health, and safety of individuals

living in the community. Each model addresses eight Key Features specified in the funding

opportunity announcement for Living Well grants (refer to Exhibit 1).

Project Objectives

Grantees are implementing model approaches of a coordinated and comprehensive system

that include two interrelated core components for enhancing and assuring the independence,

integration, safety, health, and well-being of individuals living in the community: (1) Community 

Monitoring, and (2) Community Capacity Building. Each model1 will work towards:

• Enhancing the health and safety of individuals by increasing their independence, social

capital, self-determination, community integration, productivity, and participation.

Exhibit 1. Eight Key Features 

of Living Well Models

1. Partnerships

2. Meaningful and active engagement with

self-advocates and families

3. Evidence based practices for service

improvements

4. Building capacity of DSPs and HCBS

providers

5. Reducing abuse and neglect through

community monitoring

6. Addressing health and safety with data

tools

7. Program and outcome evaluation

8. Sustainability

1 Living Well-Model Approaches for Enhancing the Quality, Effectiveness and Monitoring of Home and Community Based Services for

Individuals with Developmental Disabilities; HHS-2017-ACL-AOD-DNIQ-0221. https://acl.gov/grants/living-well-model-approachesenhancing-quality-effectiveness-and-monitoring-home-and-1
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 3

• Strengthening the knowledge, skills, specialization, economic/career prospects, workforce 

stability, retention, and advancement of direct support professionals.

• Incentivizing the effective adoption and use of evidence based and/or promising practices 

related to supported-decision making, person-centered thinking, competitive integrated 

employment, and similar strategies.

• Strengthening and elevating the leadership roles that self-advocacy organizations, Centers 

for Independent Living, and other peer support networks, in collaboration with families and others, 

must play in improving and assuring the quality of home and community-based services (HCBS).

• Enhancing the effectiveness and coordination of efforts by the state, the Developmental 

Disabilities Assistance and Bill of Rights Act (DD Act) network and others to anticipate, 

avoid, investigate, reduce and remediate abuse, neglect and other rights violations in the 

HCBS service delivery system for people with developmental disabilities.

• Increasing the capacity of states to effectively implement the HCBS settings rules by 

providing such services in the most integrated setting appropriate to the needs of 

individuals with developmental disabilities of all ages.

Evaluation Questions

A set of questions guide the cross-site evaluation (Exhibit 2).

Exhibit 2. Living Well Cross-Site Evaluation Questions

Focus Evaluation Questions

Quality and 

Effectiveness

1. What are the performance measures for determining how the models contribute 

to improving the quality of life of individuals with intellectual and developmental 

disabilities (I/DD)?

2. How effective are each of the approaches to Living Well?

3. What is the best approach for AoD’s future work in this area?

Scalability and 

Replicability

1. Are the models scalable nationally and/or replicable in other states?

2. Is the grantee model at the right level of scale to test the model? Are there 

elements missing?

3. What demographic factors impact or are a factor in the ability to scale the 

Living Well models?

4. What is core to the model, or can parts of the model be scaled? Is the whole 

model needed to be effective?

5. What changes to policy and practice in the service delivery systems are needed 

to replicate the Living Well model?

Sustainability 1. Are these models sustainable?

2. What features are necessary to sustain the projects?

3. How do systems need to change and/or what systems must be in place in 

order to implement one of the models being tested under the Living Well 

Grant Program?
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 4

Data Collection and Analysis

During the 2020-2021 evaluation year, Lewin collected qualitative, quantitative, and 

observational data through interviews, virtual stakeholder meetings, and semi-annual reports 

submitted by grantees to ACL in the spring of 2021. Due to the impact of the COVID-19 

pandemic, Lewin modified its data collection approach by collecting all data virtually and using 

grantees’ semi-annual reports to inform data collection efforts to reduce reporting 

burden on grantees. A standard guide was developed for all grantee interviews 

conducted in the spring of 2021. Data collection tools are available in Appendices A, 

B, C, and D.

Lewin met bi-weekly with ACL and quarterly the Mission Analytics Group technical 

assistance (TA) team to coordinate and streamline activities, where possible. For 

example, the evaluation and TA teams jointly participated in the initial planning 

interviews and shared materials (e.g., meeting and interview notes) with each 

other. The evaluation uses a participatory approach, in which grantees provide 

recommendations on grant meetings or activities for observation, the structure 

and content of grantee profiles documenting grantee’s progress, and the overall 

evaluation approach. Lewin adapted materials and processes in response to 

grantee feedback. 

Lewin modified 

its data collection 

approach by 

collecting all data 

virtually and using 

grantees’ semiannual reports 

to inform data 

collection efforts 

to reduce reporting 

burden on grantees

Initial Planning Interviews. Lewin held one-hour, semi-structured interviews with each 

grantee’s project team and one member of the Mission Analytics Group TA team. 

ACL staff participated as observers in a sample of the interviews. Interview questions 

focused on planned activities during the 2020-2021 grant year, including any changes 

to grant objectives, key contextual factors influencing grant implementation, and 

requests for TA. Lewin recorded each interview, took notes, and used the data to 

plan subsequent evaluation activities.

Semi-Annual Report. Grantees submit reports to ACL semi-annually documenting 

their progress toward grant goals and objectives, challenges encountered, and 

planned activities. Lewin compared its data collection tool for the grant leadership 

interview with each semi-annual report and removed any duplicate questions from 

the data collection tool. Lewin then extracted grantees’ responses from their semiannual reports and analyzed the responses to ensure the data thoroughly addressed 

questions removed from the data collection tools. Lewin documented any gaps and 

added grantee-specific questions to interview guides to collect missing data.

Virtual Stakeholder Meetings. In lieu of in-person site visits, Lewin virtually participated 

in stakeholder meetings with each grantee. These varied events, including workgroup 

meetings, trainings, planning sessions, and stakeholder meetings, ranged in length 

from 90 minutes to multiple days. Lewin documented observations in notes for each 

stakeholder meeting and provided notes to ACL and Mission Analytics Group.
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 5

Key Informant Interviews. The Lewin evaluation team conducted a series of 60-minute 

key informant interviews to characterize each grantee’s model and identify 

components or approaches that may be leading to success. The interviews were held 

with each grantee’s project leadership (e.g., project director/principal investigator, 

project manager/coordinator, evaluator). The interview focused on grant process, 

structure, roles, and contextual factors affecting grant implementation and progress.

Grant Leadership Interviews. As the final data collection activity in this evaluation cycle, 

Lewin conducted a 60-minute interview with members of each grantee’s leadership 

team. The interview focused on progress toward grant objectives, accomplishments 

to date, challenges or contextual factors influencing grant implementation, and 

specific areas of interest identified through analysis of grantee’s semi-annual reports. 

Lewin coded qualitative data from semi-annual reports, key informant interviews, 

and grant leadership interviews using the ATLAS.ti software. This allowed the team 

to triangulate information for the identified themes. Evaluators first categorized data 

using the two core components required for each grantee’s model (Community 

Monitoring and Community Capacity Building), then applied a set of pre-determined 

codes based on the eight Key Features of Living Well models. To ensure interrater reliability, evaluators double-coded a sample of the findings and resolved 

disagreements through discussion. Once discordance was resolved, emergent 

themes were identified. Lewin provided grantees an opportunity to share written and 

verbal feedback on draft annual findings in August 2021.

Timeline

Lewin initiated year two of the evaluation in October 2020, building on activities that occurred 

during years one and two of the evaluation. Exhibit 3 provides a timeline of the year two 

evaluation activities.

Exhibit 3. Timeline of Evaluation Activities 2020-2021

October 2020

Create workplan, 

finalize OY2 products, 

and present findings

October-December 2020

Revise evaluation tools 

& conduct virtual initial 

planning interviews

January 2021

Submit final 

evaluation tools

January-May 2021

Conduct virtual

site visits

March 2021

Modify data collection 

tools - grant leadership and key infomant 

interview guides

April 2021

Conduct key 

informant interviews 

and analyze data

June 2021

Conduct grant leadership 

interviews, analyze data, and 

attend stakeholder meetings

June-September 2021 

Report data
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 6

Appendix A: Initial Planning Interview Guide

The initial planning interview opens the data collection period for the third year of the Living 

Well cross-site evaluation. The interview will provide an opportunity for each grantee to highlight 

key activities planned for the 2020-2021 evaluation year and discuss upcoming meetings and 

events. Grantees will also have an opportunity to ask questions about the evaluation.

At least two members of the evaluation team will participate in each interview. One member of 

the TA team will be invited to attend. One evaluation team member will conduct the interview, 

and one member will take notes. With the grantee’s permission, the team will record the 

interview. Prior to the interview, the team will review each grantee’s most recent semi-annual 

report. Grantees should ensure that they have reviewed the final report from the 2019-2020 

evaluation year. The interviewer will conduct a semi-structured interview using each of the 

following guiding questions and supplementing with optional prompts, as needed, based on the 

information available from the grantee’s semi-annual report.

Opening:

Introduce yourself and the note taker briefly with your names and reiterate role on the project. 

Ask for or state the name(s) and job title(s)/role(s) of the interviewee(s) in the Living Well 

demonstration for the recording.

Thank you for your meeting with us today. We’re excited to hear about the high-level 

developments in your work and plans to support individuals with intellectual and developmental 

disabilities (I/DD) in Living Well.

As a reminder, we’ve asked to record this interview to ensure fidelity in the event that we need 

to clarify sections of our notes. Do we have your permission to record this interview? (Wait for 

them to confirm). Thank you.

Do you have any questions before we get started?

1. How are things going for your team? 

a. Prompt: Are there any new contextual factors that you expect to affect your 

work this evaluation year (e.g., state policy, state budget, engagement of key 

stakeholders, pandemic response)? 

b. Prompt: Do you have any major updates to your Living Well grant team or 

approach since the semi-annual report? Any staffing changes?

2. What are your priorities for this year related to Community Monitoring? We expect this 

to help us focus our evaluation efforts throughout the course of the year. 

a. Prompt: From our Final Report in 2019-2020, we identified:

i. (Activity 1 from 2019-2020 Grantee profile)

ii. (Activity 2 from 2019-2020 Grantee profile)
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 7

b. Prompt: Do you plan to continue with next steps on these activities this evaluation 

year? Are any of the activities completed? Are you planning to begin any new key 

activities related to Community Monitoring this evaluation year?

c. Prompt: Are there any key activities that you have decided to no longer pursue? 

3. What are your priorities for this year related to Community Capacity Building? 

a. Prompt: From our Final Report in 2019-2020, we identified: 

i. (Activity 1 from 2019-2020 Grantee profile)

ii. (Activity 2 from 2019-2020 Grantee profile)

iii. (Activity 3 from 2019-2020 Grantee profile)

b. Prompt: Do you plan to continue with next steps on these activities this evaluation 

year? Are any of the activities completed? Are you planning to begin any new key 

activities related to Community Capacity Building this evaluation year?

c. Prompt: Are there any key activities that you have decided to no longer pursue?

4. Thanks for sharing those details with us. Now we’d like to talk about our evaluation plans 

for the year as well as give you a chance to share any administrative details or plans you 

may already have set.

a. We plan to collect data from the semi-annual reports again and follow-up on this 

data collection through grant leadership interviews during the first half of June. We 

would also like to continue attending stakeholder meetings and/or conduct key 

informant interviews (insert more details of data collection after meeting with ACL).

b. If you know the details of any Living Well events for the upcoming year, we would 

love to hear about them, especially anything upcoming in January. Otherwise, 

we will be reaching out in January to learn more about your planned events and 

meetings.

5. Thank you for providing all of this information. Before we wrap up, do you have any 

questions for the cross-site evaluation team at Lewin?

a. Prompt: Do you have any questions about Lewin’s evaluation structure for the year?

b. Prompt: Is there anything else that you think we should know?

Closing 

Thank you for your time today. If you have any other thoughts, questions, or ideas after today’s 

call, please do not hesitate to reach out to either team by phone or email. If you need contact 

information, we can provide that now.
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 8

Appendix B: Key Informant Interview Guide

Introduce yourself and the note taker briefly with your names and roles on the project. Ask for 

or state the name(s) and job title(s)/role(s) of the interviewee(s) in the Living Well demonstration 

for the recording.

Thank you for meeting with us today. As you know, the purpose of the Living Well grants is to 

increase community integration and independence of individuals with DD and improve HCBS 

by developing and testing one or more model approaches of a coordinated and comprehensive 

system. These model approaches include two interrelated core components: Community 

Monitoring and Community Capacity Building. Our interview today will focus on these model 

approaches. When we use the term “model” throughout this interview, we are focusing on 

the following domains, which are adapted from Wendy Hirsch’s Change Implementation 

Framework:2 Structure, Process, Roles, and Context. Using these interconnected domains will 

help us to understand your model as a whole. We would like to understand how each of these 

has evolved and grown throughout the duration of the grant. In order to do that, we’d like you 

to think about the evolution of your model from when you were first applying for the Living Well 

grant to the present. 

We recognize that some of the topics discussed today may be sensitive or could have political 

implications. We ask that you try to answer our questions honestly, and we will not identify 

information that could identify particular grantees or stakeholders when discussing sensitive 

issues. As always, you will have a chance to review any part of the report that contains 

information about your grant project before it is released publicly.

As a reminder, we’ve asked to record this interview to ensure fidelity in the event that we need 

to clarify sections of our notes. Do we have your permission to record this interview? (Wait for 

them to confirm). Thank you.

Do you have any questions before we get started?

Topic 1: Process

Planning

1. How did you plan and prepare prior to beginning to implement your grant model? This 

includes writing your application and initial planning once you have received the grant. 

Note: While each grant activity/initiative may have its own implementation timeline, these 

questions focus on the holistic grant implementation.

a. In what ways is your planning and preparation facilitating the implementation of 

your grant?

b. Did you have time to adequately prepare? Please describe any barriers or 

challenges related to implementation for which you could not prepare.

1 https://wendyhirsch.com/implementation-framework#:~:text=%EF%BB%BFChange%20Implementation%20Framework,an%20art%20

and%20a%20science
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 9

Implementing

2. How would you describe your implementation process so far (e.g., more linear, more circular)? 

a. Have you made adjustments to your process as you have evaluated your progress?

b. How do you identify process improvements to be made?

c. Describe what, if any, quality assurance measures do you have in place that allow 

you to incorporate changes to your model?

Topic 2: Structure

Planning

3. How did you initially interpret the term ‘model approaches’ as it is used in the FOA?

a. How did your interpretation of the term model approaches impact the structure of 

your project?

4. Beyond your interpretation of the term model approaches, what else impacted your 

project structure (e.g., prior grant experience, existing stakeholder group)?

Implementing

5. In what ways has your project structure evolved in response to changing goals or 

objectives?

6. In what ways does your grant structure (e.g., organization of workgroups, cadence of 

meetings, team communication) facilitate or support progress toward your project goals?

7. In what ways does your grant model or structure pose challenges or barriers to progress 

toward you project goals?

Topic 3: Roles

Planning

8. As you designed your model approaches to this grant, in what ways did you envision 

stakeholders contributing to the success of your model? What roles did you envision 

stakeholders playing?

Implementing

9. How would you characterize the roles of those responsible for leading and implementing 

your grant?

a. Do roles tend to be clearly defined and distinct? Are roles evolving?

b. Does there tend to be a top-down or bottom-up approach to leadership?

10. Grantees are engaging stakeholders across various categories. How have the different 

roles played by these stakeholders contributed to the success of your grant?

i. Self-advocates

ii. Families

iii. DSPs/provider staff

iv. State staff

v. Others
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 10

Topic 4: Context

11. How did you account for contextual factors that you thought may impact your project 

while designing your model? If so, which factors did you expect?

12. How has your understanding of internal/external contextual factors prepared you to adapt 

when faced with barriers or challenges?

a. What took you by surprise?

13. How does your understanding of internal/external contextual factors make you uniquely 

equipped to carry out your grant activities?

a. To what extent are you building on prior successes? 

b. How do past efforts come into play?

Wrap Up

Finally, we want to ask you to reflect on your overall experience developing your overall model 

under this grant. 

14. Is there anything else you would want future grantees to know regarding model 

approaches?

a. What advice about grant structure, process, roles, and/or context would you give 

a future grantee based on what you have learned through this process so far?
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 11

Appendix C: Grant Leadership Interview Guide

Thank you for taking the time to meet with us and for sharing your semi-annual report. The 

purpose of today’s grant leadership interview is to learn more about the current status of 

your grant activities and your progress toward outcomes. While we spent the key informant 

interviews understanding how you have structured your model design as well as the planning 

and implementation behind your grant activities, today we will be focusing more on the content 

of your activities, how you are progressing toward grant outcomes, and we will be following up 

on some items that we have identified in the semi-annual report.

Before we begin, do we have your permission to record this interview? The recording will strictly 

be used for internal, note-taking purposes.

Standard Questions

These are questions that will be asked of all grantees.

1. As you consider the grant objectives for this project, what would you identify as your 

top achievement(s) for this past year? What resources, activities, contextual factors 

contributed to these achievements?

a. How have you leveraged partnerships, including those with self-advocates and 

family members, in the last year to achieve progress toward grant objectives (As 

opposed to individual project goals)?

Grant objectives:

i. Enhancing the health and safety of individuals by increasing their 

independence, social capital, self-determination, community integration, 

productivity and participation.

ii. Strengthening and rewarding the knowledge, skills, specialization, economic/

career prospects and work force stability, retention and advancement of direct 

support professionals.

iii. Incentivizing the effective adoption and use of evidence based or promising 

practices related to supported-decision making, person-centered thinking, 

competitive integrated employment and similar strategies. 

iv. Strengthening and elevating the leadership roles that self-advocacy 

organizations, Centers for Independent Living and other peer support 

networks, working together with families and others, must play in improving 

and assuring the quality of home and community based services.

v. Enhancing the effectiveness and coordination of efforts by the state, the DD 

Act network and others to anticipate, avoid, investigate, reduce and remediate 

abuse, neglect and other rights violations in the HCBS service delivery system 

for people with developmental disabilities
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 12

2. Are there changes that need to occur in your state context or systems to successfully 

implement your grant?

3. What are some grant activities you anticipate doing differently moving forward as a result 

of the COVID-19 pandemic (e.g., new issues gaining traction, shift in focus, continued use 

of virtual formats)?

a. What have you learned through your COVID-19 response that you can apply in the 

future to support the health and safety of individuals with I/DD?

4. You noted in your semi-annual report the following activities related to Community 

Capacity Building:

a. (Activities from semi-annual report)

i. What are your goals in this area, and to what extent are you progressing toward 

your goals? What have been the facilitators and barriers around these efforts?

5. You noted in your semi-annual the following activities related to Community Monitoring: 

a. (Activities from semi-annual report)

b. What are your goals in this area, and to what extent are you progressing toward 

your goals? What have been the facilitators and barriers around these efforts?

6. What outputs and outcomes is your evaluation measuring to determining whether your 

Living Well model contributes to improving the quality of life for individuals with I/DD?

a. How are you using the findings from your evaluation to inform your project 

activities (e.g., refining interventions)?

7. Are there core elements of your Living Well model that must be present for it to be 

effectively scaled or replicated? Is the whole model needed to be effective or can parts of 

it be scaled or replicated?

8. Please describe any progress toward project sustainability in the last year. What key 

elements (either internal or external) are necessary to sustain your project?

Follow up to Semi Annual Report

These will be individualized questions crafted for each grantee based on the data they provided 

during the Semi-Annual Report. Any questions not thoroughly answered through the SemiAnnual Report will be discussed during the grant leadership interview.

9. Question based on semi-annual report

10. Question based on semi-annual report

11. Question based on semi-annual report
LIVING WELL CROSS-SITE EVALUATION: METHODOLOGY 13

Appendix D: Performance Report Template

Recipients of discretionary grants from ACL must submit regular performance reports: A 

semi-annual report at the conclusion of each six-month reporting period and a final report, 

which takes the place of the last semi-annual report, at the conclusion of the grant. Grantees 

document their activities and accomplishments in the semi-annual report by providing a 

narrative response to the following questions:

1. What did you accomplish during this reporting period and how did these 

accomplishments help you reach your stated project goal(s) and objective(s)? Please note 

any significant project partners and their role in project activities.

2. What, if any, challenges did you face during this reporting period and what actions did 

you take to address these challenges? Please note in your response changes, if any, to 

your project goal(s), objective(s), or activities that were made as a result of challenges 

faced.

3. How have the activities conducted during this project period helped you to achieve the 

measurable outcomes identified in your project proposal?

4. What was produced during the reporting period and how have these products been 

disseminated? Products may include articles, issue briefs, fact sheets, newsletters, 

survey instruments, sponsored conferences and workshops, websites, audiovisuals, and 

other informational resources.