Living Well 2020 

Cross-Site Evaluation

Volume IV 

Methodology
Living Well Cross Site Evaluation Volume IV: Methodology 2

I. Evaluation Overview

The Administration on Disabilities (AoD), through the Administration for Community Living’s (ACL) Office of 

Performance and Evaluation (OPE), awarded an evaluation contract to New Editions Consulting, which subcontracted 

with The Lewin Group (Lewin). In 2017-2018, New Editions gathered data from grantees in Cohort One (Georgia, New 

Hampshire, and Virginia) and documented their Living Well models to inform the evaluation design. Lewin initiated the 

evaluation in 2018 and completed its second year of data collection, analysis, and reporting in 2020. The purpose of this 

cross-site evaluation is to determine:

• How the Living Well grants are being implemented across grantee sites;

• Whether Living Well grantees are meeting the goals of the project; and 

• Whether the models implemented across the sites impact the quality of life of individuals with intellectual and 

developmental disabilities (I/DD).

Exhibit 1. Eight Key Features of Living Well Models 

Partnerships Meaningful and active 

engagement with selfadvocates and families

Evidence based 

practices for service 

improvements

Building capacity 

of DSPs and HCBS 

providers

Reducing abuse 

and neglect through 

community monitoring

Addressing health and 

safety with data tools

Program and 

outcome evaluation

Sustainability

Each grantee designed at least one integrated model of community monitoring and capacity building to enhance the 

independence, integration, health, and safety of individuals living in the community. Each model addresses eight key 

features specified in the funding opportunity announcement for Living Well grants (see Exhibit 1). 

II. Project Objectives

Grantees are implementing model approaches of a coordinated and comprehensive system that include two interrelated 

core components for enhancing and assuring the independence, integration, safety, health, and well-being of individuals 

living in the community: (1) Community Monitoring, and (2) Community Capacity Building. Each model will work 

toward:

• Enhancing the health and safety of individuals by increasing their independence, social capital, self-determination, 

community integration, productivity, and participation.

• Strengthening the knowledge, skills, specialization, economic/career prospects, workforce stability, retention, and 

advancement of direct support professionals.

• Incentivizing the effective adoption and use of evidence-based and/or promising practices related to supporteddecision making, person-centered thinking, competitive integrated employment, and similar strategies.

• Strengthening and elevating the leadership roles that self-advocacy organizations, Centers for Independent Living, 
Living Well Cross Site Evaluation Volume IV: Methodology 3

and other peer support networks, in collaboration with families and others, must play in improving and assuring the 

quality of home and community-based services (HCBS).

• Enhancing the effectiveness and coordination of efforts by the state, the Developmental Disabilities Assistance and 

Bill of Rights Act (DD Act) network and others to anticipate, avoid, investigate, reduce and remediate abuse, neglect 

and other rights violations in the HCBS service delivery system for people with developmental disabilities.

• Increasing the capacity of States to effectively implement the HCBS settings rules by providing such services in the 

most integrated setting appropriate to the needs of individuals with developmental disabilities of all ages.

III. Evaluation Questions

A set of questions guide the cross-site evaluation (Exhibit 2). 

Exhibit 2. Living Well Cross-Site Evaluation Questions

Focus Evaluation Questions

Quality and 

Effectiveness

What are the performance measures for determining how the models contribute to improving 

the quality of life of individuals with I/DD?

How effective are each of the approaches to Living Well?

What is the best approach for AoD’s future work in this area?

Scalability and 

Replicability

Are the models scalable nationally and/or replicable in other states?

Is the grantee model at the right level of scale to test the model? Are there elements missing?

What demographic factors impact or are a factor in the ability to scale the Living Well models?

What is core to the model, or can parts of the model be scaled? Is the whole model needed to 

be effective?

What changes to policy and practice in the service delivery systems are needed to replicate the 

Living Well model?

Sustainability

Are these models sustainable?

What features are necessary to sustain the projects?

How do systems need to change and/or what systems must be in place in order to implement 

one of the models being tested under the Living Well Grant Program?
Living Well Cross Site Evaluation Volume IV: Methodology 4

IV. Data Collection and Analysis

During the 2019-2020 evaluation year, Lewin collected qualitative, quantitative, and observational data through interviews, 

virtual stakeholder meetings, an annual reporting tool, and semi-annual reports submitted by grantees to ACL in the 

spring of 2020. Due to the impact of the COVID-19 pandemic, Lewin modified its data collection approach as follows: 

collected all data virtually, shortened the annual reporting tool and delayed its implementation, and used grantees’ semiannual reports to reduce reporting burden on grantees. A standard guide was developed for initial planning interviews 

with all grantees in the fall of 2019, grantee-specific guides for interviews with all grantees in the spring and summer of 

2020, and an abbreviated reporting tool. Data collection tools are available in Appendices A, B, C, and D.

Lewin met regularly (i.e., at least monthly) with ACL and the Mission Analytics Group technical assistance (TA) team to 

coordinate and streamline activities, where possible. For example, the evaluation and TA teams jointly participated in the 

initial planning interviews and shared materials (e.g., meeting and interview notes) with each other. The evaluation uses 

a participatory approach, in which grantees provide recommendations on grant meetings or activities for observation, 

the structure and content of grantee profiles documenting grantee’s progress, and the overall evaluation approach. Lewin 

adapted materials and processes in response to grantee feedback. 

Initial Planning Interviews. Lewin held one-hour, semi-structured interviews with each grantee’s project team and 

one member of the Mission Analytics Group TA team. ACL staff participated as observers in a sample of the interviews. 

Interview questions focused on planned activities during the 2019-2020 grant year, including any changes to grant 

objectives, key contextual factors influencing grant implementation, and requests for TA. Lewin recorded each interview, 

took notes, and used the data to plan subsequent evaluation activities.

Semi-Annual Report. Grantees submit reports to ACL semi-annually documenting their progress toward grant goals and 

objectives, challenges encountered, and planned activities. Lewin compared its data collection tools (i.e., annual reporting 

tool and grant leadership interview guide) with the semi-annual report and removed any duplicate questions from the 

data collection tools. Lewin then extracted grantees’ responses from their semi-annual reports and analyzed the responses 

to ensure they completely addressed questions removed from the data collection tools. Lewin documented any gaps and 

added grantee-specific questions to interview guides to collect missing data.

Annual Reporting Tool. Grantees submitted their responses to the annual reporting tool, which included a series of 

closed-ended questions, using an online tool. The reporting tool gathered data on grantees’ progress toward project goals 

and included specific questions about the impact of the COVID-19 pandemic on grant activities. 

Virtual Stakeholder Meetings. In lieu of in-person site visits, Lewin virtually participated in stakeholder meetings with 

each grantee.1 These varied events, including workgroup meetings, trainings, planning sessions, and stakeholder meetings, 

ranged in length from 90 minutes to multiple days. Lewin documented observations in notes for each stakeholder meeting 

and provided notes to ACL and Mission Analytics Group.

Grant Leadership Interviews. As the final data collection activity in this evaluation cycle, Lewin conducted a 90-minute 

interview with members of each grantee’s leadership team. The interview focused on progress and accomplishments to 

date, challenges or contextual factors influencing grant implementation, and specific areas of interest identified through 

analysis of grantee’s semi-annual reports. 

Lewin manually coded qualitative data from semi-annual reports, stakeholder meeting notes, and grant leadership 

interviews, allowing the team to triangulate information for many of the identified themes. Evaluators first categorized 

data using the two core components required for each grantee’s model (Community Monitoring and Community Capacity 

Building), then applied a set of predetermined codes based on the eight key features of Living Well models. To ensure 

inter-rater reliability, evaluators double coded findings and resolved disagreements through discussion. Once discordance 

was resolved, emergent themes were identified. Lewin shared partial findings with grantees during the all-grantee meeting 

in June 2020 and provided grantees an opportunity to share written and verbal feedback on draft annual findings in August 

2020.

1 Lewin attended virtual meetings for seven grantees. One grantee (Georgia) did not hold virtual events during the data collection period.
Living Well Cross Site Evaluation Volume IV: Methodology 5

V. Timeline

Lewin initiated year two of the evaluation in November 2019, building on activities that occurred during years one and 

two of the evaluation. A timeline of activities indicates when each activity occurred (Exhibit 3).

Exhibit 3. Timeline of Evaluation Activities 2017-2020
Living Well Cross Site Evaluation Volume IV: Methodology 6

Appendix

C

D

Grant Leadership 

Interview Guide

Performance Report 

Template

B Annual Reporting Tool

A Initial Planning 

Interview Guide
Living Well Cross Site Evaluation Volume IV: Methodology 7

The initial planning interview opens the data collection period for the second year of the Living Well cross-site evaluation. The 

interview will provide an opportunity for each grantee to reflect on accomplishments from the 2018-2019 evaluation year, 

highlight key activities planned for the 2019-2020 evaluation year, and discuss upcoming meetings and events. The interview 

will also solicit suggestions for technical assistance (TA) during the year. Grantees will also have an opportunity to share 

questions or feedback on the evaluation.

At least two members of the evaluation team and one member of the TA team will participate in each interview. One member 

will conduct the interview, and one member will take notes. With the grantee’s permission, the team will record the interview. 

Prior to the interview, the team will review each grantee’s most recent semi-annual report. The interviewer will conduct a 

semi-structured interview using each of the following guiding questions and supplementing with optional prompts, as needed.

Interview Script

Opening

Introduce yourself and the note taker briefly with your names and reiterate role on the project. Ask interviewer from 

Mission to introduce themselves. Ask for or state the name(s) and job title(s)/role(s) of the interviewee(s) in the Living Well 

demonstration for the recording.

Thank you for your meeting with us today. We’re excited to hear about the developments in your work and plans to 

support individuals with intellectual and developmental disabilities (I/DD) in Living Well.

As a reminder, we’ve asked to record this interview to ensure fidelity in the event that we need to clarify sections of our 

notes. Do we have your permission to record this interview? (wait for them to confirm). Thank you.

Do you have any questions before we get started?

1. Guiding Question: We’d like to start by asking if you have any updates for us. We have reviewed the semi-annual 

report you submitted (summarize new activities briefly). Is there more you would like us to know about these 

updates? Or are there any other key accomplishments, changes, or other updates that have recently occurred on 

your project?

a. Did anything you learned at the all-grantee meeting change your immediate next steps?

b. Are there any other updates since the all-grantee meeting?

2. Guiding Question: Now we’d like to discuss your plans and goals for the upcoming evaluation year. What 

specific objectives do you plan to begin, continue, or complete this evaluation year? [The evaluation team will adapt 

interview language related to grantee plans, goals, objectives, initiatives, and activities based on the how the grantees 

describe their progress plans for the coming year.]

a. Are there elements of your state context or environment that are particularly important for our understanding 

of your Living Well grant?

b. Are there any significant changes from the context of your state or Living Well team from last year to this year? 

(e.g. funding cuts, hiring new staff member)

c. What are the leadership team’s focus areas for this year? How are they similar or different from those of last 

year?

d. What current initiatives (that have already been implemented) will you be continuing this year? 

e. Is there anything that you have decided to no longer pursue?

f. Are there any changes in how the grant leadership team is carrying out Living Well activities?

i. What drove or informed these changes?

g. What specific new initiatives do you plan to pursue this year?
Living Well Cross Site Evaluation Volume IV: Methodology 8

h. How do these new initiatives build on prior Living Well work? How do they fit into your broader goals? 

i. What challenges—either continuing from previous year(s) or new ones—do you anticipate? What strategies 

will you use to mitigate these difficulties?

j. What is your timeline in terms of prioritizing and/or ordering Living Well activities for this coming year?

3. Guiding Question: What key activities are you planning this year? Key activities may include leadership or 

stakeholder meetings, pilot site kickoffs or check-ins, self-advocacy or DSP summits, or other activities.

a. What meetings are you planning to hold this year? What is the intended purpose for these meetings? Do you 

have a proposed timeline for meetings this year?

i. Do you expect to have the same attendees across all meetings?

b. Do you have any other key activities or events planned?

i. Or scheduled pilot site visits/kickoffs?

c. What do you think will be the most productive and representative meeting, event, or activity for Lewin to 

observe on a site visit this evaluation year?

i. Will this help Lewin obtain a comprehensive understanding of (some aspect of) your Living Well Grant 

work?

ii. Would it be more helpful for Lewin to conduct interviews with specific stakeholders in addition to observing 

meetings or attending a specific event?

d. Are there any aspects of your Living Well grant model for which you would like the evaluation team to give 

attention this cycle?

4. Guiding Question (TA team): To ensure that we can provide valuable TA this year, we would like to develop a 

specific plan to support you and your team. What kinds of support or assistance would help you achieve your goals 

this year? 

a. Are there areas where you would like to know more information about best practices?

b. Are there areas where you would like some more intensive one-on-one help, potentially including TA staff 

coming to work with you in person?

c. Would you like help in better connecting and learning from specific other grantees? On what types of topics? 

d. Are there particular staff members or partners that you would like to have an opportunity to connect with their 

counterparts at other grantees? 

5. Guiding Question: Thank you for providing all of this information. Before we wrap up, do you have any 

questions for the TA team at Mission or the cross-site evaluation team at Lewin?

a. Do you have any questions about Mission’s TA structure for the year?

b. Do you have any questions about Lewin’s evaluation structure for the year?

c. Is there anything else that you think the evaluation team or TA team should know?

Closing

Thank you for your time today. If you have any other thoughts, questions, or ideas after today’s call, please do not 

hesitate to reach out to either team by phone or email. If you need contact information, we can provide that now.
Living Well Cross Site Evaluation Volume IV: Methodology 9

Appendix

C

A

D

Grant Leadership 

Interview Guide

Initial Planning 

Interview Guide

Performance Report 

Template

B Annual Reporting Tool
Living Well Cross Site Evaluation Volume IV: Methodology 10

This annual reporting tool will collect standardized information from each grantee on Living Well project structure and 

members, capacity across the Eight Key Features of the grant, and improvements made to integration and community 

monitoring. Information regarding impact of COVID-19 on grant-related activities will also be captured. Please complete each 

question to the best of your abilities and based on your understanding of the question in context of your state.

The evaluation team will follow up on information provided in this tool during upcoming interviews with grant leadership.

Question 1: In what ways are the formal body of stakeholders convened for the purpose of this grant advancing your 

project goals? Select all that apply.

☐ Aligning Living Well with other projects or initiatives in the state

☐ Facilitating access to data or information

☐ Identifying or securing needed resources

☐ Initiating partnerships with new or different stakeholders

☐ Strengthening the voice of self-advocates and families in statewide systems or policy change

☐ Providing technical expertise 

☐ Providing cultural expertise or brokerage

☐ Other, please specify: _____________________

Question 2: Have you engaged any new partners during this evaluation cycle to help you achieve your grant goals?

☐ Yes (if selected, proceed to next question)

☐ No (if selected, skip to question 4)

Question 3: Which of the following partners did you engage with during this evaluation cycle to help you achieve 

your grant goals? 

☐ Self-Advocacy Organization

☐ Family Network or Organization

☐ Advocacy Organization

☐ State DD Agency

☐ Service Provider

☐ Case Management

☐ Developmental Disabilities (DD) Council

☐ University Center for Excellence in Developmental Disabilities (UCEDD)

☐ State Protection & Advocacy Agency (P&A)

☐ Other, please specify: _____________________

Question 4: How have self-advocates engaged with your Living Well models since the last evaluation cycle? In this 

question, Living Well models refers to all aspects of your Living Well project related to community monitoring and 

community capacity building. Select all that apply.

☐ Developing Living Well models

☐ Testing Living Well models

☐ Implementing Living Well models

☐ Scaling Living Well models

☐ Other, please specify: _____________________

☐ Not applicable
Living Well Cross Site Evaluation Volume IV: Methodology 11

Question 5: How have family members of individuals with I/DD engaged with your Living Well models since the 

last evaluation cycle? In this question, Living Well models refers to all aspects of your Living Well project related to 

community monitoring and community capacity building. Select all that apply.

☐ Developing Living Well models

☐ Testing Living Well models

☐ Implementing Living Well models

☐ Scaling Living Well models

☐ Other, please specify: _____________________

☐ Not applicable

Question 6: What actions have you taken since the last evaluation cycle to advance or improve full and meaningful 

participation of self-advocates and families in your Living Well grant? Select all that apply.

☐ Provided support outside of meetings to understand materials and meeting objectives.

☐ Provided transportation, childcare, respite services, or other supports necessary to attend meetings

☐ Ensured self-advocates or families lead project activities

☐ Changed project activities based on self-advocate or family feedback

☐ Other, please specify: _____________________

☐ Other, please specify: _____________________

☐ Other, please specify: _____________________

☐ Not applicable

Question 7: For each of the following Living Well grant outcomes, please indicate the extent to which you have 

progressed toward this outcome since the last evaluation cycle. Select: To a great extent; to a moderate extent; not at all; 

not applicable.

To a great extent To a moderate extent Not at all Not applicable

Enhanced the health and safety 

of individuals by increasing their 

independence, social capital, 

self-determination, community 

integration, productivity, and/or 

participation

☐ ☐ ☐ ☐

Strengthened the knowledge, skills, 

specialization, economic/career 

prospects, workforce stability, 

retention, and/or advancement of 

direct support professionals

☐ ☐ ☐ ☐

Incentivized the effective adoption 

and use of evidence-based and/

or promising practices related to 

supported decision-making, personcentered thinking, competitive 

integrated employment, and similar 

strategies.

☐ ☐ ☐ ☐
Living Well Cross Site Evaluation Volume IV: Methodology 12

To a great extent To a moderate extent Not at all Not applicable

Elevated the leadership roles that 

self-advocacy organizations, centers 

for independent living, and other peer 

support networks play in improving 

and assuring the quality of home and 

community-based services

☐ ☐ ☐ ☐

☐ Other, please specify: _____________________

☐ Not applicable

Question 8: Please describe the extent to which grant activities in the last evaluation cycle contributed to 

improvements in the following areas? Select: To a great extent; to a moderate extent; not at all; not applicable. 

To a great extent To a moderate extent Not at all Not applicable

An infrastructure for reporting 

incidents (e.g., established procedure, 

dedicated reporting phone lines, web 

addresses)

☐ ☐ ☐ ☐

Relationships between agencies that 

receive, investigate, and follow up on 

reports

☐ ☐ ☐ ☐

Community reporting ☐ ☐ ☐ ☐

Monitoring for individuals at high risk ☐ ☐ ☐ ☐

Use of technology to collect, analyze, 

or share data ☐ ☐ ☐ ☐

☐ Other, please specify: _____________________

Question 9: Is COVID-19 affecting how you are operating your grant?

☐ Yes (if selected, proceed to next question)

☐ No (if selected, go to the end of the survey)

Question 10: Please indicate how the COVID-19 response is affecting your grant. Select all that apply.

☐ Delaying implementation of grant activities

☐ Changing how the grant is implemented (e.g., in-person to virtual)

☐ Staff time diverted away from grant activities to other priorities

☐ Staff unable to complete grant activities due to sickness, caregiving, or “stay-at-home” guidelines

☐ Funding diverted away from grant activities

☐ Funding insufficient to react to changing environment for grant activities due to COVID-19

☐ Recipients of grant activities unable to participate due to COVID-19 illness, caregiving, or “stay-at-home” guidelines

☐ Other, please specify: _____________________
Living Well Cross Site Evaluation Volume IV: Methodology 13

Appendix

C

A

B

D

Grant Leadership 

Interview Guide

Annual Reporting Tool

Initial Planning 

Interview Guide

Performance Report 

Template
Living Well Cross Site Evaluation Volume IV: Methodology 14

Thank you for the information you have provided through the Initial Planning Interview, the Semi-Annual Report, and 

various email correspondences. Today, we’re going to touch on some of the same topics to get more information and clarity. We 

want to learn more about activities in your state. We’re especially interested in what leads to positive outcomes. I’m going to ask 

a series of conversation prompts. We’ll also leave time at the end so everyone can share additional thoughts they have.

Before we jump in, could we start with introductions? Please share your name, any organization affiliation, and your role on 

the Living Well grant.

Standard Questions

These are questions that will be asked of all grantees.

Question 1: We will be asking about specific activities in a moment, but first, are there any major updates or 

milestones for your Living Well project that you would like to share? 

Question 2: During our interview with you earlier this year, you noted the several specific circumstances in your state 

as potentially impacting your grant, including: [Insert state-specific response]

• Have there been any other changes to these circumstances? 

• How has COVID affected your grant activities? Has your project done anything specific in response to COVID? 

Has COVID 19 affected your stakeholder engagement?

Question 3: Please describe any partnerships unique to the Living Well grant or which go beyond the formal body 

of stakeholders required by the grant (i.e., Self-Advocacy, Family, or Advocacy Organization, State DD Agency, DD 

Council, UCEDD). Specifically, have you formed any new partnerships within the last year to advance your project 

goals?

Question 4: You noted in the Semi-Annual Report that you [insert responses]. Please describe the factors that are 

contributing to your progress. Are you able to access needed data sets to produce your desired measurements? Have 

you adopted any data tools since we last spoke? Have you experienced any challenges and, if so, how are you working to 

mitigate them?

Question 5: How have you worked with state and local entities to collect, analyze, and share data to enhance 

community monitoring?

Question 6: How are your grant activities contributing to improved access to or quality of HCBS settings?

Question 7: Have you developed or implemented new evaluation measures to ensure that your project activities are 

impacting the intended target group and evaluating the effectiveness of your model? 

• Please describe any changes to your project design or approach as a result of your evaluation activities to date.

Question 8: Please discuss any progress toward project sustainability and/or any practices that are leading toward 

scalability or replicability of your model.

Follow up to Semi Annual Report

Please let the interviewees know that you will now ask some specific, follow-up questions to the information they provided 

in their most recent semi-annual report.

[Insert grantee-specific questions]

Question: Is there anything else that we haven’t discussed today that you’d like to share with us?
Living Well Cross Site Evaluation Volume IV: Methodology 15

Appendix

C

A

B

D

Grant Leadership 

Interview Guide

Annual Reporting Tool

Initial Planning 

Interview Guide

Performance Report 

Template
Living Well Cross Site Evaluation Volume IV: Methodology 16

Recipients of discretionary grants from ACL must submit regular performance reports: a semi-annual report at the conclusion 

of each six-month reporting period and a final report, which takes the place of the last semi-annual report, at the conclusion 

of the grant. Grantees document their activities and accomplishments in the semi-annual report by providing a narrative 

response to the following questions:

Question 1: What did you accomplish during this reporting period and how did these accomplishments help you 

reach your stated project goal(s) and objective(s)? Please note any significant project partners and their role in project 

activities.

Question 2: What, if any, challenges did you face during this reporting period and what actions did you take to 

address these challenges? Please note in your response changes, if any, to your project goal(s), objective(s), or activities 

that were made as a result of challenges faced. 

Question 3: How have the activities conducted during this project period helped you to achieve the measurable 

outcomes identified in your project proposal?

Question 4: What was produced during the reporting period and how have these products been disseminated? 

Products may include articles, issue briefs, fact sheets, newsletters, survey instruments, sponsored conferences and 

workshops, websites, audiovisuals, and other informational resources.