**ACL process evaluation interview protocol**

Appendix I.Key Informant Interviews Protocol

This page has been left blank for double-sided copying.

**Chapter #** Title of Chapter	

**DRAFT 06/14/2103/29/21****** Mathematica	2

Interview protocol

Introductory statement and consent

Thank you for speaking with me today. My name is _____________ and I am with the National Association of Area Agencies on Aging (n4a). In partnership with Mathematica, we are conducting a process evaluation of the Aging Network for the Administration for Community Living (ACL). The purpose of this study is to learn about how the Aging Network collaborates to improve the lives of older adults, and how it determines the effectiveness of these efforts.

We have several goals for this discussion. We are interested in hearing about your organization’s experiences collecting data to compare the value or benefit of program services relative to the costs; that is, what is often referred to as “return on investment (ROI) analysis.” We would also like to hear about how you have used the results of these analyses, and any lessons learned, challenges, or types of supports that would be helpful. We expect our discussion to last about one hour.[footnoteRef:2] Your participation is voluntary, and you may skip any question. Your information will be kept private to the extent permitted by law. We will not name individual staff or agencies in our study report. [2:  Priority questions that should be asked are shown in bold.] 

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is XXXX-XXXX and the expiration date is MM/DD/YYYY. If you have any comments on this collection of information, please contact Caryn Bruyere.

Would it be okay if we recorded the interview? We take notes to capture everything you say but in case we want to check something in our notes, it is useful to have a recording. We do not share recordings outside of the study team and will destroy recordings when our project ends.

[If interviewee does not wish to be recorded, acknowledge their request and do not record the interview. Otherwise, START audio recording.]

Do you have any questions before we begin? Okay, let’s get started.

I. Background

To start, please introduce yourself and describe your role at [ORGANIZATION NAME]. What are your key responsibilities? 

How long have you worked in your current role? How long have you worked at your organization? Have you previously held other roles at your organization?

We are interested in learning more about the partnerships [ORGANIZATION NAME] has with other organizations. We understand that you partner with [TYPES OF ORGANIZATIONS]. Does your organization primarily have individual relationships with partners, or do you belong to any networks, collaboratives or coalitions, or both?

1. For partnerships with individual agencies: What have your successful relationships with individual agencies accomplished? [Probe: What service needs did you and your partners help address?] 

If part of a collaborative: What has the collaborative accomplished? [Probe: What service needs did the coalition address?] 

Now I would like to talk about your organization’s experience calculating the financial value of services, including participating in or conducting return on investment (ROI) or cost benefit analyses, which calculate the net benefit of an investment compared to its cost. Sometimes this means translating something that is challenging to measure, such as quality of life, into quantifiable values. In our survey, we asked about the services, costs, and benefits that are assessed. We want to learn more about how you define each of those concepts, and also hear about your organization’s experiences and lessons learned. To start, we’ll talk about your organization’s overall experience with ROI.

**Why did your organization begin [CONDUCTING/PARTICIPATING IN] assessments to determine the value or benefit compared to the cost of program services?** [Prompts: Do any of your funders or partners require or strongly encourage it? To support a business case to potential partners? To advocate for more funding with state/local officials?]

**We understand that your organization has assessed costs and benefits for [TYPES OF SERVICES]. Is that correct? Why do you focus on these services?**

1. Are there any services for which you do not conduct ROI analyses? Why or why not?

When your organization began conducting ROI or cost benefit analyses, what resources or data system changes were required?

What other organizations do you partner with to assess costs and benefits, if any?

1. What are the roles of each organization in the ROI analysis? For example, does your organization collect and analyze the data? 

II. Cost and benefit calculation

Next I am going to ask about how you think about and calculate costs and benefits. We are interested in learning about how you typically think about and conduct ROI assessments. If it’s helpful, we can focus on your most recent or largest analysis.

**For both benefits and costs, what types or sources of data do you use? From whom do you collect or with whom do you share data?**

1. How are data analyzed? What specific software is used? Who does the analysis? 

**When you think about these financial calculations, from whose perspective do you typically calculate costs and benefits?** [Prompt: Your organization? The participants? Caregivers? Payers (e.g. ACL, Medicaid, Medicare)? Society at large (e.g. taxpayers)?] Why?

**When you calculate the value of benefits for these analyses, we understand that you typically include [TYPES OF BENEFITS]. Is that correct? We would like to learn more about each of these benefits.**

*Probe on WHY OR WHY NOT the organization considers the following benefits, including how they monetize the benefit, which outcome measures are used and why (required by funders? Following fidelity guidelines?), and whether there are barriers to measurement:*

1. Improved management of chronic conditions (e.g. diabetes, high blood pressure, high cholesterol)

Improved or maintained functional status (ADLs/IADLs)

Greater independence, or delaying or avoiding entry into long term care facilities

Increased socialization or reduced loneliness

Reduced use of costly health and social services (e.g. fewer avoidable hospital admissions and ED visits)

Improvements in self-reported physical or mental health

Increased life expectancy 

Improved quality of life	

Other (e.g. increased employment, consumer satisfaction, food security, caregiver ability to retain employment or conserve time off)

**When you calculate costs for these analyses, we understand that you typically include [TYPES OF COSTS]. Is that correct? We would like to learn more about each of these costs.**

*Probe on WHY OR WHY NOT the organization considers the following costs, including how they monetize the cost and whether there are barriers to measurement:*

1. Staff costs (e.g. salaries and fringe benefits, volunteer time)

Direct service costs (e.g. service contracts, support services, partner and provider service costs)

Supply and equipment costs (e.g. supplies and/or leases on purchase of equipment)

Overhead and operating costs (e.g. overhead, facilities, utilities, indirect cost)

Development and maintenance of data systems

III. Uses and lessons learned

Next we will discuss how your organization uses the results of ROI analyses and any lessons learned from conducting or participating in such analysis.

**How has your organization used the results of your ROI analysis**? [Prompt: How have either the results, or the measurement itself, affected your decisions to continue (or discontinue) or expand programs? Have the results been used to demonstrate value to potential partners or funders?]

1. If the analysis has identified any cost savings, to whom have the savings accrued? [Prompt: Your organization? The participants? Payers (e.g. ACL, Medicaid, Medicare)? Society at large (e.g. taxpayers)?]

Have you conducted the analyses more than once? If so, did you make any major changes in how you conducted the analyses and if so why? 

**What has your organization learned from measuring ROI?**

1. What worked well, or made it a success?

What were (or are) the biggest challenges? How did (or does) your organization address those challenges?

What additional supports, if any, could your organization have used to strengthen your ability to effectively measure ROI and to use the results to improve service delivery (or any other uses, such as demonstrating value to partners and funders)?

Finally, are there any last comments or observations you would like to share regarding your organization’s practices conducting ROI analyses?

Thank you! Those are all the questions I have for you today.

Would it be okay to contact you after today with a follow up question or two in case we missed something in our notes?

**DRAFT 06/14/21****** Mathematica	I.6