ACL FFY 2022 Evaluation Plan

This document summarizes the evaluations ACL plans to undertake to address its priority 

questions and to meet statutory requirements. ACL reserves the right to adjust this plan 

pending changes in agency priorities and funding availability.

Priority Question: What is the efficacy and effectiveness of ACL programs and initiatives? 

How does ACL prepare its staff to better support the aging and disability networks?

Background and Significance: Authorizing Legislation: Section 361 of the Older 

Americans Act (OAA) of 1965, as amended states that “SEC. 361. (a) The Assistant 

Secretary shall carry out a program for making grants to States under State plans approved 

under section 307 to provide evidence-based disease prevention and health promotion 

services and information at multipurpose senior centers, at congregate meal sites, through 

home delivered meals programs, or at other appropriate sites. In carrying out such 

program, the Assistant Secretary shall consult with the Directors of the Centers for Disease 

Control and Prevention and the National Institute on Aging. (b) The Assistant Secretary 

shall, to the extent possible, assure that services provided by other community 

organizations and agencies are used to carry out the provisions of this part.”

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated: The FY 

2012 Congressional appropriations law included, for the first time, an evidence-based 

requirement related to Title III-D funds and specifying: 

Evidence-based health promotion programs, as defined in the Older Americans Act (Title I 

section 102 (14)(D)) includes programs related to the prevention and mitigation of the 

effects of chronic disease (including osteoporosis, hypertension, obesity, diabetes, and 

cardiovascular disease), alcohol and substance abuse reduction, smoking cessation, weight 

loss and control, stress management, falls prevention, physical activity, and improved 

nutrition.

In response to the 2012 requirement, ACL developed an evidence-based definition to assist 

states in developing their own Title III-D guidance. Specifically, to be considered an 

evidence-based program, a program must:

• Have demonstrated through evaluation to be effective for improving the health 

and well-being or reducing disease, disability and/or injury among older adults; 

• Be proven effective with older adult population, using Experimental or QuasiExperimental Design; 

• Have research results published in a peer-review journal; 

• Be fully translated in one or more community site(s); and

• Include developed dissemination products that are available to the public.
Stakeholders: This work was informed by consultation with ACL stakeholders and 

requirements under the Older Americans Act. Stakeholders are ACL staff, grantees and subgrantees, older adults and their families and caregivers, and other members of the aging 

network.

Data Sources: Primary and secondary data from ACL-funded evidence-based programs 

around questions such as:

1. How do sites select appropriate evidence-based programs for their contexts (e.g., 

resources, populations)?

2. To what extent do staff overseeing and implementing the programs fully understand 

and comply with the implementation guidelines/instructions? For example, 

a. Are facilitators trained in accordance with the guidelines of the relevant 

evidence-based program(s)?

b. Are program resources in line with the guidelines of the relevant evidence-based 

program(s)?

c. Are programs being implemented with the intended populations?

d. Is the amount of training and frequency sufficient?

e. Is the content provided completely and properly per the guidelines of the 

relevant evidence-based program(s)? 

3. While not recommended, what kinds of adaptations, if any, are grantees/subgrantees making to the evidence-based programs? And, why?

4. What can ACL do to support and encourage the proper use and implementation of 

evidence-based programs? 

Methods: ACL expects to use secondary data to review and conduct primary data 

collection and analysis as part of this fidelity evaluation. 

Anticipated Challenges and Mitigation Strategies: Potential challenges in conducting 

this work may be access to secondary data, and sampling from existing evidence-based 

programs funded under the Older Americans Act. To mitigate this, ACL’s Office of 

Performance and Evaluation will work closely with program staff and the ACL funded 

resource center.

Plan for Disseminating Results: The final deliverable for this contract is a report 

documenting the information collected and providing clear, actionable recommendations 

for ensuring the effective use of evidence-based programming. Recommendations will 

address what ACL and its grantees and sub-grantees can do to improve the selection, 

implementation, and monitoring of evidence-based programming. The report will also 

include tool(s) for use by ACL and its OAA state grantees to assess fidelity after this 

contract ends. This report will be shared publicly on ACL’s website and information will be 

distributed to key stakeholders, including grantees. Further, this project will create job aids 

and other technical assistance materials, and briefings and webinars on the findings will 

also be presented for a variety of stakeholders, both internal and external to the agency.
Priority Question: What are the knowledge gaps between ACL, research, and 

stakeholders? How can ACL close the knowledge gaps? Sub-question: Which ACL evidenceinformed services and programs are addressing social determinants of health (SDOH) or 

social needs and how they are doing this?

Background and Significance: ACL wants a review of how the various OAA-funded 

programs are addressing SDOH (keeping in mind areas where aging and disability 

crosscut). The first step of this work will be theoretical, but the next piece of this work 

should choose a sample of the programs to understand where ACL is intersecting with the 

social determinants of health (SDOH) and the effect of ACL programming on the social 

determinants of health. 

Questions of interest are related to how to best measure effects of ACL services and 

programs on SDOH, if dosage makes a difference, and other underlying factors that support 

ACL programs addressing SDOH in positive ways.

Programs, Policies, Regulations, or Operations to be Analyzed: This evaluation is 

focused on how services ACL funds under the Older Americans Act specifically address the 

social determinants of health and social needs, how we might measure the agency’s impact 

in these areas, and these issues might be addressed across the board through ACL’s 

programs and grants. 

Stakeholders: The learning agenda development process informed this work. This work 

would initially be internal to ACL in terms of working with administrative data at the ACL 

level and working with ACL staff to understand programming aims. IN an anticipated 

follow on, ACL would conduct a deeper evaluative dive into how the ACL-funded network is 

working to address SDOH and how this might best be measured.

Data Sources: The initial work will start with a 24 month base year contract to conduct a 

literature review on social determinants and social needs; review ACL materials related to 

grants crosscutting SDOH and social needs issues; and map out ACL programs intersecting 

these areas. The subject matter expert interviews will be used to help develop an approach 

for ACL to measure our impact on the SDOH and social needs over time. 

Methods: Literature reviews, key informant interviews, focus groups.

Anticipated Challenges and Mitigation Strategies: Anticipated challenges are the 

complexity of mapping out the sheer number of grant programs given some of the agency 

silos. Other possible challenges might be where the program is addressing SDOH or social 

needs, but not collecting administrative data around these areas, making it harder to 

determine what impacts these grants may have on the areas of interest. ACL will mitigate 

these challenges through stakeholder involvement. 

Plan for Disseminating Results: Dissemination of this work will be through the white 

paper making a recommended approach to ACL on how to better capture information in 

order to measure the agency’s effect on SDOH and social needs, as well as briefs, blog posts, 

and possible presentations. All reports will be made available on ACL’s website.
Priority Question: What is the efficacy and effectiveness of ACL programs and initiatives? 

What are states funded by ACL doing to enhance and promote aging and disability services 

and programs? What are the current avenues for collaboration and coordination between 

the aging and disability networks that ACL works with? How can ACL encourage better 

collaboration and coordination? What new research and information are being generated, 

by both ACL and in the field, on aging and disability? How is ACL implementing it, and how 

can ACL better implement it? 

Background and Significance: The Older Americans Act (OAA) was passed in 1965 with 

the goal of supporting older Americans and helping them to live at home and in their 

communities independently and with dignity for as long as possible. The OAA supports 

many programs and services, which as congregate and home-delivered meals, in-home 

care, adult day care, caregiver support, elder abuse prevention, health and wellness 

programs, transportation, and information and referral. These services are provided 

through the national Aging Network, which is comprised of 56 State Units on Aging (SUA), 

622 Area Agencies on Aging (AAA), and more than 260 Title VI Native American aging 

programs. Additionally, thousands of community organizations and volunteers support the 

aging network.

While the Aging Network is extensive, its processes and operations are not fully 

understood. Basing policy and justifying expenditures on evidence is increasingly a priority 

in the federal government. However, without a thorough understanding of the Aging 

Network, it is not feasible to accurately plan to measure the return on the investment of the 

funds distributed through the network. Yet, the increasing demands require rigorous and 

independent assessment of progress, efficiency and effectiveness to ensure the most 

productive use of government funds for the best consumer outcomes.

Programs, Policies, Regulations, or Operations to be Analyzed: Through program 

evaluation contracts of Older Americans Act (OAA) programs, ACL seeks increased 

understanding of how these programs are structured at the State and local levels and their 

progress towards their goals and mission. The information will also aid in program 

refinement and continuous improvement. The more productive ACL’s programs, the 

greater the number of older adults will have access to a higher quality of life.

Stakeholders: This work was informed by the needs of aging network stakeholders to 

demonstrate the networks’ value to older adults, funders, and policy makers. The purpose 

of this evaluation is to contract for a comprehensive process evaluation of the Aging 

Network engaging all levels (Federal, state, and local) to answer the following questions:

1. How is the Aging Network structured and how does it operate at the local, state, 

and federal levels including who the program serves, how it is staffed, and what data 

are collected about activities and outcomes?

2. How does the Aging Network use resources to measure and improve the quality 

of services available/provided? What is the role of the Network in identifying and 

responding to emerging needs?
3. How do the various levels of the Aging Network work together, with whom do 

they partner, and how do they work with those programs? For example, do they 

partner only with organizations considered to be part of the Aging Network, or 

outside organizations as well?

4. How does the Aging Network measure successful practices and areas for 

improvement?

Data Sources: Data collection should include gathering information from:

1. Federal staff and national associations about their understanding of and role in 

the Aging Network, interaction with other parts of the Aging Network, interorganizational relationships, and use/collection of program data.

2. State Units on Aging about their understanding of and role in the Aging Network, 

interaction with other parts of the Aging Network, inter-organizational 

relationships, state mandates, main activities, funding sources, feedback to and 

monitoring of local programs, and use/collection of program data. It should also 

include a discussion of unique features of local programs/structures, specifically 

interactions with AAAs and Native American aging programs as well as federal 

entities.

3. Area Agencies on Aging about their understanding of and role in the Aging 

Network, interaction with other parts of the Aging Network, inter-organizational 

relationships, state mandates, main activities, funding sources, feedback to and 

monitoring of local programs, and use/collection of program data. It should also 

include a discussion of unique features of local programs/structures, specifically 

interactions with SUAs, Native American aging programs, and federal entities.

4. Native American aging programs about their understanding of and role in the 

Aging Network, interaction with other parts of the Aging Network, interorganizational relationships, Tribal mandates, main activities, funding sources, 

feedback to and monitoring of local programs, and use/collection of program data. 

It should also include a discussion of unique features of local programs, specifically 

interactions with SUAs, AAAs, and federal entities.

5. Other entities identified through the literature review and the other data 

collection efforts as important parts of the Aging Network.

Methods: Data collection methods may include literature reviews, questionnaires, 

stakeholder interviews, site visits, and/or focus groups. 

Anticipated Challenges and Mitigation Strategies: Possible challenges in this work will 

be around scope of the evaluation and choosing a purposeful sample as the aging network 

is vast. Other challenges may lie with access to and availability of administrative data that 
allow ACL to determine measures for examining return on investment. The information 

collected through this contract is intended to lay a foundation for a possible future outcome 

evaluation and/or cost study. ACL will mitigate these challenges through contracting with 

an outside organization with a proven track record with similar work.

Plan for Disseminating Results: Findings from the Return on Investment study will be 

published on ACL’s website and information may be shared with key stakeholders in the 

form of public presentations. In particular, ACL is looking to produce a suite of highly visual 

products to be shared widely with stakeholders to better illustrate the composition of the 

aging network and how it works.
Priority Question: To what extent are NIDILRR’s grantees generating new knowledge and 

products (outputs and short-term outcomes)? 

Background and Significance: This question is directly related to the first part of 

NIDILRR’s core mission—the generation of new knowledge and products. This missioncritical function is integrated into our agency logic model, operations, and systems. All of 

our agency components work together to support the fulfillment of this mission-critical 

function. 

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated: The

proposed analysis or evaluation will focus on grantees who have been funded by one of our 

program mechanisms. NDILRR-funded programs are authorized to operate under 45 CFR 

Part 1330.

Stakeholders: NIDILRR management and NIDILRR Project Officers employ a variety of 

methods (e.g., teleconferences, email, webinars via WebEx, etc.) to communicate with 

researchers and members of other stakeholder groups within our NIDILRR grantee 

community. Our stakeholders have consistently re-affirmed the importance of generating 

new knowledge that is relevant to the problems and challenges that people with disabilities 

face. 

Data Sources: The data sources for addressing this question are: (1) NIDILRR’s online 

Annual Reporting System (APR); and (2) citation analyses of grantee-produced 

publications.

Methods: For the APR data source, methods include but are not limited to: (1) data mining 

and data extraction techniques, i.e., select queries, conditional queries, parameter queries; 

use of regular expressions, etc.; (2) data analysis techniques that are appropriate to the 

type of data being examined, i.e., descriptive and inferential statistics; content analysis; and 

(3) data visualization, i.e., infographics, graphs, etc. For the citation analyses, the results are 

analyzed using various bibliometric methods which include but are not limited to counts of 

citations; frequency distributions of citations and calculation of the journal impact factor. 

Anticipated Challenges and Mitigation Strategies: To mitigate challenges of missing 

data or incomplete data, through the online data submission system NIDILRR Project 

Officers have the ability to send comments to grantees about the data they have entered in 

response to particular questions. Grantees and can make the requested changes within the 

system.. NIDILRR also runs various quality checks on the data that are already in the 

system. These checks identify and remediate missing data, duplicate records, and broken 

hyperlinks.

Plan for Disseminating Results: Possible uses of the results include but are not limited to: 

informing and improving agency policy formulation and decision-making; informing and 

improving the quality of agency Funding Opportunity Announcements; informing and 

improving the design and delivery of agency service-delivery programs; and informing and 
improving the ability of persons with disabilities and their families to make better selfdirected choices and decisions.
Priority Question: To what extent are NIDILRR-generated knowledge and products being 

used to shape policy and practices that are important in the lives of people with 

disabilities?

Background and Significance: This question is directly related to the second part of 

NIDILRR’s core mission—the promotion of the use and adoption of the new knowledge and 

products. This mission-critical function is integrated into our agency logic model, 

operations, and systems. All of our agency components work together to support the 

fulfillment of this mission-critical function. Without evidence of use and adoption of new 

knowledge and products, changes in policy and practice that will benefit the lives of 

persons with disabilities cannot happen.

Programs, Policies, Regulations, or Operations to be Analyzed or Evaluated: The 

proposed analysis or evaluation will focus on grantees who have been funded by one of our 

program mechanisms. NDILRR-funded programs are authorized to operate under 45 CFR 

Part 1330.

Stakeholders: NIDILRR management and NIDILRR Project Officers employ a variety of 

methods (e.g., teleconferences, email, webinars via WebEx, etc.) to communicate with 

researchers and members of other stakeholder groups within our NIDILRR grantee 

community. Our stakeholders have consistently re-affirmed the importance of promoting 

the use and adoption of new disability knowledge and products. It is only through the 

processes of use and adoption of these outputs that changes in policy and practice can 

occur.

Data Sources: The data source for this question is the use and adoption module of 

NIDILRR’s online Annual Reporting System (APR) that our grantees complete. This module 

contains questions that track the use and adoption of new knowledge and products by our 

various stakeholder groups. 

Methods: For the use and adoption module of the APR data source, methods employed 

include but are not limited to: (1) data mining and data extraction techniques, i.e., select 

queries, conditional queries, parameter queries; use of regular expressions, etc.; (2) data 

analysis techniques that are appropriate to the type of data being examined, i.e., descriptive 

and inferential statistics; content analysis, etc.; and (3) data visualization, i.e., infographics, 

graphs, etc.

Anticipated Challenges and Mitigation Strategies: To mitigate challenges of missing data 

or incomplete data, through the online data submission system NIDILRR Project Officers 

have the ability to send comments to grantees about the data they have entered in 

response to particular questions. Grantees and can make the requested changes within the 

system.. NIDILRR also runs various quality checks on the data that are already in the 

system. These checks identify and remediate missing data, duplicate records, and broken 

hyperlinks.
Plan for Disseminating Results: Possible uses of the results include but are not limited to: 

informing and improving agency policy formulation and decision-making; informing and 

improving the quality of agency Funding Opportunity Announcements; informing and 

improving the design and delivery of agency service-delivery programs; and informing and 

improving the ability of persons with disabilities and their families to make better selfdirected choices and decisions.