Administration for Community Living

Fidelity Evaluation of ACL’s 

Evidence-Based Programs

Appendices

September 19, 2022 

This report was funded by the US Department of Health and Human Services’ (HHS) Administration for 

Community Living (ACL). The views expressed are those of the authors and should not be attributed to HHS or 

ACL. Funders do not determine research findings or the insights and recommendations of the authors.
Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page i

Administration for Community Living 

Table of Contents

A. Appendix A: Methodology ............................................................................................................ A-1

Grantee and Implementation Organization Surveys ....................................................................... A-1

Survey Questionnaire Design and Implementation ......................................................................... A-2

Survey Data Collection ................................................................................................................... A-2

Response Rates .......................................................................................................................... A-2

Survey Data Analysis ...................................................................................................................... A-3

Comparisons .............................................................................................................................. A-4

Qualitative Analysis: Key Informant Interviews ............................................................................. A-4

Grantee Materials (Documents) ................................................................................................ A-4

Limitations ....................................................................................................................................... A-5

B. Appendix B: Tables ....................................................................................................................... B-1

Program Selection ........................................................................................................................... B-1

Program Fidelity ............................................................................................................................ B-13

General Fidelity ....................................................................................................................... B-13

Fidelity in Training .................................................................................................................. B-22

Fidelity in Program Resources ................................................................................................ B-31

Fidelity in Target Population ................................................................................................... B-39

Fidelity in Content ................................................................................................................... B-43

Program Adaptations ..................................................................................................................... B-49

Adaptations not related to the COVID-19 PHE ...................................................................... B-49

Adaptations Related to the COVID-19 PHE ........................................................................... B-52

Program Support ............................................................................................................................ B-57

C. Appendix C: Crosswalk of Research Questions and Knowledge Objectives ........................... C-1
Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-1

Administration for Community Living 

A. Appendix A: Methodology

To address ACL’s research questions and knowledge objectives, HSAG developed a multi-pronged 

research effort involving the following:

1. Collecting fidelity-related materials provided by ACL and ACL grantees 

2. Conducting surveys of grantees and I/Os 

3. Conducting interviews with ACL staff involved in making and administering grants 

4. Conducting interviews with EBP developers and administrators

The survey and data collection and analysis plan for the survey was approved by the OMB under the 

title “Fidelity Evaluation of the Administration for Community Livings [sic] EBPs,” OMB Control 

Number 0985-0073, effective 1/26/2022 through 1/31/2025. 

Grantee and Implementation Organization Surveys

The population of potential respondents for the grantee and implementation organization survey 

consisted of the following: 

• Discretionary Grantees that receive awards from ACL to provide evidence-based Falls 

Prevention or Chronic Disease Self-Management Education (CDSME) programs. New threeyear grants are awarded annually and are competitive. All organizations that have been awarded 

discretionary grants between 2018 and 2020 are part of this population. 

• State Units on Aging (SUAs) in all 50 states, Washington, D.C., and 5 U.S. territories, which 

receive mandatory Older Americans Act (OAA) Title III-D funding from ACL to deliver 

evidence-based health promotion programs. In addition to their mandatory funding, SUAs can 

compete for ACL discretionary grants. 

• I/Os directly deliver evidence-based program (EBP) services to eligible members of the public 

under contracts or agreements with SUAs or discretionary grantees. I/Os may in some instances 

be divisions of discretionary grantee organizations or SUAs rather than independent entities. 

The grantee populations, including both discretionary grantees and SUAs, are small but diverse. 

Discretionary grantees may offer only Falls Prevention programs, only Chronic Disease SelfManagement programs, or both, and may provide programs directly or through other organizations or 

both. SUAs from large coastal states serve populations that are orders of magnitude larger and typically 

more diverse than SUAs from small mountain and plains states. Therefore, the sample design called for 

collecting data from the full population, which is inherently representative of the population. 
APPENDIX A. METHODOLOGY

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-2

Administration for Community Living

Like grantees, I/Os are expected to be heterogenous. They are located in urban and rural areas, are 

associated with large and small SUAs or discretionary grantee organizations, serve communities with 

different cultural and linguistic backgrounds, and implement a wide variety of EBPs. Each of these 

characteristics may have an impact on approaches to implementation and fidelity. In contrast to the 

grantees, however, the number of I/Os is unknown; while individual grantees may know which 

organizations are implementing their ACL-funded programs, there is no central directory of I/Os from 

which to sample. Since the population size is unknown, the sampling fraction is also unknown. 

However, a sample could still be drawn with the assistance of the grantees.

Lacking a central directory, I/Os were selected in a two-stage process, with both stages requiring the 

cooperation of grantees. The design was a multi-stage cluster design where the clusters were EBPs. 

EBPs were sampled from lists provided by grantees, and I/Os delivering the sampled programs were 

sampled, from additional lists provided by grantees. An important objective of the sampling process was 

to ensure that organizations providing services in rural/non-metropolitan areas were represented. 

Survey Questionnaire Design and Implementation

Grantee and implementation organization surveys were designed by HSAG. Survey items, including 

single-answer multiple choice items, check-all-that-apply items, and matrix-style rating items, were 

developed to answer ACL’s research questions and advance and help inform their knowledge objectives. 

The items were extensively reviewed by HSAG for content and readability, and HSAG engaged with 

ACL and stakeholder representatives to ensure that key topics were addressed adequately and in a 

manner appropriate to the target populations.

The surveys were developed in tandem and are substantially similar to the extent permitted by the 

differences in organization type. The largest difference between the grantee and implementation 

organization surveys is that grantee respondents were asked to “generalize your experience across all 

currently offered EBPs provided with funding from ACL,” while implementation organization 

respondents were asked to “think only about [a selected specific program] as you respond to questions.”

Once approved by ACL and by OMB, survey questionnaires were entered into a proprietary internet 

survey tool. 

Survey Data Collection

The surveys were fielded online from March 16, 2022, through May 6, 2022. 

Response Rates

Of 102 grantee surveys administered, responses were received for 91 (89 percent). Among the grantee 

surveys administered, responses were received for 49 of 55 SUA surveys (89 percent), and 42 of 47 (89

percent) of discretionary grantee surveys. Among I/Os that were sampled from lists provided by 
APPENDIX A. METHODOLOGY

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-3

Administration for Community Living

cooperating grantees, 69 of 131 contacted organizations (53 percent) responded to the survey. The 

response rate for surveys administered to rural I/Os (18 of 35, 51 percent) was slightly lower compared 

to the response rate for surveys administered to urban I/Os (43 of 79, 54 percent.)

High response rates among grantees do not guarantee valid and reliable data, but they raise our 

confidence in findings that emerge from data analysis. Conversely, the relatively low response rate 

among I/Os does not necessarily point to bias or other validity or reliability issues in the data collected. 

Taken in the context of the selectivity of the sample, however—contact lists were provided by little 

more than one-third of grantees (37 of 96, or 39 percent), meaning that there was no possibility of 

collecting data from the I/Os employed by the other 63 percent of grantees—caution is needed in 

drawing conclusions from the implementation organization data.

Survey Data Analysis

Once the survey data collection window closed, data were extracted from the Oracle databases used by 

the survey package and imported into the SAS statistical analysis and data management package. 

Univariate frequency tables were created for all closed response items—that is, all “multiple choice” or 

“check all that apply items,” which comprise virtually all of the items in the surveys. Some items are 

based on ordinal (ordered) scales with three or more levels, such as “Not adequate to meet guidelines,” 

“Adequate to meet guidelines,” and “More than necessary to meet guidelines.” For these items, tables 

with the following summary statistics were computed: mean, median, mode, minimum and maximum 

(range). Some of these items also included non-ordered categories such as “Not sure” or “We do not 

track this”; those categories were not included in computing the summary statistics. This means that Ns 

(frequencies) for the summary statistics may be lower than the Ns for univariate frequencies.

Multi-item indexes can be useful for summarizing a set of related survey items, and can be particularly 

useful for making between-group comparisons. Grantee survey items Q41, Q43, Q45, Q47, Q49, Q51, 

and Q53 (Q38, Q40, Q42, Q44, Q46, Q48, and Q50 in the implementation organization survey) concern 

how often various aspects of EBPs, such as program content and necessary materials, are provided to 

participants with fidelity to program guidelines. An index representing a rating of fidelity to program 

guidelines was constructed from these seven survey items. The index counts the number of items where 

respondents indicated that guidelines are “Always” adhered to, so its range is 0 to 7. Grantees may not 

have been able to provide meaningful responses to all seven survey items, however. For example, a 

program may not have guidelines regarding the frequency of sessions. To represent grantees’ fidelity 

practices as accurately as possible, items with missing responses do not affect the grantees’ score on the 

index. When there are missing responses, the index is rescaled to the 0 to 7 range using the calculation: 

INDEX = 7 * (<ALWAYS RESPONSES> / < NON-MISSING RESPONSES>).
APPENDIX A. METHODOLOGY

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-4

Administration for Community Living

Comparisons

Inferential statistics (statistical tests) were used to detect statistically significant differences between 

groups on the index of fidelity to program guidelines described above and on a selected set of 

measures/survey items. The groups that compared are:

1. Discretionary vs. formula grantee (grantee type)

2. Urban vs. rural implementation organization location (implementation organization urban/rural 

location)

The statistical tests are generally associated with crosstabulations, which show the general form of 

differences between groups, if any. The specific test varies depending on the type of item:

• For items with “Yes/No” responses, two-sample Z tests are used to test whether the groups differ 

in the proportion of “Yes” responses. 

• For items with nominal (unordered) response categories, such as “Funding,” “Staffing,” 

“Equipment,” and “Facilities,” Chi square tests of independence are computed to assess whether 

an association exists between the group type and the variable that the survey item represents.

• For items with ordinal response scales with 3 or more categories, we use the Mann-Whitney U 

test to identify statistically significant group differences. Although this test is commonly thought 

of a test for differences in medians, it is sensitive to differences in the distributions of responses 

and can yield a significant result even when medians are equal.

Qualitative Analysis: Key Informant Interviews

To obtain the qualitative and subjective information needed to address some of ACL’s research 

questions and knowledge objectives, eight ACL staff and representatives of eight EBP 

developers/administrators were interviewed using standardized protocols developed with input from 

ACL. HSAG used qualitative content analysis and an approach rooted in grounded theory to identify and 

describe key themes and emergent concepts.

Grantee Materials (Documents)

HSAG reached out to grantees on behalf of ACL, requesting:

• Lists of the EBPs being implemented by grantees under ACL discretionary grants for Falls 

Prevention and Chronic Disease Self-Management or Title III-D grants for health promotion 

under the Older Americans Act

• Written plans for monitoring fidelity of implementation of EBPs to program models

• Examples of administrative materials, documentation, and reports that describe or reflect how 

grantees supervise or monitor fidelity of EBPs to program models
APPENDIX A. METHODOLOGY

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-5

Administration for Community Living

The program lists are treated as statements of fact not requiring qualitative analysis. The other materials 

were analyzed to identify distinct types of and characteristics of fidelity monitoring plans and activities 

and identify themes across individual grantees, grantee types, and program types. They were also used to 

develop an understanding of the breadth of issues facing stakeholders and to help formulate specific 

survey questions. 

Limitations

This report has limitations, some stemming from the unfortunate timing of the study, which overlapped 

substantially with the COVID-19 PHE, and others are related to the extent of ACL’s prior knowledge 

about the practices of grantees, especially Title III-D mandatory grantees.

The COVID-19 PHE disrupted activities everywhere, but it hit organizations accustomed to providing 

face-to-face services to the public especially hard. Services had to be stopped or modified. Agencies had 

to scramble to find ways to effectively deliver services remotely. Agencies faced staffing shortages due 

to illness, childcare needs, and other correlates of the emergency. Some agencies might never have 

responded to requests for documents, interviews, or participation in a survey, but these circumstances 

surely made cooperation with researchers less likely than it might have been.

Notable limitations to the survey include:

• The data used in this study were voluntarily self-reported by grantees, program 

developers/administrators, and ACL staff. Voluntary participation is subject to self-selection bias, 

where those who provide data may differ systematically from those who do not; those who choose to 

participate may be more likely to represent larger, better-resourced organizations than those who do 

not, for example. Self-reporting is also subject to a number of biases; respondents may be prone to 

over-report desirable behaviors and under-report undesirable ones, for example. These biases can 

affect the reliability and validity of conclusions drawn from the data.

• For the grantee survey, respondents were asked to generalize across all relevant ACL-funded grant 

programs that they deliver or administer. This reduced the burden of response for grantees that 

deliver/administer multiple programs. It also affected the extent to which grantee responses can be 

meaningfully compared to implementation organization data, because I/Os were instructed to restrict 

their responses to the implementation of one specific program. Also, since several discretionary 

grantees hade awards for both Falls Prevention programs and Chronic Disease Self-Management 

programs, having grantees generalize across all their programs made it difficult to detect differences 

in how organizations approach program selection and fidelity for these two program types.

• The survey was administered in the first half of the third year of the COVID-19 PHE. The 

constraints on in-person interactions imposed by the PHE have forced grantees to stop offering 

programs or transform how the programs are offered. For this reason, grantees and I/Os that were 

delivering or administering programs prior to the PHE were asked to focus on those times in most of 

their survey responses. Nevertheless, the timing of this study may affect the degree to which the 

conclusions of this study will be generalizable to later time periods.
APPENDIX A. METHODOLOGY

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page A-6

Administration for Community Living

Limitations related to response rates include:

• As response rates decrease from an ideal 100% to a desirable 80% and down to as low as 50%, 25%, 

or even 5%, the level of confidence the researcher can have in the quality of the data also decreases 

considerably. A survey with a high response rate can suffer from selection bias (where some 

segments of the target population are badly underrepresented), and a high response rate does not 

overcome design flaws such as sampling frames that do not represent the target population or 

questions with low face validity. All else being equal, however, the lower the response rate the more 

caution that must be taken in generalizing from the survey respondents to the target population.

• Because the survey sampling frame for the I/Os was very selective, pulling organizations from an 

especially cooperative minority of grantees, and because the survey response rate was relatively low 

for the implementation organizations, extra care must be taken when generalizing from the survey 

respondents to the full population of implementation organizations.

Limitations specific to the methods used in analyzing the survey data include:

• The data analysis methods include univariate frequency distributions, univariate summary statistics, 

and bivariate crosstabulations, which are regarded as descriptive statistical methods. These methods 

are appropriate for the questions posed by ACL, but they are also subject to the potential biases and 

generalizability issues discussed above.

• Inferential methods (statistical testing) are used for comparing results for discretionary grantees to 

those for mandatory grantees, and for comparing I/Os located in urban ZIP Codes to those with rural 

ZIP Codes. Inferential methods generally make stronger assumptions about the nature of the data 

than do descriptive methods. Data issues such as those discussed above can undermine the capacity 

of inferential methods to yield correct conclusions, so results will need to be interpreted cautiously.

• This study was designed to gather basic knowledge about grantee and implementation organization 

practices, experiences, and perceptions. The analyses developed for this study are not intended to 

detect or quantify causal relationships between, for example, organizational characteristics, program 

characteristics, and the fidelity of program implementation. Instead, this study may provide 

information that can be used to formulate hypotheses about causal mechanisms, and these 

hypotheses could be tested in appropriately designed future research.
Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-1

Administration for Community Living 

B. Appendix B: Tables

Program Selection

To learn about how grantees and I/Os of various kinds and at different levels of organization approach 

the selection of appropriate EBPs, the survey examined how decisions are made and by whom, what 

information sources are important, what factors are important to the process and how they are weighed. 

The results for each question are presented in tabular form, showing both grantee and implementation 

organization results where possible. If the questions presented in the two surveys were different, results 

are presented in separate tables.

Input in program selection: To learn about the selection process, respondents were asked to identify 

who had input in the selection process and how various factors were weighed in decision-making. The 

results for grantees and I/Os are presented in Table 2-1. Input from leadership and in-house staff 

dedicated to the selection of programs were frequently selected among both groups of respondents, and 

state or local government guidance was also a source of input frequently selected by grantees. 

Table B-1. Grantee and Implementation Organization Question 3 — 

Who has input in the selection of programs? (Check all that apply)

Input Source Grantee I/O

Funder dictates 39% 25%

Leadership determines which programs to select 51% 52%

One or more in-house staff members oversee selection of programs 49% 64%

Committee or multidisciplinary process seeks input from across the organization 14% 12%

External partners, subject matter experts, or consultants 39% 18%

State or local government guidance (e.g. state units on aging) 61% 36%

Participation with peers and partners in planning or learning network 39% 31%

Recommendations/requests of public 31% 30%

Other 11% 3%

Grantee N = 90; Implementation Organization N = 67. Percentages may not total 100 as users can select multiple 

responses. 

Research question: How do grantees and I/Os select appropriate evidence-based programs for their 

contexts? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-2

Administration for Community Living

Among both grantees and implementation organizations, the EBP program registries provided by ACL, 

its partner NCOA, and interested stakeholders such as the EBLC were a prominent source of 

information for program selection, as were community needs and past organizational experience, as 

shown in Table 2-2. 

Table B-2. Grantee and Implementation Organization Question 4 — 

What information sources do you consider in selecting appropriate EBPs for 

your community and organization? (Check all that apply)

Information Source Grantee I/O

Internal policies 32% 19%

The Aging and Disability Evidence-Based Programs and Practices initiative (ADEPP), 

the National Council on Aging’s (NCOA) list of EBPs funded through the Older 

Americans Act Title III-D and approved through the Evidence-Based Program Review 

Process, the Evidence-Based Leadership Collaborative (EBLC) program menu and/or 

some other evidence-based program registry or webpage 88% 69%

Funder direction 42% 30%

Past experience 67% 43%

Community needs/interests 78% 81%

Other 9% 9%

Grantee N = 88. Implementation organization N = 67. Percentages may not total 100 as users can select multiple 

responses.

Importance of Availability of Resources in Selection of EBPs: The survey examined how important 

various factors are to the selection process, looking separately at three domains: general resource factors, 

program characteristics, and population characteristics. Respondents were asked to weigh factors in 

value from 1, the factor is not at all important, to 5, the factor is extremely important. The results for 

rating general resource factors are presented separately in Table 2-3 for grantees and Table 2-4 for 

implementation organizations. All of the options presented were important considerations. The item 

most frequently rated as extremely important, or 5 on the scale, was the same for both grantees and 

implementation organizations, with 56 percent of the former and 72 percent of the latter indicating that 

staffing is an extremely important resource issue. Conversely, fewer than 30 percent of respondents 

among grantees and I/Os identified any of these factors as less than moderately important. 

For grantees, 87 percent rated the importance of staffing as a 4 or 5, closely followed by training, rated a 

4 or 5 by 78 percent of grantees. Similarly, 90 percent of I/Os rated staffing a 4 or 5, and 76 percent 

rated training a 4 or 5. Among grantees, facility space, participant recruitment and data collection and 

reporting were all ranked lower in importance, with less than half of grantees rating them 4 or 5 (44 

percent, 48 percent, and 47 percent, respectively). For implementation organizations, only one resource 

factor, participant recruitment costs, fell to the level that less than half of them (49 percent) rated the 

factor a 4 or 5.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-3

Administration for Community Living

Table B-3. Grantee Question 5 — 

Please indicate how important the following resource factors are in the selection process. 

Importance

Resource Factor 1 2 3 4 5 Median N

Licensing costs 1% 13% 18% 29% 39% 4 85

Staffing required 0% 1% 12% 31% 56% 5 85

Training costs 0% 5% 18% 39% 39% 4 85

Special equipment costs 1% 13% 25% 31% 29% 4 83

Availability of facility space 4% 23% 30% 27% 17% 3 84

Participant recruitment costs 7% 20% 25% 29% 19% 3 85

Data collection and reporting costs 4% 19% 31% 26% 21% 3 85

 

Table B-4. Implementation Organization Question 5 — 

Please indicate how important the following resource factors are in the selection process.

Importance

Resource Factor 1 2 3 4 5 Median N

Licensing costs 8% 9% 21% 27% 35% 4 66

Staffing required 0% 6% 3% 18% 72% 5 65

Training costs 3% 5% 17% 34% 42% 4 65

Special equipment costs 6% 14% 22% 31% 28% 4 65

Availability of facility space 3% 15% 25% 34% 23% 4 65

Participant recruitment costs 3% 18% 29% 23% 26% 3 65

Data collection and reporting costs 5% 20% 23% 31% 22% 4 65

Statistical testing identified no significant differences in the distribution of responses between Title III-D 

and discretionary grantees or rural and urban implementation organizations. The complete results of 

statistical testing are presented in Table C-1 for grantees and Table C-24 for implementation 

organizations.

Importance of Program-Specific Characteristics in Selection of EBPs: The next set of factors 

considered for their importance in selecting EBPs included 12 program-specific characteristics. The 

results are presented for grantees in Figure 2-1 and for I/Os in Figure 2-2. There was very little variation 

in median scores across all 12 categories and both types of survey respondents, with a median of 4 

(meaning at least 50 percent of respondents rated it 4 or higher) for all with the exception of one 

characteristic: program content, which received a median score of 5 (at least 50 percent of respondents 

rated it 5) for both types of respondents. These results are illustrated in Figure 2-1 in descending order 

by the percentage of grantees rating the item extremely important. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-4

Administration for Community Living

Figure 2-1. Grantee Question 6 —

Please indicate how important the following program-specific characteristics are in the selection process.

22%

0%

1%

1%

1%

1%

0%

0%

0%

0%

2%

0%

11%

13%

8%

17%

17%

7%

2%

8%

2%

2%

2%

3% 0%

33%

30%

30%

24%

24%

24%

24%

21%

16%

20%

19%

19%

7%

22%

41%

42%

36%

36%

43%

42%

40%

44%

40%

40%

35%

34%

11%

16%

19%

21%

21%

24%

31%

31%

37%

38%

40%

41%

59%

OTHER

SUPPORTS FOR DISSEMINATION 

SUPPORT PROVIDED BY THE PROGRAM 

DEVELOPER/ADMINISTRATOR

PROGRAM DESIGN

STAFF/VOLUNTEER FAMILIARITY

PROGRAM LENGTH 

SUPPORTS FOR IMPLEMENTATION 

READINESS FOR REMOTE DELIVERY 

APPLICABILITY/CULTURAL APPROPRIATENESS TO 

THE COMMUNITY

PROGRAM FLEXIBILITY/ADAPTABILITY

AVAILABILITY OF TRAINING

STRENGTH OF THE EVIDENCE BASE FOR THE 

PROGRAM

PROGRAM CONTENT

PERCENTAGE OF GRANTEES RATING

PROGRAM-SPECIFIC CHARACTERISTIC

1 Not at all Important 2 3 4 5 Extremely Important
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-5

Administration for Community Living

Data Table for Figure 2-1. Grantee Question 6— 

Please indicate how important the following program-specific characteristics are in the selection process.

Program Characteristic 

1 

Not at all 

Important

2 3 4 

5 

Extremely 

Important

Program content 0% 0% 7% 34% 59% 

Strength of the evidence 

base for the program 2% 3% 19% 35% 41% 

Availability of training 0% 2% 19% 40% 40% 

Program 

flexibility/adaptability 0% 2% 20% 40% 38% 

Applicability/cultural 

appropriateness to the 

community

0% 2% 16% 44% 37% 

Supports for 

implementation 0% 2% 24% 42% 31% 

Readiness for remote 

delivery 0% 8% 21% 40% 31% 

Program length 1% 7% 24% 43% 24% 

Program design 1% 17% 24% 36% 21% 

Staff/Volunteer familiarity 1% 17% 24% 36% 21% 

Support provided by the 

program 

developer/administrator

1% 8% 30% 42% 19% 

Supports for dissemination 0% 13% 30% 41% 16% 

Other 22% 11% 33% 22% 11% 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-6

Administration for Community Living

Figure 2-2. Implementation Organization Question 6 —

Please indicate how important the following program-specific characteristics are in the selection process.

41%

2%

2%

5%

2%

3%

6%

6%

6%

8%

5%

5%

6%

8%

3%

3%

5%

2%

24%

30%

29%

27%

27%

27%

22%

19%

20%

20%

21%

14%

5%

12%

39%

39%

37%

34%

34%

33%

33%

33%

31%

29%

39%

27%

18%

23%

24%

24%

34%

34%

37%

38%

44%

45%

46%

47%

67%

Other

Support provided by the program developer/administrator

Supports for dissemination

Readiness for remote delivery

Program design

Staff/volunteer familiarity

Program length

Applicability/cultural appropriateness to the community

Program flexibility/adaptability

Supports for implementation

Strength of the evidence base for the program

Availability of training

Program content

Percentage of Implementation Organizations Rating

Program-Specific Characteristics

1 Not at all Important 2 3 4 5 Extremely Important
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-7

Administration for Community Living

Data Table for Figure 2 2. Implementation Organization Question 6 — 

Please indicate how important the following program-specific characteristics are in the selection process.

Program Characteristics 

1 

Not at all 

Important

2 3 4 

5 

Extremely 

Important

Program content 0% 2% 5% 27% 67% 

Availability of training 0% 0% 14% 39% 47% 

Strength of the evidence base for the 

program 

0% 5% 21% 29% 46% 

Supports for implementation 0% 3% 20% 31% 45% 

Program flexibility/adaptability 0% 3% 20% 33% 44% 

Applicability/cultural appropriateness 

to the community 3% 8% 19% 33% 38% 

Program length 2% 6% 22% 33% 37% 

Program design 0% 5% 27% 34% 34% 

Staff/volunteer familiarity 0% 5% 27% 34% 34% 

Supports for dissemination 2% 6% 29% 39% 24% 

Readiness for remote delivery 5% 8% 27% 37% 24% 

Support provided by the program 

developer/administrator 2% 6% 30% 39% 23% 

Other 41% 6% 24% 12% 18% 

Statistically significant differences in the distribution of responses to these questions were found 

between Title III-D and discretionary grantees for three program characteristics, with a tendency to be 

ranked more importantly among Title III-D grantees. These differences were found in the importance 

placed on: 

• The existence of supports for program dissemination

• Program design (e.g., discussion group, class structure, hands-on activities) 

• The degree to which staff or volunteers were already familiar with a program 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-8

Administration for Community Living

There were also significant differences in the distribution of answers between urban and rural I/Os for 

several factors, with a pattern of being rated more important by rural I/Os. This was true for several 

factors:

• Program design 

• The existence of supports for dissemination (e.g., contains instructions/materials for reaching out 

to population)

• The degree to which respondents have staff/volunteers who already know how to implement the 

program

• The existence of supports for implementation (e.g., contains instructions/materials for 

maintaining quality assurance, fidelity checklists)

• Program flexibility or adaptability 

• The availability of training 

Importance of Population-Specific Characteristics in Selection of EBPs: The relative importance of 

several population-specific characteristics to the selection process were examined, and the results are

presented for grantees in Figure 2-3. 

Figure 2-3. Grantee Question 7 —

Please indicate how important the following population-specific characteristics are in the selection process

24%

3%

4%

1%

1%

1%

2%

0%

6%

5%

8%

5%

9%

3%

36%

36%

31%

31%

23%

28%

15%

28%

35%

38%

33%

40%

29%

38%

12%

20%

24%

27%

31%

32%

41%

Other

Languages spoken

Urban/rural status

Disability status

Cultural diversity

Special populations

Specific health needs

Percentage of Grantees Rating

Population Characteristic

1 Not at all Important 2 3 4 5 Extremely Important
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-9

Administration for Community Living

Data Table for Figure 2-3. Grantee Question 7 — 

Please indicate how important the following population-specific characteristics are in the selection process. 

Population Characteristics

1 

Not at all 

Important

2 3 4 

5 

Extremely 

Important

Specific health needs 2% 3% 15% 38% 41% 

Special populations 1% 9% 28% 29% 32% 

Cultural diversity 1% 5% 23% 40% 31% 

Disability status 1% 8% 31% 33% 27% 

Urban/rural status 4% 5% 31% 38% 24% 

Languages spoken 3% 6% 36% 35% 20% 

Other 24% 0% 36% 28% 12% 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-10

Administration for Community Living

Implementation organizations’ rating of the importance of various population characteristics in their 

selection process is illustrated in Figure 2-4. 

Figure 2-4. Implementation Organization Question 7 —

Please indicate how important the following population-specific characteristics are 

in the selection process

57%

13%

10%

6%

8%

6%

3%

7%

20%

8%

6%

10%

5%

0%

29%

28%

25%

32%

24%

25%

25%

7%

25%

40%

38%

37%

35%

39%

0%

14%

17%

17%

22%

29%

33%

OTHER

LANGUAGES SPOKEN

CULTURAL DIVERSITY 

DISABILITY STATUS

URBAN/RURAL STATUS

SPECIAL POPULATIONS

SPECIFIC HEALTH NEEDS

PERCENTAGE OF IMPLEMENTATION ORGANIZATIONS RATING

POPULATION CHARACTERISTICS

1 Not at all Important 2 3 4 5 Extremely Important
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-11

Administration for Community Living

Data Table for Figure 2-4. Implementation Organization Question 7 — 

Please indicate how important the following population-specific characteristics are in the selection process

Population Characteristics

1 

Not at all 

Important

2 3 4 

5 

Extremely 

Important

Specific health needs 3% 0% 25% 39% 33% 

Special populations 6% 5% 25% 35% 29% 

Urban/rural status 8% 10% 24% 37% 22% 

Cultural diversity 10% 8% 25% 40% 17% 

Disability status 6% 6% 32% 38% 17% 

Languages spoken 13% 20% 28% 25% 14% 

Other 57% 7% 29% 7% 0% 

Statistical testing identified differences in the distribution of responses for most of these population 

characteristics between Title III-D and discretionary grantees, with each factor rated higher in 

importance among the former, as shown in Table 2-5

Table B-5. Statistical Tests for Differences in Responses, Grantee Question 7 — 

Please indicate how important the following population-specific characteristics are 

in the selection process.

Title III-D Grantees

Importance

Discretionary Grantees

Importance

Population Characteristics 1 2 3 4 5 1 2 3 4 5

Specific health needs 0% 2% 20% 30% 48% 5% 5% 10% 48% 33%

Languages spoken 2% 0% 39% 37% 22% 5% 13% 33% 33% 18%

Cultural diversity (e.g. racial/ethnic 

groups, tribes)**

0% 2% 15% 46% 37% 3% 8% 33% 33% 25%

Special populations (e.g. veterans, 

low income)**

0% 4% 24% 35% 37% 3% 15% 33% 23% 26%

Disability status* 0% 2% 33% 33% 33% 3% 15% 30% 33% 20%

Urban/rural status** 2% 2% 24% 42% 29% 5% 8% 38% 33% 18%

Other 19% 0% 44% 25% 13% 33% 0% 22% 33% 11%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses 

by Title III-D grantees compared to discretionary grantees for this population characteristic.

Question text: Please indicate how important the following population-specific characteristics are in the selection 

process: Please rate these factors between 1 and 5, where 1 = not at all important and 5 = extremely important.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-12

Administration for Community Living

Note: A Mann-Whitney U test was used to assess differences in the distribution of responses between Title III-D 

grantees and discretionary grantees for each population characteristic. Across all population characteristics, N ranges 

from 16 to 46 for Title III-D grantees and from 9 to 40 for discretionary grantees. Percentages within each grantee 

type may not sum to 100 within each row due to rounding.

There were minimal differences in distribution of responses by type of implementation organization, 

with only cultural diversity receiving a different distribution of scores in urban and rural organizations 

(p<0.1). A total of 67 percent of I/Os located in urban areas rated this factor 4 or 5 in importance, 

compared to 41 percent of rural implementation organizations. 

At the highest level, these results indicate that grantees and I/Os must balance a wide slate of competing 

but important priorities in making program selections. Overall, program content, the strength of the 

evidence base, and the availability of training tended to be the most important program characteristics. 

The health needs of the population and special populations were assigned the highest importance ratings 

among population characteristics. 

Opportunities for Assistance in Selection of EBPs: Survey respondents indicated receptiveness to all 

of the suggested types of assistance for improving the program selection process offered, with the clear 

favorite being more help in making local adaptations to programs. The results are presented in Table 

2-6. 

Table B-6. Grantee and Implementation Organization Question 8 — 

Which of the following could be a significant help to your organization in the program selection process? 

(Check all that apply)

Helpful in Selection Grantee I/O

Comprehensive and comparable estimates of program costs 80% 63%

Simpler program guidelines 58% 46%

Standardized program guidelines 62% 54%

Greater information about program flexibility—the capacity to accommodate local 

adjustments without affecting fidelity and effectiveness 93% 87%

Other 2% 2%

Grantee N = 86. Implementation Organization N = 63. Percentages may not total 100 as users can select multiple 

responses.

When provided with the opportunity to explain further what “other” items might be helpful, one SUA 

requested information on the number of facilitators required for implementation and whether there were 

options for variation if the program was done in some form of hybridization between online instructor 

and congregated participants. One discretionary grantee requested clear information on the return on 

investment (ROI) and health impact and outcomes of programs and whether it had been proven effective 

in different languages and with different populations. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-13

Administration for Community Living

Program Fidelity 

ACL’s ability to fulfill its mission to maximize the independence, well-being, and health of older adults, 

people with disabilities across the lifespan, and their families and caregivers rests on the quality of the 

evidence-based programs it supports. The results of implementing these EBPs, in turn, are contingent on 

the extent to which their program models are followed. Fidelity for purposes of this report is the extent 

to which delivery of the EBPs supported by ACL consistently adhere to the program models. 

Maintaining fidelity to the program model is essential to ensure that programs achieve the desired results 

– in this case, improvements in health outcomes for program participants. 

General Fidelity

More than half of grantees (53 percent), and two thirds of I/Os (68 percent), provided EBP programming 

through subcontractors/subgrantees, but many also provided services directly. The results are presented 

in Table 2-7. These results indicated that a great number of grantees and I/Os are relying on others to 

oversee and implement guidelines. 

Table B-7. Grantee Question 2 — 

Does your organization deliver ACL-funded programs directly to the public, 

indirectly through subcontractors/partners, or both?4-1

Service Delivery Grantee I/O

Directly 14% 18%

Indirectly through subcontractors/subgrantees/partners 53% 68%

Both 32% 15%

Grantee N = 90; Implementation Organization N = 68. 

Although many of these respondents did not directly implement programing, there is evidence that the 

majority of I/Os surveyed were actually required to provide documentation of their fidelity practices as 

4-1 Implementation Organization Question 2 — Your organization is included in this survey because it is receiving grant 

funding from ACL to deliver <PROGRAM NAME>. Does your organization receive grant funding directly from ACL 

or is it a subcontractor/subgrantee/or partner to another organization that receives grant funds from ACL?

Research Question: To what extent do staff overseeing and implementing the programs fully 

understand and comply with the implementation guidelines/instructions?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-14

Administration for Community Living

shown in Table 2-8. Unlike the grantees who participated in the survey, these organizations were 

selected because they were identified as providing a specific program directly to participants.

Table B-8. Implementation Organization Question 9 — 

Does <GRANTEE NAME> require that you provide documentation that you are implementing 

<PROGRAM NAME> with fidelity?

Fidelity Documentation I/O 

Yes 80%

No 20%

N = 60. 

Clarity of Guidelines: In order to translate program models into outcomes, it is necessary for the 

program developer/administrator to provide clear guidelines that can be uniformly interpreted by the end 

users to produce consistent results. Respondents were asked to rate the clarity of standards for training, 

program resources, target population and program content as “very clear,” “somewhat clear,” somewhat 

unclear,” or “very unclear.” Majorities of more than 50 percent of respondents indicated that program 

guidelines for were very clear for three of the four items. Guidelines for resources were rated as very 

clear by only 36 percent of grantees. The results of this rating of clarity were collected in several tables 

that were presented in a preliminary report to ACL, and are not reproduced here. The results from these 

tables are compared in Figure 2-5. 

 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-15

Administration for Community Living

Figure 2-5 — Rating of Guideline Clarity by Content Area and Type of Respondent

51%

58%

36%

57%

64%

69%

58%

66%

40%

33%

52%

40%

31%

29%

38%

31%

5%

3%

9%

2%

4% 0%

5%

2%

0%

3%

1%

2%

2%

0%

2%

4%

2%

2%

0%

1%

0%

0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Grantee

I/O

Grantee

I/O

Grantee

I/O

Grantee

I/O

Training Resources Target population Content

Very clear Somewhat clear Somewhat unclear Very unclear No guidelines
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-16

Administration for Community Living

Data Table for Figure 2-5— Rating of Guideline Clarity by Content Area and Type of Respondent

Resource Respondent Very clear Somewhat 

clear

Somewhat 

unclear Very unclear No guidelines

Training Grantee 51% 40% 5% 0% 4%

I/O 58% 33% 3% 3% 2%

Resources Grantee 36% 52% 9% 1% 2%

I/O 57% 40% 2% 2% 0%

Target 

population Grantee 64% 31% 4% 0% 1%

I/O 69% 29% 0% 2% 0%

Content Grantee 58% 38% 5% 0% 0%

I/O 66% 31% 2% 2% 0%

Strategies for Updating Program Guidelines: A majority of I/Os that responded to this question, 

eighty percent (48 of 60), reported that the grantee they worked with required documentation that they 

implemented their chosen EBP with fidelity. Grantees and I/Os reported using several strategies to stay 

current on best practices for establishing and maintaining program fidelity, with the most frequent being 

receiving updates from program developers/administrators via a group email or list. Grantees were much 

more likely to check ACL/NCOA websites than implementation organizations. The results are presented 

in Table 2-9. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-17

Administration for Community Living

Table B-9. Grantee Question 9 — 

How does your organization stay informed about approaches and methods for establishing

and maintaining program fidelity with regard to the programs you select or deliver?

(Check all that apply)4-2

Information Source Grantee I/O

Participate in program networking group 67% 60%

Participate in broad health related networking group (e.g. falls prevention) 60% 51%

Receive updates from program developer/administrator (e.g., via email group or list) 81% 84%

Check ACL/NCOA websites 73% 38%

Other 8% 11%

Grantee N = 85. Implementation Organization N = 63. Percentages may not total 100 as users can select multiple 

responses. 

In general, then, the respondents reported they found guidelines to be clear, that the majority used 

several strategies to stay current with best practices in the field, and although many programs are 

delivered indirectly by both grantees and implementation organizations, most of the I/Os indicated that 

they were required to document fidelity processes. 

Next, we consider how grantees and I/Os verify that programs are implemented with fidelity over time, 

addressing the following knowledge objective. 

4-2 Implementation Organization Question 10 — How does your organization stay informed about approaches and methods 

for establishing and maintaining program fidelity with regard to <PROGRAM NAME>?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-18

Administration for Community Living

Maintaining Fidelity: Both grantees and I/Os embraced a broad menu of strategies to assure fidelity of 

their EBPs, as reflected in Table 2-10. The most frequently selected strategies for both types of 

respondents were feedback from trainers, leaders, and participants. More than half of respondents 

indicated they regularly observed trainers/leaders during sessions (60 percent of grantees, 57 percent of 

implementation organizations). Nearly half of grantees (44 percent) and about one quarter of I/Os (26 

percent) reported undertaking regular standardized review of program fidelity. 

Table B-10. Grantee Question 10 — 

What strategies does your organization use to assure faithful delivery of EBPs? (Check all that apply)4-3

Fidelity Strategy Grantee I/O

Regularly observe trainers/leaders during sessions 60% 57%

Collect feedback from trainers/leaders 78% 75%

Collect feedback from participants 74% 89%

Use services provided by program developer/administrator 38% 46%

Hold regular management strategy sessions regarding fidelity 36% 18%

Hold regular staff meetings and/or trainings that stress fidelity 44% 34%

Regular standardized review of program fidelity 44% 26%

Informal review of program fidelity as needed 67% 59%

Other 9% 2%

Grantee N = 85. Implementation Organization N = 61. Percentages may not total 100 as users can select multiple 

responses. 

There was little difference in the distribution of responses among I/Os in urban or rural locations, with 

only one strategy, collecting feedback from participants, showing a statistically significant difference 

(urban I/Os 92 percent, rural organizations 75 percent (p<0.1). 

However, significant differences in the distribution of responses were identified between Title III-D and 

discretionary grantees and are shown in Table 2-11. For all of the strategies with significant differences, 

discretionary grantees reported higher rates of utilization. This is probably to be expected, given the 

agreements and responsibilities that are built into the discretionary grant structure. It is encouraging, 

though, that the results show that a fairly large percentage of Title III-D grantees also embrace these 

practices. In addition, 43 percent of Title III-D grantees and 44 percent of discretionary grantees 

4-3 Implementation Organization Question 11 — What strategies does your organization use to assure faithful delivery of 

<PROGRAM NAME>?

Knowledge Objective: How do grantees implement programs and verify that they are being 

implemented with fidelity to their original models? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-19

Administration for Community Living

indicated they conduct regular standardized review of program fidelity. A full two thirds (67 percent) of 

all grantees reported they conducted informal reviews of program fidelity as needed. 

Table B-11. Statistical Tests for Differences in Responses by Type of Grantee, Question 10 — 

What strategies does your organization use to assure faithful delivery of EBPs? (Check all that apply)

Fidelity Strategy Title III-D 

Grantees

Discretionary 

Grantees

Regularly observe trainers/leaders during sessions** 50% 72%

Collect feedback from trainers/leaders*** 63% 95%

Collect feedback from participants*** 61% 90%

Use services provided by program developer/administrator 41% 33%

Hold regular management strategy sessions regarding fidelity** 26% 49%

Hold regular staff meetings and/or trainings that stress fidelity 37% 51%

Regular standardized review of program fidelity 43% 44%

Informal review of program fidelity as needed 67% 67%

Other 13% 5%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses by 

Title III-D grantees compared to discretionary grantees for this fidelity strategy. 

Note: A two-sample z-test was used to assess the existence of a relationship between grantee type and each response to 

this question. N = 46 for Title III-D grantees and N = 39 for discretionary grantees. Percentages may not total 100 as 

users could select multiple responses.

Those respondents that reported regular standardized review of program fidelity were asked to identify 

the sources for guidelines for that fidelity review, which are presented in Table 2-12. The most common 

sources for guidelines were internal to the organization for both grantees and implementation 

organizations, with the second most common source being the program developers/administrators. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-20

Administration for Community Living

Table B-12. Grantee Question 11, Implementation Organization Question 12 — 

Who created the guidelines for the review process? (Check all that apply)

Guideline Source Grantee I/O

Your organization 68% 69%

Partners at the local or state level 38% 19%

Association or collaborative of EBP implementers 22% 13%

The program developer/administrator 54% 56%

The state/SUA 49% 13%

Other organization such as the National Council on Aging (NCOA), Evidence Based 

Leadership Collaborative (EBLC) 32% 19%

Other 0% 6%

Grantee N = 37. Implementation Organization N = 16. Percentages may not total 100 as users can select multiple 

responses. 

Most survey respondents reported confirming fidelity on a regular basis, by selecting one of the periods 

suggested in the survey. In aggregate, at least 80 percent of grantees described confirming fidelity on at 

least an annual basis, as shown in Table 2-13. This is likely a conservative measure of the total use of 

reviews, since some of the 20 percent of respondents who selected “other” might be conducting regular 

reviews but using different time periods. Nearly all of the I/Os indicated they utilized one of the 

suggested periodic review cycles, with only 6 percent selecting “other” as shown in Table 2-14.

Table B-13. Grantee Question 12 — 

How often do you confirm that the EBPs are being implemented according to the program models?

Confirmation Frequency Grantee

Each time the program is offered 25%

Monthly 8%

Quarterly 19%

Annually 28%

Other 20%

Grantee N = 85.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-21

Administration for Community Living

Table B-14. Implementation Organization Question 13 — 

How often do you confirm that <PROGRAM NAME> is being implemented according to 

the program models?

Confirmation Frequency I/O

When the program was first offered 8%

Each time the program is offered 55%

Monthly 6%

Quarterly 8%

Annually 16%

Other 6%

Implementation Organization N = 62.

Grantees identified time burden as their major impediment to maintaining fidelity, while I/Os reported 

fewer challenges, with nearly half (43 percent) reporting no challenges as shown in Table 2-15. 

Table B-15. Grantee Question 13 — 

What challenges have you encountered to maintaining fidelity to program models? 

(Check all that apply)4-4

 

Fidelity Challenge Grantee I/O

Cost of monitoring fidelity 36% 8%

Time burden of monitoring fidelity 72% 38%

Lack of internal expertise on maintaining fidelity 31% 11%

Inadequate guidance from program developer/administrators 15% 8%

Lack of agreement among stakeholders on the importance of fidelity 9% 5%

Other 16% 15%

None 16% 43%

Grantee N = 85. Implementation Organization N = 61. Percentages may not total 100 as users can select multiple 

responses.

There were almost no differences between grantee types on this question, with lack of internal expertise 

maintaining fidelity more frequently selected among Title III-D grantees (39 percent) than discretionary 

grantees (21 percent)(p<0.1). There were no statistically significant differences in the distribution of 

responses between I/Os in urban and rural areas. 

4-4 Implementation Organization Question 14 — What challenges have you encountered to maintaining fidelity for 

<PROGRAM NAME>?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-22

Administration for Community Living

Fidelity in Training 

The survey examined training at two levels – that of the immediate facilitators, and that of the more 

removed grantee or organization that supervises or trains master trainers or higher-level trainers. 

1. Training of Facilitators

Responsibility for training facilitators: Among survey respondents, most training of facilitators was 

handled either by the program developer/administrator or, if the program permitted master trainers, by 

the respondents’ staff members that had been certified to provide training by the 

developer/administrator, as shown in Table 2-16. 

Table B-16. Grantee Question 15, Implementation Organization Question 16 — 

Who is primarily responsible for training facilitators?

Training Source Grantee I/O

External—Program developer/administrator or training entity 43% 59%

Internal—Program-certified trainers on staff (if applicable) 49% 31%

Internal—Other staff members with program experience (e.g. facilitators) 2% 8%

Other 6% 2%

Grantee N = 82. Implementation Organization N = 61.

There were no statistically significant differences between responses based on type of grantee or 

location of implementation organization. Some insight was provided in response to the invitation to 

provide a text response. Grantees and I/Os both sometimes contracted with partners to provide and 

monitor training of facilitators, with the results that they were dependent on their schedules for training 

availability. Several SUA grantees described using sub-grantees or other organizations to conduct 

formal observations of classes documented with program-specific fidelity check lists and have built that 

requirement into their signed agreements. Some required and monitored regular reports of the number of 

sessions observed or other fidelity metrics, but practices appear to vary widely.

Guidelines for training facilitators: Fidelity to the program model is widely ensured by the fact that 

both grantees and I/Os draw training materials directly from program developer/administrators, as 

indicated by 98 percent of the former, and 83 percent of the latter. A number of respondents also 

provided their own guidelines for training facilitators as shown in Table 2-17.

Research Questions: Are facilitators trained in accordance with the guidelines of the relevant 

evidence-based program(s)? Is the amount of training and frequency sufficient? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-23

Administration for Community Living

Table B-17. Grantee Question 16, Implementation Organization Question 17 — 

What is the source of guidelines and materials used in training facilitators? (Check all that apply)

Guideline Source Grantee I/O

Program developer/administrator, either directly or through certification of facilitators 98% 83%

Own organization 33% 42%

Peer organizations 14% 4%

State agency other than own organization 7% 8%

Other 0% 0%

Grantee N = 42. Implementation Organization N = 24. Percentages may not total 100 as users can select multiple 

responses. 

Confirmation of certification of facilitators: Most respondents required and verified that facilitators 

were properly certified either on a regular schedule or each time the program is offered, providing some 

confidence that their training should be up to date and in accordance with program guidelines. Majorities 

of grantees (59 percent) and I/Os (62 percent) reported regular or annual confirmation as shown in 

Table 2-18. In contrast, only a small percentage of respondents reported that they did not track regular 

confirmation of certifications, 13 percent of grantees, and 5 percent of implementation organizations. 

A larger percentage of Title III-D grantees than discretionary grantees indicated they confirmed training 

certification of facilitators during the application or approval process (18 percent for the former, 

compared with 5 percent of the latter, p<0.1). However, Title III-D grantees also indicated more 

frequently that they did not track certification of facilitators (20 percent, as compared to 5 percent of 

discretionary grantees, p<0.05). No significant differences in the distribution of responses were found 

between urban and rural implementation organizations.

Table B-18. Grantee Question 18, Implementation Organization Question 19 — 

For programs that require certification for facilitators, when do you confirm that certifications are current? 

(Check all that apply)

Certification Timing Grantee I/O

During grant application/approval process 12% 8%

When first offer program 16% 13%

Each time program is offered 31% 38%

Annually or regularly 59% 62%

We do not track this 13% 5%

Grantee N = 83. Implementation Organization N = 61. Percentages may not total 100 as users can select multiple 

responses.

Assessment of fidelity skills: Roughly two thirds of respondents verified that facilitators implemented 

programs with fidelity to program models by formally observing group sessions (66 percent of grantees 

and 68 percent of implementation organizations) and collecting feedback from participants or trainees 

(60 percent of grantees and 62 percent of implementation organizations). Only ten percent or less of 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-24

Administration for Community Living

respondents reported they had no system for assessing facilitator’s fidelity skills post training. The 

results are presented in Table 2-19. Statistical testing identified differences in the frequency of “other” 

responses by both type of grantee and location of implementation organization, however it is difficult to 

draw any infrences from these differences. 

Table B-19. Grantee Question 19, Implementation Organization Question 20 — 

How are facilitators’ fidelity skills assessed after training is completed? (Check all that apply)

Assessment Method Grantee I/O

Formal observation of group/workshop sessions 66% 68%

Formal tests of skills or knowledge 12% 7%

Collection of feedback from participants or trainees 60% 62%

By monitoring process metrics to identify potential issues 30% 18%

Other (please specify) 11% 5%

No assessment system in place 9% 10%

Grantee N = 82. Implementation Organization N = 60. Percentages may not total 100 as users can select multiple 

responses. 

Among respondents who provided additional explanation in text, some mentioned relying on 

documentation or regular reporting of observation conducted by AAAs. Others relied on local agencies 

to assess skills, acknowledging that there is a variety in practices among them. Several did not track this 

type of assessment. One implementation organization reported that only two co-workers faciliated all 

classes, and another described using a self-evaluation form to assess fidelity skills.

Supervision of facilitators: Fidelity in delivery of program content appears to be supported by 

employing systematic methods to address any shortcomings discovered in the review process. Most 

respondents reported systematic follow-up upon identification of issues with facilitators’ program 

delivery, ranging from coaching to replacement of facilitators who deviated from program guidelines, as 

shown in Table 2-20. The additional text responses provided by respondents largely either followed a 

combination of the methods discussed in this question, or reported that this oversight was conducted by 

sub-grantees or agencies. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-25

Administration for Community Living

Table B-20. Grantee Question 20, Implementation Organization Question 21 — 

What actions are taken if facilitators are found to be delivering content that does not meet program 

guidelines? (Check all that apply)

Corrective Actions Grantee I/O

Provide coaching 81% 86%

Create a corrective action plan 51% 40%

Issue a warning 33% 17%

Replacement 23% 22%

Other (please specify) 12% 2%

None of the above 7% 12%

Grantee N = 83. Implementation Organization N = 58. Percentages may not total 100 as users can select multiple 

responses. 

Fidelity to the program guidelines for items such as the number, order, or length of sessions was treated 

much the same as content delivery, with most respondents reporting systematic follow-up upon 

identification of issues with facilitators’ delivery of program per guidelines regarding order or length of 

sessions or making unauthorized adaptations, as shown in Table 2-21. 

Table B-21. Grantee Question 21, Implementation Organization Question 22 — 

What actions are taken if facilitators are found to be delivering the program in ways that do not meet 

program guidelines (e.g., changing the order or length of sessions, making unauthorized program 

adaptations)? (Check all that apply) 

Corrective Actions Grantee I/O

Provide coaching 80% 81%

Create a corrective action plan 53% 41%

Issue a warning 34% 31%

Replacement 28% 26%

Other (please specify) 20% 9%

Grantee N = 83. Implementation Organization N = 58. Percentages may not total 100 as users can select multiple 

responses. 

The additional text provided by some respondents generally either followed a combination of the 

methods discussed in Table 2-21, above, or reported that oversight was conducted by sub-grantees or 

agencies. Some noted that they have not experienced a situtation where facilitators failed to follow 

guidelines with fidelity. Some I/Os acknoledged that their fidelity had been checked by master trainers 

who have the ability to remove certification if a facilitator does not comply with guidelines.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-26

Administration for Community Living

2. Fidelity in Training of Trainers

The results for the higher administrative level of training the trainers were analogous to those for 

training facilitators, and reflect recognition of the integral importance of fidelity to program design at 

every level. 

Responsibility for training trainers: A higher percentage of respondents resorted to external training 

of trainers or master trainers than facilitators, with most utilizing the program developer/administrator or 

a training entity (58 percent of grantees and 79 percent if implementation organizations) as shown in 

Table 2-22. In the optional text field, one SUA respondent added the explanation that if the program 

developers were the only trainers available, they used them, but they primarily used a contractor or AAA 

program certified staff or partners. A significant difference (p<0.1) was found in the pattern of answers 

for grantees, with Title III-D grantees making greater use of external trainers than internal trainers by a 

ratio of 67 percent to 24 percent, while discretionary grantees are more evenly divided with roughly half 

(47 percent) external and half (50 percent) internal. No difference was found between urban and rural 

implementation organizations.

Table B-22. Grantee Question 22, Implementation Organization Question 23 — 

Who is primarily responsible for training program trainers? (Check all that apply)

Training Source Grantee I/O

External—Program developer/administrator or training entity 58% 79%

Internal—Program-certified trainers on staff (if applicable) 36% 20%

Internal—Other staff members with program experience 1% 2%

Other (please specify) 5% 0%

Grantee N = 83. Implementation Organization N = 61. Percentages may not total 100 as users can select multiple 

responses. 

Guidelines for training trainers: Among those respondents who trained trainers directly, training 

materials were overwhelmingly adopted from program developer/administrators, although individual 

organizations at the grantee or implementation organization level also provided guidelines for training 

facilitators as shown in Table 2-23. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-27

Administration for Community Living

Table B-23. Grantee Question 23, Implementation Organization Question 24 — 

What is the source of guidelines and materials used in training program trainers? (Check all that apply)

Guideline Source Grantee I/O

Program developer/administrator 90% 77%

Own organization 29% 38%

Peer organizations 10% 8%

State agency other than own organization 6% 23%

Organization providing funding 6% 8%

Grantee N = 31. Implementation Organization N = 13. Percentages may not total 100 as users can select multiple 

responses. 

Confirmation of trainer certification: The majority of grantees take some steps to ensure that program 

trainer certifications are current; only 10 percent of grantees reported they did not systematically track 

current certification of trainers as shown in Table 2-24. About one fifth (21 percent of grantees) relied 

on program implementers to use only certified facilitators. 

Table B-24. Grantee Question 24 — 

For programs that require certification for program trainers, when do you confirm that 

certifications are current? (Check all that apply)

Certification Timing Grantee

During grant application/approval process 18%

Each time program is offered 27%

Annually or periodically 58%

We rely on program implementer to use only certified facilitators 21%

We do not track this 10%

N = 84. Percentages may not total 100 as users can select multiple responses.

There were significant differences in the distribution of response by type of grantee, with the results 

presented in Table 2-25. Title III-D grantees indicated they confirmed certification during the grant 

approval process more frequently than discretionary grantees (26 percent vs 8 percent respectively, 

p<0.05). The most frequent item selected by both types of grantees was annual or periodic confirmation, 

although the percentage was lower among Title III-D grantees (50 percent) than discretionary grantees 

(68 percent) (p<0.1). Twenty eight percent of Title III-D grantees relied on program implementers to 

perform this function, and fifteen percent did not track it themselves, as compared to 13 percent and 3 

percent among discretionary grantees. Conversely, this appears to mean that 97 percent of I/Os track 

confirmation that program trainers are currently certified in some way. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-28

Administration for Community Living

Table B-25. Statistical Tests for Differences in Responses by Type of Grantee, Question 24 — 

For programs that require certification for program trainers, when do you confirm 

that certifications are current? (Check all that apply)

Certification Timing

Title III-D 

Grantees

Discretionary 

Grantees

During grant application/approval process** 26% 8%

Each time program is offered 28% 26%

Annually or periodically* 50% 68%

We rely on program implementer to use only certified facilitators* 28% 13%

We do not track this* 15% 3%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses 

by Title III-D grantees compared to discretionary grantees for this certification timing. 

Note: A two-sample z-test was used to assess the existence of a relationship between grantee type and each response 

to this question. N = 46 for Title III-D grantees and N = 38 for discretionary grantees. Percentages may not total 100 

as users could select multiple responses.

I/Os also overwhelmingly tracked current confirmation of certification as seen in Table 2-26. There 

were no significant differences found in responses by urban and rural implementation organizations. 

Table B-26. Implementation Organization Question 25 — 

For programs that require certification for program trainers, when do you confirm that 

certifications are current? (Check all that apply)

Certification Timing I/O

During grant application/approval process 18%

Each time program is offered 30%

Annually or periodically 67%

We do not track this 7%

N = 60. Percentages may not total 100 as users can select multiple responses.

Assessment of trainers: Most respondents described adopting a variety of strategies aimed at verifying 

that trainers demonstrated fidelity to program design as shown in Table 2-27. Half or more of grantees 

indicated they did so by formal observation of sessions (55 percent) and/or collecting feedback from 

participants or trainees (55 percent). The percentages were even higher for implementation 

organizations, 66 percent, and 64 percent respectively. Roughly a third of grantees (32 percent) 

monitored program metrics to identify potential fidelity issues arising from training, as did 19 percent of 

implementation organizations.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-29

Administration for Community Living

Table B-27. Grantee Question 25, Implementation Organization Question 26 — 

How are program trainers assessed for fidelity practices after training is completed?

(Check all that apply)

Assessment Method Grantee I/O

Formal observation of group/workshop sessions 55% 66%

Formal tests of skills or knowledge 9% 5%

Collection of feedback from participants or trainees 55% 64%

By monitoring program metrics to identify potential issues 32% 19%

Other (please specify) 12% 3%

No assessment system in place 16% 12%

Grantee N = 82. Implementation Organization N = 58. Percentages may not total 100 as users can select multiple 

responses. 

There was a significant difference in the distribution of responses by grantee type with regard to the use 

of formal observation of sessions to assess trainers, with 45 percent of Title III-D grantees and 66 

percent of discretionary grantees selecting that option (p<0.1). There were no significant differences in 

distribution of responses between urban and rural implemenation organizations. 

Few new approaches were identified in the free text responses to question. Some mentioned using 

annual monitoring of certification and quarterly program specific workgroup calls to monitor ongoing 

fidelity. Some noted the task was performed by a master trainer upon recertification. Some described 

delegating this task to subrecipients and/or did not directly assess trainers. 

Supervision of subcontractors or partners: Those grantees that reported overseeing EBPs delivered 

by a subcontractor or partner (supervising grantees) were asked several questions about whether they set 

standards and verified compliance with standards for training by their partners. The majority of those 

supervising grantees reported setting standards for training for their subcontractors or partners delivering 

EBPs, although there was a statistically significant difference in the responses by type of grantee as 

shown in Table 2-28. Title III-D grantees were fairly evenly divided, with 53 percent setting standards, 

and 47 percent not. Discretionary grantees, however, were much more likely to indicate they set 

standards for training for partners delivering EBPs (83 percent). I/Os were not asked a corresponding 

question, As the immediate facilitators, it was assumed they were not involved in supervising training 

for other entities for that particular program. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-30

Administration for Community Living

Table B-28. Statistical Tests for Differences in Responses by Type of Grantee, Question 26 — 

Does your organization set standards for training for partners delivering EBPs?

Set Standards

Title III-D 

Grantees

Discretionary 

Grantees

Yes 53% 83%

No 47% 17%

Note: A two-sample z-test was used to assess the existence of a relationship between grantee type and responses to 

this question. The result was statistically significant at p < 0.05 (p = 0.01). N = 43 for Title III-D grantees and N = 

29 for discretionary grantees. Percentages within each grantee type may not sum to 100 due to rounding.

Among grantees who indicated they supervised others in providing programming, a large majority (81 

percent) drew standards from program developers/administrators, and about half (49 percent) also had 

standards of their own, as presented in Table 2-29. 

Table B-29. Grantee Question 27 — 

What is the source of those standards? (Check all that apply)

Standards Source Grantee

Program developer/administrators 81%

Own organization 49%

Peer organizations 9%

State agency other than own organization 28%

ACL 34%

N = 47. Percentages may not total 100 as users can select multiple responses.

A large majority of these supervising grantees (87 percent) verified that standards for training trainers 

were met by requiring reporting from their partners who implemented programing. More than half (57 

percent) also periodically observed activities at partner sites. The results are presented in Table 2-30. 

Those respondents that provided additional text information described internal processes ranging from 

regular check-in meetings to monthly calls, to annual audits. One discretionary grantee explained that 

I/Os operated under their license and supervision, and were treated the same as their own facilitators. 

One SUA mentioned a combination of direct monitoring by themselves and through appropriate AAAs. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-31

Administration for Community Living

Table B-30. Grantee Question 28 — 

How does your organization verify that standards are met? (Check all that apply)

Training Verification Grantee

Reporting from partners 87%

Periodic observation at partner sites 57%

Other (please specify) 11%

N = 47. Percentages may not total 100 as users can select multiple responses.

These results present a picture of the broad adoption of systematic attention to training that is needed to 

maintain fidelity to program models. Certification of trainers, incorporation of program standards, 

assessment and supervision of their skills and performance over time, and plans for addressing 

noncompliance all appear to be woven into the system of management for most grantees and 

implementation organizations.

Fidelity in Program Resources 

Adequacy of Resources: The survey examined the sufficiency of several types of resources to meet 

program guidelines. Most respondents described funding, staffing, equipment, and facilities were all 

adequate to meet guidelines, as shown for grantees in Table 2-31 and I/Os in Table 2-32. From 5 percent 

to 9 percent, however, did not believe there were relevant guidelines. There were no significant 

differences in responses based on type of grantee. 

Table B-31. Grantee Question 30 — 

For each of the following, do the resources allocated to your programs typically meet program guidelines?

Resource

Not 

Adequate 

To Meet 

Guidelines 

1

Adequate 

To Meet 

Guidelines 

2

More 

Than 

Necessary 

To Meet 

Guidelines 

3

There Are 

No 

Guidelines 

4 Median N

Funding 0% 81% 11% 9% 2 57

Staffing 0% 85% 10% 5% 2 59

Equipment 0% 86% 7% 7% 2 73

Facilities 0% 84% 8% 8% 2 76

Research Question: Are program resources in line with the guidelines of the relevant evidence-based 

program(s)? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-32

Administration for Community Living

Table B-32. Implementation Organization Question 28 — 

For each of the following, do the resources available for <PROGRAM NAME> meet program guidelines?

Resource

Not 

Adequate 

To Meet 

Guidelines 

1

Adequate 

To Meet 

Guidelines 

2

More 

Than 

Necessary 

To Meet 

Guidelines 

3

There Are 

No 

Guidelines 

4 Median N

Funding 0% 73% 18% 8% 2 49

Staffing 0% 73% 23% 4% 2 52

Equipment 0% 77% 19% 4% 2 57

Facilities 0% 73% 24% 3% 2 59

There was a significant difference in the distribution of responses for urban compared with rural I/Os on 

the item of facilities. I/Os located in rural areas more frequently described facilities as more than 

necessary (43 percent) compared to those in urban areas, (19 percent) (p<0.1) as shown in Table 2-33. 

Table B-33. Significant Differences by Type of Implementation Organization, Question 28 — 

For each of the following, do the resources available for <PROGRAM NAME> meet program guidelines?

Urban Implementation Organizations Rural Implementation Organizations

Resource 1 2 3 1 2 3

Funding 18% 71% 12% 20% 53% 27%

Staffing 17% 67% 17% 7% 60% 33%

Equipment 0% 86% 14% 7% 60% 33%

Facilities* 0% 81% 19% 0% 57% 43%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses 

by I/Os located in urban areas compared to rural areas for this resource. 

Question text: For each of the following, do the resources available for <PROGRAM NAME> meet program 

guidelines: Please indicate for each of the factors whether resources are: 1 = Not adequate to meet guidelines, 2 = 

Adequate to meet guidelines, 3 = More than necessary to meet guidelines, or 4 = There are no guidelines.

Note: A Mann-Whitney U test was used to assess differences in the distribution of responses between I/Os located in 

urban and rural areas for each resource. Across all resources, N ranges from 34 to 37 for urban I/Os and ranges from 

14 to 15 for rural implementation organizations. The category “4 = There are no guidelines” was omitted from this 

analysis as it does not contribute any information on if the resources allocated typically meet program guidelines. 

Percentages within each location of implementation organization may not sum to 100 within each row due to 

rounding.

Cost challenges: When individual elements of cost were examined, the results were similar, with most 

rated as no challenge, or moderate challenge. The item most frequently rated an extreme challenge was 

the same for both types of respondents, with nearly one third of grantees (32 percent) and I/Os (29 

percent), reporting an extreme challenge providing staffing at recommended levels, as shown in Table 

2-34 and Table 2-35 respectively. A small percentage of respondents, ranging from 0 to 6 percent, 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-33

Administration for Community Living

indicated there were no guidelines for various items. There was no statistically significant difference in 

distribution of responses between Title III-D and discretionary grantees. 

Table B-34. Grantee Question 31 — 

How much of a challenge is it to meet program resource guidelines or requirements for the

following because of COST?

Guideline/Requirement

No 

Challenge 

1

Moderate 

Challenge 

2

Extreme 

Challenge 

3

No 

Guidelines 

4

Median N

Program licensing 30% 56% 14% 0% 2 79

Participant materials 28% 59% 13% 1% 2 80

Recommended staffing level 16% 49% 32% 3% 2 79

Recommended staffing credentials 28% 56% 14% 3% 2 79

Recommended minimum space and/or 

accessibility 51% 41% 1% 6% 1 80

Recommended general equipment (chairs, 

tables, weights) 51% 40% 4% 5% 1 80

Recommended program-specific equipment or 

supplies 31% 61% 4% 4% 2 80

Recommended administrative personnel or 

equipment (e.g. data entry systems) 29% 56% 13% 3% 2 79

Other 65% 22% 4% 9% 1 23
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-34

Administration for Community Living

Table B-35. Implementation Organization Question 29 — 

How much of a challenge is it to meet program resource guidelines or requirements for the following 

because of COST?

Guideline/Requirement

No 

Challenge 

1

Moderate 

Challenge 

2

Extreme 

Challenge 

3

No 

Guidelines 

4 Median N

Program licensing 52% 41% 3% 3% 1 58

Participant materials 43% 47% 9% 2% 2 58

Recommended staffing level 24% 45% 29% 2% 2 58

Recommended staffing credentials 45% 40% 14% 2% 2 58

Recommended minimum space and/or 

accessibility 57% 38% 5% 0% 1 58

Recommended general equipment (chairs, 

tables, weights) 64% 34% 0% 2% 1 58

Recommended program-specific equipment or 

supplies 57% 38% 3% 2% 1 58

Recommended administrative personnel or 

equipment (e.g. data entry systems) 36% 48% 12% 3% 2 58

Other 64% 21% 7% 7% 1 14

There were significant differences in the distribution of responses by urban and rural I/Os regarding the 

challenge of attaining recommended staffing levels (p<0.05) and staffing credentials (p<0.05), as shown 

in Table 2-36. Nearly half (47 percent) of rural I/Os reported an extreme challenge attaining 

recommended staffing levels, and 27 percent reported an extreme challenge providing staff with 

appropriate credentials, compared with only 19 and 8 percent of urban I/Os for the same items. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-35

Administration for Community Living

Table B-36. Significant Differences in Responses by Type of 

Implementation Organization, Question 29 — 

How much of a challenge is it to meet program resource guidelines or requirements

for the following because of COST?

Urban Implementation 

Organizations 

Rural Implementation 

Organizations 

Guideline/Requirement 1 2 3 1 2 3

Program licensing 61% 36% 3% 43% 50% 7%

Participant materials 56% 31% 14% 20% 80% 0%

Recommended staffing level** 31% 50% 19% 13% 40% 47%

Recommended staffing credentials** 56% 36% 8% 27% 47% 27%

Recommended minimum space and/or 

accessibility 59% 35% 5% 60% 33% 7%

Recommended general equipment 

(chairs, tables, weights) 67% 33% 0% 60% 40% 0%

Recommended program-specific 

equipment or supplies 64% 31% 6% 40% 60% 0%

Recommended administrative 

personnel or equipment (e.g. data 

entry systems)

37% 46% 17% 33% 60% 7%

Other 67% 22% 11% 100% 0% 0%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of 

responses by I/Os located in urban areas compared to rural areas for this guideline/requirement.

Question text: How much of a challenge is it to meet program resource guidelines or requirements for the 

following because of COST: Please select among the following: 1 = No Challenge, 2 = Moderate Challenge, 3 = 

Extreme Challenge, 4 = There are no guidelines.

Note: A Mann-Whitney U test was used to assess differences in the distribution of responses between I/Os 

located in urban and rural areas for each guideline/requirement. Across all guidelines/requirements, N ranges 

from 9 to 37 for urban I/Os and ranges from 2 to 15 for rural implementation organizations. The category “4 = 

There are no guidelines” was omitted from this analysis as it does not contribute any information on the 

challenge to meet program guidelines or requirements due to cost. Percentages within each location of 

implementation organization may not sum to 100 within each row due to rounding.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-36

Administration for Community Living

Challenges not related to cost: Staffing was also the most challenging non-cost challenge, identified as 

an extreme challenge by 23 percent of grantees and 20 percent of I/Os as shown below in Table 2-37.

and Table 2-38, respectively. 

Table B-37. Grantee Question 32 — How much of a challenge is it to meet program resource guidelines or 

requirements for the following NON-COST reasons? 

Guideline/Requirement

No 

Challenge 

1

Moderate 

Challenge 

2

Extreme 

Challenge 

3

There Are 

No 

Guidelines 

4 Median N

Program licensing 54% 42% 1% 3% 1 78

Participant materials 62% 31% 5% 1% 1 77

Recommended staffing level 21% 52% 23% 4% 2 77

Recommended staffing credentials 35% 48% 14% 3% 2 77

Recommended minimum space and/or 

accessibility 59% 33% 4% 4% 1 78

Recommended general equipment (chairs, 

tables, weights) 63% 35% 0% 3% 1 78

Recommended program-specific equipment 

or supplies 54% 42% 1% 3% 1 78

Recommended administrative personnel or 

equipment (e.g. data entry systems) 36% 53% 9% 3% 2 78

Other 58% 21% 4% 17% 1 24
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-37

Administration for Community Living

Table B-38. Implementation Organization Question 30 — 

How much of a challenge is it to meet program resource guidelines or requirements for the following 

NON-COST reasons?

Guideline/Requirement

No 

Challenge 

1

Moderate 

Challenge 

2

Extreme 

Challenge 

3

No 

Guidelines 

4 Median N

Program Licensing 73% 25% 2% 0% 1 55

Participant materials 64% 25% 8% 4% 1 53

Recommended staffing level 28% 50% 20% 2% 2 54

Recommended staffing credentials 51% 34% 15% 0% 1 53

Recommended minimum space and/or 

accessibility 69% 29% 2% 0% 1 52

Recommended general equipment (chairs, 

tables, weights) 75% 23% 0% 2% 1 52

Recommended program-specific equipment 

or supplies 73% 21% 4% 2% 1 52

Recommended administrative personnel or 

equipment (e.g. data entry systems) 57% 30% 11% 2% 1 53

Other 85% 8% 8% 0% 1 13

Among grantees, the only item with a significant difference in the distribution of responses between 

Title III-D and discretionary grantees was that presented by program licensing, (p<0.1), with fewer 

discretionary grantees rating this no challenge (67 percent) compared to Title III-D grantees (45 

percent). 

However, there were significant differences in the distribution of responses to two items between urban 

and rural implementation organizations, and the results are presented in Table 2-39. While only 11 

percent of I/Os located in urban areas rated finding staff with recommended credentials an extreme 

challenge, that was true for 29 percent of those located in rural areas. While 82 percent of urban I/Os 

reported no challenge to providing participant materials per guidelines, that was true of only 42 percent 

of rural organizations, and more of them, 57 percent, identified that as a moderate challenge. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-38

Administration for Community Living

Table B-39. Statistical Differences in Responses by

Type of Implementation Organization, Question 30 — 

How much of a challenge is it to meet program resource guidelines or requirements

for the following NON-COST reasons?

Urban Implementation 

Organizations 

Rural Implementation 

Organizations 

Guideline/Requirement 1 2 3 1 2 3

Program Licensing 83% 17% 0% 64% 36% 0%

Participant materials** 82% 9% 9% 43% 57% 0%

Recommended staffing level 35% 50% 15% 21% 50% 29%

Recommended staffing credentials*** 66% 23% 11% 21% 50% 29%

Recommended minimum space and/or 

accessibility 68% 29% 3% 79% 21% 0%

Recommended general equipment (chairs, tables, 

weights) 79% 21% 0% 79% 21% 0%

Recommended program-specific equipment or 

supplies 79% 18% 3% 71% 21% 7%

Recommended administrative personnel or 

equipment (e.g. data entry systems) 56% 29% 15% 64% 29% 7%

Other 78% 11% 11% 100% 0% 0%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses 

by I/Os located in urban areas compared to rural areas for this guideline/requirement. 

Question text: How much of a challenge is it to meet program resource guidelines or requirements for the following 

NON-COST reasons: Please select among the following: 1 = No Challenge, 2 = Moderate Challenge, 3 = Extreme 

Challenge, 4 = There are no guidelines.

Note: A Mann-Whitney U test was used to assess differences in the distribution of responses between I/Os located in 

urban and rural areas for each guideline/requirement. Across all program guidelines/requirements, N ranges from 9 

to 36 for urban I/Os and ranges from 3 to 14 for rural implementation organizations. The category “4 = There are no 

guidelines” was omitted from this analysis as it does not contribute any information on the challenge to meet 

program guidelines or requirements due to non-cost reasons. Percentages within each location of implementation 

organization may not sum to 100 within each row due to rounding.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-39

Administration for Community Living

Self-report of compliance with resource guidelines: Despite challenges, three-fourths of grantees (73 

percent) reported they always followed the resource guidelines in EBPs, and nearly all I/Os (98%) did 

likewise, as shown in Table 2-40. 

Table B-40. Grantee Question 51, Implementation Organization Question 48 — 

How often are the resource allocations (e.g., appropriately credentialed staff, equipment, locations) 

prescribed by the guidelines of the EBP followed? 

Frequency Followed Grantee I/O

Always 73% 98%

Usually 9% 2%

Sometimes 5% 0%

Never 1% 0%

No relevant program guidelines 0% 0%

We do not track this 12% 0%

Grantee N = 82. Implementation organization N = 58. Columns total 100%.

Fidelity in Target Population

In order to establish the evidence base for social programs, program developers test outcomes in 

specified populations. These target populations may be defined very generally, such as “older adults,” or 

more precisely, such as “persons with arthritis or other chronic condition.” While use in a tested target 

population is an important element of fidelity to program design, at the same time the entities providing 

programming to the populations served by ACL are often working in public community settings that 

serve a range of populations. It may not be feasible or even desirable to limit who can participate in 

these publicly funded opportunities for improving health. The need to balance these competing concerns 

is reflected in survey responses. 

Limitations on participation: All respondents were fairly evenly divided between those that strictly 

limited participation to the target populations and those that did not, as shown in Table 2-41. 

Research Question: Are programs being implemented with the intended populations? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-40

Administration for Community Living

Table B-41. Grantee Question 34, Implementation Organization Question 32 — 

Do you permit participants who are not strictly within the target population to participate in the program?

Permit Non-Target Populations Grantee I/O

Yes 56% 59%

No 44% 41%

Grantee N = 80. Implementation Organization N = 58. Columns total 100%.

However, there was a significant difference in responses for Title III-D and discretionary grantees 

(p<0.01) as shown in Table 2-42, where discretionary grantees were more likely to admit participants 

not technically within the target population (76 percent compared to 40 percent for Title III-D grantees). 

There was no difference in responses between urban and rural implementation organizations, with the 

majority (61 percent of urban respondents and 67 percent of rural respondents), permitting participants 

not strictly within target populations. 

Table B-42. Significant Differences by Type of Grantee, Question 34 — 

Do you permit participants who are not strictly within the target population to 

participate in the program?

Permit Non-Target Populations Title III-D Grantees Discretionary Grantees

Yes 40% 76%

No 60% 24%

Note: A two-sample z-test was used to assess the existence of a relationship between grantee type and responses to 

this question. The result was statistically significant at p < 0.01 (p = 0.00). N = 43 for Title III-D grantees and N = 

37 for discretionary grantees. Percentages within each grantee type may not sum to 100 due to rounding.

Verification of population: Respondents generally trusted participants to identify whether they were 

members of the target population as shown in Table 2-43.

Table B-43. Grantee Question 38, Implementation Organization Question 36 — 

How do you verify that potential participants fall in the target population? (Check all that apply)

Verification Method Grantee I/O

Referral from trusted organization 36% 36%

Self-identification 86% 97%

Require some form of documentation 23% 12%

Other (please specify) 2% 0%

Grantee N = 44. Implementation Organization N = 33. Percentages may not total 100 as users can select multiple 

responses. 

Self-reporting of compliance with target population guidelines: Respondents indicated that the 

majority of participants were within target populations, as shown in Table 2-44. There were no 

significant differences among grantee or I/O responses.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-41

Administration for Community Living

Table B-44. Grantee Question 35, Implementation Organization Question 33 — 

If enrollment is not restricted to targeted populations, approximately what percentage of the program 

population is comprised of the intended populations? 

Percent Intended Population Grantee I/O

75 to 100% 64% 71%

50 to 74% 14% 15%

25 to 49% 5% 3%

Under 25% 18% 12%

Grantee N = 44. Implementation Organization N = 34. Columns total 100%.

Modficiations to target populations: Respondents who admitted to permitting people outside the target 

population to participate in programing were asked to provide further information about what 

modifications they had made and why. The majority of respondents indicated they had expanded their 

programs to include companions or caretakers, people with other health conditions, or other age groups, 

as shown in Table 2-45. 

Table B-45. Grantee Question 36, Implementation Organization Question 34 — 

What adaptations, if any, have been made with respect to the population served? (Check all that apply)

Adaptation Type Grantee I/O

Expanded to populations with different health conditions 68% 60%

Expanded to different ages 68% 50%

Expanded to include companions/caretakers 78% 80%

Other (please specify) 5% 0%

Grantee N = 40. Implementation organization N = 30. Percentages may not total 100 as users can select multiple 

responses.

Responses differed significantly based on grantee type, with discretionary grantees far more likely to 

expand populations to include different ages (88 percent) than Title III-D grantees (38 percent) (p<0.01), 

as shown in Table 2-46. Among implementation organizations, those located in rural areas were more 

likely to expand to populations with different health conditions, doing so in 89 percent versus 53 percent 

for those located in urban areas (p<0.1). 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-42

Administration for Community Living

Table B-46. Statistical Tests for Differences in Response by Type of Grantee, Question 36 — 

What adaptations, if any, have been made with respect to the population served? (Check all that apply)

Adaptation Type Title III-D Grantees Discretionary Grantees

Expanded to populations with different health conditions 69% 67%

Expanded to different ages*** 38% 88%

Expanded to include companions/caretakers 88% 71%

Other (please specify) 6% 4%

* p<0.1, ** p<0.05, *** p<0.01. Each represents a statistically significant difference in the distribution of responses 

by Title III-D grantees compared to discretionary grantees for this adaptation type. 

Note: A two-sample z-test was used to assess the existence of a relationship between grantee type and each response 

to this question. N = 16 for Title III-D grantees and N = 24 for discretionary grantees. Percentages may not total 100 

as users could select multiple responses.

Respondents were asked whether they had encountered any of several potential reasons for permitting 

additional people to participate in the program beyond the target population. The majority of both types 

of respondents indicated they had identified other populations that would benefit from the program (67 

percent of grantees, 76 percent of implementation organizations) and/or had allowed others to join on 

request (74 percent of grantees, 72 percent of implementation organizations). The results are presented 

in Table 2-47. There were no statistically significant differences by type of grantee or location of 

implementation organization. 

Table B-47. Grantee Question 37, Implementation Organization Question 35 — 

Why has the population been enlarged beyond the target population defined by the program 

developer/administrator? (Check all that apply)

Reason Enlarged Grantee I/O

Target population not large enough to sustain program 33% 28%

Funding sources other than ACL require inclusion of other populations 26% 7%

Identified other populations that would benefit from the program 67% 76%

At the request of individuals or groups outside the target population 74% 72%

Other (please specify) 15% 14%

Grantee N = 39. Implementation organization N = 29. Percentages may not total 100 as users can select multiple 

responses. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-43

Administration for Community Living

Fidelity in Content

Program content includes several domains, which for this survey included providing materials 

prescribed by guidelines, the frequency, and length of sessions, the allocation of resources, and group 

size depending on the program. Grantees, who frequently provide programming through partners or 

subcontractors were asked to describe how they verified that specific sites were implementing EBPs 

with fidelity to these aspects of program models. The majority of grantees (65 percent) required that 

sites report data such as checklists or attendance counts, and more than half (57 percent) conducted site 

visits to monitor fidelity, as shown in Table 2-48. 

Table B-48. Grantee Question 40 — 

How do you verify that sites implementing EBPs are following EBP guidelines with respect to 

content and content delivery? (Check all that apply)

Verification Method Grantee

Sites attest that they are observing EBP guidelines faithfully 48%

Sites report data on program fidelity (e.g. checklists, attendance counts) 65%

We hold regular meetings with sites to discuss implementation fidelity 38%

We perform site visits to monitor fidelity of program delivery 57%

Other (please specify) 9%

N = 81. Percentages may not total 100 as users can select multiple responses.

Most of the grantees who selected the “other” option and provided additional detail were Title III-D 

grantees. Some indicated that they monitored all EBPs, others that this was done by other state agencies 

or subcontractors, and others that they did not have the staff needed to monitor these items. 

Index of Fidelity

HSAG calculated an index of fidelity to summarize the degree to which respondents were implementing 

programs with fidelity, based on their self-reported actions. The questions that make up the Fidelity 

Index are presented in Table 2-49. 

Research Question: Is the content provided completely and properly per the guidelines of the 

relevant evidence-based program(s)? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-44

Administration for Community Living

Table B-49. Questions Comprising Fidelity Index

Survey and 

Question# Question Text

G 41 

I/O 38

How often are all key components of the program content provided per the guidelines of the 

EBP? 

G 43

I/O 40

How often are materials prescribed by guidelines (e.g., exercise bands, handouts, web 

pages, videos) provided to participants according to the EBP guidelines? 

G 45

I/O 42

How often is the content delivery mode (e.g., in-person vs. remote, lecture, discussion) 

prescribed by the guidelines of the EBP followed? 

G 47

I/O 44

How often is the frequency of sessions prescribed by the guidelines of the EBP followed?

G 49

I/O 46

How often is the length of sessions prescribed by the guidelines of the EBP followed? 

G 51

I/O 48

How often are the resource allocations prescribed by the guidelines of the EBP followed? 

G 53

I/O 50

How often is the group/session size within the limits (minimum and maximum) prescribed 

by the guidelines of the EBP? 

Note: G = grantee survey, I/O = implementation organization survey.

Possible responses to these items are “Always,” “Usually,” “Sometimes,” “Never,” “No relevant 

program guidelines,” and “We do not track this.” To create the index score for each respondent, one 

point was added to the index for each “always” response to these items, for a maximum of 7 points. The 

results were adjusted for missing responses, as described more fully in Appendix A: Methodology. 

Grantee respondents said they “Always” follow guidelines when implementing EBPs in 5.3 of 7 areas 

on average. Discretionary grantee respondents reported always following slightly more guidelines (5.5) 

compared to SUAs (5.1), but this difference was not statistically significant. The median number of 

“Always” responses was 6.0 across both groups of grantee respondents; that is, at least half of grantee 

respondents reported they always follow guidelines in at least 6 of the 7 areas. These results are 

presented in Table 2-50. 

Table B-50. Index of Fidelity by Grantee Type

Index of Fidelity

Grantee Type Mean Median N

Discretionary 5.5 6.0 37

SUA 5.1 6.0 36

All Grantees 5.3 6.0 73

Note: A t-test was used to assess the existence of a relationship between grantee type and the fidelity index. The result was 

not statistically significant. Higher index values indicate greater fidelity to EBP guidelines.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-45

Administration for Community Living

Respondents representing rural I/Os said that they “Always” followed guidelines when implementing a 

specific EBP in 5.9 of 7 areas on average. Rural implementation organization respondents reported 

always following slightly fewer guidelines (5.8) compared to urban I/Os (6.0), but this difference was 

not statistically significant. The median number of always responses was also slightly lower for rural 

compared to urban I/Os (5.6 vs. 6.0), indicating that while half or more of the urban respondents said 

they always followed guidelines, less than half of urban respondents said they always followed that 

many guidelines. The results are presented in Table 2-51.

Table B-51. Index of Fidelity by Location of Implementation Organization

Index of Fidelity

Location of Implementation 

Organization Mean Median N 

Rural 5.8 5.6 15

Urban 6.0 6.0 35

All Implementation Organizations 5.9 6.0 50

Note: A t-test was used to assess the existence of a relationship between location of the implementation organization and the 

fidelity index. The result was not statistically significant. I/Os where location was unknown (N = 7) were not included in the 

statistical test. Higher index values indicate greater fidelity to EBP guidelines.

Description of Past Adaptations Made

The seven questions that made up the fidelity index were paired with questions that asked respondents to 

describe what they changed when they felt they had to change some element of content to address the 

next of ACL’s knowledge objectives. 

A relatively small number of respondents, (15 grantees and 6 implementation organizations) described 

making some changes to program content. Of these, nearly two thirds (67 percent for all respondents) 

modified some original content, as illustrated in Table 2-52. 

Knowledge Objective: Identification of intentional and unintentional adaptations 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-46

Administration for Community Living

Table B-52. Grantee Question 42, Implementation Organization Question 39 — 

Please describe the kind of adaptations made to components of the program content. (Check all that apply)

Adaptation Type Grantee I/O

Added new content 13% 17%

Dropped some original content 7% 0%

Modified some original content 67% 67%

Other (please specify) 20% 17%

Grantee N = 15. Implementation Organization N = 6. Percentages may not total 100 as users can select multiple 

responses. 

Some respondents (13 grantees and 6 implementation organizations) described adaptations they had 

made to program materials. About half of grantees (46 percent) modified existing materials. Half of 

implantation organizations (50 percent, 3 of 6 implementation organizations) described adding new 

materials. The results are presented in Table 2-53. 

Table B-53. Grantee Question 44, Implementation Organization Question 41 — 

Please describe the kind of adaptations made to program materials. (Check all that apply) 

Adaptation Type Grantee I/O

Added new materials 23% 50%

Dropped some original materials 15% 17%

Modified some original materials 46% 33%

Other (please specify) 31% 0%

Grantee N = 13. Implementation Organization N = 6. Percentages may not total 100 as users can select multiple 

responses. 

Thirteen grantees and six I/Os described adaptations they had made to content delivery. Sixty two 

percent of grantees changed the extent of remote content delivery. Eighty three percent of 

implementation oraganizations changed the extent of group discussions. Both of these changes are 

consistent with the responses necessitated by the COVID-19 PHE, and are presented in Table 2-54. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-47

Administration for Community Living

Table B-54. Grantee Question 46, Implementation Organization Question 43 — 

Please describe the kind of adaptations made to content delivery mode. (Check all that apply)

Adaptation Type Grantee I/O

Increased/decreased extent of remote content delivery 62% 17%

Increased/decreased extent of group discussions 23% 83%

Increased/decreased extent of group exercise 31% 33%

Other (please specify) 23% 17%

Grantee N = 13. Implementation Organization N = 6. Percentages may not total 100 as users can select multiple 

responses. 

Nine grantees and three I/Os described making adaptations to the frequency of sessions, with the 

majority decreasing the frequency. This item was selected by 78 percent of this small group of grantees 

and 100 percent of implementation organizations, as presented in Table 2-55. 

Table B-55. Grantee Question 48, Implementation Organization Question 45 — 

Please describe the kind of adaptations made to the frequency of sessions. (Check all that apply)

Adaptation Type Grantee I/O

More frequent 44% 0%

Less frequent 78% 100%

Other (please specify) 11% 0%

Grantee N = 9. Implementation Organization N = 3. Percentages may not total 100 as users can select multiple 

responses. 

Twelve grantees and thirteen I/Os described making adaptations to the length of sessions, with the 

majority of both (83 percent and 85 percent respectively) shortening sessions, as shown in in Table 2-56. 

Table B-56. Grantee Question 50, Implementation Organization Question 47 — 

Please describe the kind of adaptations made to the length of sessions. (Check all that apply)

Adaptation Type Grantee I/O

Longer 42% 54%

Shorter 83% 85%

Other (please specify) 8% 0%

Grantee N = 12. Implementation Organization N = 13. Percentages may not total 100 as users can select multiple 

responses. 

Eleven grantees and a single implementation organization described adaptations to resource allocations. 

The grantee results indicate a reluctance to relax staff credentialling or certification requirements, but 

some need to permit fewer staff per participant or to allow substitution of different equipment. The 

implementation organization’s adaptations were also in the staff ratio and equipment substitution areas, 

as presented in Table 2-57. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-48

Administration for Community Living

Table B-57. Grantee Question 52, Implementation Organization Question 49 — 

Please describe the kinds of adaptations made to resource allocations (e.g. staff, equipment, locations). 

(Check all that apply)

Adaptation Type Grantee I/O

Fewer staff per participant 36% 100%

Relaxation of staff credential/training requirements 18% 0%

Substitutions/omission of certain types of equipment 36% 100%

Other (please specify) 27% 0%

Grantee N = 11. Implementation Organization N = 1. Percentages may not total 100 as users can select multiple 

responses. 

A larger group of grantees (41) and I/Os (24) indicated they had made adaptations to group or session 

size. For both types of respondents, smaller groups smaller than the prescribed minimum had been 

permitted by over 80 percent of respondents (85 percent of grantees and 88 percent of implementation 

organizations). At the same time, nearly half of respondents (46 percent) and one third of I/Os (33 

percent) had also permitted larger group than the maximum prescribed in program materials. These 

results are presented in Table 2-58. 

Table B-58. Grantee Question 54, Implementation Organization Question 51 — 

Please describe the kinds of adaptations made to group/session size. (Check all that apply)

Adaptation Type Grantee I/O

Permit group/session smaller than prescribed minimum 85% 88%

Permit group/session larger than prescribed maximum 46% 33%

Other (please specify) 7% 4%

Grantee N = 41. Implementation Organization N = 24. Percentages may not total 100 as users can select multiple 

responses. 

First, the majority of respondents indicated that they did not make any adaptations, and would not have 

been given these questions in the survey. These adaptations were very much the exception, rather than 

the rule. The changes are frequently consistent with responding to the COVID pandemic, but at the same 

time, may be just as attributable to other constraints, such as resource issues. 

The theme across all types of adaptations described by respondents is that when an item was changed, it 

was more often reduced resulting in less frequent sessions, shorter or fewer sessions, and fewer staff per 

participant. Each of these has the potential to reduce fidelity. It is unclear how different these results 

might have been prior to the COVID-19 PHE, since most are consistent with the need to limit exposure 

of individuals by reducing group sizes and moving to remote sessions whenever possible. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-49

Administration for Community Living

Program Adaptations

Adaptations not related to the COVID-19 PHE 

Grantees were asked to generalize across all EBPs they were currently delivering using funds from 

ACL, while I/Os were asked to consider the specific program that was the impetus for their invitation to 

take the survey. This difference in focus must be kept in mind when interpreting and comparing their 

results. 

Grantees were asked what directions they give their subcontractors or partners on adaptations, and only 

11 percent indicated they had not provided at least some guidance on adaptations. About half (52 

percent) had brought up the subject proactively, and over one third (37 percent) had responded to 

requests for guidance as shown in Table 2-59. 

Table B-59. Grantee Question 55 — 

When do you provide guidance to subcontractors/partners on adaptations?

Guidance Timing Grantee

Proactively give guidance on adaptations to all subcontractors/partners 52%

Provide guidance in response to specific requests for advice from subcontractors/partners 37%

No guidance provided 11%

N = 71.

Only 20 percent of grantees and 31 percent of I/Os indicated they had not considered making 

adaptations, as shown in Table 2-60. Respondents considered a broad range of factors in determining 

whether adaptations were necessary, with several options selected by 50 percent or more of grantees. 

Implementation organization results seemed to be approximately 20 percentage points lower than 

grantees, with most between 26 and 31 percent. A relatively few respondents selected the “other” option, 

and fewer identified additional considerations. One added text explaining they also took into 

consideration the need to build new programs and would consider allowing classes to proceed with less 

than the minimum recommended number of participants to develop interest in new programs. 

Research Question: How and for what reasons do grantees work with the developers of the 

evidence-based program(s) to identify acceptable program adaptations that maintain the integrity 

of the evidence-based nature of the program? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-50

Administration for Community Living

Table B-60. Grantee Question 56 — 

What factors do you consider when determining whether adaptations to EBPs are warranted?

(Check all that apply)5-5

Factor Considered Grantee I/O

Funding constraints 32% 19%

Availability of staff needed for guideline adherence 48% 34%

Availability of other resources needed for guideline adherence (e.g., facilities, 

equipment) 39% 26%

Accommodations for accessibility (disabilities, etc.) 50% 29%

Accommodations for lower-income and rural participants (transportation, hours, etc.) 51% 31%

Cultural inclusivity, including language and religion 55% 26%

To increase appeal to local populations 37% 29%

Other (please specify) 7% 3%

Have not considered adaptations 20% 31%

Grantee N = 82. Implementation organization N = 58. Percentages may not total 100 as users can select multiple 

responses. 

No significant difference in the distribution of responses to this question was identified between Title 

III-D and discretionary grantees. The single significant difference that appeared among I/Os related to 

accomodations considered for lower-income and rural participants. This was an identified as an issue by 

42 percent of urban I/Os and only 13 percent of rural I/Os (p<0.1). Urban I/Os may face a greater 

diversity of circumstances that require making adaptations or may be more likely than rural I/Os to 

consider making accommodations for issues such as difficulties accessing services due to lack of 

transportation or long distances between the sites where programs are offered and the residences of 

eligible populations. 

For all respondents, the most widely selected source of information on adaptations was program 

developers/administrators, with 76 percent of grantees and 53 percent of I/Os considering their input 

important in deciding whether and how to make adaptations. Again, grantees reported a higher 

frequency of considering all of the different input sources presented in the question, as shown in Table 

2-61.

5-5 Implementation Organization Question 52 — What factors does your organization consider when determining the need 

for adaptations to <PROGRAM NAME>?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-51

Administration for Community Living

Table B-61. Grantee Question 57, Implementation Organization Question 53 — 

Whose input is important in determining whether adaptations are warranted and how they should be made? 

(Check all that apply)

Input Source Grantee I/O

Program developer/administrator 76% 53%

Program participants 46% 38%

Partner organizations 52% 40%

Peer organizations 20% 13%

Local government, civic organizations, social, and/or religious organizations 19% 9%

State government 25% 13%

Funders 31% 24%

Other 2% 4%

Minimal/no external input 1% 7%

Do not make such determinations 11% 22%

Grantee N = 83. Implementation Organization N = 55. Percentages may not total 100 as users can select multiple 

responses. 

About half of grantees (53 percent) and more than one third of I/Os (38 percent) reported working with 

developers/administrators to identify acceptable adaptations, as shown in Table 2-62. 

Table B-62. Grantee Question 58, Implementation Organization Question 54 — 

Have you worked with program developer/administrators to identify acceptable program adaptations that 

maintain the integrity of the evidence-based nature of program?

Worked with Developers Grantee I/O

Yes 53% 38%

No 47% 62%

Grantee N = 83. Implementation organization N = 58. 

Respondents were asked about their work with program developers/administrators to identify acceptable 

program adaptations, and the majority had obtained advice when they requested it, as shown in Table 

2-63. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-52

Administration for Community Living

Table B-63. Grantee Question 59 — 

If you have received guidance from the program developer/administrator on adaptations, 

when was it provided?5-6

Guidance Timing Grantee I/O

Program developer/administrator proactively provided guidance on adaptations for 

programs 34% 41%

Provided in response to request for guidance 64% 59%

Do not provide guidance on adaptations 2% 0%

Grantee N = 44. Implementation Organization N = 22. 

Many respondents also reported having received guidance from an entity with oversight authority either 

regarding allowed adaptations or in response to requests for advice, as shown in Table 2-64.

Table B-64. Grantee Question 60 — When have you received guidance from other entities with oversight 

authority on adaptations?5-7

Guidance Timing Grantee I/O

General guidance on adaptations allowed/recommended 30% 28%

Guidance only in response to specific requests for advice 40% 23%

No guidance provided 29% 49%

Grantee N = 82. Implementation organization N = 53. 

In summary, most grantees and I/Os had considered making adaptations to their EBPs. They considered 

a variety of factors in making decisions about adaptations, turning predominantly to the 

developers/administrators, but also to peers and partners. Many had requested and received guidance on 

adaptations from program developers/administrators or other entities with oversight on their operations, 

and many had worked with developers/administrators to identify appropriate adaptation for their needs. 

Still, about one third of grantees and half of I/Os indicated they had not received any guidance about 

making adaptations from an entity with oversight over their operations. There may be an opportunity for 

ACL to step in and provide additional guidance on the topic. 

Adaptations Related to the COVID-19 PHE 

This project was conceived and initiated prior to the COVID-19 PHE. However, the PHE had multiple 

major impacts on how grantees and I/Os were able to fulfill their mission to provide EBP programming 

5-6 Implementation organization Question 55 — If you have received guidance on adaptations for <PROGRAM NAME> 

from the program developer/administrator, when was it provided? 5-7 Implementation Organization Question 56 — If you have received guidance on adaptions from the source of your ACL 

funding, <GRANTEE NAME>, when was it provided?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-53

Administration for Community Living

to promote health. ACL was able to leverage the design and execution of the survey and analysis of 

results to gain insight into those impacts to provide insight into the impact and response to the pandemic. 

Focus on grantees: Grantees were asked how much of their pre-COVID-19 PHE programming they 

were able to continue through the PHE, and results showed a broad range of responses as presented in 

Table 2-65. There were no significant differences in distribution of responses by type of grantee. 

Table B-65. Grantee Question 61 — What proportion of your previous in-person ACL-funded EBPs for disease 

prevention and health promotion have you maintained during the COVID-19 PHE?

Percent EBPs Maintained Grantee

75 to 100% 23%

50 to 74% 14%

25 to 49% 25%

1 to 24% 27%

None 9%

Not Applicable (no in-person service delivered prior to COVID-19 PHE) 2%

N = 81.

For grantees, the predominant source for advice on adaptations during the PHE for I/Os was program 

developers/administrators, with many identifying ACL and NCOA as important sources of information 

as shown in Table 2-66. 

Table B-66. Grantee Question 63 — 

Whose input has been important in determining whether COVID-19-related adaptations 

are warranted and how they should be made? (Check all that apply)

Input Source Grantee

Program developer/administrators 86%

ACL 64%

NCOA 59%

Peer organizations 26%

Participants and other stakeholders 43%

CDC and other public health agencies 53%

Other (please specify) 4%

No guidance received 1%

N = 80. Percentages may not total 100 as users can select multiple responses.

Among grantees funded by mandatory grants under Title III-D, many (40 percent) indicated that some 

funds had been transferred by the state to COVID-related disaster relief activities which may have 

impacted the EBPs that are the subject of this study. The results are presented in Table 2-67. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-54

Administration for Community Living

Table B-67. Grantee Question 65 — 

Have Title III-D funds been transferred by the state to COVID-related disaster relief activities?

and/or other OAA Title III services?

Funds Transferred Grantee

Yes 40%

No 38%

Not sure 22%

N = 45.

Focus on implementation organizations: I/Os indicated that nearly 90 percent of their EBPs were 

delivered in person prior to the PHE, as shown in Table 2-68. There was no statistically significant 

difference between I/Os located in urban and rural areas. 

Table B-68. Implementation Organization Question 57 — 

Prior to the COVID-19 PHE, what proportion of <PROGRAM NAME> was provided in person?

Proportion in Person I/O

All in person 89%

Most in person, some remote 4%

Some in person, most remote 2%

All remote 0%

Not sure 5%

N = 56.

I/Os reported a pronounced shift to remote programming, as shown in Table 2-69. There was no 

statistically significant difference between I/Os located in urban and rural areas. 

Table B-69. Implementation Organization Question 58 — 

During the COVID-19 PHE, what proportion of <PROGRAM NAME> content was provided in-person?

Proportion in Person I/O

All in person 14%

Most in person, some remote 10%

Some in person, most remote 12%

All remote 48%

This program has been halted during the COVID-19 PHE 16%

N = 58.

The predominant source for advice on adaptations during the PHE for I/Os was program 

developers/administrators as shown in Table 2-70. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-55

Administration for Community Living

Table B-70. Implementation Organization Question 61 — 

Who provided guidance and other materials that helped you to adapt programs? 

successfully during the COVID-19 PHE? (Check all that apply)

Guidance Source I/O

Program developers 71%

NCOA 25%

Peer organizations 27%

Participants and other stakeholders 9%

CDC and other public health agencies 18%

Other 9%

No guidance received 7%

N = 55. Percentages may not total 100 as users can select multiple responses.

It appeared that funding for the specific EBP program asked about in the survey was not reduced for 

reasons other than diminished participant attendance, as shown in Table 2-71. 

Table B-71. Implementation Organization Question 62 — 

Has your <PROGRAM NAME> funding decreased during the COVID-19 PHE for reasons

other than decreased demand for services?

Funding Decrease I/O

Yes 13%

No 88%

N = 56.

Comparing Grantees and Implementation Organizations: All respondents were asked which 

program elements they had modified due to the PHE, and the most frequently selected component was 

delivery mode, with grantees changing 98 percent of programming and I/Os changing 80 percent as 

shown in Table 2-72. Group size was also frequently modified. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-56

Administration for Community Living

Table B-72. Grantee Question 62 — 

In your ACL-funded EBPs, which of the following program components have been modified 

in response to the COVID-19 PHE? (Check all that apply)5-8

Program Component Grantee I/O

Program Content 11% 4%

Materials (e.g., exercise bands, handouts, web pages, videos) 34% 29%

Delivery mode (e.g., in-person vs. remote, lecture, discussion) 98% 80%

Frequency of sessions 25% 13%

Duration of sessions 19% 7%

Resources allocated (ex: staff, equipment, locations, etc.) 38% 25%

Group/session size (including number of clients per facilitator) 68% 41%

Other (please specify) 4% 5%

Grantee N = 80. Implementation organization N = 56. Percentages may not total 100 as users can select multiple 

responses. 

There were differences in the frequency of response based on type of grantee for two items: Frequency 

of sessions (Title III-D grantees 35 percent, discretionary 14 percent, p<0.05); and Duration of sessions 

(Title III-D grantees 26 percent, discretionary grantees 11 percent, p<0.1). 

For I/Os, there was one significant difference in modifications in response to COVID-19, with 20 

percent of rural I/Os making modifications to duration of sessions, compared to only 3 percent of those 

located in urban areas. 

Respondents were asked about additional topics for which they sought guidance related to the PHE, and 

the results are presented in Table 2-73. The most frequently raised concerns were related to remote 

delivery, as would be expected. 

5-8 Implementation organization Question 59 — Which of the following program components have been modified in 

response to the COVID-19 PHE?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-57

Administration for Community Living

Table B-73. Grantee Question 64, Implementation Organization Question 60 — 

What topics, if any, have you sought guidance on related to the COVID-19 PHE? (Check all that apply)

Guidance Topic Grantee I/O

Safe in-person service delivery 50% 24%

How to redesign a program remotely that had been designed for in-person delivery 67% 48%

Additional training needed for remote delivery 71% 50%

How to ensure that clients practiced the program safely without direct oversight 44% 26%

How to reach clients and recruit them for remote delivery 66% 39%

Other (please specify) 2% 0%

Have not sought guidance on COVID-related adaptations 7% 15%

Grantee N = 82. Implementation organization N = 54. Percentages may not total 100 as users can select multiple 

responses. 

Program Support

Recipients of discretionary grants from ACL were asked about their experience completing several of 

the cooperative terms of the standard agreements included in the grant application process.B-9 Generally, 

the agreements require grant recipients to pay license and training fees, provide staff trained to gather 

and report basic data, and to communicate regularly with ACL. Grantees were asked to rate each item on 

a scale from 1-5, where 1 = very easy and 5 = very difficult. None of the items stood out as particularly 

difficult or more problematic than the others, with no more than 5 percent of discretionary grantees 

indicating that any of the items was very difficult as shown in Table 2-74. Three discretionary grantees 

identified a component not included among the items listed, and one in particular found the semi-annual 

reporting and required monitoring calls with ACL and NCOA to be very time intensive, rating that item 

a 5 for very difficult. There might be an opportunity to further simplify data collection requirements, as 

that was considered very easy by only 20 percent of respondents.

B-9 See, for example, ACL’s Notice of Funding Opportunity for 2018 Empowering Older Adults and Adults with 

Disabilities through Chronic Disease Self-Management Education Programs Financed by the Prevention and Public 

Health Fund HHS-2018-ACL-AOA-CSSG-0256.

Knowledge Objective: What are the strengths and weakness of the current award and 

implementation process?
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-58

Administration for Community Living

Table B-74. Grantee Question 66 — 

Please think about the process of implementing a discretionary grant from ACL. How would you rate the 

difficulty or ease of completing the following operational components?* 

Operational Component

Very 

Easy 

1 2 3 4

Very 

Difficult 

5 Median N

Complying with the data collection requirements, 

use of the required forms 20% 23% 30% 23% 5% 3 40

Maintaining staff properly trained in data entry 

procedures 28% 30% 20% 18% 5% 2 40

Sending two staff people yearly to professional 

development conferences 43% 28% 20% 8% 3% 2 40

Paying licensing and training fees 28% 43% 15% 10% 5% 2 40

Monthly communication with Project Officer 63% 23% 8% 8% 0% 1 40

Other (please specify) 0% 0% 67% 0% 33% 3 3

*This question was addressed to discretionary grantees only.

Grantees were generally satisfied with the level of support they had received from ACL on program 

fidelity. As might be expected, the responses to this question were significantly different among 

discretionary grantees, who have a contractual expectation of regular communication and support from 

ACL, compared with Title III-D grantees who have no such relationship. A combined total of 97 percent 

of discretionary grantees rated ACL support for fidelity as good or very good, as did 78 percent of Title 

III-D grantees. The results are presented in Table 2-75. 

Table B-75. Statistical Tests for Differences in Responses by Type of Grantee, Question 68 — 

What level of support have you received from ACL to help you ensure 

fidelity in program implementation?

Level of Support Title III-D Grantees Discretionary Grantees

Very good 15% 50%

Good 63% 47%

Poor 15% 0%

Very poor 7% 3%

Note: A Mann-Whitney U test was used to assess differences in the distribution of responses between Title III-D 

grantees and discretionary grantees. The result was statistically significant at p < 0.01 (p = 0.00). N = 41 for Title 

III-D grantees and N = 38 for discretionary grantees. Percentages within each grantee type may not sum to 100 due 

to rounding.
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-59

Administration for Community Living

Grantees were generally satisifed with current levels of communication with ACL regarding EBPs and 

fidelity, as shown in Table 2-76. For most of the items, more than 70 percent of grantees preferred 

keeping communicaitons at about the same level. The item that received the highest percentage of 

requests for “more” was the range of topics communicated, selected by 42 percent of grantees. Among 

the seven grantees that selected “other” there were requests for interactive webinars, regional 

workgroups with networking opportunities, and regular meetings with other SUAs or Title III-D 

contacts. 

Table B-76. Grantee Question 71 — 

Please select the options below that indicate your preferences regarding

communications with ACL regarding EBPs and fidelity.

Communication Aspect More Same Less Median N

Frequency of communication frequency 27% 72% 1% 2 81

Detailed communications 27% 71% 3% 2 79

Range of topics communicated 42% 54% 4% 2 81

Two-way communications 24% 71% 5% 2 80

Electronic communications 27% 71% 3% 2 79

Telephone communications 8% 81% 12% 2 77

Other (please specify) 43% 43% 14% 2 7

Respondents indicated a willingness to make use of help from ACL across a variety of subjects. Just 

under half of respondents (45 percent of grantees and 48 percent of implementation organizations) 

indicated that user friendly fidelity checklists would be the most helpful tool for their use in overcoming 

challenges to fidelity, as shown in Table 2-77. 

Research Question: What can ACL do to support and encourage the proper use and implementation 

of evidence-based programs? 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-60

Administration for Community Living

Table B-77. Grantee Question 14, Implementation Organization Question 15 — 

What materials or tools would most help your organization overcome fidelity challenges?

Fidelity Help Grantee

User friendly fidelity maintenance guidance 20%

Fidelity checklists 45%

Fidelity training courses for managers 19%

Trainer registry 11%

Other 5%

Grantee N = 84. Implementation Organization N = 60.

All respondents, but especially grantees, endorsed some options for improving the selection of available 

EBPs. The most frequently selected option was an increase in the number of EBPs that are easy to 

implement as designed, selected by 71 percent of grantees and 54 percent of implementation 

organizations. The results are presented in Table 2-78. Grantees also showed strong interest in 

increasing the number of EBPs that can be delivered remotely (66 percent of grantees), and those that 

have been tested with particular sub-populations such as members of racial and ethnic minority groups, 

or individuals with disabilities (61 percent of grantees). 

Table B-78. Grantee Question 67, Implementation Organization Question 63 — 

In what ways could the selection of available EBPs be improved? (Check all that apply)

Selection Improvement Grantee I/O

Update more frequently with newer programs 52% 38%

Increase the number of EBPs that are easy to implement as designed 71% 54%

Increase the number of EBPs that address other aspects of Falls or CDSME 49% 23%

Increase the number of EBPs in languages other than English 48% 21%

Increase the number of EBPs that have been tested with additional populations 

(members of racial and ethnic minority groups, individuals with disabilities, etc.) 61% 34%

Increase the number of EBPs that can be delivered remotely 66% 41%

Other (please specify) 14% 11%

None 6% 18%

Grantee N = 83. Implementation Organization N = 56. Percentages may not total 100 as users can select multiple 

responses. 

Several respondents offered additional requests in the text field provided. Grantees requested updates to 

relevant research, reducing costs and barriers to delivery of programs, more EBPs that address social 

determinants of health, increased technical support for establishing centers or coordinating hubs for 

EBPs to optimize shared funds, resources, and staffing, more physical activity EBPS, training for remote 

delivery of programs, and easier access to information. 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-61

Administration for Community Living

I/Os requested programs that have less arduous trainings, and identified a need for programs that deal 

with grief. There was a request for an EBP using yoga. One implementer described its ideal for 

programs that would meet their needs: Session length 90 minutes to 2 hours, only require one facilitator, 

facilitator training remain virtual as has been done during the pandemic and be shortened. 

Agreement was less pronounced, but still strong with approaching half or more of respondents 

requesting assistance with several aspects of fidelity, as shown in Table 2-79. I/Os less frequently 

indicated a desire for such assistance. 

Table B-79. Grantee Question 69, Implementation Organization Question 64 — 

What types of support would improve your organization’s ability to implement programs 

with fidelity and/or maintain fidelity over time? (Check all that apply)

Type of Support (Other) Grantee I/O

Assistance with selecting EBPs appropriate to your local context 27% 16%

Guidance on how to make local adaptations without seriously threatening fidelity 50% 36%

Guidance and tools for effective monitoring of program implementations 55% 24%

Guidance and tools for maintaining fidelity over time 49% 24%

Guidance on controlling costs while maintaining fidelity 44% 20%

Assistance with evaluating the evidence base for EBPs 30% 16%

Other types of technical assistance regarding EBP programs 18% 9%

Feedback on reports submitted by your organization 16% NA

Other 1% 0%

No additional support required 17% 42%

Grantee N = 82. Implementation organization N = 55. Percentages may not total 100 as users can select multiple 

responses. 

Both surveys provided an opportunity for respondents to describe needed support in a free text field in 

response to the following question: “If you believe that any additional support would improve your 

organization’s ability to implement programs with fidelity and/or maintain fidelity over time, please 

elaborate on your response.” Grantee Question 70, Implementation Organization Question 65. 

Twenty-six grantees and fourteen I/Os responded to this question. In addition to several requests for 

additional funding, respondents asked for: 

• A simple fidelity monitoring tool

• simple user-friendly guidelines with examples 

• Quality improvement tools for monitoring fidelity, leaders, and master trainers

• Updates to the evidence base for older EBPs

• Advice on recruiting participants, especially in rural areas 
APPENDIX B. TABLES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page B-62

Administration for Community Living

• Guidance on allowable program flexibilities

• Information sharing on best practices such as for tracking fidelity, expanding programs, methods for 

low-cost monitoring for leaders and program implementers

• A clear statement of ACL’s requirements for fidelity

• Clarification of the SUAs’ role in supervising fidelity

• More technical assistance regarding programs

• Issues presented by remote delivery that go beyond participation in programs to

– Remote training

– Facilitate payment for facilitators providing virtual sessions to broader population of participants
Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page C-1

Administration for Community Living 

C. Appendix C: Crosswalk of Research Questions and Knowledge Objectives

Table B-1 contains a crosswalk of the major research topics of this report to the research questions and 

knowledge objectives set out in the contract and used in framing the survey tools. The numbering for 

research questions and knowledge objectives follows the contract language, but is not strictly in order, 

since some knowledge objectives applied to numerous questions, or were ordered differently. 

Research Topic Research Question Knowledge Objective

I. Program selection: 

How do grantees select

appropriate evidence-based 

programs for their contexts 

(e.g., resources, populations)?

1. How do sites select appropriate 

evidence-based programs for their 

contexts (e.g., resources, 

populations)?

2. How grantees select the evidencebased programs that they implement 

or, for discretionary grantees, how 

they determine for which grants they 

will apply

II. Program fidelity:

What processes do ACL staff 

and grantees use to verify that 

grantees are implementing 

EBPs as designed? 

2. To what extent do staff 

overseeing and implementing the 

programs fully understand and 

comply with the implementation 

guidelines/instructions? For 

example:

1. The process ACL staff use when 

making awards (i.e., discretionary 

grants) or otherwise verifying that 

grantees (e.g., formula grantees) are 

using funds for evidence-based 

programing

a. Are facilitators trained in 

accordance with the guidelines of 

the relevant evidence-based 

program(s)? 

2. How grantees implement those 

programs and verify that they are 

being implemented with fidelity to 

their original models 

b. Are program resources in line 

with the guidelines of the relevant 

evidence-based program(s)?

2. How grantees implement those 

programs and verify that they are 

being implemented with fidelity to 

their original models 

c. Are programs being implemented 

with the intended populations?

2. How grantees implement those 

programs and verify that they are 

being implemented with fidelity to 

their original models
APPENDIX C: CROSSWALK OF RESEARCH QUESTIONS AND 

KNOWLEDGE OBJECTIVES

Final Report Appendices

Fidelity Evaluation of Evidence-Based Programs Final Report September 2022 Page C-2

Administration for Community Living

Research Topic Research Question Knowledge Objective

d. Is the amount of training and 

frequency sufficient?

2. How grantees implement those 

programs and verify that they are 

being implemented with fidelity to 

their original models

e. Is the content provided 

completely and properly per the 

guidelines of the relevant evidencebased program(s)?

2. How grantees implement those 

programs and verify that they are 

being implemented with fidelity to 

their original models

III. Program Adaptations: 

Recognizing that adaptations to 

EBPs are not recommended, are

there adaptations that 

grantees/subgrantees seek to 

make to evidence-based 

programs?

3. While not recommended, what 

kinds of adaptations, if any, are 

grantees/subgrantees making to the 

evidence-based programs? And, 

why?

5. How and for what reasons 

grantees work with the developers of 

the evidence-based program(s) to 

identify acceptable program 

adaptations that maintain the 

integrity of the evidence-based 

nature of the program 

2. [I]dentification of intentional and 

unintentional adaptations 

IV. Program Support:

What can ACL do to support 

and encourage the proper use 

and implementation of evidencebased programs?

4. What can ACL do to support and 

encourage the proper use and 

implementation of evidence-based 

programs?

4. The strengths and weakness of the 

current award and implementation 

process