Evidence Based Program Fidelity Survey

Site Survey

*[Note to ACL: Each survey recipient will receive a customized survey invitation e-mail that includes a survey link (URL) specific to them. This link will be indexed to information about the survey respondent, including their organization name and whether they are an SUA, grantee, or subcontractor/subgrantee/partner delivering EBP content to the public (i.e., implementation organization). For the latter, the URL will also link to information about the specific program that they have been selected to provide information about.*

*The survey will be web-based. Instructions about skip patterns are included in italics and brackets. Questions are sequentially numbered for ease of review, with a maximum of 65 questions. However, 13 of these are conditional on prior responses, and may be skipped by some implementation organizations depending on their status. The minimum number of questions any respondent might complete is 52.*

*We estimate it will take less than 40 minutes to complete the survey, but that will be revised depending on the final form and number of questions.]*

# General Instructions

**Thank you for participating in this important effort to collect data on behalf of the Administration for Community Living (ACL) on the ways in which ACL grantees, State Units on Aging (SUA), and subcontractors/partners ensure that evidence based programs (EBPs) are implemented and sustained with fidelity to the program developers’/administrators’ models. By fidelity, we mean the extent to which organizations take steps to be sure that their offered programs are implemented according to the directions of the program developer/administrator. This survey will take approximately 30-40 minutes to complete.**

**Some sections of the survey have specific instructions about how the questions should be interpreted and responded to. Please read these instructions in order to ensure that we are able to accurately capture your responses describing practices and experiences with regard to selecting, administering, and/or delivering EBPs with fidelity.**

# Section 1. General/Screening Items

## Instructions

**Your organization may be involved in delivering one or more EBPs to the public. For the purposes of this survey, please think only about <PROGRAM NAME> as you respond to questions.**

**In this section, please focus on your general practices prior to the Coronavirus Disease 2019 (COVID-19) Public Health Emergency (PHE), unless your organization began providing <PROGRAM NAME> during the PHE. In that case, please focus on your current practices.**

## Questions

Which of the following best describes your organization’s history of providing EBPs funded by ACL:

Offered prior to the PHE, but stopped due to the PHE

Offered prior to the PHE, and continued at least some programming during the PHE

Began offering services during or after the PHE

Unable to begin services due to PHE

Your organization is included in this survey because it is receiving grant funding from ACL to deliver <PROGRAM NAME>. Does your organization receive grant funding directly from ACL or is it a subcontractor/subgrantee/or partner to another organization that receives grant funds from ACL?

1. Receives funds directly from ACL

Does not receive funds directly from ACL 

Not sure

# Section 2. Selecting Evidence Based Programs

## Instructions

**We are interested in knowing how you select evidence-based programs. ACL requests that each question be answered by someone with a good working knowledge of the topic at hand to ensure that the information you supply is complete and accurate. For sections where someone else’s input would be most appropriate, please enlist their assistance.**

## Questions

Who has input in the selection of programs? (Check all that apply)

1. Funder dictates

Leadership determines which programs to select

One or more inhouse staff members oversee selection of programs 

Committee or multidisciplinary process seeks input from across the organization 

External partners, subject matter experts, or consultants

State or local government guidance (e.g., state units on aging)

Participation with peers and partners in planning or learning network

Recommendations/requests of public 

Other

What information sources do you consider in selecting appropriate EBPs for your community and organization? (Check all that apply)

1. Internal policies 

The Aging and Disability Evidence-Based Programs and Practices initiative (ADEPP), the National Council on Aging’s (NCOA) list of EBPs funded through the Older Americans Act Title III-D and approved through the Evidence-Based Programs Review Process, the Evidence-Based Leadership Collaborative (EBLC) program menu and/or some other evidence-based program registry or webpage

Funder direction

Past experience

Community needs/interests

Other

Please indicate how important the following **resource factors** are in the selection process:

*Please rate factors between 1 and 5, where 1 = not at all important and 5 = extremely important*

1. Licensing costs

Staffing required

Training costs 

Special equipment costs	

Availability of facility space

Participant recruitment costs

Data collection and reporting costs

Please indicate how important the following **program-specific** characteristics are in the selection process:

*Please rate factors between 1 and 5, where 1 = not at all important and 5 = extremely important*

1. Program design (e.g. discussion group, class structure, hands-on activities) 

Program content	 

Program flexibility/adaptability

Strength of the evidence base for the program (e.g., highest tier, strength of literature) 

Availability of training

Supports for implementation (e.g., contains instructions/materials for maintaining quality assurance, fidelity checklists)

Supports for dissemination (e.g., contains instructions/materials for reaching out to population,) 

Readiness for remote delivery (contains instructions, materials for on-line, telephonic programming)	 

Support/technical assistance provided by the program developer/administrator 

The applicability or cultural appropriateness of the program to the community we serve

The degree to which we have staff/volunteers who already know how to implement the program

Program length (e.g. number of sessions, length of sessions)

Other 

Please indicate how important the following **population-specific** characteristics are in the selection process:

*Please rate factors between 1 and 5, where 1 = not at all important and 5 = extremely important*

1. Specific health needs 

Languages spoken

Cultural diversity (e.g. racial/ethnic groups, tribes)

Special populations (e.g. veterans, low income)

Disability status

Urban/rural status

Other 

Which of the following could be a significant help to your organization in the program selection process? (Check all that apply)

1. Comprehensive and comparable estimates of program costs

Simpler program guidelines

Standardized program guidelines

Greater information about program flexibility—the capacity to accommodate local adjustments without affecting fidelity and effectiveness

Other (Please specify)

# Section 3. Ensuring Fidelity to the EBP

## Instructions

**We are interested in knowing how you maintain fidelity to different aspects of the program model for the <PROGRAM NAME>. ACL requests that each question be answered by someone with a good working knowledge of the topic at hand to ensure that the information you supply is complete and accurate. For sections where someone else’s input would be most appropriate, please enlist their assistance.**

## Questions

Does <GRANTEE NAME> require that you provide documentation that you are implementing <PROGRAM NAME> with fidelity? 

1. Yes

No

How does your organization stay informed about approaches and methods for establishing and maintaining program fidelity with regard to <PROGRAM NAME>? (Check all that apply)

1. Participate in program networking group

Participate in broad health related networking group (e.g. falls prevention)

Receive updates from program developer/administrator (e.g., via email group or list) 

Check ACL/NCOA Websites

Other

What strategies does your organization use to assure faithful delivery of <PROGRAM NAME> ? (Check all that apply)

1. Regularly observe trainers/leaders during sessions

Collect feedback from trainers/leaders

Collect feedback from participants

Use services provided by program developer/administrator

Hold regular management strategy sessions regarding fidelity

Hold regular staff meetings and/or trainings that stress fidelity

Regular standardized review of program fidelity

Informal review of program fidelity as needed

Other

*[If Question 11 = g, answer Question 12. All others will skip to Question 13.]*

Who created the guidelines for the review process? 

1. Your organization

Partners at the local or state level

Association or collaborative of EBP implementers

The program developer/administrator

The state/SUA

Other organization such as the National Council on Aging (NCOA), Evidence Based Leadership Collaborative (EBLC) 

Other

How often do you confirm that <PROGRAM NAME> is being implemented according to the program models?

1. When the program was first offered

1. Each time the program is offered

Monthly

Quarterly

Annually

Other

What challenges have you encountered to maintaining fidelity for <PROGRAM NAME>? (Check all that apply)

1. Cost of monitoring fidelity

Time burden of monitoring fidelity

Lack of internal expertise on maintaining fidelity

Inadequate guidance from program developers/administrators

Lack of agreement among stakeholders on the importance of fidelity 

Other

None 

What materials or tools would most help your organization overcome fidelity challenges? 

1. User friendly fidelity maintenance guidance

Fidelity checklists

Fidelity training courses for managers

Trainer registry 

Other

## Fidelity in Training

### Instructions

**Please limit your responses to information about <PROGRAM NAME>.**

**Programs may be licensed and distributed by a third party rather than the initial developer/administrator, and that is true of program-related training as well. In the following questions, please read “program developer/administrator” as also referring to such third parties.**

**Terminology:**

**Facilitators include staff members, volunteers, peers, or others who deliver program content to participants. They may also be referred to by other names such as “leaders” or “coaches.”**

**Trainers are staff members who instruct facilitators on how to faithfully deliver program content to participants.**

**Program trainers includes those identified as Master trainers or who have participated in a Train the Trainer or similar program, certifying that they are able to instruct and/or mentor trainers in the education of program facilitators. Program trainers may perform the role of facilitators as well.**

### Questions

**We are going to ask you some questions about how facilitators are trained.**

Who is primarily responsible for training facilitators?

1. External—Program developer/administrator or training entity 

Internal—Program-certified trainers on staff (if applicable)

Internal—Other staff members with program experience (e.g., facilitators) 

Other (Please specify)

*[If Question 16 = b or c, go to Question 17. All others skip to Question 19.]*

What is the source of guidelines and materials used in training facilitators? (Check all that apply)

1. Program developer/administrator

Own organization

Peer organizations

State agency other than own organization

Other 

In your experience, how clear are the guidelines from training facilitators provided by the program developer/administrator?

1. Very clear

1. Somewhat clear

1. Somewhat unclear

1. Very unclear

1. No guidelines for training

For programs that require certification for facilitators, when do you confirm that certifications are current? (Check all that apply)

1. During grant application/approval process 

When first offer program

Each time program is offered

Annually or regularly

We do not track this 

How are facilitators’ fidelity skills assessed after training is completed? 

1. Formal observation of group/workshop sessions

Formal tests of skills or knowledge

Collection of feedback from participants or trainees

By monitoring program metrics to identify potential issues

Other (Please specify)

No assessment system in place

What actions are taken if facilitators are found to be delivering **content** that does not meet program guidelines? (Check all that apply)

1. Provide coaching

Create a corrective action plan

Issue a warning

Replacement

Other (Please specify)

None of the above

What actions are taken if facilitators are found to be delivering the program in **ways** that do not meet program guidelines (e.g., changing the order or length of sessions, making unauthorized program adaptations)? (Check all that apply)

1. Provide coaching

1. Create a corrective action plan

1. Issue a warning

1. Replacement

1. Other (Please specify)

****

**We are now going to ask about how your program trainers are trained.**

Who is primarily responsible for training program trainers?

1. External—Program developer/administrator or training entity 

Internal—Program-certified trainers on staff (if applicable)

Internal—Other staff members with program experience 

Other (Please specify)

*[If Question 23 = b or c, answer Question 24. All others skip to Question 25.]*

What is the source of guidelines and materials used in training program trainers? (Check all that apply)

1. Program developer/administrator 

Own organization

Peer organizations

State agency other than own organization

Organization providing funding

For programs that require certification for program trainers, when do you confirm that certifications are current? 

1. During grant application/approval process 

Each time program is offered

Annually or periodically 

We do not track this

How are program trainers assessed for fidelity practices after training is completed? 

1. Formal observation of group/workshop sessions

Formal tests of skills or knowledge

Collection of feedback from participants or trainees

By monitoring program metrics to identify potential issues

Other (please specify)

No assessment system in place

## Fidelity in Program Resources

### Instructions

**In this section, “resources” refer to funding, staffing, equipment, and facilities. Other types of resources may also be required for faithful implementation of some EBPs.**

### Questions

In your experience, how clear are the guidelines for program resources provided by the program developer/administrator? 

1. Very clear 

Somewhat clear 

Somewhat unclear

Very unclear

No guidelines for program resources

For each of the following, do the resources available for <PROGRAM NAME> meet program guidelines?

*Please indicate for each of the factors whether funding is: 1 = Not adequate to meet guidelines, 2 = Adequate to meet guidelines, 3 = More than necessary to meet guidelines, or 4 = There are no guidelines.*

1. Funding

Staffing

Equipment

Facilities 

How much of a challenge is it to meet program resource guidelines or requirements for the following because of COST?

*Please select among the following: 1 = No Challenge, 2 = Moderate Challenge, 3 = Extreme Challenge, 4 = There are no guidelines*

1. Program licensing

Participant materials

Recommended staffing level

Recommended staffing credentials 

Recommended minimum space and/or accessibility 

Recommended general equipment (chairs, tables, weights) 

Recommended program-specific equipment or supplies	 

Recommended administrative personnel or equipment (e.g. data entry systems) 

Other

How much of a challenge is it to meet program resource guidelines or requirements for the following NON-COST reasons?

*Please select among the following: 1 = No Challenge, 2 = Moderate Challenge, 3 = Extreme Challenge, 4 = There are no guidelines*

1. Program Licensing

Participant materials

Recommended staffing level

Recommended staffing credentials 

Recommended minimum space and/or accessibility 

Recommended general equipment (chairs, tables, weights) 

Recommended program-specific equipment or supplies	 

Recommended administrative personnel or equipment (e.g. data entry systems) 

Other 

## Fidelity in Target Population

### Instructions

**In this section, target population is the group the program was intended for, as specified in the model for <PROGRAM NAME>. These may be very general, such as “older adults,” or more specific, such as “persons with arthritis or other chronic condition.”**

### Questions

In your experience, how clear are the guidelines for the target population provided by the program developer/administrator? 

1. Very clear 

Somewhat clear 

Somewhat unclear

Very unclear

No guidelines for target population 

Do you permit participants who are not strictly within the target population to participate in the program? 

1. Yes

No

*[If Question 32 = a, answer Question 33. If Question 32 = b, skip to Question 35.]*

What adaptations, if any, have been made with respect to the population served? (Check all that apply)

1. Expanded to populations with different health conditions

1. Expanded to different ages

1. Expanded to include companions/caretakers

1. Other (please specify)

*[If Question 33 = a-c, answer Question 34. If Question 33 = d, skip to Question 35.]*

Why has the population been enlarged beyond the target population defined by the program developer/administrator? (Check all that apply)

1. Target population not large enough to sustain program

1. Funding sources other than ACL require inclusion of other populations

1. Identified other populations that would benefit from the program

1. At the request of individuals or groups outside the target population

1. Other (please specify)

How do you verify that potential participants fall in the target population? (Check all that apply)

1. Referral from trusted organization

Self-identification

Require some form of documentation 

Other (Please specify)

If enrollment is not restricted to targeted populations, approximately what percentage of the program population is comprised of the target populations? 

1. 75 to 100%

50 to 74%

25 to 49%

Under 25%

## Fidelity in Content Provision

### Instructions

**When responding to the following questions, please focus on <PROGRAM NAME>.**

### Questions

In your experience, how clear are the guidelines for content provided by the program developer/administrator?

1. Very clear

1. Somewhat clear

1. Somewhat unclear

1. Very unclear

1. No guidelines for content

How often are all key components of the program content provided per the guidelines of the EBP?

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 38 = b-f, answer Question 45, then return to answer Question 39. If Question 38 = a, proceed to Question 39.]*

How often are materials prescribed by guidelines (e.g., exercise bands, handouts, web pages, videos) provided to participants according to the EBP guidelines? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 39 = b-f, answer Question 46, then return to answer Question 40. If Question 39 = a, proceed to Question 40.]*

How often is the content delivery mode (e.g., in-person vs. remote, lecture, discussion) prescribed by the guidelines of the EBP followed? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 40 = b-f, answer Question 47, then return to answer Question 41. If Question 40 = a, proceed to Question 41.]*

How often is the frequency of sessions prescribed by the guidelines of the EBP followed? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines 

We do not track this

*[If Question 41 = b-f, answer Question 48, then return to answer Question 42. If Question 41 = a, proceed to Question 42.]*

How often is the length of sessions prescribed by the guidelines of the EBP followed? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 42 = b-f, answer Question 49, then return to answer Question 43. If Question 42 = a, proceed to Question 43.]*

How often are the resource allocations (e.g., appropriately credentialed staff, equipment, locations) prescribed by the guidelines of the EBP followed? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 43 = b-f, answer Question 50, then return to answer Question 44. If Question 43 = a, proceed to Question 44.]*

How often is the group/session size within the limits (minimum and maximum) prescribed by the guidelines of the EBP? 

1. Always

Usually

Sometimes

Never

No relevant program guidelines

We do not track this

*[If Question 44 = b-f, answer Question 51. If Question 44 = a, skip to Question 52.]*

# Section 4. Program Adaptations

## Adaptations NOT related to the Coronavirus Disease 2019 (COVID-19) Public Health Emergency (PHE)

### Instructions

**We are interested in knowing whether you have made any adaptations to the program model for <PROGRAM NAME>, and if so, how that might impact fidelity. ACL requests that each question be answered by someone with a good working knowledge of the topic at hand to ensure that the information you supply is complete and accurate. For sections where someone else’s input would be most appropriate, please enlist their assistance.**

**In this section, “adaptations” are intentional deviations from the program model and recommended program guidelines.**

**NOTE: Please do NOT consider adaptations made to address the COVID-19 PHE when answering these questions.**

### Questions

*[Questions 45-51 are based on answers to questions 38-44, i.e. if respondent answers that some adaptations have been made, then they describe the kind of adaptation here.]*

Please describe the kind of adaptations made to components of the program content. (Check all that apply)

1. Added new content

Dropped some original content

Modified some original content

Other (Please specify)

Please describe the kind of adaptations made to program materials. (Check all that apply)

1. Added new materials

Dropped some original materials

Modified some original materials

Other (Please specify)

Please describe the kind of adaptations made to content delivery mode. (Check all that apply)

1. Increased/decreased extent of remote content delivery

Increased/decreased extent of group discussions

Increased/decreased extent of group exercise

Other (Please specify)

Please describe the kind of adaptations made to the frequency of sessions. (Check all that apply)

1. More frequent

Less frequent

Other (Please specify) 

Please describe the kind of adaptations made to the length of sessions. (Check all that apply)

1. Longer

Shorter

Other (Please specify)

Please describe the kind of adaptations made to resource allocations (e.g., staff, equipment, locations). (Check all that apply)

1. Fewer staff per participant

Relaxation of staff credential/training requirements

Substitution/omission of certain types of equipment

Other (Please specify)

Please describe the kind of adaptations made to group/session size.

1. Permit group/session smaller than prescribed minimum

Permit group/session larger than prescribed maximum

Other (Please specify)

**When responding to the following questions, please focus on <PROGRAM NAME>.**

What factors does your organization consider when determining the need for adaptations to <PROGRAM NAME>? (Check all that apply)

1. Funding constraints

Availability of staff needed for guideline adherence

Availability of other resources needed for guideline adherence (e.g., facilities, equipment)

Accommodations for accessibility (disabilities, etc.)

Accommodations for lower-income and rural participants (transportation, hours, etc.)

Cultural inclusivity, including language and religion

To increase appeal to local populations

Other (Please specify)

Have not considered adaptations

Whose input is important in determining whether to make adaptations and how they should be made? (Check all that apply)

1. Program developer/administrator

Program participants

Partner organizations

Peer organizations

Local government, civic organizations, social, and/or religious organizations

State government

Funders

Other

Minimal/no external input

Do not make such determinations

Has your organization worked with the program developer/administrator to identify acceptable program adaptations that maintain the integrity of the evidence-based nature of the program?

1. Yes

No

If you have received guidance on adaptations from the program developer/administrator, when was it provided? 

1. Program developer/administrator proactively provided guidance on adaptations for <PROGRAM NAME>

Provided in response to request for guidance 

Did not provide guidance on adaptations

If you have received guidance on adaptions from the source of your ACL funding, <GRANTEE NAME>, when was it provided?

1. General guidance on adaptations allowed/recommended

Guidance only in response to specific requests for advice

No guidance provided

## Adaptations related to the COVID-19 PHE

### Instructions

**NOTE: The following section relates to changes in your operation of ACL-funded evidence-based programs made in response to the COVID-19 PHE.**

### Questions

Prior to the COVID-19 PHE, what proportion of <PROGRAM NAME> was provided in person?

1. All in person 

Most in person, some remote

Some in person, most remote

All remote

Not sure

During the COVID-19 PHE, what proportion of <PROGRAM NAME> content was provided in-person?

1. All in person

Most in person, some remote

Some in person, most remote

All remote

This program has been halted during the COVID-19 PHE

Which of the following program components have been modified in response to the COVID-19 PHE?

1. Program Content

Materials (e.g., exercise bands, handouts, web pages, videos)

Delivery mode (e.g., in-person vs. remote, lecture, discussion)

Frequency of sessions

Duration of sessions

Resources allocated (e.g., staff, equipment, locations)

Group/session size (including number of clients per facilitator)

Other (Please specify)

What topics, if any, have you sought guidance on related to the COVID-19 PHE? (Check all that apply)

1. Have not sought guidance on COVID-related adaptations

Safe in-person service delivery

How to redesign a program for remote delivery 

Additional training needed for remote delivery

How to ensure that clients practiced the program safely without direct oversight

How to reach clients and recruit them for remote delivery

Other (Please specify)

Who provided guidance and other materials that helped you to adapt programs successfully during the COVID-19 PHE? (Check all that apply)

1. Program developers

NCOA

Peer organizations

Participants and other stakeholders

CDC and other public health agencies

Other (Please specify)

No guidance received

Has your <PROGRAM NAME> funding decreased during the COVID-19 PHE for reasons other than decreased demand for services?

1. Yes

No

# Section 5. Support from ACL

## Instructions

**ACL requests that each question be answered by someone with a good working knowledge of the topic at hand to ensure that the information you supply is complete and accurate. For sections where someone else’s input would be most appropriate, please enlist their assistance.**

## Questions

In what ways could the list of eligible EBPs be improved? (Check all that apply)

1. Update more frequently with newer programs

Increase the number of EBPs that are easy to implement as designed 

Increase the number of EBPs that address other aspects of Falls or CDSME

Increase the number of EBPs in languages other than English

Increase the number of EBPs that have been tested with additional populations (members of racial and ethnic minority groups, individuals with disabilities, etc.)

Increase the number of EBPs that can be delivered remotely

Other (Please specify)

None

What types of support would improve your organization’s ability to implement programs with fidelity and/or maintain fidelity over time? (Check all that apply)

1. Assistance with selecting EBPs appropriate to your local context 

Guidance on how to make local adaptations without threatening fidelity

Guidance and tools for effective monitoring of program implementations

Guidance and tools for maintaining fidelity over time

Guidance on controlling costs while maintaining fidelity

Assistance with evaluating the evidence base for EBPs

Other types of technical assistance regarding EBP programs

Other

No additional support required

If you believe that any additional support would improve your organization’s ability to implement programs with fidelity and/or maintain fidelity over time, please elaborate on your response. [OPEN TEXT]

# Conclusion

Thank you for participating in this survey!

Your responses will help ACL determine how to improve its grant-making process and its support and monitoring of EBP fidelity, ensuring high quality delivery of proven programs to individuals and families supported by the OOA.

If you have questions about this survey please contact [PERSON, ORGANIZATION, TITLE, E-MAIL, TEL]

Administration for Community Living 								              Page | 1

Fidelity Evaluation – Draft Program Leader Survey 							             May 2021

# image1.jpg