Independent Living Technical Assistance Evaluation 

Discussion of Results

Jennifer Howard, PhD

Dustin Williams, MA

RTI International

September 21, 2023

1

Discussion Agenda

Study Background

Introduction

Research Questions

Project Overview

Results by Research Question

Methods

Key Informant Interviews

Document Review

Literature Review

Program Performance Report Data Analysis

TA Survey

Conclusions

2

Study Background

3

Introduction

The Administration on Disabilities (AoD) provides grants to support independent living (IL) services through the IL network—which includes CILs, SILCs, and DSEs—as part of its mission to give individuals with disabilities of all ages the opportunities, tools, and supports to lead the lives of their choice in their community.

Under two cooperative agreements with AoD, the Independent Living Research Utilization (ILRU) and its partners provide technical assistance (TA) to support the CILs, SILCs, and DSEs.

AoD contracted with RTI International to evaluate the TA strategy and how it might be improved to better meet the needs of IL grantees.

4

4

Research Questions

What are the elements of effective program TA?

What do the DSEs, SILCs, and CILs get with regard to TA? 

What are DSEs’, SILCs’, and CILs’ needs when it comes to TA? 

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? 

How is the TA not meeting their needs? What is additionally needed? 

What are AoD’s TA needs and priorities for the DSE, SILC, and CIL programs? 

How is the current TA meeting AoD’s needs? 

What are the lessons learned that might be applicable to other AoD programs?

5

Project Overview

To answer the research questions, we did the following: 

Interviewed IL grantees, TA providers, and AoD staff to give us information to help us develop survey questions

Reviewed available literature and resources from various organizations about what works well in TA

Reviewed documents to better understand the current TA approach

Analyzed IL grantee PPR data

Surveyed grantees about their TA experiences

6

6

Results by Research Question

7

What are the elements of effective program TA?​

8

What are the elements of effective program TA?Key Informant Interview (KII) Results
| What makes TA effective according to KII participants? |
|--------------------------------------------------------|
| TA provider is credible                                |
| Information available is timely                        |
| Information balances specificity and generality        |
| TA recipients are informed of TA availability          |
| TA is offered in a variety of formats                  |
| Questions and comments are kept confidential           |
| Teaching style is engaging                             |

9

What are the elements of effective program TA?Literature Review Results
| What are elements of effective TA according to the literature?            |
|---------------------------------------------------------------------------|
| Relationship between TA provider and recipient/TA provider is a SME       |
| Baseline needs assessment/establishing TA goals                           |
| Accessibility/delivering TA in multiple formats                           |
| Tailoring TA to recipient’s needs                                         |
| Flexibility of TA provider                                                |
| Knowledge of the TA recipients’ context                                   |
| Working with staff at various levels within the TA recipient organization |

10

10

TA Survey Participation

RTI fielded the survey in June–August 2023 and received 234 responses.
| Organization Type | Number of Survey Responses | Organizations on Frame | Response Rate (%) |
|-------------------|----------------------------|------------------------|-------------------|
| CIL               | 180                        | 286                    | 62.6              |
| SILC              | 27                         | 56                     | 44.6              |
| DSE               | 27                         | 55                     | 50.1              |
| Total             | 234                        | 397                    | 58.9              |

11

What are the elements of effective program TA? ​Survey Results
| What would make TA most effective for your organization? | All Respondents (%) |
|----------------------------------------------------------|---------------------|
| Convenience                                              | 74.8                |
| Relevance                                                | 72.2                |
| Personalized Content                                     | 50.9                |
| Accessibility                                            | 50.4                |
| Timeliness                                               | 44.4                |
| Other                                                    | 3.4                 |

12

What do the DSEs, SILCs, and CILs get with regard to TA? ​

13

What do the DSEs, SILCs, and CILs get with regard to TA? ​KII Results/Document Review Results
| Who provides the TA and what types of TA are received according to the KII and document review results? |
|---------------------------------------------------------------------------------------------------------|
| TA Providers Used                                                                                       |
| ILRU                                                                                                    |
| APRIL and NCIL                                                                                          |
| Types of TA Received                                                                                    |
| Webinars & courses/rapid courses                                                                    |
| Peer-to-peer                                                                                            |
| Conferences                                                                                             |
| Intensive support                                                                                       |
| Products (e.g., newsletters, directory updates)                                                         |

14

What do the DSEs, SILCs, and CILs get with regard to TA? ​Survey Results (1 of 5)
| Grantee Type | Number of Survey Responses | Received ILRU TA N (%) | Received non-ILRU TA N (%) |
|--------------|----------------------------|------------------------|----------------------------|
| CIL          | 180                        | 130 (72.2)             | 108 (60.0)                 |
| DSE          | 27                         | 12 (44.4)              | 8 (29.6)                   |
| SILC         | 27                         | 26 (96.3)              | 24 (88.9)                  |
| Total        | 234                        | 168 (71.8)             | 140 (59.8)                 |

15

15

What do the DSEs, SILCs, and CILs get with regard to TA? ​Survey Results (2 of 5)
| TA Type                                              | % Receiving TA Type |
|------------------------------------------------------|---------------------|
| Online, live trainings or webinars                   | 91.9                |
| On-demand (recorded) trainings or webinars           | 91.5                |
| On-demand publications and resources                 | 85.0                |
| Other                                                | 82.1                |
| Peer Group Technical Assistance Discussions          | 73.1                |
| In-person trainings or webinars                      | 67.5                |
| One-on-one technical assistance or intensive support | 65.8                |
| Technical Assistance Office Hours                    | 63.2                |
| Peer-to-Peer Mentoring                               | 62.4                |

16

What do the DSEs, SILCs, and CILs get with regard to TA? ​Survey Results (3 of 5)
| TA Topic                                                                     | % Receiving TA on Topic |
|------------------------------------------------------------------------------|-------------------------|
| Other                                                                        | 91.9                    |
| COVID-19 and emergency preparedness resources                                | 80.8                    |
| Disability, diversity, intersectionality, and outreach to underserved groups | 79.5                    |
| Transition of youth                                                          | 77.8                    |
| Independent living history and philosophy                                    | 76.1                    |
| Home and community-based services                                            | 74.4                    |
| Community and institutional transition or diversion                          | 73.9                    |
| Individual or systems advocacy                                               | 73.9                    |
| SILC and DSE roles and responsibilities; State Plan for Independent Living   | 73.9                    |

17

What do the DSEs, SILCs, and CILs get with regard to TA? ​Survey Results (4 of 5)
| TA Topic                                                                                                         | % Receiving TA on Topic |
|------------------------------------------------------------------------------------------------------------------|-------------------------|
| Peer counseling and peer support                                                                                 | 69.7                    |
| Performance measurement, program performance reporting, compliance reporting, and evaluation/outcome measurement | 69.7                    |
| Transportation                                                                                                   | 68.4                    |
| Expanding housing options                                                                                        | 67.5                    |
| Independent living skills training                                                                               | 67.5                    |
| Information and referral support                                                                                 | 65.8                    |
| Succession planning, staff development, benefits, and remote working                                             | 65.8                    |

18

What do the DSEs, SILCs, and CILs get with regard to TA? ​Survey Results (5 of 5)
| TA Topic                                                                                                                               | % Receiving TA on Topic |
|----------------------------------------------------------------------------------------------------------------------------------------|-------------------------|
| Board training and development                                                                                                         | 65.8                    |
| Business acumen, financial management, fundraising and grant writing                                                                   | 65.4                    |
| Consumer information files or consumer service records (for example, release authorizations, goal forms, and independent living plans) | 64.5                    |
| Assistive device or technology support                                                                                                 | 61.5                    |

19

What are DSEs’, SILCs’, and CILs’ needs when it comes to TA? ​

20

What are DSEs’, SILCs’, and CILs’ needs when it comes to TA? KII Results
| What are the IL Grantees’ TA needs according to KII participants?   |
|---------------------------------------------------------------------|
| Programmatic                                                        |
| Administrative or management                                        |
| New or upcoming regulation guidance                                 |
| IL philosophy or history                                            |
| Technology                                                          |
| Innovative or “hot topics”                                          |
| Post-COVID                                                          |

21

21

What are DSEs’, SILCs’, and CILs’ needs when it comes to TA? PPR TA Needs Data Analysis Results (1 of 2)

What are the CILs’ reported TA needs? 
| 2019 (N = 347)                                                                                                                                                                                                                                                                                                                                                                        | 2020 (N = 339)                                                                                                                                                                                                                                                                                   | 2021 (N = 335)                                                                                                                                                                                                                                                                                                                |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Financial: Resource Development - Fee-for-Service Approaches (60)Advocacy/Leadership Development - Systems Advocacy (60)Innovative Programs - Best Practices (49)Data Collecting and Reporting - PPR/704 Reports (45)Financial: Resource Development - Diversification of Funding Base (42)Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (42)     | Financial: Resource Development - Fee-for-Service Approaches  (55)Innovative Programs - Best Practices (43)Financial: Resource Development - Diversification of Funding Base (42)Advocacy/Leadership Development - Systems Advocacy (42)Data Collecting and Reporting: PPR/704 Reports (38)      | Financial: Resource Development - Fee-for-Service Approaches (50)Financial: Resource Development - Diversification of Funding Base (48)Advocacy/Leadership Development - Systems Advocacy (43)Innovative Programs - Best Practices (32)Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (32) |

22

22

What are DSEs’, SILCs’, and CILs’ needs when it comes to TA? PPR TA Needs Data Analysis Results (2 of 2)

What are the SILCs’ reported TA needs?
| 2019 (N = 54)                                                                                                                                                                                                                                                                                                                                                                  | 2020 (N = 54)                                                                                                                                                                                                                                                                                                                                                                                                                                           | 2021 (N = 51)                                                                                                                                                                                                                                                                                                                                                                    |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (8)Advocacy/Leadership Development - Community/Grassroots Organizing (7)Financial: Resource Development - Fund-Raising Events of Statewide Campaigns (7)SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6) Evaluation - Community Needs Assessment (6)      | Advocacy/Leadership Development - Systems Advocacy (8)Advocacy/Leadership Development - Community/Grassroots Organizing (7)Advocacy/Leadership Development - General Overview (7)Innovative Programs - Best Practices (6)SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6) Financial: Resource Development - Diversification of Funding Base (6)Applicable Laws - Medicaid/Medicare/PAS/waivers/long-term care (6)    | Advocacy/Leadership Development - Systems Advocacy (10)SILC Roles/Relationship to CILs - Development of State Plan for Independent Living (9)Data Collecting and Reporting - Performance Measures contained in PPR/704 Reports (7)Advocacy/Leadership Development - Legislative Process (6)SILC Roles/Relationship to CILs - Implementation (monitor & review) of SPIL (6)   |

23

23

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​

24

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​ KII Results
| Overall Experience with TA                                              |
|-------------------------------------------------------------------------|
| Positive, especially for newer grantee staff                            |
| Mixed experiences with timeliness of TA topics                          |
| Experience Requesting TA                                                |
| Paula is well known as the “go-to” person for TA requests at ILRU       |
| Unclear whether IL grantees know how to request TA                      |
| Grantee organizations refer members to ILRU for TA                      |
| ILRU responds quickly to TA requests                                    |

25

25

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​Survey Results (1 of 5)

How satisfied was your organization with the types of TA received?
| TA Type                   | Very Dissatisfied | Dissatisfied | Neither Satisfied or Dissatisfied | Satisfied | Very Satisfied |
|---------------------------|-------------------|--------------|-----------------------------------|-----------|----------------|
| Peer-to-peer mentoring    | 2.9%              | 1.9%         | 7.8%                              | 44.7%     | 42.7%          |
| TA office hours           | 2.4%              | 1.2%         | 4.8%                              | 48.8%     | 42.9%          |
| Peer group TA discussions | 3.1%              | 2.3%         | 7.7%                              | 46.9%     | 40.0%          |

26

26

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​Survey Results (2 of 5)

How satisfied was your organization with the types of TA received?
| TA Type                            | Very Dissatisfied | Dissatisfied | Neither Satisfied or Dissatisfied | Satisfied | Very Satisfied |
|------------------------------------|-------------------|--------------|-----------------------------------|-----------|----------------|
| One-on-one TA or intensive support | 3.0%              | 2.0%         | 9.0%                              | 41.0%     | 45.0%          |
| In-person trainings or webinars    | 4.4%              | 0.9%         | 13.3%                             | 46.0%     | 35.4%          |
| Online, live trainings or webinars | 2.8%              | 1.4%         | 6.9%                              | 55.3%     | 33.6%          |

27

27

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​Survey Results (3 of 5)

How satisfied was your organization with the types of TA received?
| TA Type                                                  | Very Dissatisfied | Dissatisfied | Neither Satisfied or Dissatisfied | Satisfied | Very Satisfied |
|----------------------------------------------------------|-------------------|--------------|-----------------------------------|-----------|----------------|
| On-demand (recorded) trainings or webinars               | 3.1%              | 1.0%         | 8.7%                              | 52.8%     | 34.4%          |
| On-demand publications and resources (e.g., fact sheets) | 2.8%              | 0.6%         | 7.7%                              | 51.4%     | 37.5%          |
| Other                                                    | 0.0%              | 0.0%         | 20.0%                             | 20.0%     | 60.0%          |

28

28

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​ Survey Results (4 of 5)

Respondent rating of ILRU and non-ILRU TA in terms of helping organization fulfill its mission
| Grantee Type | Poor # | Poor % | Fair # | Fair % | Good # | Good % | Very Good # | Very Good % | Excellent # | Excellent % | Total # |
|--------------|--------|--------|--------|--------|--------|--------|-------------|-------------|-------------|-------------|---------|
| CIL          | 2      | 0.9    | 14     | 6.0    | 69     | 29.4   | 94          | 40.0        | 56          | 23.8        | 235     |
| SILC         | 0      | 0.0    | 4      | 8.3    | 7      | 14.6   | 18          | 37.5        | 19          | 39.6        | 48      |
| DSE          | 0      | 0.0    | 0      | 0.0    | 8      | 42.1   | 8           | 42.1        | 3           | 15.8        | 19      |
| All Grantees | 2      | 0.7    | 18     | 6.0    | 84     | 27.8   | 120         | 39.7        | 78          | 25.8        | 302     |

29

29

How is the current approach to TA meeting DSEs’, SILCs’, and CILs’ needs? ​ ​Survey Results (5 of 5)
| What are the benefits of TA for the organization? | All Grantees (%) |
|---------------------------------------------------|------------------|
| Improved operational efficiency                   | 86.3             |
| Improved performance on key outcomes              | 80.8             |
| Improved financial performance                    | 41.9             |
| Improved outreach                                 | 41.9             |
| Staff retention                                   | 39.7             |
| Other                                             | 4.7              |
| There are no benefits                             | 1.7              |

30

30

How is the TA not meeting their needs? What is additionally needed? ​

31

How is the TA not meeting their needs? What is additionally needed?​ Survey Results (1 of 2)
| Barriers preventing organizations from using additional ILRU and non-ILRU TA      | CILN (%)   | SILCN (%) | DSEN (%) | All GranteesN (%) |
|-----------------------------------------------------------------------------------|------------|-----------|----------|-------------------|
| No additional TA needed                                                           | 143 (60.1) | 31(13.0)  | 11(4.6)  | 185(77.7)         |
| Limited staff and could not make time for anyone to participate in TA             | 45(18.9)   | 7(2.9)    | 2(0.8)   | 54(22.7)          |
| TA types did not occur frequently enough or at times my organization could attend | 32(13.4)   | 8(3.4)    | 3(1.3)   | 43(18.1)          |
| Did not know about it                                                             | 27(11.3)   | 5(2.1)    | 2(0.8)   | 34(14.3)          |
| Other                                                                             | 10(4.2)    | 9(3.8)    | 3(1.3)   | 22(9.2)           |

32

32

How is the TA not meeting their needs? What is additionally needed?​ Survey Results (2 of 2)
| Barriers preventing organizations from using additional ILRU and non-ILRU TA          | CIL      N (%)    | SILC N (%) | DSE     N (%) | All Grantees N (%)          |
|---------------------------------------------------------------------------------------|-------------------|------------|---------------|-----------------------------|
| Past TA has not been appropriately tailored for my organization and was not relevant  | 11(4.6)           | 3(1.3)     | 1(0.4)        | 15 (6.3)                    |
| TA was not timely                                                                     | 5(2.1)            | 2(0.8)     | 1(0.4)        | 8(3.4)                      |
| TA was poor quality                                                                   | 7           (2.9) | 0    (0.0) | 1     (0.4)   | 8                     (3.4) |
| TA is not accessible for people with disabilities                                     | 3(1.3)            | 0(0.0)     | 1(0.4)        | 4                     (1.7) |

33

33

How is the TA not meeting their needs? What is additionally needed? KII Results
| Recommendations for TA improvements according to KII participants                       |
|-----------------------------------------------------------------------------------------|
| No need to change to TA (2)                                                             |
| Direct a greater portion of available funds for direct support and peer-to-peer (2)     |
| Better identify grantees’ TA needs and desired approaches and tailor accordingly (2)    |
| Elevate the focus on disability (2)                                                     |
| Require TA providers to work with national associations (1)                             |
| Change and increase ILRU staff (1)                                                      |
| Increase personal engagement and networking (1)                                         |
| Reduce TA information email volume (1)                                                  |

34

34

methods

35

Key Informant Interview Methods

Conducted 11 KIIs between October 2022 and January 2023 with representatives from AoD (2), CILs (3), SILCs (1), DSEs (2), and TA providers (3).

Interviews covered the following topics:

Types of TA received 

TA providers used

Awareness of TA

Experience with TA

TA effectiveness

TA needs

Recommendations for improvements

36

Document Review Methods

Document types reviewed and information pulled:

Funding Opportunity Announcements (FOAs)—TA requirements

Performance reports—time frame, performance metrics, and partners

Contracts and related materials—TA requirements and planned approach

37

Literature Review Methods

Developed search strategies for both the peer-reviewed literature and organizations’ websites.

Published in English since January 2012 

Search terms—technical assistance and service providers and disab* 

Search locations—7 databases

Search locations—22 federal government agency, research organization, and service provider websites

Team members reviewed the literature and determined that 40 publications met inclusion criteria.

38

38

Project Performance ReportData Analysis Methods

Grantees were asked to select from 78 TA topics to answer the following: 

Please identify the CIL’s and SILC’s TA needs; the needs identified in this chart will guide the priorities set by ACL for the TA provided to CILs and SILCs.

Choose up to 10 Priority Needs—Rate items 1–10, with 1 being most important.

Frequencies

Number of grantees that responded to the question

Number of grantees that reported a 1–3 for a TA topic

39

39

TA Survey Methods

Developed a web survey about TA needs and effectiveness.

Conducted six cognitive interviews and nine pilot surveys to test the survey.

The full survey was distributed to 397 grantees, using a combination of email, mail, and phone outreach. 

Organizations were asked questions on:

Their use or non-use of TA

The benefits/utility of current TA types and topics

Their TA needs

Barriers to using TA

Ways to improve future TA

40

40

Conclusions (1 of 2)

There is a great deal of agreement in what constitutes effective TA.

Grantees are getting TA from a mix of TA providers.

Webinars are the most commonly used type of TA.

Out of all the topics covered for TA received, the following topped the list: (1) COVID-19 and emergency preparedness, and (2) Disability, diversity, intersectionality, and outreach to underserved groups.

These topics differed from the reported needs in the 2021 PPR, which indicated that the top TA needs focused on finances, including resource development related to Fee-for-Service approaches or diversification of funding base.

These topics also differed from the reported needs by KII participants, which indicated that the top TA needs were related to programmatic, administrative or management, IL philosophy or history, technology, or post-COVID.

Grantees are overall satisfied with the TA providers and the TA they provide.

41

41

Conclusions (2 of 2)

Areas of opportunity may include: 

Providing TA offerings more frequently and with more flexibility to allow greater participation among grantees with limited staff availability

Increasing grantees’ awareness of and engagement in TA 

Identifying grantees’ TA needs and tailoring TA accordingly

42

42

Thank you!

43

43

# image1.png

# image3.png

# image4.png

# image5.png