NIDILRR Survey Support for 

Evaluation Activities: Year 1 

Report

Contract # ACL155092

Final

January 31, 2016

Prepared for: 

Maggie Hatton

Contract Technical Representative

National Institute for Disability, Independent 

Living, and Rehabilitation Research

Prepared by:

Luba Katz, PhD

Abt Associates Inc.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

TABLE OF CONTENTS 

Executive Summary.....................................................................................................................iii
 

Introduction ................................................................................................................................1
 

Methods......................................................................................................................................3
 

Data Sources ................................................................................................................................ 3
 

Study Participants ........................................................................................................................ 3
 

Data Collection Procedures.......................................................................................................... 3
 

Data Analysis................................................................................................................................5
 

Findings.......................................................................................................................................6
 

Disability and Business Technical Assistance Centers.................................................................. 6
 

Findings from the PI Survey...............................................................................................6
 

Findings from User Interviews........................................................................................... 9
 

Knowledge Translation Program................................................................................................11
 

Findings from the PI Survey.............................................................................................11
 

Findings from User Interviews.........................................................................................15
 

Analysis of Grantee Publications.....................................................................................16
 

Small Business Innovation Research Program...........................................................................17
 

Findings from the PI Survey.............................................................................................17
 

Products Mentioned by SBIR Grantees ...........................................................................22
 

Findings from User Interviews.........................................................................................25
 

NIDILRR Application Process......................................................................................................26
 

Summary of Scoring Data ................................................................................................26
 

Summary of Qualitative Comments ................................................................................ 29
 

Recommendations for Improving the Application Process.............................................33
 

Study Limitations ....................................................................................................................... 33
 

Summary and Conclusions..........................................................................................................34
 

Disability and Business Technical Assistance Centers................................................................34
 

Knowledge Translation Program................................................................................................34
 

Small Business Innovation Research Program...........................................................................35
 

Application Process....................................................................................................................35
 

Appendix A: Study Protocols ......................................................................................................36
 

Survey of Principal Investigators on Americans with Disabilities Act Grants ............................36
 

Survey of Principal Investigators on Knowledge Translation Grants.........................................38
 

Survey of Principal Investigators on Small Business Innovation Research Grants ....................42
 

Survey of Applicants...................................................................................................................45
 

Customer Interview Questions ..................................................................................................46
 

Appendix B: Interviewee Information.........................................................................................47
 

Appendix C: Implications for Furture Evaluation .........................................................................48
 

Abt Associates NIDILRR Evaluation Report Year 1
 i 
 

 

 

FIGURES 

FIGURE 1: A SCREEN FROM THE SBIR SURVEY........................................................................................................4
 

FIGURE 2: FINDINGS FROM PI DBTAC SURVEY ON ADVANCEMENTS DUE TO DBTAC FUNDING (N=10) ...............7
 

FIGURE 3: FINDINGS FROM THE PI DBTAC SURVEY ON BENEFIT AREAS (N=10) ....................................................7
 

FIGURE 4: ACTIVITIES SINCE THE END OF GRANT REPORTED BY KT PIS (N=6).....................................................12
 

FIGURE 5: OUTCOMES OF NIDILRR FUNDING REPORTED BY KT PIS (N=6)...........................................................14
 

FIGURE 6: COLLABORATIONS RESULTING FROM NIDILRR FUNDING REPORTED BY KT PIS (N=6)........................15
 

FIGURE 7: BIBLIOMETRIC ANALYSIS OF PUBLICATIONS LINKED TO KT CENTERS FOR 2005-2011........................17
 

FIGURE 8: PERCENT OF SBIR I/II GRANTEES RECEIVING ADDITIONAL DEVELOPMENT (N=26) COMPARED TO
 

OTHER FEDERAL SBIR PROGRAMS...............................................................................................................18
 

FIGURE 9: LIKELIHOOD OF UNDERTAKING THE PROJECT WITHOUT SBIR FUNDING REPORTED BY NIDILRR SBIR 

I/II PIS (N=26) COMPARED TO OTHER FEDERAL SBIR PROGRAMS ..............................................................19
 

FIGURE 10: COMMERCIALIZATION OUTCOME OF SBIR II FUNDING REPORTED BY NIDILRR PIS (N=10) 

COMPARED TO OTHER FEDERAL SBIR PROGRAMS .....................................................................................19
 

FIGURE 11: CHALLENGES EXPERIENCED BY GRANTEES REPORTED IN THE SBIR PI SURVEY (N=24).....................21
 

FIGURE 12: EXAMPLES OF ABLELINK PRODUCTS..................................................................................................22
 

FIGURE 13: TOUCH GRAPHICS PRODUCT .............................................................................................................23
 

FIGURE 14: PAGEFLIP PRODUCTS .........................................................................................................................23
 

FIGURE 15: KOESTER PERFORMANCE RESEARCH PRODUCTS ..............................................................................24
 

FIGURE 16: CHARMTECH PRODUCT .....................................................................................................................24
 

FIGURE 17: COGNITOPIA PRODUCT .....................................................................................................................25
 

FIGURE 18: MEAN SATISFACTION SCORES WITH GRANT APPLICATION PROCESS, FUNDING LEVEL, AND 

FUNDING DURATION REPORTED IN THE APPLICANT SURVEY (N=153) .......................................................27
 

FIGURE 19: APPLICANT SATISFACTION SCORES BY PROGRAM (N=153)...............................................................28
 

FIGURE 20: GRANTEE PI SURVEY COMMENTS ABOUT APPLICATION PROCESS (N=210) .....................................30
 

TABLES 

TABLE 1: LAWS AND POLICIES REVISED OR CREATED AS A RESULT OF DBTAC FUNDING MENTIONED BY SURVEY 

RESPONDENTS (N=10) ...................................................................................................................................8
 

TABLE 2: RESPONSE RATES FOR SBIR SURVEY FOR GRANTEES BETWEEN 2007 AND 2014 (N=58) .....................17
 

TABLE 3: FOLLOW-UP FUNDING SOURCES FOR NIDILRR SBIR GRANTEES (N=26)................................................18
 

TABLE 4: FINDINGS FROM SBIR SURVEY RE: MOST IMPORTANT COMMERCIALIZATION OUTCOMES REPORTED 

BY SBIR I/II PIS (N=26)..................................................................................................................................20
 

TABLE 5: NUMBER AND PERCENT OF NIDILRR APPLICANTS DISSATISFIED OR SOMEWHAT DISSATISFIED GRANT 

APPLICATION PROCESS, FUNDING LEVEL, AND FUNDING DURATION (N=153)...........................................27
 

Abt Associates NIDILRR Evaluation Report Year 1
 ii 
Abt Associates NIDILRR Evaluation Report Year 1 iii

Executive Summary

The National Institute for Disability, Independent living, and Rehabilitation Research (NIDILRR) 

supports a diverse portfolio of research, development, technical assistance, and training programs

to improve the lives of people with disabilities. To assess the effectiveness if its grant-making 

processes and to document program outcomes, NIDILRR initiated a 5-year evaluation study in the 

spring of 2015. The first evaluation year included (1) a review of three programs—the Disability and 

Business Technical Assistance Centers, Knowledge Translation centers, and Small Business 

Innovation Research program—and (2) an assessment of applicant satisfaction with recent funding 

competitions. 

The focus of the programmatic reviews was on the benefits of funding to individuals with 

disabilities. Because of the gap between the activities funded under the grants and the emergence 

of benefits resulting from these activities, only long-running projects (funded for 10 years or more)

and completed projects were included in the study. Several data collection strategies were used to 

obtain outcome data. First, we conducted on-line surveys of Principal Investigators (PIs) for the

three programs to document the funding status of the projects, use of resources of products 

generated with NIDILRR funding, and improvements in the lives of people with disabilities that could 

be attributed at least in part to these grants. Because these data were self-reported and may be 

positively biased, we also asked PIs to recommend a few individuals who could speak to us about 

the benefits of funding. Nearly 50 contacts were provided across the three programs and 

approximately half agreed to an interview.

All surveyed PIs indicated that their grants led to improvements in the lives of people with 

disabilities and to better understanding by society of their needs and challenges. Many examples of 

direct and indirect benefits were provided by PIs, including new laws and policies related to 

disability access, training of architecture students in accessibility design, assistive technologies, and 

others. We also found that commercialization outcomes at NIDILRR were similar or better than at 

other agencies with SBIR programs and that the papers citing KT grants were published in higher 

impact journals than average in the rehabilitation field. 

We obtained many direct testimonials of benefits from the users of NIDILRR-sponsored products 

and services. DBTACs, in particular, were highly praised by representatives of state and local 

governments, non-profit organizations, and businesses that serve people with disabilities. Several of 

the users self-identified as having a disability. The findings indicate that NIDILRR funding is 

benefiting its ultimate customers—individuals with disabilities.

In addition to the programmatic reviews, we examined participant satisfaction with recent grant 

competitions in an on-line survey. Approximately 150 applicants in 2015 participated in the study. 

Content of funding announcements, assistance from NIDILRR, and time for submission emerged as 

relative strengths. In contrast, the applicants were less satisfied with the quality and usefulness of 

reviewer feedback and with the length of time from proposal submission to award notification. 

Based on the survey findings, we identify several changes to the application procedures which might 

enhance participant satisfaction. 
Abt Associates NIDILRR Evaluation Report Year 1 1

Introduction 

The mission of the National Institute for Disability, Independent living, and Rehabilitation Research 

(NIDILRR) is to “generate new knowledge and promote its effective use to improve the abilities of 

people with disabilities to perform activities of their choice in the community, and also to expand 

society’s capacity to provide full opportunities and accommodations for its citizens with disabilities.”1

To fulfill this mission, NIDILRR funds a portfolio of research, development, technical assistance, and 

training programs, which include Rehabilitation Research and Training Centers (RRTC), Rehabilitation 

Engineering Research Centers (RERC), NIDILRR Scholars Program, Disability and Rehabilitation Research 

Projects (DRRT), Knowledge Translation (KT), Field Initiated Projects (FIP), Advanced Rehabilitation 

Research Training Projects (ARRT), Mary E. Switzer Fellowships (Switzer), Small Business Innovation 

Research (SBIR), and Model Systems of Care (MS). In addition, NIDILRR administers 10 regional Disability 

and Business Technical Assistance Centers (DBTAC). 

In 2009 NIDILRR commissioned the Nasional Academies of Sciences to conduct an evaluation of its 

funding programs and grant-related process. One of the recommendations made by the Academies to 

NIDILRR was to establish a longer-term, systematic evaluation program. NIDILRR responded to this 

suggestion by contracting with Abt Associates to develop a 5-year evaluation plan and a toolkit to 

implement this plan. The planning phase of the project has been successfully completed in 2014 and in 

February 2015 we began the evaluation of the NIDILRR program portfolio. 

NIDILRR already collects extensive outcome data on the funded projects while they are active. 

Consequently, the evaluation is focused on longer-term outcomes of NIDILRR programs as well as on the 

effectiveness and efficiency of grant-making process. Specifically, the objective of the evaluation is to 

address the following questions:

• What are the longer-term benefits of NIDRR funding for individuals with disabilities and their 

caregivers, the research community, federal partners, and other stakeholders?

• How can NIDILRR administer its programs most efficiently?

Each year, the evaluation will focus on a portion of the NIDILRR program portfolio and on an assessment 

of some aspect of the grant-making process, such as strategic planning, internal management, or peer 

review. In year 1 of the 5-year evaluation we examined three programs: the Disability and Business 

Technical Assistance Centers, Knowledge Translation, and Small Business Innovation Research. In 

addition, we evaluated participant satisfaction with the entire application process and with the funding 

level and duration. 

Our findings were based on several data collection activities. First, we conducted an on-line survey of all 

Principal Investigators (PIs) funded by three programs being reviewed—to determine whether they had 

been able to continue the work funded under the grant and to document the benefits of the grants. 

Second, we interviewed the users of NIDILRR funding, other than the grantees—to validate and 

supplement the information on the benefits provided by PIs. Third, we conducted bibliometric analysis 

 

1 http://www.acl.gov/Programs/NIDILRR/About/About.aspx#mission
Abt Associates NIDILRR Evaluation Report Year 1 2

of publications citing KT grants—to characterize PI productivity and influence. Fourth, we performed 

web searches for the companies cited by SBIR grantees—to collect additional information on 

commercialization. Finally, we surveyed all recent NIDILRR applicants—to determine the strengths and 

weaknesses of the application process.

Our methodology, findings, and conclusions are presented in the rest of this report. The report also 

includes data collection instruments (Appendix A), the list of users interviewed (Appendix B), and 

lessons learned related to the evaluation design (Appendix C).
Abt Associates NIDILRR Evaluation Report Year 1 3

Methods

Our procedures for data collection and analysis are described below. For additional details on the 

evaluation design, please refer to the Evaluation Plan and Toolkit for the National Institute on Disability 

and Rehabilitation Research: Final Report submitted by Abt Associates Inc. to NIDILRR on January 29, 

2015.

Data Sources

Grant data. Grant data were obtained from the National Rehabilitation Information Center (NARIC) and

included: PI name, institution, and contact information; program name; and funding year. For the SBIR 

program, several grants had to be excluded because the same individual was a PI on multiple grants and 

we assumed that they will not be willing to complete more than one survey. In cases when choices had 

to be made, SBIR Phase II was selected over Phase I and older grants over newer ones. NIDILRR staff 

provided us with the names and contact information for all individuals who submitted an application in 

2015.

Publications. Web of Science and PubMed databases were queried using NIDILRR grant numbers to 

obtain publication and citation information. SCI Journal Rank Indicator was used as a measure of journal 

impact.

Surveys. The surveys were a combination of multiple-choice and open-ended questions. In PI surveys, 

we explored funding status of the projects, their longer-term outcomes, and challenges experienced by 

PIs. In ADA and KT surveys we asked respondents to provide us with the names and contact information 

for individuals who could speak to us about the benefits of NIDILRR funding. In the SBIR survey, we 

asked respondents to refer these individuals to us, because OMB reviewers were concerned about

providing contact information for someone without his/her knowledge. In the applicant survey, we 

examined satisfaction with various aspects of the process and willingness to apply to NIDILRR in the 

future. We closely collaborated with NIDILRR program managers in developing all instruments. The 

surveys were pilot tested by Abt researchers to ensure the clarity of the questions and the logic of flow.

Interviews. User interviews included questions about how NIDILRR funding had helped them, their 

organizations, and/or individuals with disabilities. 

Study Participants

The study included PIs on the DBTAC grants funded in 2001-2011 (N=10), KT grants funded in 2005-2011 

(N=7), and SBIR I and II grants funded in 2007-2014 (N=58). DBTAC grants are still active; SBIR and KT 

grants have been completed.

PI surveys yielded contact information for 49 users (15 for KT, 2 for SBIR, and 32 for DBTAC), all of whom 

were contacted for an interview.

Data Collection Procedures

All data collection protocols and procedures were approved by OMB and by Abt IRB.
Abt Associates NIDILRR Evaluation Report Year 1 4

Surveys. Because some NIDILRR grantees apply to multiple programs, respondents were asked to 

consider only the specific grant presented to them when answering the questions. For purposes of 

verification, respondents were also asked to confirm that they were the PI on the listed grant and those 

who said that they were not exited the survey. The applicant survey was conducted in two stages. In the 

first stage we surveyed half of the applicants before they had been notified on the status of their 

submission and in the second stage the other half after they had been notified. This approach was used 

out of concern that respondent satisfaction with the application process might be colored by the 

competition outcome. The applicants were randomly assigned to each group, and minor adjustments 

were made to ensure consistent programmatic distribution.

The surveys were programmed in FluidSurveys software and tested by several Abt staff. Any errors and 

glitches identified were corrected. A survey screenshot is shown in Figure 1.

Figure 1: A Screen from the SBIR Survey

An initial invitation explaining the survey goals and containing a link to the website was sent to all 

subjects using the emails on file at NIDILRR. Web searches were made in an attempt to find alternative 

addresses for the emails that could not be delivered. A specially created email 

(NIDRR_Evaluation@abtassoc.com) was provided to the survey subjects who wished to contact us, and 
Abt Associates NIDILRR Evaluation Report Year 1 5

the account was monitored daily. We received several communications2 about the survey, but no 

problems were reported. The surveys took about 10 minutes to complete.

Four to five reminders were necessary to increase response rate. The surveys were closed when it 

became apparent that additional reminders would produce zero or very few responses. Data collection 

dates were as follows:

• KT PIs: August 3 – September 18, 2015

• SBIR PIs: August 3 – September 17, 2015

• DBTAC PIs: August 3 – September 10, 2015

• Applicant Phase I: September 9 – September 30, 2015

• Applicant Phase II: October 7 – November 2, 2015

Interviews. We emailed all individuals who were suggested by PIs or who contacted us directly and 

interviewed everyone who responded to our request. All interviews were conducted by telephone and 

took approximately 15 minutes. 

Data Analysis

Survey data were cleaned, coded, and prepared for analysis. Descriptive statistics were used for 

numerical data. Published outcome data for SBIR programs across the federal government were used for 

comparison. Statistical tests (t-test and χ2

) were performed to examine the differences between groups. 

Qualitative survey and interview data were coded and analyzed using NVivo 10.0.

 

2 One PI said that it has been too long since the project was active; a few applicants asked whether the survey 

meant that they will/will not receive the grant; and a few respondents referred us to someone else.
Abt Associates NIDILRR Evaluation Report Year 1 6

Findings 

Findings for the evaluation Year 1 for the DBTAC, KT, and SBIR programs and the application process are 

presented in this chapter. It is important to note that the evaluation was not designed to be 

comparative across programs, because of the differences in goals, duration, size, and funding amounts. 

Consequently, all findings are presented by program, beginning with an overview of the PI survey 

findings, followed by results of the user interviews. For the SBIR program, we supplement these data 

with information on several products in the marketplace obtained from web searches. For the KT 

program, we include bibliometric analysis. We conclude the chapter with the findings from the applicant 

survey. 

Disability and Business Technical Assistance Centers

DBTACs provide information, training, and technical assistance to state and local governments and to 

private businesses to facilitate compliance with the Americans with Disabilities Act (ADA). All 10 regional 

DBTACs managed by NIDILRR were included in the study, and all responded to the survey (100% 

response rate). 

Findings from the PI Survey

When asked whether they were able to obtain additional funding to support their activities, half of

DBTACs responded in the affirmative (data not shown). The funders included City of Houston, National 

Disability Institute, Rehabilitation Services Administration, ADA25 Chicago, US Access Board, and JFK 

Center for Performing Arts. These grants supported training and technical assistance services to respond 

to hurricanes, training mid-level managers, improving partnerships between employers and placement 

providers, and increasing knowledge of ADA at non-profit and arts/culture organizations. In most cases, 

respondents were not PIs on these grants, however (data not shown).

Survey participants were offered a choice of four types of advancements and asked to indicate whether 

any of these had occurred as a result of dissemination, technical assistance or training activities 

provided by DBTAC. Figure 2 shows that improved understanding by the public of the needs/challenges 

experienced by people with disabilities was the most frequent outcome, chosen by almost all PIs. This 

was followed by new/improved policies, reported by 50–80% depending on the type of activity, and by 

tools/devices/approaches to identify/confirm a disability, reported by 40%. Creation of new laws 

appeared to be the least common outcome, reported by 20–30% DBTACs. As can be seen from Figure 3, 

the benefits for individuals with disabilities were reported in all areas, but were especially common in 

communication and community/social/civic life.
Abt Associates NIDILRR Evaluation Report Year 1 7

Figure 2: Findings from PI DBTAC Survey on Advancements due to DBTAC Funding (N=10)

Have any of the following advancements occurred, at least in part, as a result of NIDILRR funding? If yes, 

please indicate which grant activity (training, TA, and/or dissemination) brought about this change.

Select all that apply.

Figure 3: Findings from the PI DBTAC Survey on Benefit Areas (N=10)

In which of the following areas have the advancements occurred? Select all that apply.

DBTAC PIs offered many examples to illustrate the value of the centers to people with disabilities. Table 

1 lists laws and policies that were revised or created as a result of DBTAC activities. Other contributions 

included accessibility audit checklist for correction and detention facilities, community engagement 

initiative to improve access to healthcare services in rural areas, capacity building to enable people with 

disabilities to better advocate for what they need, training of architects and employers, and improved 

emergency preparedness. DBTAC PIs indicated that the work of DBTACs resulted in greater access to the 
Abt Associates NIDILRR Evaluation Report Year 1 8

community, both physically and in terms of communication; full participation in civic responsibilities; 

enjoyment of entertainment and food; and neutral and safe access to information.

Table 1: Laws and Policies Revised or Created as a Result of DBTAC Funding Mentioned by 

Survey Respondents (N=10) 

Law or Policy Goal

Rehabilitation Act Section 503 Modification of state policy to support 

employment services

California Civil Codes Implementation of ordinance to create on-street 

parking options

Arizona State Revised Statues Modification of law that governs placing placards

Service Animal Policies Employer policies on reasonable accommodation 

and access to technology in Sonoma and Yolo 

Counties

Each item was mentioned by one respondent

PIs reported several challenges in running DBTACs. One of these, mentioned by 3 of 10 respondents, 

was being able to reach a large geographic area under their jurisdiction and become known as a 

resource. In addition, DBTACs find it difficult to determine how the resources that they provide are used 

and to measure their impact:

The greatest challenge to accomplishing the goals we established for the ADA Center has actually 

been around getting people to report on and quantify the impact that our work, specifically, has had 

on the changes that they make in policy, practice and behavior.

Finally, three PIs experienced challenges in finding and recruiting qualified staff.

At the conclusion of the survey, we encouraged PIs to share with us any other comments about the 

importance of DBTACs. Two points were made. The first was that the Americans with Disabilities Act was 

enacted 25 years ago and yet the need remains for continued outreach, training, and technical 

assistance to individuals, businesses, and organizations about their roles and responsibilities in 

complying with this complex law. This is how one PI characterized the utility of his DBTAC:

The design community relies on us for accurate information affecting the built environment, the 

government policy makers at the local, state, and national level require our assistance in creating 

inclusive programming, the business community demands our services for a myriad of issues from 

facility access to communication access to all the ramifications of employment.

The second point was that DBTAC’s regional infrastructure enabled each Center to tailor their services:

Because the ADA Centers are regionally based, we are able to develop partnerships with key 

stakeholders across our region to actually focus on the issues that are important to them. For 

example, right now in the Virgin Islands they are really struggling to ensure that transportation is 

accessible to everyone. We were able to put together fact sheets and training that focused on 

transportation, to equip the trainer network to provide training locally and offer the support of our 

affiliate to both parties to ensure that someone was there who understood the culture and climate 
Abt Associates NIDILRR Evaluation Report Year 1 9

who could provide practical information. In NY and NJ, which is the hub for so many large businesses, 

we are able to tailor solutions and interventions that will meet their needs.

Findings from the DBTAC User Interviews

In response to our request, DBTAC PIs provided us with the names and contact information for 32

individuals who could speak with us about the benefits of DBTAC activities. Of these, 19 agreed to be 

interviewed.3

In terms of their professional roles, interviewees fell into three groups. One group was employed at the 

organizations exclusively serving the needs of people with disabilities, such as Bay Area Center for 

Independent Living or Office of Vocational Rehabilitation. The second group was independent 

consultants serving businesses or people with disabilities (architectural consultant and accommodation 

consultant). The third group included employees at non-dedicated organizations, but who were 

responsible for issues related to special accommodation (Regions Bank and Kennedy Center for 

Performing Arts). Finally, one respondent was an architecture professor, who teaches special 

accommodation design. Several individuals interviewed self-identified as having disabilities. Complete 

list of respondents and their roles can be found in Appendix B. 

In the interviews, we asked in what way DBTACs had helped the subjects, their organizations, and/or

individuals with disabilities. While each interviewee discussed specific assistance they had received, 

several themes emerged.

Information

Almost all respondents use DBTACs as a source of information to help interpret the provisions and 

interpretation of the Americans with Disabilities Act, which was described as wide-ranging, complicated, 

and ambiguous in certain areas. DBTACs were viewed as a reliable and unbiased resource, which was 

necessary even for someone with many years of experience in the disability field. Information sought 

from DBTACs was related to employment, technologies, and accessibility/accommodation requirements.

When asked about alternative sources, all respondents said that DBTACs were the best, and perhaps the 

only place to obtain information. We were told that while the Department of Justice should, in theory,

provide similar information and guidance, their opinions could be biased, answers inconsistent, and 

materials difficult to obtain. In contrast, interviewees praised the knowledge and experience of DBTAC

staff, as well as their helpfulness and responsiveness. For example, staff would research the question for 

which they did not have an immediate answer and get back to the requester with information within a 

day. Respondents had also referred their colleagues to DBTACs, who had similarly positive experience. 

Interviewees offered several examples of the questions they posed to the Centers, which included 

responsibilities of libraries to provide headsets, requirements for routs and automatic doors, and 

medical documentation for service animals. 

In addition, some respondents have encouraged people with disabilities/their families to contact 

DBTACs. In one case, a man with cognitive disabilities had been refused an oral exam to enter the Coast 

 

3 Of these, 4 bounced back.
Abt Associates NIDILRR Evaluation Report Year 1 10

Guard. At the urging of our respondent (a disability advocate), the family contacted DBTAC, which 

advised them about their rights, and a reasonable accommodation was made by the Coast Guard for 

their son, who was able to join. This experience was described by the family as “life changing.”

Education and Training

The second most frequently mentioned service provided by DBTACs was education and training. Several 

interviewees praised the annual conference organized by the centers, which gave them the opportunity 

to learn about the latest developments related to the Act and to network with like-minded colleagues. 

One respondent said that this was the best conference she has ever attended among numerous others, 

that it inspired her work, and that she looked forward to it every year. Several respondents mentioned 

training to become a certified ADA Coordinator offered by the centers, which they had completed. This, 

in turn, helped them become knowledgeable about the disability issues and obtain employment in this 

field. One respondent (an individual with disability) completed a paid internship at DBTAC, which 

enabled her to get a job as a disability advocate. She also described her experience with the center as 

“life changing.”

Training materials received from DBTACs were characterized by respondents as high quality and useful. 

For example, we were told of the materials which not only included a PowerPoint presentation, but also 

a guide which described how it can be tailored to the needs of different audiences, what to focus on in 

each segment, and how much time to spend on each topic. An employment specialist from the 

Pennsylvania Office of Vocational Rehabilitation recounted presenting DBTAC materials about disability 

etiquette to a local business group. He said that it was clear that the listeners had never considered how 

to best assist someone with a disability and found the seminar very helpful. An Executive Director of Bay 

Area Center for Independent Living told us how he uses the modules created by DBTAC to train local 

businesses, government agencies, and educational institutions about accommodation needs of people 

with disabilities.

A design professor who uses DBTAC for experiential student training described how the center staff in 

wheelchairs came to talk to the students about special accommodation design and how important it was 

for the students to meet someone who had personal experience with disability. This respondent said 

that DBTAC was essential to her training program and that there was no other resource to teach 

accessibility design.

Technical Assistance

Finally, over half of the interviewees spoke of the technical assistance received from the centers, which 

was typically related to the issues of accessibility of facilities (for example, advice where to put a ramp 

or a sign) as well as websites and other online materials (for example, tests for 508 compliance). In 

addition, DBTACs helped respondents revise organizational policies and procedures to better meet the 

needs of people with disabilities. For example, the Kennedy Center for Performing Arts regularly offers 

free tickets to events, but to receive them requires waiting in line, which might be difficult for someone 

with a disability. The Kennedy Center consulted its local DBTAC, which advised them to give away two 

tickets instead of one, so that people with disabilities can send a proxy. The Center also began making it 

clear to the visitors that they do not need to stand in line, but can bring or borrow a chair. The DBTAC

also helped the Kennedy Center revise a “stand only” policy for elderly volunteers working in a gift shop. 
Abt Associates NIDILRR Evaluation Report Year 1 11

In conclusion, all of the 19 interviewees were grateful to DBTACs, saw them as partners and as an 

invaluable and unique resource. For example: 

I can’t think of a better use of funding. There is a constant influx of new people with disabilities, and 

to have a knowledgeable and credible source of information is critical, and the impact is huge.

The Center has an outstanding reputation in the design community which is unequaled. Staff is 

timely, responsive and proactive. They consult with faculty and students on a regular basis as well as 

our design community. Staff regularly comes to classes and lead experiential training. In all cases 

they are professional, knowledgeable and a valuable resource to all people. They help all of us 

become more sensitive to needs for persons with disabilities. 

Knowledge Translation Program

The Knowledge Translation (KT) Centers conduct research on strategies to successfully disseminate and 

commercialize disability and rehabilitation research. Between 2005 and 2011, NIDILRR funded seven 

centers. A survey was sent to all PIs on these grants; five completed and one partially completed the 

survey (response rate of 71%). Answers from all six were included in analysis.

Findings from the PI Survey

Four of six respondents (67%) received additional funding to support the activities that have begun 

under the KT grant (data not shown). Sources of support included NIDILRR (N=3) and MS Society (N=1).

We asked respondents whether they engaged in various knowledge translation and technology transfer 

(TT) activities related to their grant since the end of the grant. According to the PIs, they continue to 

maintain resources developed with NIDILRR funding (N=5); disseminate information about NIDILRRfunded research (N=4) and about challenges/best practices in KT/TT (N=4); and provide advice, training, 

and assistance related to KT/TT (N=4, Figure 4).

Responding PIs were asked to provide examples of these activities. One PI mentioned graduate course 

on ADA implementation. Another wrote that they took tailored knowledge packages generated within 

three RCT studies performed in collaboration with NIDILRR grantee and shared them with companies 

operating within the relevant fields. Four of the six PIs continue to provide advice and technical 

assistance to researchers. Examples included: suggestions and feedback to RERCs on their mandatory TT 

plans, service as an advisor to RRTC on KT issues; assistance to SBIR and RERC with intellectual property 

issues; and providing sample commercialization, value proposition, and contextualization knowledge 

packages to grantees. In addition, PIs have given KT and TT advice non-NIDILRR US researchers and to 

international researchers at King’s College, UK; Ben Gurion University, Israel; Hanze University, 

Netherlands; and Malmo University, Sweden. One respondent mentioned assisting NIH and PCORI.
Abt Associates NIDILRR Evaluation Report Year 1 12

Figure 4: Activities since the End of Grant Reported by KT PIs (N=6)

Since the end of your KT grant, have you engaged in any of the following activities related to that grant? 

Select all that apply.

Five of the PIs said that they have maintained the resources developed with NIDILRR funding, which 

included:

• NEARP database,4 which is being updated every four years and currently includes nearly 500,000 

closed investigations of ADA discrimination

• Model Systems Knowledge Translation Center web page,5 which is being updated as new 

products are developed

• Database that supports Need to Knowledge model,6 a guide to innovation for technology-based 

commercial devices and services that connects the academic research process with industry 

standards

• Repository of Resources for the Center for Psychiatric Rehabilitation7

• CIRRIE Database of International Rehabilitation Research,

8 the CIRRIE International Encyclopedia 

of Rehabilitation,9 and other resources on the CIRRIE web site. 

 

4 Web site was not found

5 http://www.msktc.org/

6 http://kt4tt.buffalo.edu/knowledgebase/model.php

7 http://cpr.bu.edu/resources/recovery-repository

8

 http://cirrie.buffalo.edu/database/ 
Abt Associates NIDILRR Evaluation Report Year 1 13

The PIs continue to disseminate information about NIDILRR-funded research. Two examples provided in 

the survey were 400+ downloads of publications resulting from the KT project and a DVD entitled Facts 

and Stubborn Things: ADA Amendments and the Employment of Americans with Disabilities, of which 

several thousand copies had been distributed at no cost to employment organizations and rehabilitation 

providers. Other activities that have built on the NIDILRR funding were participation in a working group 

on reimbursement issues in assistive technologies and a workshop on strategies to link related initiatives 

across sectors and countries.

Finally, 4 (67%) respondents said that the resources developed with NIDILRR funding have been used in 

areas other than disability and rehabilitation. For example, research methods in rehabilitation

developed for the Equal Employment Opportunity Commission have been applied to the study of race, 

gender, and age. Another respondent wrote that “the materials have been useful to policy makers and 

managers at the NSF’s Center for Science and Engineering Statistics, Department of Transportation, nonprofit foundations (e.g. Center for American Progress & Information Technology Innovation Foundation), 

and private sector organizations (e.g. the Product Development Manager's Association and the 

Technology Transfer Society). The European Union, Brazil and Australia have also been applying the 

concepts within their own science, technology and innovation policies.”

Respondents were given a choice of several outcomes that could have resulted from their projects and 

asked to indicate which, if any, had occurred at least in part due to the KT funding. Four of the six PIs 

reported increased familiarity with systematic reviews and other rigorous research methods and three 

selected increased understanding of KT/TT by NIDILRR grantees and other researchers (Figure 5). Other 

outcomes included increased awareness of NIDILRR-funded R&D (N=4); better understanding of the 

commercial landscape and commercialization process (N=2 and N=1, respectively); and increased use of 

valid and relevant findings to inform policy and practice (N=2). Respondents were asked to provide

specific examples of these outcomes. These included application of KT and TT methods by other US 

agencies and international organizations, more persuasive discussion of the commercial landscape by 

NIDILRR grantees in applications, evidence that increasing percentage of NIDILRR products are reaching 

a marketplace, use of findings in the passage and regulatory development process around the ADA 

Amendments Act if 2008, full compliance in presenting a TT plan by 2015 RERCs, and a review of peerled interventions adapted for SAMHSA.

 

9 http://cirrie.buffalo.edu/encyclopedia/
Abt Associates NIDILRR Evaluation Report Year 1 14

Figure 5: Outcomes of NIDILRR Funding Reported by KT PIs (N=6)

Have any of the following outcomes occurred since the end of your KT grant and at least in part as the 

result of the work on this grant? Select all that apply.

All KT PIs had maintained collaborations established during the funding period (Figure 6). The most 

common partners were health care providers (N=6), followed by researchers and advocates (N=5). PIs 

also continue to collaborate with vocational rehabilitation professionals, business organizations, 

federal/state/local agencies, and individuals with disabilities/their caregivers (N=4). The least frequent 

partners included employers (N=3), educators (N=3), and funders (N=2). As examples of the most 

important collaborations established during the grant period respondents included the Equal 

Employment Opportunity Commission, NIDILRR management and staff, and US Business Leaders 

Network. 
Abt Associates NIDILRR Evaluation Report Year 1 15

Figure 6: Collaborations Resulting from NIDILRR Funding Reported by KT PIs (N=6)

Please indicate with which of the following groups you have maintained collaborations established 

during the funding period. Select all that apply.

KT PIs were asked to share with us the challenges to accomplishing their grant goals and four responses 

were provided. Three individuals reported interacting with NIDILRR grantees as a challenge. It appears 

that the leadership of DBTACs and Model System centers showed little interest in the KT projects. PIs 

also felt that grantees, peer reviewers, and NIDILRR staff did not appreciate the time, money, and effort 

requited for knowledge translation. One respondent noted that publications are the ultimate measure 

of success in academia, and thus commercialization was a low priority for grantees. Finally, a KT PI said 

that NIDILRR grantees were not aware of the services and assistance that can be provided by the KT 

centers. 

Findings from the KT User Interviews

In total, 15 contacts were provided10 by the KT PIs and all were invited to participate in a short 

interview. Eleven individuals agreed and were interviewed. Below are examples of benefits reported by 

individuals who used the resources or expertise offered by the centers.

The Director of Research at the US Equal Employment Opportunity Commission that EEOC maintains a 

database of discrimination claims, which is being analyzed by one of the KT PIs. His analysis revealed 

several important trends, for example that people with cognitive disabilities are more likely to file 

harassment charges. This type of analysis had a big impact on the strategic planning process at EEOC, 

including what policy and practice issues to focus on in the future. Another investigator working on this 

grant believed that KT research informed the 2009 amendments to the ADA.

 

10 Of these, one email bounced back
Abt Associates NIDILRR Evaluation Report Year 1 16

One of the KT grants sponsored a WHO summit in Washington, DC about the state of the art in 

rehabilitation. Based on the proceeding from the summit, the United Arab Emirates designed a new 

school for children with disabilities.

An SBIR grantee received significant commercialization assistance from a KT center, which included

focus groups with potential users of technology organized by the center. The SBIR grantee said that 

without the KT center his company “would have made very expensive and perhaps disastrous mistakes.”

KT Center linked the Assistive Technology Industry Associates (ATTC) to SBIR grantees. The Association

continues to work with grantees to disseminate information about technologies and puts them in touch 

with manufacturers to help bring technologies to market. These relationships are especially important in 

the disability areas whether the market is not well established. In addition, ATTC provides funding to 

NIDILRR grantees to independently determine the benefits of specific technologies.

A university professor collaborated with a KT grantee to survey HR managers to understand their 

attitudes to hiring individuals with epilepsy. These data were published and highly cited. The PI has

received follow-up funding to develop a program to train employers.

Three researchers received a travel grant through the program called CIRRIE, run by the University of 

Buffalo received a Knowledge Translation Center. The grant covers airfare for US rehabilitation 

researchers to travel internationally and for foreign researchers to visit the United States to exchange 

information and expertise. All three participants interviewed said that the program broadened their 

experience, enabled them to make international connections, and led to papers or other published 

materials. For example, one respondent received a travel award to East Asia to conduct a needs 

assessment of programs and services available for individuals with spinal cord injuries. Based on this 

information, he developed training materials for patient councilors, which include information on 

councilor qualifications, skills, and advice for how to deal with clinically difficult situations.11

Analysis of Grantee Publications

We used KT grant numbers to query PubMed and Web of Science for papers that cited these grants. In 

total, 23 papers were obtained; of these, 11 were published after the official end date of the grant, 

although some of the grants may be under a no-cost extension (Figure 7A). All but two papers were 

cited between 11 and 18 times per year (Figure 7B). A paper called Treatment for Depression after 

Traumatic Brain Injury: A Systematic Review published in 2009 received a total of 107 citations.12 SCI 

Journal Rank Indicator (SJR) is a measure of journal impact, influence, and prestige. We obtained SJRs for

the KT journals in our sample and compared them to all journals in the rehabilitation subject category.13

 

11 In 2010, an external evaluation was conducted to determine the benefits of the grant to the participants and 12 

of the 18 researchers funded were interviewed. All participants stated that the experience broadened their 

horizons and several had added international content to their lectures and seminars. Many of the students of 

these respondents subsequently also sought international experience. CIRRIE participants also reported that the 

trips led to collaborations, including with government agencies at foreign countries.

12 Fann JR, Hart T, Schomer KG (2009). J Neurotrauma. 2009 Dec;26(12):2383-402.

13 http://www.scimagojr.com/journalrank.php 
Abt Associates NIDILRR Evaluation Report Year 1 17

We found that the KT grantees publish in significantly better journals than the comparisons: SJR of 1.4 

versus 0.4 (p<.0001, data not shown).

Figure 7: Bibliometric Analysis of Publications Linked to KT Centers for 2005-2011

A. B.

Small Business Innovation Research Program

SBIR is a federally mandated program to assist researchers in commercializing their invention. The 

program has two tracks—Phase I and Phase II. The goal of Phase I ($75,000 for six months at NIDILRR) is 

to develop and test prototype technologies and products. Grantees who meet the benchmarks of Phase 

I program are eligible to apply for Phase II ($250,000 per year for two years) to bring their products to 

market. 

Findings from the PI Survey

Surveys were sent to 58 SBIR Phase I and/or Phase II grantees who participated in the program between 

2007 and 2014 and 26 responses were received. Response rates were highest among the grantees who 

received both Phase I and II awards (69%, Table 2). 

Table 2: Response Rates for SBIR Survey for Grantees between 2007 and 2014 (N=58)

Sent Responded Response Rate

SBIR I 39 16 41%

SBIR II 6 1 17%

SBIR I/II 13 9 69%

Total 58 26 46%

Of 26 SBIR PIs who responded to the survey, 18 (70%) had received additional development funding 

(Figure 8). Comparison to other SBIRs across the federal government revealed that NIDILRR grantees 

were more successful than those at NIH, DOD, and NASA. By far the most frequent funding source for 

NIDILRR grantees was NIDILRR SBIR Phase II, reported by 56% of PIs, with non-SBIR federal funding being 

a distant second at 17% (Table 3). At other agencies, approximately one-third of grantees reported 

receiving another SBIR award and between 7% and 50% a non-SBIR federal grant (data not shown).

NIDILRR grantees may be more reliant on funding from NIDILRR because of the relatively small size of 

the disability/rehabilitation market and correspondingly limited interest from private investors. 
Abt Associates NIDILRR Evaluation Report Year 1 18

Figure 8: Percent of SBIR I/II Grantees Receiving Additional Development (N=26) Compared to 

other Federal SBIR Programs

Source of comparison data: An Assessment of the Small Business Innovation Research Program at the National Institutes of 

Health (NRC, 2009), the Department of Defense (NRC, 2009), the National Aeronautics and Space Administration (NRC, 2009), 

and the National Science Foundation (NRC, 2007).National Survey to Evaluate the NIH SBIR Program (NIH OER, 2009). 

Table 3: Follow-up Funding Sources for NIDILRR SBIR I/II Grantees (N=26)

Source Percent of PIs

State, local, or tribal government 0 

Private investment 0

Contract 6

Device manufacturer 6

SBIR Phase II from another agency 11

Investment from the company selling the product 11

Personal funds 11

Non-SBIR federal funds 17

SBIR Phase II from NIDILRR 56

Survey respondents were asked whether they would have undertaken the project without NIDILRR 

funding. Twenty three percent reported that they definitely would not and an additional 42% that they 

probably would not; the rest were uncertain (Figure 9). The fraction of “definitely nots” at NIDILRR was 

the lowest across all federal agencies considered, although when combined with “probably not” these

differences largely disappeared. Between 13 and 18% of grantees from other agencies said that would 

definitely or probably undertake the project, while none at NIDILRR selected these options. These data 

indicate that NIDILRR grantees were not confident that they could support commercialization of their 

products without NIDILRR funding.
Abt Associates NIDILRR Evaluation Report Year 1 19

Figure 9: Likelihood of Undertaking the Project without SBIR Funding Reported by NIDILRR 

SBIR I/II PIs (N=26) Compared to other Federal SBIR Programs

Would you have undertaken the project without NIDILRR funding? Select one.

NSF=National Science Foundation, NASA=National Aeronautics and Space Administration, DOD=Department of Defense

NIH=National Institutes of Health

Seven of 10 SBIR II PIs reported developing a product as a result of NIDILRR funding, three of which were 

in the marketplace; none of the products had been discontinued at the time of the survey (Figure 10).

These outcomes compared favorably to other agencies. Sales to date for three NIDILRR products were 

<$100K (N=2) and $100K-$1M (N=1). Two of the companies had 1–5 employees and one 6–10 

employees (data not shown). 

Figure 10: Commercialization Outcome of SBIR II Funding Reported by NIDILRR PIs (N=10) 

Compared to other Federal SBIR Programs
Abt Associates NIDILRR Evaluation Report Year 1 20

Finally, SBIR I and II grantees were asked to list their most important commercialization outcomes. Of 

the 26 respondents, 8 (31%) described a product, 7 (27%) a concept, 6 (23%) a prototype, and 5 (19%) 

feasibility/usability testing (Table 4). Examples of commercialization outcomes are included in Table 4. 

Table 4: Findings from SBIR Survey re: Most Important Commercialization Outcomes 

Reported by SBIR I/II PIs (N=26)

Program

Concept (N=7)

Gained expertise in wheelchairs Phase I

Showed a correlation between simulated changes in residual limb volume and 

pressures measured at strategic locations

Phase I

Uniting GPS and text/audio content to enhance travel experience for blind people Phase I

System of online supervision and training toward BCBA credential Phase I

Technology to help scoliosis patients know that their wearing their brace properly Phase II

Better understanding of what makes synthetic speech sound natural Phase I

Proof-of-concept prosthetic hand Phase II

Prototype (N=6)

Software Phase I

Behavior assessment tool Phase I

System to protect wheelchair seated individuals in large transit vehicles Phase I

Device for visually impaired people to operate office equipment Phase I

Virtual, interactive, social interaction tool Phase I

Development of watercraft transfer access prototypes Phase I

Portable rehabilitation device with intelligent control to stretch spastic joints Phase I

Feasibility and usability testing (N=5)*

Feasibility testing of crowed-sourced database to aid blind individuals using 

manual wheelchairs in route determination

Phase I

Validated feasibility of using mobile device and remote computer to provide 

automated image recognition-base assistance to blind people

Phase I

Confirmed that telecoil hearing aid solution is better optimized for both 

telephone usage and public induction loops

Phase I

Feasibility and benefits of being able to change the drive wheel position to 

optimize power wheelchair

Phase I

Product (N=8)**

Sensory feedback system for persons with lower-limb sensory deficits (FARUS LLC) Phase II

Cognitive support technology (AbleLink Technologies) Phase II

STEM Binder (Touch Graphics Inc) Phase II

Dialysis regeneration system (Chemica Technologies) Phase I

Automatic book scanner for universal access (PageFlip) Phase I

Software tool for the automatic configuration of computer keyboard (Koester 

Performance Research)

Phase I

Web screen reader (Charmtech Labs LLC) Phase I

Picture Planner (Cognitopia) Phase II

*One respondent did not specify the nature of the testing

**Business name shown in parenthesis
Abt Associates NIDILRR Evaluation Report Year 1 21

Respondents reported several challenges to accomplishing the goals of the SBIR grant. The most 

common (N=10 or 42%) was related to funding, including low amounts for Phase I and obtaining funding 

for Phase II (Figure 11). Funding concerns were closely followed by technical challenges (N=9 or 28%). 

Three respondents reported that duration for Phase I was too short to develop a prototype. Other 

challenges included developing a system that fully complies with ADA regulations, obtaining insurance 

code, limited support for setting up a business, and using the online system to get project funds 

released. One respondent reported no challenges.

Figure 11: Challenges Experienced by Grantees Reported in the SBIR PI Survey (N=24) 

At the conclusion of the survey, we asked respondents to describe any other benefits of NIDILRR 

funding. We received 18 comments, of which 17 were positive (the remaining respondent wrote that 1-

year funding duration would be helpful). Grantees said that the program enabled them to apply their 

experience in a different field, develop and test products, and ultimately serve people with disabilities. 

Below are examples of testimonies about the importance of NIDILRR funding:

We have been able to change many lives for the better as a result of funding provided by NIDILRR. 

NIDILRR funding has been the difference between continuing on in business and going out of 

business altogether—more than once! Niche markets for disability populations take much longer to 

establish than mainstream markets. Therefore, NIDILRR plays an extremely critical part in helping 

small businesses survive.

The NIDILRR support really was a gating factor in starting the work on Handsight. We then went on 

to Phase II with the NEI, and created a number of apps for iPhone and Android, which had over 

100,000 downloads.

The NIDILRR SBIR program has been crucial to helping us establish a company and go through the 

risky stage in which few investors would ever commit a sufficient amount of funding.

NIDILRR funding has been crucial to us and has produced real benefits for the individuals with 

disabilities whom we've reached directly through our products as well as those whom we believe 

we've influenced through our dissemination efforts.
Abt Associates NIDILRR Evaluation Report Year 1 22

Products Mentioned by SBIR Grantees

We examined the web sites for eight products mentioned by grantees (Table 4). Two—FARUS and

Chemica Technologies—had websites, but no information was contained within these. The remaining six 

are presented below.

AbleLink Technologies (www.ablelinktech.com) 

Established in 1997, the company develops cloud-based technologies to support individuals with 

cognitive disabilities. AbleLink received 24 SBIR I and II grants between 2007 and 2014 and NIDILRR 

funding is acknowledged on the company’s web site. AbleLink technologies help the users with 

navigation, scheduling, skill development, and other activities of daily living and functioning. For 

example, WayFinder helps people with intellectual disabilities travel safely and Everyday Skills and Living 

Safely has self-directed learning sessions to teach the users how to live independently (Figure 12). The 

company also receives development funding from DOD, NIH, NSF, universities, and foundations.

Figure 12: Examples of AbleLink Products
Abt Associates NIDILRR Evaluation Report Year 1 23

Touch Graphics Inc. (touchgraphics.com)

Established in 1997, the company develops tactile products and exhibits for visually impaired people.

Touch Graphics received an SBIR I grant in 2008 and SBIR II grant in 2009. While a number of products 

are available from the company, Stem Binder was attributed to NIDILRR in the survey and on the web 

site. Stem Binder enables people with limited vision to touch an object on a map, a diagram, or game 

with a stylus and to hear information about it.

Figure 13: Touch Graphics Product

PageFlip (www.pageflip.com)

Established in 2002, the company develops, manufactures, and commercializes automatic page turning 

technology for people with disabilities. PageFlip received an SBIR I grant in 2007

Figure 14: PageFlip Products
Abt Associates NIDILRR Evaluation Report Year 1 24

Koester Performance Research

Established in 1996, the company designs and sells software products for the automatic configuration of 

computer keyboards to assist people who cannot easily use computer keyboard and mouse. Koester 

received an SBIR I grant in 2013.

Figure 15: Koester Performance Research Products

Charmtech Labs LLC (www.captivoice.com/capti-site/public/entry/about_us) 

Established in 2010, the company sells software that allows people with vision impairments to use the 

web by listening to what they want to read. Initially focusing on individuals with disabilities, the 

company is bringing its screen-reading technology to the mainstream market. CharmTech received 3 

SBIR I grants and 2 SBIR II grants between 2009 and 2013.

Figure 16: Charmtech Product
Abt Associates NIDILRR Evaluation Report Year 1 25

Cognitopia (www.cognitopia.com)

The company develops and markets computer-based task management application to help individuals 

with cognitive disabilities live more independent lives. A

common database of information about an individual’s life 

and needs is developed and accessible data delivered to

either the consumer or family/support staff. Cognitopia 

received 4 SBIR I and 3 SBIR II grants between 2008 and 2014.

Figure 17: Cognitopia Product

Findings from SBIR User Interviews

Because of the privacy concerns expressed by OMB, we did 

not ask survey respondents to provide us with contact 

information for individuals who could speak about the 

benefits of NIDILRR funding. Instead, we asked the PIs to 

forward our request to the users of their products with a 

request to contact us. We do not know how many grantees 

made this request; only one person emailed us and was interviewed. She referred us to another user of 

the technology, who was also interviewed.

The first respondent worked with the SBIR PI to develop the software tool, which aims to measure social 

information processing skills in children. Rather than relying on a traditional provider-administered 

assessment tool, a child interacts with a character that he or she had created. Originally, this tool was 

meant for diagnosis of autistic children, but the respondent hopes that it can also be used to help the 

children develop the skills that they need. The tool was completed in 2013 and the respondent is 

working with two community partners to use it: Playwork, a national organization that runs enrichment 

programs and a therapeutic group in NY called ASPIRE. Both partners are using the tool to monitor their 

intervention programs. The second respondent told us that the tool is currently used as part of a 

research study with a community partner, a non-profit clinical service. Lots of interest has been 

expressed by parents and schools in using this tool. It is becoming better known because of enrolling 

control subjects, who can be any child of the same age. Respondent is also working with a school in NYC 

that specializes in teaching children with social challenges. The tool has already had a real impact at that 

school as well. 

This section concludes our findings related to NIDLRR programs. In the remainder of this chapter, we 

present our findings from the survey of applicants.
Abt Associates NIDILRR Evaluation Report Year 1 26

NIDILRR Application Process

With continuing decline in federal funding,14 an applicant’s research program and possibly his/her entire 

career may depend on the outcome of grant competition. Therefore, we hypothesized that survey 

responses may be influenced by the review outcome, especially if an applicant was unsuccessful. If the 

survey were administered before the competition results are known, we would be unable to examine

the applicants’ views of the quality and usefulness of peer review, as this information is provided with

the award status notification. Consequently, we randomly divided all applicants into two groups to 

control for potential bias connected to the review process; one group received the survey before and 

the other after the award announcement. Response rates for two groups were 78% (80 of 102) and 73% 

(73 of 100). Quantitative answers to the survey questions were compared for two groups using a t-test 

or a χ2 test, depending on the question. Since no differences were found in responses of the two groups

(data not shown), the data were combined and reported as one sample (N=153).

The survey included three questions (Appendix A). In Question 1 we asked the applicants to indicate 

their level of satisfaction with various aspects of the process. All respondents who indicated that they 

were dissatisfied or somewhat dissatisfied were offered an opportunity to explain their views using a 

text box. In Question 2 respondents could indicated whether they would apply for NIDILRR funding in 

the future and in Question 3 share any other comments about the application process.

We present the quantitative data on the applicant satisfaction followed by the synthesis of comments. 

Note that only the respondents expressing dissatisfaction could provide comments to Question 1. 

Consequently, these data are negatively biased by design.

Summary of Satisfaction Scoring Data Reported by PIs

Figure 18 shows respondent satisfaction with different application steps, funding level, and duration.

The length of time from application submission to award decision notification was the only variable that 

received a score below neutral (<3.0); quality and usefulness of proposal feedback were the second 

lowest, with 3.2. Most scores were in a 4.0–4.5 range (somewhat satisfied). Table 5 presents the same 

data by percent respondents who were dissatisfied or somewhat dissatisfied with each application step.

While some applicants had clearly experienced challenges, the majority (74%) would apply for NIDILRR 

funding in the future (25% were unsure and 3% would not, data not shown).

 

14 http://www.aaas.org
Abt Associates NIDILRR Evaluation Report Year 1 27

Figure 18: Mean Satisfaction Scores with Grant Application Process, Funding Level, and 

Funding Duration Reported in the Applicant Survey (N=153) 

1=Dissatisfied; 2=Somewhat Dissatisfied; 3=Neutral; 4=Somewhat Satisfied; 5=Satisfied

What is your level of satisfaction with the following aspects of the application process? If you are not 

satisfied, please explain why in the rightmost column which will automatically appear

Bars indicate standard deviation

Table 5: Percent of NIDILRR Applicants Dissatisfied or Somewhat Dissatisfied with Grant 

Application Process, Funding Level, and Funding Duration (N=153)

Percent of respondents selecting dissatisfied 

or somewhat dissatisfied

Funding duration 5

Helpfulness of NIDILRR staff 9

Usability of grants.gov application upload system 11

Usefulness of the pre-funding meeting 11

Suitability of application format to convey R&D plan 11

Clarity of the solicitation requirements 14

Amount of time allowed for proposal submission 17

Clarity of the evaluation criteria 17

Funding amount 21

Quality of feedback on proposal 35

Usefulness of feedback on proposal 37

Length of time from submission to award decision 50

The sample contained the applicants to several NIDILRR programs, including ARRT (N=10), DRRP (N=63), 

RERC (N=13), RRTC (N=6), SBIR (N=32), and Switzer (N=26, data not shown). Variation in satisfaction 

scores by program was examined and several significant differences emerged (Figure 19). RERC 

applicants appeared to be less satisfied than all other applicants with pre-funding meeting and clarity of 
Abt Associates NIDILRR Evaluation Report Year 1 28

solicitation requirements, and along with DRRP applicants, with the usefulness of reviewer feedback. 

The reasons for these differences are unclear. SBIR and Switzer program applicants gave lower scores 

for funding amount and duration, which was not surprising as these programs are the smallest. While 

some variation across programs was observed for other application steps, these differences were not 

statistically significant.

Figure 19: Applicant Satisfaction Scores by Program (N=153)

Bars indicate standard deviation. ANOVA tests were used to determine significance (p values are shown).

RRTC=Rehabilitation Research and Training Centers, RERC=Rehabilitation Engineering Research Centers, DRRT=Disability and 

Rehabilitation Research Projects, KT=Knowledge Translation, ARRP=Advanced Rehabilitation Research Training Projects, 

Switzer=Mary E. Switzer Fellowships, SBIR=Small Business Innovation Research. 
Abt Associates NIDILRR Evaluation Report Year 1 29
Abt Associates NIDILRR Evaluation Report Year 1 30

Summary of Qualitative Comments in Applicant Survey

In total, 210 comments were received to Questions 1 and 3, which were coded and analyzed (Figure 20). 

The themes that emerged are presented below organized in descending order by the number of 

comments. 

Figure 20: Grantee PI Survey Comments about Application Process (N=210)

Time from Submission to Notification

This step in the application process was the lowest rated (score of 2.9 out of 5.0) and received the most 

comments (48 or 23% of the total). Respondents appeared frustrated that they did not know when to 

expect the decision, and that the decision took a long time was announced shortly before the start date. 

They wrote that these delays made it difficult to plan projects, decide whether to apply for other grants, 

and arrange career moves. Several respondents pointed out that the reviews were dated in May, but

were not shared until September or October.

Reviewer Feedback

Quality and usefulness of reviewer feedback received the second lowest scores and the second highest 

number of comments (40 or 19% of all comments). Several weaknesses were identified. Four out of 40 

asserted that some of the scores were not accompanied by an explanation or that the explanation was 

perfunctory. An additional 12 indicated that scoring was arbitrary, that the comments did not match 

point deduction, and that opinions expressed by reviewers were inaccurate. Several applicants asserted

that reviewers did not have the requisite or simply had not fully read the application. Two applicants 

thought that reviewers were biased. Seven respondents said that the low quality and inconsistency of 

the feedback left no path to resubmission. The applicants also said that proposal process has to be more 

transparent, that at a minimum a reviewer roster should be provided to enable the applicants exclude 

reviewers who might be in conflict. Another common view was that NIDILRR should have a mechanism 

in place to address reviewer critiques and that resubmitted applications should be reviewed by the same 

panel. Below are a few representative comments.
Abt Associates NIDILRR Evaluation Report Year 1 31

The feedback is of limited usefulness since the review cycle is so long and the same reviewers may 

not be back. Many reviewers do not provide comments, so the fact that they gave you 40/50 on one 

criterion does not give information on how to improve or what the problem is.

There seems to be little accountability for reviewers that are not thorough in their work.

Peer review comments often indicate that the reviewers are not familiar with the state-of-thescience.

The range of scores is always so extreme that it is difficult to imagine that the reviewers are actually 

reading the same application... I frequently believe that my applications have not actually been read. 

I have no idea why this is not caught by NIDILRR staff. Such things never occur at other federal 

agencies. 

Time for Submission

Nine percent of comments were received related to the time to submission. The applicants claimed that 

60 days was insufficient to prepare good applications, especially for research centers that involve 

multiple partners. They also noted that competitions are held at random times and are sometimes close 

to each other forcing the applicants to choose among programs. Respondents suggested having a 

standing funding announcement with multiple application cycles (the system used by NIH) to help the 

applicants plan their submissions. One comment was made that the current system favors repeat 

applicants familiar with the process.

Solicitation 

Seventeen comments (8%) were related to program solicitations, of which four were about NIDILRR’s 

funding priorities. These were seen as too broad by two respondents and too restrictive by the third. 

The last applicant said that it was unclear why some topics had dedicated competitions and others were 

investigator-initiated. An additional 13 comments was on the clarity of solicitations. The applicants 

wrote that they were long, vague, confusing, and difficult to read. The following specific problems were 

mentioned: combining two DRRP competitions for KT was confusing, it was unclear that ADA had to be 

addressed in the application, it was unclear whether fellowships were individual or institutional, and 

application requirements were mentioned in different places and were hard to find. An SBIR respondent 

recommended that NIDILRR follow a standard format used by NIH.

Grants.gov

We received 15 comments (7% of all) recounting problems with grants.gov. The system was described as 

cumbersome and slow, and some applicants had to submit an application several days in advance “to be 

safe.” It was compared unfavorably to what is used by NSF. Several respondents also pointed out that it 

was impossible to track review progress and one respondent found it challenging to use multiple 

passwords.

Funding Amount and Duration

Five of 14 comments about funding amount were made by SBIR I grantees, all of whom said that $75K 

was insufficient to produce significant results. These applicants pointed out that SBIR I funding levels at 

other agencies are typically $150K, and in some cases include an additional $5K for commercialization 
Abt Associates NIDILRR Evaluation Report Year 1 32

assistance. The applicants wrote that funding amounts were not keeping pace with the economic 

environment and that they could not offer postdocs competitive salaries. The applicants said that 

NIDILRR should either decrease the scope of work or increase funding costs. Four comments were also 

made about funding duration, of which three came from SBIR I grantees.

Assistance from NIDILRR Staff

Applicants had varied experience obtaining assistance from NIDILRR staff. Of the 13 comments received, 

one was positive, four mixed, and the rest negative. The most common complaint, voiced by five 

respondents, was that NIDILRR staff could not or would not answer questions.

I find NIDILRR to be like a black hole. No real information is ever provided no matter the request. 

Answers, if provided, are not answers but non-answers. It is very frustrating.

With NIH grant officers, you feel like the agency really wants you to submit the best possible grant. 

They give you constructive feedback and encouragement, but are also blunt about the likelihood of 

funding and whether the idea is worth pursuing. With NIDRR staff, I often feel like I am asking for 

state secrets when I ask for a timeline.

Several other respondents said that their emails went unanswered or required several follow-ups.

Application Format

Twelve comments (6% of the total) indicated problems with application format. Of these, two said that 

double-spacing was not appropriate, especially for tables. Three others pointed out that SBIR I and II 

applications were the same length, even though SBIR II award was much larger and the applicants for 

this phase were required to include the results for SBIR I. The RRTC applicants said that the new limit of 

100 pages was too short and that any limit should be a requirement, not a recommendation. Finally, one 

respondent wrote that page limits for different sections were not aligned with the priorities outlined in

the RFP, and two others that it was a challenge to address all of the evaluation criteria within the 

relatively short page limit.

Evaluation Criteria

Eleven respondents pointed out problems with the evaluation criteria. One said that there were too 

many dimensions (research, development, utilization, collaboration, etc.) for a small project and that it 

was difficult to address each of these in a non-repetitive way. Another noted that the criteria as worded 

have poor inter-rater reliability. Six additional respondents thought the criteria inappropriate or 

confusing. For example, a DRRP applicant said that a research grant should not be scored for training 

and technical assistance. A RERC applicant pointed out that the criteria reflected only a small fraction of 

program requirements. Finally, a comment was made that the evaluation criteria were not well 

explained and that proposal outline, scoring criteria, and funding priorities were not aligned.

Pre-Funding Meeting

We received five comments suggesting that the meeting was not useful. Respondents said that they 

were simply read the solicitation, and that NIDILRR staff did not provide answers to specific questions.
Abt Associates NIDILRR Evaluation Report Year 1 33

Positive Comments

A number of applicants (N=11) provided positive comments about the application process. They said

that RFPs were clearly written and that NIDILRR staff was helpful and had a good grasp of the field’s 

needs. One respondent wrote that the review process was getting increasingly better and that in this 

person’s experience as a reviewer, other panelists were sincere and competent.

Recommendations for Improving the Application Process

Based on the results of the survey, we suggest the following changes to the application process: 

• Revise program announcements to make them more succinct and better organized. Include the 

timeline for review and award notification.

• Have a standard application deadline well publicized to the community, ideally at a different 

time for each program.

• Share reviewer comments with the applicants as soon as they are available even if funding 

decisions had not yet been made.

• Notify the applicants of the award status a few weeks in advance of the project start date, in 

particular for shorter grants such as SBIR I.

• Provide clearer scoring guidance to reviewers and applicants and emphasize to reviewers that 

point reduction must be commensurate with the level of weakness.

• Make the reviewer roster public and allow the applicants to request that their application is not 

assigned to reviewers who they see as being in conflict.

• Consider collapsing similar evaluation criteria and make sure that the evaluation criteria are 

consistent with program goals.

Study Limitations

The study had three limitations. First, response rate for SBIR grantees was low, and therefore, our 

findings may not be generalizable to the entire program. Second, since the contacts provided by PIs 

were the source of the customer interviews, the findings may be more positively biased since the 

individuals are likely to be satisfied, frequent users. Third, we could not properly implement the 

customer interview task for the SBIR program. 
Abt Associates NIDILRR Evaluation Report Year 1 34

Summary and Conclusions

This report presents findings from the assessment of three NIDILRR programs: KT, DBTAC, and SBIR. In 

addition, we examined the NIDILRR applicants’ satisfaction with the proposal process. The strengths and 

weaknesses of the evaluation approach, with recommended changes for years 2–5, are presented in 

Appendix C. 

Disability and Business Technical Assistance Centers

According to the grantees, NIDILRR funding has enabled advancements in public understanding of needs 

and challenges of individuals with disabilities, resulted in new laws and policies, and led to new 

approaches to identify, confirm, and manage disability. These advancements occurred in all areas, 

including health, education, employment, communication, transportation, and social/community life.

Many specific examples of benefits were provided in the survey. Challenges experienced by PIs included

capturing the impact of the materials and services that they provide and becoming known as a resource.

The contribution of DBTACs conveyed in the survey was strongly reinforced in the conversations with 

the customers. All of the 19 individuals interviewed reported the importance of DBTACs to their 

professional responsibilities. The Centers were praised as a source of unbiased and reliable information 

about ADA as well as for customer orientation of the staff. All respondents who were asked said that 

there was no comparably good alternative. Finally, the customers spoke positively about training 

opportunities available at DBTACs.

Knowledge Translation Program

In contrast to the DBTACs, KT grants included in this study are no longer active. However, at least half of 

the former grantees continue to provide advice, training, and technical assistance to other grantees and 

to non-NIDILRR researchers, and maintain and disseminate the resources developed under the KT 

grants. For example, one center distributed several thousand DVDs related to ADA to employer 

organizations at no cost. KT PIs continue to help grantees develop technology transfer plans and provide 

commercialization related materials. Grantees also reported that they maintain various collaborations 

established during the funding period. Challenges experienced by grantees included interaction with 

other NIDILRR PIs and lack of awareness that commercialization takes time and effort.

Several examples of KT Center benefits were provided by the users. For instance, the US Equal 

Employment Opportunity Commission relied on the analysis performed by the KT Center to develop its 

strategic plan. An SBIR grantee described guidance and help he received from the center as instrumental 

to the success of his business. Several researchers who received a travel award through a KT center 

described the experience as broadening their horizons and influencing career direction.

As KT program is expected to produce peer-reviewed articles, we conducted bibliometric analysis to 

characterize publication impact. We found 23 publications referencing KT grants. All but two of these 

received at least one citation and one (a systematic review) received 107 citations. The journals in which 

the papers were published had significantly higher prestige scores than average for all journals in the

rehabilitation subject category.
Abt Associates NIDILRR Evaluation Report Year 1 35

Small Business Innovation Research Program

SBIR is a mandatory program across the federal government. Consequently, we designed the survey to 

enable a comparison between SBIR at NIDILRR to other agencies. We found that NIDILRR performed 

equally well or better on all indicators for which data were available. For example, 70% of NIDILRR PIs 

had received additional funding, compared to 44–58% at NASA, DOD, and NIH. Only NSF did better, at 

93%. Seven of 10 SBIR II grantees reported developing a product with NIDILRR funding, and three of 

these products were in a marketplace, also exceeding other agencies. Six of the products listed by

grantees in the survey had websites advertising them for sale.

When offered an opportunity to comment about the benefits of NIDILRR funding, 57% of the PIs were 

very positive, saying that the program helped them gain experience, develop and test products, and 

ultimately serve people with disabilities. Challenges reported by grantees included low funding 

level/short project duration for Phase I obtaining funding for Phase II.

Application Process

Participant satisfaction with the application process varied depending on the step in the process. The 

strongest negative feedback, given by half of the applicants, was about time to award notification, which 

was seen as unreasonably slow especially because the reviews were dated weeks before they were 

shared. One-third of respondents were also dissatisfied with proposal scoring and reviewer feedback.

We were told that large point deductions were accompanied by disproportionally minor comments. 

Several applicants claimed that reviewers had not even fully or carefully read the proposal because they 

were asking for information already included. The feedback was also seen as unhelpful for resubmission. 

In contrast, the applicants were reasonably satisfied with the help from NIDILRR staff, clarity of program 

announcement, and the amount of time to submit an application.

Statistically significant differences in satisfaction were observed across six programs (ARRT, RERC, RRTC, 

SBIR, DRRP, and Switzer). For example, RERC applicants were less satisfied than others with usefulness 

of pre-funding meeting and clarity of solicitation requirements and DRRP and RERC were less satisfied 

with usefulness of proposal feedback. Despite the challenge conveyed in the survey, 74% of respondents 

would apply for NIDILRR funding again.

In conclusion, based on self-reporting by PIs and on the evidence by the users of NIDILRR products and 

services we identified many tangible benefits of NIDILRR funding. The DBTACs, in particular, are seen as 

an indispensable resource to many organizations that serve people with disabilities. Among the 

customers referred to us by PIs several self-identified as having disabilities. All of these individuals spoke 

of a positive impact of NIDILRR funding, and one described it as life-changing. 

The application process can be enhanced. Based on the nature of the comments, we proposed several 

changes that should improve satisfaction, including sharing reviewer feedback as soon as it is available, 

establishing fixed application due dates, providing more scoring guidance to reviewers, and a few 

others.
Abt Associates NIDILRR Evaluation Report Year 1 36

Appendix A: Study Protocols 

Survey of Principal Investigators on Americans with Disabilities Act Grants 

1. Have you been able to obtain additional, non-NIDILRR funding to support the training, technical 

assistance (TA), dissemination, and other activities initiated with NIDILRR funding? Select all that 

apply.

□ Yes  GO TO 1A

□ No.  GO TO 2

1A. Which funding sources supported these activities? Please list all funding sources and goals of 

the project and indicate whether you are a Principal Investigator (PI) or a (Co-PI).

Funding agency/branch Goals of the grant As a PI or Co-PI?

If yes, click button

□

□

□

□

2. Have any of the following advancements occurred for the intended target audience(s) of your grant, 

at least in part, as a result of NIDILRR funding? If yes, please indicate which grant activity (training, 

TA, and/or dissemination) brought about this change. Select all that apply. 

Training TA Dissemination

New or improved laws relevant to 

individuals with disabilities or their 

caregivers

□ □ □

New or improved policies relevant to 

individuals with disabilities or their 

caregivers

□ □ □

New or improved tools, devices, tests, 

measures, or screening approaches to 

identify, confirm, or manage disability

□ □ □

Improved understanding by the public of 

the needs and challenges of individuals 

with disabilities

□ □ □

Other ___________________________ □ □ □

Other ___________________________ □ □ □

Other ___________________________ □ □ □

3. In which of the following areas have the advancements occurred? Select all that apply.

□ Personal care

□ Health 

□ Transportation

□ Communication
Abt Associates NIDILRR Evaluation Report Year 1 37

□ Living arrangement

□ Education

□ Employment

□ Economic life 

□ Community, social life, and civic life

□ I do not know

□ Other type of improvement. Please describe_______________________

3A. Please describe the most important benefit(s) for individuals with disabilities.

 ___________________________

3B. Can you suggest individuals who will be willing to further speak with us about the benefits of 

your work? These persons can be researchers, policymakers, care providers, disability 

advocates, educators, individuals with disabilities, or other appropriate persons.

Name Position and 

affiliation

Email and phone 

number

Nature of the 

benefit

1.

2.

3.

4.

5.

[If any individuals are suggested] In order to speak about the benefits of your work, we will need to 

mention your name and your work to these individuals. Do we have your permission to do so? 

□ Yes. You can mention my name and my work

□ No. You cannot mention my name and/or my work

4. What have been the greatest challenges to accomplishing your ADA grant goals?

________________________________________

5. Please feel free to make any other comments about the importance of NIDILRR programs to your 

work and to individuals with disabilities.

_____________________________________________
Abt Associates NIDILRR Evaluation Report Year 1 38

Survey of Principal Investigators on Knowledge Translation Grants 

1. Were you a Principal Investigator or Project Director on a Knowledge Translation (KT) grant 

Number [xxx] Title [yyy]15? 

□ Yes  2 

□ No  Exit the survey

2. Since the end of your KT grant, have you engaged in any of the following activities related to 

that grant? Select all that apply.

□ Disseminate information acquired during the funding period about best practices and 

challenges in KT and/or technology transfer (TT)

If selected  Please elaborate [text box]

□ Provide advice, training, and/or technical assistance to NIDILRR-funded researchers related 

to KT and/or TT 

If selected  Please elaborate [text box] 

□ Provide advice, training, and/or technical assistance to non-NIDILRR-funded researchers 

related to KT and/or TT

If selected  Please elaborate [text box]

□ Maintain KT and/or TT-related databases or other information systems and resources 

developed with NIDILRR funding.

If selected  Please elaborate [text box]

□ Disseminate information about NIDILRR-funded research and development to researchers, 

individuals with disabilities, service providers, advocates, employers, and other 

stakeholders.

If selected  Please elaborate [text box]

□ Link disability and rehabilitation researchers with industry, service providers, and other

potential users of their work.

If selected  Please elaborate [text box]

□ Other

If selected  Please elaborate [text box]

□ I have not engaged in any of the activities related to the KT grant

3. Please indicate with which of the following groups you have maintained collaborations 

established during the funding period. Select all that apply.

□ Researchers 

□ Health care providers/professionals

□ Educators

□ Vocational rehabilitation professionals

 

15 Pre-loaded, so that respondent can see
Abt Associates NIDILRR Evaluation Report Year 1 39

□ Employers

□ Business organizations

□ Advocates

□ Funders

□ Other federal, state, or local agencies

□ Individual with disabilities

□ Caregivers or family members of individuals with disabilities

□ Other ________________

□ I have not maintained any collaborations

Please describe the most important collaboration from the items indicated above, including 

its nature and purpose.

 _______________________

4. Have any of the following outcomes occurred since the end of your KT grant and at least in part 

as the result of the work on this grant? Select all that apply.

Knowledge

□ Increased understanding of KT and/or TT process/practice by NIDILRR grantees

If selected  Please elaborate [text box]

□ Increased understanding of KT and/or TT process/practice by individuals other than NIDILRR 

grantees

If selected  Please elaborate [text box]

□ Increased understanding of commercial landscape by NIDILRR grantees

If selected  Please elaborate [text box]

□ Increased familiarity with systematic reviews and other rigorous research synthesis methods 

applied to disability and rehabilitation research

If selected  Please elaborate [text box]

□ Increased familiarity with high quality evidence in disability and rehabilitation research

If selected  Please elaborate [text box]

□ Increased awareness of findings and/or products resulting from NIDILRR-funded research 

and development among individuals with disabilities, services providers, advocates, and 

researchers 

If selected  Please elaborate [text box]

□ Other contributions to knowledge

If selected  Please elaborate [text box]

Practice/System Capacity

□ Commercialization of products by NIDILRR grantees

If selected  Please elaborate [text box]
Abt Associates NIDILRR Evaluation Report Year 1 40

□ Dissemination of research and development findings/products by NIDILRR grantees beyond 

journal publications

If selected  Please elaborate [text box]

□ Increased use of valid and relevant disability and rehabilitation research findings to inform 

policy and/or practice

If selected  Please elaborate [text box]

□ Publication of systematic reviews and other items requiring rigorous research synthesis

If selected  Please elaborate [text box]

□ Improvements in care and/or services provided to individuals with disabilities

If selected  Please elaborate [text box]

□ Improved understanding by the public of the needs and challenges of individuals with 

disabilities 

If selected  Please elaborate [text box]

□ Other contributions to practice/system capacity

If selected  Please elaborate [text box]

5. Since the end of your KT grant, have you been able to use knowledge, methods, training 

materials, tools, or other resources developed under the grant to advance KT and/or TT in areas 

other than disability and rehabilitation?

□ No

□ Yes

If yes is selected  Please elaborate [text box]

6. Since the end of your KT grant, have you obtained additional funding to support/continue the 

activities that have begun under the KT grant? Please select one. 

□ Yes  GO TO 6A

□ No

6A. Please list all funding sources and goals of the project and indicate whether you are a PI or a 

co-PI.

Funding agency/branch

(e.g. Department of 

Education/NIDILRR)

Goals of the grant As a PI or co-PI?

If yes, click button

□

□

□

7. To help us further document the impacts of your work, can you suggest an individual or individuals 

who can speak with us about the benefits of your work to other grantees, individuals with 

disabilities, or other disability stakeholders? This person can be another researcher, policymaker, 

care provider, disability advocate, educator, individual with disabilities or his/her caregiver, or any 

other appropriate person.
Abt Associates NIDILRR Evaluation Report Year 1 41

Name Position and 

affiliation

Email and phone 

number

Nature of the 

benefit

1.

2.

3.

7A. [If any individuals are suggested] In order to speak about the benefits of your work, we will 

need to mention your name and your work to these individuals. Do we have your 

permission to do so? 

□ Yes. You can mention my name and my work

□ No. You cannot mention my name and/or my work

8. In retrospect, what were the greatest challenges to accomplishing the goals of your NIDILRRfunded KT grant?

________________________________

9. Before you go, please feel free to share any other thought about the KT program.

________________________________
Abt Associates NIDILRR Evaluation Report Year 1 42

Survey of Principal Investigators on Small Business Innovation Research Grants

1. Which of the following grants have you been awarded by the National Institute for Disability, 

Independent Living, and Rehabilitation Research (NIDILRR)? Select one.

□ SBIR Phase I  GO TO 2

□ SBIR Phase II  SKIP 2, GO TO 

□ SBIR Phase I and II  GO TO 2

□ None of the above  Exit the survey

2. What were the most important results of your NIDILRR SBIR Phase I funding?

___________________________

3. Did you receive additional funding to commercialize your idea? Select one.

□ No  GO TO 3A

□ Yes  GO TO 3B

3A. Which of the following statements are correct? Select one.

□ As a result of SBIR Phase I research, we decided that our idea did not have commercial 

potential

□ We applied for a SBIR Phase II grant, but were not successful in receiving it

□ We applied for non-SBIR funding, but were not successful in receiving it 

3B. What was the source of the additional funding? Select all that apply. 

□ SBIR Phase II from NIDILRR

□ SBIR Phase II from another agency

□ Non-SBIR federal funds 

□ State, local, or tribal government

□ Private investment

□ Investment from the company selling the product

□ Your personal funds

□ Other funding sources. Please list____________________

SKIP 4–9 for SBIR PHASE I

4. Which of the following products resulted from your NIDILRR SBIR Phase II funding? Select all that 

apply.

Type of products Click button if “yes”

No product(s) resulted from NIDILRR funding □

Industry standard/guideline □

Software/netware □

Hardware □

Assistive technology □

Durable medical equipment □

Research tool □

Educational material □
Abt Associates NIDILRR Evaluation Report Year 1 43

5. Please describe the most important commercialization outcome and include a citation, if possible:

_____________________________________

6. What is the current sales status of your product? Select one.

□ No product resulted from SBIR program  GO TO 10

□ Product discontinued  GO TO 8

□ No sales to date; sales are expected  GO TO 10

□ No sales to date; sales are not expected  GO TO 10

□ No sales, but outcome is used by intended target population  GO TO 10

□ The product is being sold in a marketplace  GO TO 7

7. Please estimate the sales resulting from the product(s) to date. Select one.

□ <$100K

□ $100K to $1M

□ >$1M to $5M

□ >$5M to $10M

□ >$10M to $50M

□ >$50M

8. Approximately, how many units of the product have been sold since the company was established? 

Please enter number.

__________

9. How many persons are currently employed by the company that benefited from NIDILRR funding?

Select one.

□ There is no longer a company

□ 1–5 

□ 6–10

□ 11–20

□ 21–50

□ 51–100

□ >100

10. Would you have undertaken the project without this NIDILRR funding? Select one.

□ Definitely yes  GO TO 12

□ Probably yes  GO TO 12

□ Uncertain  GO TO 12

□ Probably not GO TO 10A

□ Definitely not GO TO 10A

10A. Why not? What has NIDILRR funding enabled you to do?

 _______________________________________________

11. If NIDRR could change the funding amount, which funding option do you prefer? Select one.

□ Double the funding for SBIR Phase I from $75,000 to $150,000 and reduce the number of awards 

by half 

□ Keep the number of awards and the funding amount the same

□ No opinion
Abt Associates NIDILRR Evaluation Report Year 1 44

12. We would like to further document the impact of your work by speaking with the individuals who 

have benefits from it. Appropriate persons might include researchers, policymakers, care providers, 

disability advocates, educators, individuals with disabilities, and their caregiver(s). 

Because of privacy concerns, we cannot ask you to share these persons’ email, so instead we would 

very much appreciate it if you would share our email, NIDRR_Evaluation@abtassoc.com with any 

individual who you believe would be willing to speak with us regarding the benefits of your work. 

We will take no more than 15 minutes of their time.

13. In retrospect, what were the greatest challenges to accomplishing the goals of your SBIR grant?

 ________________________________

14. Please use the space below to make any additional comments about the importance of NIDILRR 

programs to your work and to individuals with disabilities.

____________________________________________________
Abt Associates NIDILRR Evaluation Report Year 1 45

Survey of Applicants 

1. What is your level of satisfaction with the following aspects of the application process? If you are 

not satisfied, please explain why in the rightmost column which will automatically appear. Dissatisfied Somewhat dissatisfied Neutral Somewhat satisfied Satisfied Too early to tell

NA

Clarity of the solicitation requirements

Clarity of the evaluation criteria

Usefulness of the pre-funding meeting

Helpfulness of NIDILRR staff on logistical 

and other issues

Suitability of required proposal format to 

convey research and development plans

Amount of time allowed for proposal 

submission

Usability of grants.gov application upload 

system

Quality of feedback on proposal

Usefulness of feedback on proposal

Length of time from proposal submission to 

award decision notification

Funding duration

Funding amount

1A. If dissatisfied or somewhat dissatisfied is selected for any category:

Why were you not satisfied with ______________?

 ____________________________________________________

2. Will you apply for NIDILRR funding in the future? Select one.

□ No

□ May be

□ Yes

3. Please feel free to share any other thoughts related to the application process. NIDILRR values your 

input!

___________________________________
Abt Associates NIDILRR Evaluation Report Year 1 46

Customer Interview Questions

1. Please describe your role and the nature of your involvement with the ADA (or KT) grant.

[Question added to the original protocol]

2. What were the benefits resulting from this involvement to you? Your organization? People with 

disabilities? 

3. Would you like to add anything else about the value of NIRILRR programs?
Abt Associates NIDILRR Evaluation Report Year 1 47

Appendix B: Interviewee Information

Name Referencing 

Program

Position

1 Paul Ackerman KT International consultant, parent advocacy coalition 

for educational rights

2 Meg Conger ADA ADA Compliance Manager, Kansas City

3 Brenda Dawes ADA Project Affiliate, Oklahoma ABLE Tech

4 Stanley Ducharme KT Professor, Boston University

5 Ron Edwards KT Director of Research, EEOC

6 Robert Fraser KT Professor, University of Washington

7 Michael Gibbens ADA Architect and Author, Private business

8 Rob Hodapp ADA Western PA Employment Specialist, 

PA Vocational Rehabilitation

9 Caroline Van 

Howe

KT Chief Operating Officer, 

Assistive Technology Industry Association 

10 Margot Imdieke ADA Accessibility Specialist, 

Minnesota Council on Disability

11 Dan Johnson ADA Director, State Council on Physical Disability

12 Maya Kalyanpur KT Associate Professor, University of San Diego

13 Gary Kidd ADA ADA Specialist, The Disability Network

14 Matthew Lerner SBIR Assistant Professor, Stony Brook University

15 Kathy Lovell ADA ADA Manager at Regions Bank

16 Janice Marrero ADA Disability Consultant, Private business

17 Kerri Morgan KT OT Program Instructor, Washington University

18 James Rimmer KT Professor, University of Alabama

19 Phillip Rumrill KT Professor, Kent State University

20 Nicole Russo SBIR Assistant Professor, Rush University

21 Robye Scott ADA Sr. Manager-Compliance, Kraft Foods

22 Betty Siegel ADA Director of Accessibility, 

The Kennedy Center for the Performing Arts

23 Vicki Simpson ADA Attorney General's Office, State of Illinois

24 Emmanuel Smith ADA Person with a disability, Disability Rights Iowa

25 Steve Sutter KT President, Creatabilty Concepts

26 Patty Tingle ADA Director Bay Area Center for Independent Living

27 Ruth Tofle ADA Chair of Environmental Sciences-University of 

Missouri

28 Wendy Wilkinson ADA Former director of the ADA Center

29 Victoria Willard ADA EEO Manager and ADA Coordinator, 

County of Sonoma
Abt Associates NIDILRR Evaluation Report Year 1 48

Appendix C: Implications for Furture Evaluation

This evaluation year was a testing ground for some of the approaches and instruments that will be used 

again for other programs. Therefore, it is helpful to reflect on what went well and what can be 

improved.

What Went Well

All survey instruments are working as intended; based on the data, respondents appeared to have 

understood the questions and could answer them. None of the emails that we received while the survey 

was in the field indicated any problems with the instruments. The only minor adjustment that we made 

during data collection was to the customer interview protocol. We found that it was helpful to first ask 

respondents about their roles and responsibilities, so that the benefits of the ADA and KT centers could 

be put in context. We recommend adding this question to the protocol.

The applicant survey was also successful. Since we found that respondents’ satisfaction has not been 

influenced by the review outcome, we recommend conducting the survey once after the award is 

announced.

Challenges and Proposed Changes

We encountered two challenges. The first one was low response rate for SBIR program. Our attempts to 

recruit additional PIs by sending multiple reminders (including by NIDILRR staff) and by verifying email 

addresses were only partially successful. It appeared that most non-respondents did not open the 

survey. We are unsure how this type of problem can be addressed in the future, other than by limiting 

respondent pool to more recent grantees.

The second challenge was with obtaining contacts for customer interviews. While our proposed 

approach of asking for a contact in the survey worked very well (a total of 47 contacts were provided by 

16 respondents), relying on the customers to reach out to us was ineffective (only one person did so). 

Therefore, we recommend asking for the contacts in future years.