**Executive Summary**

**Modular Grants Application Process Outcome Evaluation**

The modular grants application process was initiated June 1, 1999 with
the intent of reinforcing the grant-in-aid philosophy of supporting
research, as opposed to the contract mentality of "buying" research.
More specifically, the long-term goals of the modular grant application
process include focusing the efforts of key stakeholders on the science
of the applications (rather than on the itemized budget), reducing
administrative burden for all stakeholders and simplifying the grant
administration process, and accommodating the principal investigators'
need for flexibility.

At the outset of its implementation, the research community was told
that NIH would evaluate the modular grant application process several
years after its implementation so that everyone involved with the
process would have some experience with it. In April, 2003, a two-phase
evaluation began. The Phase I Feasibility Study resulted in the
development of five web surveys for the five stakeholder groups involved
with the modular grants application process. These five stakeholder
groups include principal investigators, peer reviewers, NIH scientific
review administrators, NIH program and grants management staff, and
institutional officials. The Phase II Outcome Evaluation took place
between April 2004 and April 2005. The intent of the outcome evaluation
was to determine whether the modular grant application process met its
intended long-term goals and whether stakeholders understood the modular
grant application process.

The sampling strategy the outcome evaluation involved using a census for
two of the stakeholder groups and drawing a systematic random sample for
the other three stakeholder groups. The NIH scientific review
administrators and the institutional officials were not sampled. Instead
every person in these two populations was surveyed. A systematic random
sample was drawn from the principal investigator, peer reviewer, and NIH
program and grants management staff populations. The number of surveys
administered was based on a confidence level of 95% and a confidence
interval (margin of error) of + or - 5%, taking into account an
estimated 60% response rate for the sampled groups and a 15% email
"bounce back" rate for non-NIH employees.

The data collection procedures included up to five contacts with each
stakeholder. First, the stakeholders were sent a letter from NIH that
explained the purpose of the study and provided information on the
upcoming survey notification email. Next, stakeholders were sent an
email that contained a unique log-in name, password, and web link to
their specific stakeholder survey. Approximately 10 days after the
initial email was sent, all non-respondents were sent an email reminder.
A second email reminder was sent to those who had still not responded 10
days after the first reminder. Following the second reminder, a Westat
staff person made telephone follow-up calls to non-respondents, asking
them to complete the survey online. These data collection efforts
resulted in response rates of 79% for institutional officials, 74% for
peer reviewers, 73% for NIH scientific review administrators, 70% for
NIH program and grants management staff, and 66% for principal
investigators.

The results indicate that the modular grant application process is well
received by the extramural community but somewhat less well received by
NIH staff. More than 80% of principal investigators, 76% of
institutional officials, and 73% of peer reviewers report they were
satisfied overall with the modular grant application process. In
comparison, only 57% of scientific review administrators and 50% of
program and grants management staff reported overall satisfaction with
the modular application process.

When asked if listing dollar value totals for major categories
(personnel, travel, etc.) would be a good compromise between a detailed
budget and a modular budget, the results were opposite those found for
overall satisfaction. For this survey item, 62% of scientific review
administrators and 60% of program and grants management staff agreed
that listing dollar value totals for major categories would be a good
compromise compared to only 50% of peer reviewers, 40% of institutional
officials, and 39% of principal investigators.

It was clear from asking stakeholders about whether or not they would
prefer the modular grant limit be higher than \$250,000, that peer
reviewers, program and grants management staff, and scientific review
administrators did not think this was a particularly good idea. Although
a fairly high percentage of institutional officials (76%) and principal
investigators (61%) said they would prefer the modular limit be higher
than \$250,000, only 46% of peer reviewers, 31% of program and grants
management staff, and 28% of scientific review administrators felt the
same way.

Since reducing administrative burden is one of the long-term goals
associated with the modular grant application process, stakeholders were
asked whether their job responsibilities had increased, stayed the same,
or decreased as a direct result of the modular grant application
process. More than one-third of the peer reviewers (37%) reported their
job responsibilities had decreased, followed by 29% of scientific review
administrators, 24% of program and grants management staff, and 17% of
institutional officials.

Perhaps one reason only 17% of institutional officials report a decrease
in their job responsibilities is because most institutional officials
(86%) report their institution still requires principal investigators to
submit a detailed budget even when applying for an NIH modular grant.
Another reason may be that close to one-third (30%) of the institutional
officials that work with auditors say they spend extra time with the
auditors explaining modular grants.

The modular grants application process also produced additional work for
program and grants management staff, as evident by 61% indicating that
since there are no "Other Support" pages time is spent trying to find
out this type of information. On the positive side, however, 54% of
program and grants management staff indicated the modular grant
application process saves time because cost analysis doesn't have to be
done.

The modular grant application process also appears to save time for peer
reviewers and scientific review administrators. When asked how much
study section meeting time was devoted to discussing an applicant's
proposed budget before and after the implementation of modular grants,
responses across all peer reviewers indicate the median discussion time
per application was 5 minutes before modular grants and 3 minutes after
modular grants, potentially saving hours of study section meeting time
focused on the applicants' budgets. In addition, 70% of peer reviewers
and 80% of scientific review administrators agreed that as a direct
result of the modular grant application process, discussions about
budgets in their study section are much more limited.

Despite the fact that modular applications do not include a detailed
budget, 87% of peer reviewers felt they could still provide an
assessment of the scientific merit of an application without the
detailed budget. A similar percentage of principal investigators (85%)
and scientific review administrators (79%) also felt that peer reviewers
could assess the scientific merit without a detailed budget. However,
only 62% of peer reviewers and 58% of scientific review administrators
believe peer reviewers can assess the scientific merit of a modular
application without the "Other Support" pages. Principal investigators
were somewhat more optimistic, with 73% indicating peer reviewers could
assess the scientific merit without the "Other Support" pages.

To determine stakeholders' understanding of the modular grant
application process, all stakeholders were asked about the extent to
which they understood each of 14 modular grant features. A higher
percentage of institutional officials than the other stakeholder groups
indicated the majority of the modular features were clear. Across all
stakeholder groups the best understood features included the fact that
all modular grant application forms are available on the NIH web site,
biographical sketches need to be prepared for all key personnel, and
individual salary information is not required. Less well-understood
features included those related to consortium/contractual costs and
administrative supplements. Even though some features are not well
understood, the survey data show a high percentage of external
stakeholders (82% of peer reviewers, 92% of principal investigators, and
92% of institutional officials) report being satisfied with the
information received from NIH staff about the modular grant application
process.

The survey data also show there are some misperceptions about modular
grants. For example, 40% of principal investigators and 34% of
institutional officials believe a modular grant is more likely to be
funded than a nonmodular grant. This is contrary to findings based on
NIH administrative data.

Finally, the modular grant outcome study was used as an opportunity to
ask stakeholders about Just-in-Time. Similar to the findings on overall
satisfaction with modular grants, Just-in-Time is also better received
by the extramural community than by NIH staff with 76% of principal
investigators and 72% of institutional officials reporting overall
satisfaction with Just-in-Time compared to only 48% of program and
grants management staff.