# Presentation: 355302

## Outcome Study for the Evaluation of the Modular Grants Program

- May 2005

- Prepared by Westat under Contract # GS-23F-8144H

## Background on the Modular Grant Application Process

- June 1, 1999 NIH initiated the modular grant application process
- Intent was to reinforce the grant-in-aid philosophy of supporting research (as opposed to the contract mentality of “buying” research)

## Goals of the Modular Grant Application Process

- Goals:
- Focus the efforts of peer reviewers, principal investigators, institutional officials, and NIH staff on the science of the applications (rather than on the itemized budget)
- Reduce administrative burden for all stakeholders and simplify the grant administration process
- Accommodate principal investigators’ need for flexibility

## History of the Modular Grant Application Process Evaluation

- Nov 2001 – Modular Evaluation Workgroup developed a report on guidelines for the evaluation of the modular process
- Feb 2002 – Peer Review Oversight Group (PROG) approved report; Phase I Feasibility Study funds made available
- April 2003 – Phase I feasibility study began
- Jan/Feb 2004 – PROG members review Phase I results and approve survey instruments
- March 2004 – Phase I presentation at EPMC meeting, requesting feedback on surveys
- April 2004 – Phase II outcome evaluation study began
- May 2005 – Phase II presentation and materials delivered to NIH

## Phase II Outcome Study Purpose

- Overall purpose
- Determine if the modular grants application process met its intended long-term program goals
- Determine if stakeholders understand the modular grant application process

## Phase II Outcome Study Activities

- Developed and tested five web surveys
- Administered web surveys
- Conducted quantitative and qualitative data analysis
- Produced slide presentations, executive summary, summary reports, methodology report, quantitative and qualitative results, data files

## Samples: Census Groups

- Used a census for two stakeholder groups
- NIH Scientific Review Administrators (N=372)
- CSR and Non-CSR Scientific Review Administrators in NIH Health Scientist Administrator (HSA) database as of August 2004
-  Institutions (Institutional Officials) (N=327)
- Institutions that applied for 10 or more modular grants, excluding foreign institutions (but including Canada); used National Council for University Research Administrators (NCURA) database or institution web site to find highest ranking person (e.g., Director of Office of Sponsored Research); gave highest ranking person option to let another staff member more knowledgeable take the survey

## Samples: Non-census Groups

- Used a systematic random sample for three stakeholder groups
- Principal investigators
- Principal investigators that had submitted a modular grant application between September 1999 and March 2004
- Peer reviewers
- NIH Peer reviewers serving as of December 2003 and those who had rotated off during the previous 2 years
- NIH program and grants management staff
- Program staff in NIH Health Scientist Administrator (HSA) database as of August 2004
- Grants management officers and grants management specialists listed in Grants Management Advisory Committee (GMAC) Directory
- Oversampled non-census groups to account for:
- 60% response rate
- 15% “bounce back” (undeliverable) emails for non-NIH employees

## Sample Sizes

| Stakeholder Group | Population Size | Sample Size
(# of surveys administered) |
| --- | --- | --- |
| Principal Investigators | 31,180 | 1,676 |
| Peer Reviewers | 2,836 | 753 |
| Scientific Review Administrators | 372 | 372 |
| Program and Grants Management Staff | 1,213 | 488 |
| Institutional Officials | 327 | 327 |

## Data Collection Schedule: Fall 2004

## Final Sample Sizes and Response Rates

| Stakeholder Group | Final Sample Size* | Completed Surveys | Response Rate |
| --- | --- | --- | --- |
| Institutional Officials | 288 | 227 | 78.8% |
| Peer Reviewers | 671 | 497 | 74.1% |
| Scientific Review Administrators | 285 | 209 | 73.3% |
| Program and Grants Management Staff | 355 | 247 | 69.6% |
| Principal Investigators (PIs) | 1,439 | 949 | 65.9% |
| 	PIs – Recipients of Modular Grants | 879 | 644 | 73.3% |
| 	PIs – Unfunded Applicants | 560 | 305 | 54.5% |

- *Includes only eligible respondents

## Overall Satisfaction with Modular Grants

- Overall, how satisfied are you with the modular grant application process?

## Raising the Modular Grant Limit

- Would you prefer the modular grant limit be higher than $250,000?

## Assessing Scientific Merit

- I/Peer reviewers can assess the scientific merit of a modular grant application without...

## Changes in Job Responsibility Due to Modular Grants

- As a direct result of the modular grant application process, my responsibilities have decreased

## Institutions Require Detailed Budgets

- Does your office or any other office at your university/research center require principal investigators to submit detailed budgets for review even if they are applying for an NIH modular grant?

## Study Section Meeting Time

- On average, how much study section meeting time was devoted to discussing an applicant’s proposed budget before and after the implementation of the modular grant application process?

## Satisfaction with Information Received from NIH Staff

## Best and Least Understood Features of the Modular Grant Application Process

- Best understood features:
- All forms for modular grant applications are available on the NIH web site
- Biographical sketches need to be prepared for all key personnel
- Individual salary information is not required for personnel
- Least understood features:
- Total consortium/contractual costs need to be estimated for each year, rounded to the nearest $1,000
- Modular grant awards are eligible for administrative supplements (i.e., noncompeting supplemental funding)

## Perception of Funding Likelihood for Modular Compared to Nonmodular Grant Applications

- I think/PIs believe a modular grant is more likely to be funded than a nonmodular grant

## Overall Satisfaction with Just-in-Time

- Overall, how satisfied are you with Just-in-Time?*

- *Just-in-Time survey items were not included on peer reviewer or scientific review administrator surveys

## Conclusions

- Modular grant application process...
- is generally well received in the extramural community, somewhat less well received by NIH staff
- has achieved the goals of...
- reducing administrative burden (most for peer reviewers and scientific review administrators than others)
- focusing peer reviewers on the scientific content of the application
- has resulted in less study section meeting time devoted to budget discussions
- Peer review can be performed well without a detailed budget or IRB/IACUC approval
- Extramural community is very satisfied with the information provided on the modular grant application process by NIH staff

## Conclusions and Recommendations for Modular Grant Application Process

- While the modular grant application process solved some problems, it created others
- Principal investigators required to prepare two budgets
- Practices used to reduce burden of two budgets could be shared among institutional officials and/or learned and disseminated by NIH
- Peer reviewers less able to conduct a good assessment without “Other Support” pages
- Consider re-implementing “Other Support” pages or obtaining similar information at time of application submission
- Lack of understanding and misconceptions about some aspects of the modular grant application process
- Use Phase II findings as a guide to developing more targeted on-line training and educational modules

## Conclusions and Recommendations for Just-in-Time

- Just-in-Time is better received by extramural community than by NIH staff
- Procedures for requesting and receiving materials are problematic
- Assign one contact person for working with principal investigators and institutional officials to get Just-in-Time materials in
- Request materials from only one point person (PI? IO?) at an institution
- Routinely test NIH “Commons” electronic submission and receipt functioning
- Develop better document tracking system for materials that are received
- Too many people are notified that they may receive an award
- Consider decreasing cutoff  percentage
- Turnaround time for applicants to submit materials is too short
- Consider notifying applicants earlier and/or providing them with more time when possible

## On the Modular Grant Outcome Evaluation Web Site

- Introduction/Background
- Methodology
- Surveys
- Results
- Overview
- Highlight
- Specific Topics
- Main Themes from Open-Ended Comments
- All Open-Ended Comments
- All Survey Responses