# Presentation: 633648

## Outcome Study for the Evaluation of the Modular Grants Program

- May 2005

- Prepared by Westat under Contract # GS-23F-8144H

## Introduction

## Background on the Modular Grant Application Process

- June 1, 1999 NIH initiated the modular grant application process
- Intent was to reinforce the grant-in-aid philosophy of supporting research (as opposed to the contract mentality of “buying” research)

## Goals of the Modular Grant Application Process

- Goals:
- Focus the efforts of peer reviewers, principal investigators, institutional officials, and NIH staff on the science of the applications (rather than on the itemized budget)
- Reduce administrative burden for all stakeholders and simplify the grant administration process
- Accommodate principal investigators’ need for flexibility

## History of the Modular Grant Application Process Evaluation

- Nov 2001 – Modular Evaluation Workgroup developed a report on guidelines for the evaluation of the modular process
- Feb 2002 – Peer Review Oversight Group (PROG) approved report; Phase I Feasibility Study funds made available
- April 2003 – Phase I feasibility study began
- Jan/Feb 2004 – PROG members review Phase I results and approve survey instruments
- March 2004 – Phase I presentation at EPMC meeting, requesting feedback on surveys
- April 2004 – Phase II outcome evaluation study began
- May 2005 – Phase II presentation and materials delivered to NIH

## Evaluation

- Phase I: Feasibility Study
- Phase II: Outcome Study

## Phase I Feasibility Study Purpose

- Overall purpose: To develop survey instruments for five NIH stakeholder groups
- Principal Investigators (PIs)
- Peer Reviewers (PRs)
- Scientific Review Administrators (SRAs)
- Program and Grants Management Staff (PGMS)
- Institutional Officials (IOs)

## Phase I Feasibility Study Activities

- Conducted 5 focus groups (one with each stakeholder group) to identify content for the surveys
- Conducted 55 cognitive interviews during two rounds of pre-testing the surveys to refine instructions, survey questions, and response options
- Drew samples of stakeholder populations

## Phase II Outcome Study Purpose

- Overall purpose
- Determine if the modular grants application process met its intended long-term program goals
- Determine if stakeholders understand the modular grant application process

## Phase II Outcome Study Activities

- Developed and tested five web surveys
- Administered web surveys
- Conducted quantitative and qualitative data analysis
- Produced slide presentations, executive summary, summary reports, methodology report, quantitative and qualitative results, data files

## Methodology

## Samples: Census Groups

- Used a census for two stakeholder groups
- NIH Scientific Review Administrators (N=372)
- CSR and Non-CSR Scientific Review Administrators in NIH Health Scientist Administrator (HSA) database as of August 2004
-  Institutions (Institutional Officials) (N=327)
- Institutions that applied for 10 or more modular grants, excluding foreign institutions (but including Canada); used National Council for University Research Administrators (NCURA) database or institution web site to find highest ranking person (e.g., Director of Office of Sponsored Research); gave highest ranking person option to let another staff member more knowledgeable take the survey

## Samples: Non-census Groups

- Used a systematic random sample for three stakeholder groups
- Principal investigators
- Principal investigators that had submitted a modular grant application between September 1999 and March 2004
- Peer reviewers
- NIH Peer reviewers serving as of December 2003 and those who had rotated off during the previous 2 years
- NIH program and grants management staff
- Program staff in NIH Health Scientist Administrator (HSA) database as of August 2004
- Grants management officers and grants management specialists listed in Grants Management Advisory Committee (GMAC) Directory
- Oversampled non-census groups to account for:
- 60% response rate
- 15% “bounce back” (undeliverable) emails for non-NIH employees

## Sample Sizes

| Stakeholder Group | Population Size | Sample Size
(# of surveys administered) |
| --- | --- | --- |
| Principal Investigators | 31,180 | 1,676 |
| Peer Reviewers | 2,836 | 753 |
| Scientific Review Administrators | 372 | 372 |
| Program and Grants Management Staff | 1,213 | 488 |
| Institutional Officials | 327 | 327 |

## Data Collection

- Each stakeholder (respondent) was contacted up to five times using the following methods:
- Contact 1: Pre-notification letter mailed
- Contact 2: Survey notification emailed
- Contact 3: First reminder emailed
- Contact 4: Second reminder emailed
- Contact 5: Telephone call made to respondent

## Data Collection Schedule: Fall 2004

## Final Sample Sizes and Response Rates

| Stakeholder Group | Final Sample Size* | Completed Surveys | Response Rate |
| --- | --- | --- | --- |
| Institutional Officials | 288 | 227 | 78.8% |
| Peer Reviewers | 671 | 497 | 74.1% |
| Scientific Review Administrators | 285 | 209 | 73.3% |
| Program and Grants Management Staff | 355 | 247 | 69.6% |
| Principal Investigators (PIs) | 1,439 | 949 | 65.9% |
| 	PIs – Recipients of Modular Grants | 879 | 644 | 73.3% |
| 	PIs – Unfunded Applicants | 560 | 305 | 54.5% |

- *Includes only eligible respondents

## Confidence Intervals

- Overall Results

## Overall Satisfaction with Modular Grants

- Overall, how satisfied are you with the modular grant application process?

## Reasons Stakeholders Like Modular Grants

- Themes from open-ended comments
- Don’t have to submit detailed budget, budget justification, or Other Support pages (PIs, IOs)
- Saves time...
- in preparing the application (PIs)
- during review and study section discussions (PRs, SRAs)
- since there is less paperwork to fill out (e.g., don’t have to complete cost analysis) (PGMS)
- because less time is spent reviewing budgets for proposals (IOs)
- Allows PIs to concentrate on science (PIs); focuses study section meetings on science (PRs, SRAs)

## Reasons Stakeholders Do Not Like Modular Grants

- Themes from open-ended comments
- Institutions often require a detailed budget in addition to a modular budget which means preparing a modular grant application is actually more time consuming (PIs, IOs)
- Adding subcontractor/consortium indirect costs to project direct costs (PIs, PRs, IOs)
- Limit of $250,000 (PIs, IOs)
- Can’t assess overlap (no Other Support pages) (PIs, PRs, PGMS)
- Difficult to evaluate true cost of research (PRs, SRAs, PGMS)
- Difficult to make recommended cuts/justify budget reductions (PRs, SRAs, PGMS)
- Budgets are inflated (PRs, SRAs, PGMS)
- Lack of detail prevents NIH staff from doing their job properly (oversight is lost) (PGMS)
- Can’t be sure F&A is calculated correctly (PGM)
- Modular process confuses PIs (PGMS, IOs)
- PIs don’t monitor funds appropriately (IOs)

## Raising the Modular Grant Limit

- Would you prefer the modular grant limit be higher than $250,000?

## Reasons Stakeholders Want the Modular Grant Limit Raised

- Themes from open-ended comments
- Cost of research has increased since implementation, dollar amount hasn’t kept up with inflation, cost of living (all stakeholders)
- Would facilitate research collaboration (PIs, PRs, IOs)
- Currently certain types of research are inherently excluded (e.g., clinical trials, large animals, transgenic animals, large pieces of equipment, multi-site) (all stakeholders)
- Easier to have just one budget system (IOs)
- PIs will no longer restrict their research to $250,000 (to avoid having to submit a detailed budget) (IOs)

## Reasons Stakeholders Do Not Want the Modular Grant Limit Raised

- Themes from open-ended comments
- Higher limit would lead to fewer awards (PIs, PRs)
- Raising the limit would encourage PIs to ask for more money (since many simply ask for the largest amount available) (PIs, PRs, SRAs, IOs)
- Proposed budgets greater than $250,000 should stay detailed because..
- PIs should be held more accountable for more expensive research (PIs, PRs, IOs)
- Reviewers need this information to conduct quality reviews of more expensive studies (PRs, SRAs)
- Raising the limit would not allow for proper stewardship of more expensive projects (PGMS)
- Proposals with larger budgets can always use nonmodular format (all stakeholders)

## Compromise Between a Detailed and Modular Budget

- Listing dollar value totals for major categories (personnel, travel, etc.) would be a good compromise between a detailed budget and a modular budget

## Focusing Peer Reviewers on Science

## Assessing Scientific Merit

- I/Peer reviewers can assess the scientific merit of a modular grant application without...

## Reducing Administrative Burden for All Stakeholders and Simplifying the Grant Administration Process

## Changes in Job Responsibility Due to Modular Grants

- As a direct result of the modular grant application process, my responsibilities have decreased

## Institutions Require Detailed Budgets

- Does your office or any other office at your university/research center require principal investigators to submit detailed budgets for review even if they are applying for an NIH modular grant?

## Reasons Institutions Require a Detailed Budget

- Themes from open-ended comments made by institutional officials
- Information needed for accounting system, finance department, internal planning
- PIs need to develop a detailed budget to help plan their research; will help PIs determine dollar amount needed
- Ensures all costs are accounted for, no financial risk to the institution, make sure the PI includes all necessary costs
- Helps departments monitor their expenditures; need to know what commitments were made by the PI, need to be sure PIs don’t overcommit personnel and resources
- Need detailed budget to calculate F&A correctly; need to make sure proposals are compliant with A21 regulations

## Institutional Officials and Auditors

## Budget Discussions

- As a direct result of the modular grant application process, discussions about the budget in my study section are much more limited

## Study Section Meeting Time

- On average, how much study section meeting time was devoted to discussing an applicant’s proposed budget before and after the implementation of the modular grant application process?

## Program and Grants Management Staff Experiences with Modular Grants

## Providing Principal Investigators with Flexibility

## PI Flexibility

- A modular grant provides me with more flexibility in terms of...

## Understanding the Modular Grant Application Process

## Satisfaction with Information Received from NIH Staff

## Knowledge about Features of the Modular Grant Application Process

## Best and Least Understood Features of the Modular Grant Application Process

- Best understood features:
- All forms for modular grant applications are available on the NIH web site
- Biographical sketches need to be prepared for all key personnel
- Individual salary information is not required for personnel
- Least understood features:
- Total consortium/contractual costs need to be estimated for each year, rounded to the nearest $1,000
- Modular grant awards are eligible for administrative supplements (i.e., noncompeting supplemental funding)

## Training for Peer Reviewers

- Since the modular grant applications only list total costs, new reviewers should receive training on how to determine whether or not proposed costs are reasonable

## Perceptions of Average Award Size

- As a direct result of the modular grant application process, my perception is that the average size of an award (dollar amount) has increased

## Perceptions of Average Amount of Funding Requested

- As a direct result of the modular grant application process, my perception is that the average amount of funding requested has increased

## Perception of Funding Likelihood for Modular Compared to Nonmodular Grant Applications

- I think/PIs believe a modular grant is more likely to be funded than a nonmodular grant

## Just-in-Time Results

## Overall Satisfaction with Just-in-Time

- Overall, how satisfied are you with Just-in-Time?*

- *Just-in-Time survey items were not included on peer reviewer or scientific review administrator surveys

## Reasons Stakeholders Like Just-in-Time

- Themes from open-ended comments made by principal investigators, program and grants management staff, and institutional officials
- Information submitted by PIs is up to date (PIs, PGMS, IOs)
- Eases administrative burden and relieves stress – don’t have to worry about IRB/IACUC approvals unless grant may be funded; gives PIs more time to prepare materials (PIs, PGMS, IOs)
- Saves everyone time if grant is unlikely to be funded (including IRB and IACUC committee members) (PIs)
- Less paperwork to deal with at time of submission (PIs, PGMS, IOs)
- Ability to submit materials electronically through NIH Commons (PGMS, IOs)
- Allows PIs to focus on science and not deal with other administrative aspects (e.g., Other Support pages, approvals)

## Reasons Stakeholders Do Not Like Just-in-Time

- Themes from open-ended comments made by principal investigators, program and grants management staff, and institutional officials
- Notice from NIH to get materials in is too short, resulting in “big rush” ”mad scramble” to obtain approvals, funding can get held up (PIs, PGMS, IOs)
- Don’t like pushing and pressuring PIs (PGMS)
- Institution requires approvals be obtained earlier than NIH (negates Just-in-Time benefits) (PIs)
- Lack of coordination, lack of communication, and poor document tracking at NIH – consistently have to submit materials more than once (PIs, IOs); Sometimes more than one NIH staff member will contact the PI, problems tracking the information internally once received, makes NIH look disorganized (PGMS)
- PIs receive Just-in-Time requests then are not funded – very disappointing and demoralizing for PIs (IOs); automated reminder is sent to too high a percentage of grant applicants, PIs get their hopes up (PGMS)
- Problems with NIH “Commons” electronic submissions and NIH access – results in also having to submit hard copies (IOs)
- NIH staff have to follow-up with PIs to track down missing materials (PGMS)

## Satisfaction with Just-in-Time Procedures 

## Conclusions

- Modular grant application process...
- is generally well received in the extramural community, somewhat less well received by NIH staff
- has achieved the goals of...
- reducing administrative burden (most for peer reviewers and scientific review administrators than others)
- focusing peer reviewers on the scientific content of the application
- providing PIs with increased flexibility
- has resulted in less time reviewing grant applications and less study section meeting time devoted to budget discussions
- Peer review can be performed well without a detailed budget or IRB/IACUC approval
- Extramural community is very satisfied with the information provided on the modular grant application process by NIH staff
- Mixed reaction to raising the limit and using dollar value totals for major categories

## Conclusions and Recommendations for Modular Grant Application Process

- While the modular grant application process solved some problems, it created others
- Principal investigators required to prepare two budgets
- Practices used to reduce burden of two budgets could be shared among institutional officials and/or learned and disseminated by NIH
- Peer reviewers less able to conduct a good assessment without “Other Support” pages
- Consider re-implementing “Other Support” pages or obtaining similar information at time of application submission
- Lack of understanding about some of the nuances of the modular grant application process
- Educate stakeholders by posting Phase II results regarding modular grant features on NIH web site, and/or by other means
- Misperceptions exist about modular grant application process
- Educate stakeholders by including facts next to Phase II perception findings

## Conclusions and Recommendations for Just-in-Time

- Just-in-Time is better received by extramural community than by NIH staff
- Procedures for requesting and receiving materials are problematic
- Assign one contact person for working with principal investigators and institutional officials to get Just-in-Time materials in
- Request materials from only one point person (PI? IO?) at an institution
- Routinely test NIH “Commons” electronic submission and receipt functioning
- Develop better document tracking system for materials that are received
- Too many people are notified that they may receive an award
- Consider decreasing cutoff  percentage
- Turnaround time for applicants to submit materials is too short
- Consider notifying applicants earlier and/or providing them with more time when possible

## On the Modular Grant Outcome Evaluation Web Site

- Introduction/Background
- Methodology
- Surveys
- Results
- Overview
- Highlight
- Specific Topics
- Main Themes from Open-Ended Comments
- All Open-Ended Comments
- All Survey Responses