# Presentation: 670188

## Shortening the Review Cycle for New Investigators (R01*)  Preliminary Evaluation 

***February 7, 2007***
***Eileen W. Bradley, DSc.***
***For the evaluation committee***

## Goals of Pilot

***    ***
- Allow amended applications for new investigators (R01*) to be submitted for the very next cycle
- Compress referral, review, and SRA processing while maintaining core review principles
- Test feasibility for expansion to all applications reviewed by CSR

## PILOT DESIGN

***New receipt dates for amended***
***    ******July 20, Nov 20, March 20***
***Rapid summary statement release  (1 week)***
***Return to same study section***

## Cycle 1 (Feb – Oct)

**7/20 R01* Amended**

**11/20 R01* Amended**

_**Cycle 1 (Feb – Oct)**_

_**Cycle 2 (Jun – Feb)**_

_**Cycle 3 (Oct – Jun)**_

## EVALUATION

***SURVEY OF STAKEHOLDERS***
***IMPAC ll DATA***

## IMPAC ll Questions

- Who resubmitted
- How did they  fare?

## Who resubmitted?

**How many eligible R01* applicants resubmitted?**
- 				(Resubmitted/ Eligible)
- Cycle 1:  		  83/628    (13%)
- Cycle 2:  		  79/579    (13%)
- __________________________
- Total to date:	162/1207  ( 13%)

**Notes:**
“Eligible R01 applicant – any new investigator with an initial or A1 amended R01 application reviewed in the pilot study sections and who was not funded.

## Non Pilot CONTROLS

***New Investigator APPLICATIONS RESUBMITTED  NOVEMBER 1***
***     ******(RATHER THAN JULY 20)***
***New Investigator APPLICATIONS SUBMITTED IN 2005 AND REVIEWED IN SAME SS AS R01* (RESUBMITTED  JULY 1)***

## DATA FROM CYCLE 1 ONLY

***CONTROLS --- SUBMITTED***
***AS AMENDED FOR JULY 1, 2006***
***CONTROLS AND PILOT R01* ***
***REVIEWED AT THE SAME SS***

## Cycle 1 (Feb – Oct)

**7/20 R01* Amended**

**11/20 R01* Amended**

_**Cycle 1 (Feb – Oct)**_

_**Cycle 2 (Jun – Feb)**_

_**Cycle 3 (Oct – Jun)**_

## How did the re-submitters fare?

- 30% of pilot R01* resubmitters’ new scores fell in <15th  percentile range, compared to 16% of non-pilot new PIs
- About one third of pilot R01* resubmitters were unscored, compared to one half of non-pilot

**Notes:**
Table with data follows this slide

## How did the re-submitters fare? Comparison of Pilot vs. Non-Pilot Percentiles of Re-submitters

| Percentile | Pilot PIs |  | Non-Pilot PIs |  |
| --- | --- | --- | --- | --- |
|  | N | % | N | % |
| < 15 
               (sig. Chi Sq., p<.01) | 25 | 30 | 33 | 16 |
| >15 | 32 | 39 | 71 | 33 |
| Unscored | 26 | 31 | 107 | 51 |
|    TOTAL | 83 | 100 | 211 |    100 |

**Notes:**
Notes:

## Movement in Resubmitters Percentiles (Green improved, red got worse)

| Percentile 
Group –
 initial application | # Apps | Percentile group – 
revised application |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | <10th | 10-25th | >25-50th | >50 | UNS |
| 10-25th | 21 | 62% | 33% | 5% |  |  |
| >25-50th | 31 | 23% | 23% | 29% | 10% | 16% |
| >50th | 5 | 20% | 2% | 20% | 0% | 40% |
| UNS | 26 | 0% | 15% | 8% | 4% | 73% |

## Current status of R01*

- App Status           #       %
- Awd/TBP              7      8.4%
- Pending Awd       5       6.0%
- Pending Council  37    44.6%
- Unscored            22     26.5%
- Withdrawn          12     14.5%
- Total                   83     100.0%

## Analysis OF SURVEYS

*** ******OF STAKEHOLDERS***
***REFERRAL OFFICERS, SRAs***
***POs,   and REVIEWERS***
***Data just coming in***

## What are the barriers to full implementation?

**Receipt & Referral:**
- Too slow– _**all**_ applications must be received in the IRG by the deadline
- Only 81% of applications referred by deadline
**Review:**
- Recruitment problems – Difficult to recruit reviewers when meetings clustered close to their own application receipt dates
- Since SRAs needed to do DEAS work,  full ramp-up could overwhelm system
- Reviewers need to send edits to SRAs sooner
**IMPAC II:**
- Old version of application not retained & system too slow

**Notes:**
Committee informational note on first bullet:  
SRA survey data --
38% reported not receiving all applications by deadline 
15% reported receiving all applications by deadline
47% reported that they could not remember whether they received all applications by deadline

## Next Steps for Evaluation

- Collect additional data via stakeholder surveys in next council round
- Complete analyses of stakeholder surveys; results available in March 2007
- Complete analyses of IMPAC II data
- Present results to NIH committees and PRAC in early to mid 2007

## The Evaluation Subcommittee

**Co-Chairs:  **
**Bettie J. Graham, Ph.D., NHGRI and  **
**Eileen Bradley, D.Sc., CSR**
**Members:**
**  ****Teresa Levitin, Ph.D., NIDA**
**  ****Philip Smith, Ph.D., NIDDK**
**  ****Jane Steinberg, Ph.D., NIMH**
**  ****Susan Streufert, Ph.D,                                  ****	****NICHD**

**Staff:**
**Teresa Lindquist, M.S., NIDDK**
**Terra Vinson, M.PA, NIDDK**
**CSR Staff:**
**Charles Dumais, M.S.**
**Cheryl J. Oros, Ph.D.**
**Elliot Postow, Ph.D.**
**Consultants: **
**Robert Hammond, Ph.D. **
**Georgine Pion, Ph.D. **