# Presentation: 425264

## Improving Peer Review

**Improving Peer Review**

**CSR Initiatives**
***toni scarpa***

**PRAC,  April 30, 2008 **

**National Institutes of Health****U.S. Department of Health and Human**** ****Services**

## New Data

Improving Peer Review: CSR Initiatives

Changes In CSR Organization

Realigning CSR Peer Review

**New Data**
**Improving Peer Review: CSR Initiatives**
**Changes In CSR Organization**
**Realigning CSR Peer Review**

**Peer Review at CSR**

## New Data

## Number of Applications Submitted

**Number of Applications Submitted**

**Historical Growth**

## SBIR  Applications

**SBIR  Applications**

- 7000

- 6000

- 5000

- 4000

- 3000

- 2000

- 1000

- 0

- 1999

- 2000

- 2001

- 2002

- 2003

- 2004

- 2005

- 2006

- 2007

**INST**
**CSR**

## Success Rates of Competing SBIR Applications

- 60

- 0

- 2000

- 2001

- 2002

- 2003

- 2004

- 2005

- 2006

- 2007

- Fiscal Year

- 50

- 40

- 30

- 10

- 1998

- 1999

- Percent Success Rate

- 20

- SBIR Phase I
- SBIR Phase II

## R01 Applications Reviewed at CSR vs. Other ICs

## R21 Applications Reviewed at CSR vs. Other ICs

## % of RO1 Application Submitted, Type 1

## % of RO1 Application Submitted, Type 2

## Reviewer’s Load

**Reviewer’s Load**

- Applications Per Reviewer

**Oct Council Rounds**

## Colleagues Reviewing for CSR

## CSR Budget for 2008

**CSR Budget                                                   $ 63 M**
**SREA (Reviewers Expenses Budget)****	****$ 39 M**
**Review Cost, including travel for 17,000 reviewers,**
**    ****is less than 0.2% of the budget requested in the applications reviewed**

## SREA Budget Progression

**Millions**

## Travel

**Purchase non-refundable (restricted) coach tickets**
**instead of unrestricted tickets**

| Preliminary Pilot Data – June 2006 through July 2007 |  |
| --- | --- |
|  |  |
| # of tickets issued   | 28,493 |
| Changes requested  | 2,544 |
| cost of changes + fees   | $550,274 |
|  |  |
| YTD Fare Savings | $14,522,877 |

## Additional Savings

|  | Airfare | Hotel | Meeting Average Cost |
| --- | --- | --- | --- |
| Washington |      $ 7,682 |     $ 19,166 |     $ 26,848 |
| West Coast |      $ 6,486 |     $   8,974 |     $ 15,464 |

## Improving Peer Review

**CSR Initiatives**

## Improving Peer Review

**Matrix ****A****: Changes in CSR Operations**
** **
**Matrix ****B****: Changes in Peer Review**
**Matrix ****C****: System Changes?**

## Matrix A: Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
**Review Enabling Committees**
**Increase Efficiency**
**Division Directors, Chiefs and SROs Recruitment**
**	****	**

**Matrix ****A****: Changes in CSR Operations**

## Matrix A: Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
**	****	**

**Matrix ****A****: Changes in CSR Operations**

## Matrix A. Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
** ****Realign Divisions and IRGs with the changes in science**
**Increase efficiency and effectiveness**
**Increase consistency**
**Recruitment of Scientific Review Officers**

**Matrix A. Changes in CSR Operations**

## New Division Neuroscience, Aging and Development

**BDCN/Brain Disorders & Clinical Neuroscience**
**	****(est. 11 SRAs)**
**IFCN/Integrative, Functional, & Cognitive Neuroscience**
**	****(est. 9 SRAs)**
**MDCN/Molecular, Cellular & Developmental Neuroscience**
**	****(est. 10 SRAs)**
**ENST/Emerging Neuroscience & Training**
**	****(est. 8 SRAs)**
**BDA/Biology of Development & Aging  (7 SRAs)**

## Organization of Peer Review at CSR

**Divisions**

**Study Sections**

**I.R.G.s**
**(Integrated Review Groups)**

## New CSR Review Divisions

**Scientific **
**Review **
**Groups=**
**48**

**Scientific **
**Review **
**Groups=**
**50**

**Scientific **
**Review **
**Groups=**
**43**

**Scientific **
**Review **
**Groups=**
**44**

**Scientific **
**Review **
**Groups=**
**55**

## Don Schneider

## Matrix A: Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
**Review Enabling Committees**
**	****	**

**Matrix ****A****: Changes in CSR Operations**

## Review Enabling Committees

**Policy (Martin)**
**Scientific Community Outreach (Schneider)**
**Recruitment of SROs (Gibson)**
**Recruitment of Reviewers (Stassi)**
**Knowledge Management (Many)**
**Peer Review Electronic (Many)**
**SRA Training (Freund)**
**Mentoring and staff development (Many)**
**Best review practices (Kitt)**

## Matrix A: Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
**Review Enabling Committees**
**Increase Efficiency**
**	****	**

**Matrix ****A****: Changes in CSR Operations**

## Increase Efficiency 

**How we distribute 80,000 applications?**
**Retooled for electronic submission**
**Applications are now submitted electronically**
**Assign applications using text fingerprinting**
**	****	****	**** ****Implementation by August 2008**

## Matrix A: Changes in CSR Operations

**CSR Reorganization of Division and IRGs**
**Review Enabling Committees**
**Increase Efficiency**
**Division Directors, Chiefs and SROs Recruitment**
**	****	**

**Matrix ****A****: Changes in CSR Operations**

## Matrix A: Changes in CSR Operations

**Recruitment**
**      ****3    Division Directors**
**	****	****First Search (Richard Nakamura, Chair)**
**6    Integrated Review Chiefs**
**40  Scientific Review Officers**
** **

**Matrix ****A****: Changes in CSR Operations**

## Matrix B: Changes in Peer Review

**Improve study section alignment and performance**
**Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**
**Improve the identification of significant, innovative and high-impact research**

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**

## Second Round of Internal IRG Review

**July 24, 2008:  BDCN and MDCN**
**September 15, 2008:  ETTN, BDA and IFCN**
**January 2009: BBBP and HOP 2 (HDM)**
**March 2009:  RPHB and HOP 1 (EPS)**
**May 2009:  IMM and IDM**
**July 2009:  ONC 1-Basic and ONC 2-Clinical**
**September 2009:  BST and SBIB**
**November 2009:  AARR and EMNR**
**January 2010:  VH and CVR**
**March 2010:  MOSS and DRUS**
**May 2010:  GGG and BCMB**
**July 2010:  IMST and CB**

## Two Major Challenges

**Percentiling or Ranking Applications Reviewed in SEPS**** **
**Locus of Review of “Orphan Applications”**

## Don Schneider 

**Clustering of grant application for review**

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**
** ****Shorten the review cycle**

## Shortening the Review Cycle The Goal

**To provide applicants a review and score within 3 months of application submission. This will permit resubmission of applications (when doable and desirable) 4 months earlier than in the past.**
**To permit 3 reviews within one year**

## R01 A1 Resubmissions Reviewed in the Cycle Immediately following the Original Application's Review

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**
** ****Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**

## Colleagues Reviewing for CSR

## Academic Rank of All CSR Reviewers

**Academic Rank of All CSR Reviewers**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**	****Remake peer review a learning experience**
**Develop a national registry of volunteer reviewers**
**Searchable Database with 2700 reviewers**
**Move a meeting/year to the west coast**
**50 % this year, 90% next year**
**Provide tangible rewards for reviewers**
**No submission deadlines for permanent members of Study Sections (effective February 2008)**
**    ****Shorten applications  **
**Require less travel by expanding peer review platforms**

## Expanding Peer Review’s Platforms

**Study Sections**

**Necessity****	****	****    ****●****  **** ****Clinical reviewers**
** **
**Preference     ****              ****●  **** Physicists, computational biologists**
**New Opportunities****      ****● ** **Fogarty, International Reviewers**

**Goal:  15% of all reviews to be electronic in 2008****	**

## Matrix C: System Changes?

**The Advisory Committees of NIH Director**

## Advisory Committees to the NIH Director

**The Charge from Dr. Zerhouni: **
***	******“******Fund the best science, by the best scientists, ***
***	******  ******with the least administrative burden...”***

**http://enhancing-peer-review.nih.gov**

**A Self-Study by the NIH in Partnership with the Scientific Community **
**to Strengthen Peer Review in Changing Times**

## This is CSR