# Presentation: 662093

## Improving Peer Review

**Improving Peer Review**

**CSR Initiatives**
***toni scarpa***

**PRAC, December 3, 2007 **

**National Institutes of Health****U.S. Department of Health and Human**** ****Services**

## New Data

CSR Organization Initiatives

CSR Recent Activities

Realigning CSR Peer Review

**New Data**
**CSR Organization Initiatives**
**CSR Recent Activities**
**Realigning CSR Peer Review**

**Peer Review at CSR**

## New Data

## Number of Applications Submitted

**Number of Applications Submitted**

**Historical Growth**

## CSR Reviewed Application by FY

**CSR Reviewed Application by FY**

## R01 Applications Reviewed at CSR vs. Other ICs

## R21 Applications Reviewed at CSR vs. Other ICs

## Locus of NIH Reviews

**Locus of NIH Reviews**

## Locus of Review by Application Type

**Locus of Review by Application Type**

## Slide 10

## CSR Review of Women Applications, Women Scoring 10 and 20 %

## NIH 20%tile or Better Scoring (R01s, CSR reviews)

| Councils | New Applications New Investigators
(Type 1) | New Applications Established Investigators (Type 1) | Renewals From Established Investigators
(Type 2) |
| --- | --- | --- | --- |
| May 2001 | 15% | 17% | 34% |
| May 2002 | 15% | 17% | 34% |
| May 2003 | 15% | 18% | 33% |
| May 2004 | 14% | 18% | 34% |
| May 2005 | 15% | 18% | 33% |
| May 2006 | 14% | 18% | 34% |
| May 2007  | 14% | 18% | 32% |

**CSR New and Established Investigators Score%**

## Percent Distribution of Type 1 and Type 2 R01 Applications: 2003 vs. 2007

**Percent Distribution of Type 1 and Type 2 ****R01 Applications: 2003 vs. 2007**

## Percent Distribution of Amended Applications by Type

**Percent Distribution of Amended Applications by Type**
**FY 2003 vs. 2007**

## CSR Organization Initiatives

## Changes in CSR Operations

**Realign Divisions and IRGs with the changes in science**
**Increase efficiency and effectiveness**
**Increase consistency**
**Recruitment of Scientific Review Officers**

**Changes in CSR Operations**

## Organization of Peer Review at CSR

**Divisions**

**Study Sections**

**IRGs**
**(Integrated Review Groups)**

## New IRG

**ETTN/Emerging Technologies and Training in Neurosciences**

## New Division Neuroscience, Aging and Development

**BDCN/Brain Disorders & Clinical Neuroscience**
**	****(est. 11 SROs)**
**IFCN/Integrative, Functional, & Cognitive Neuroscience**
**	****(est. 9 SROs)**
**MDCN/Molecular, Cellular & Developmental Neuroscience**
**	****(est. 10 SROs)**
**ETTN/Emerging Technologies and Training in Neurosciences**
**	****(est. 8 SROs)**
**BDA/Biology of Development & Aging  (7 SROs)**

## Organization of Peer Review at CSR

**Divisions**

**Study Sections**

**IRGs**
**(Integrated Review Groups)**

## Anita Miller Sostek

**PRAC Subcommittee Report on A Second Division Realignment**

## Review Enabling Committees

**Policy (Martin)**
**Scientific Community Outreach (Schneider)**
**Recruitment of SROs (Gibson)**
**Recruitment of Reviewers (Stassi)**
**Knowledge Management (Tatham)**
**Electronic Reviews (Many)**
**SRO Training (Freund)**
**Mentoring and staff development (Many)**
**Best review practices (Kitt)**

## Increase Efficiency 

**How we distribute 80,000 applications?**
**Retooled for electronic submission**
**Applications are now submitted electronically**
**Assign applications using text fingerprinting**
**	****	****	**** ****Implementation by February 2008**

## CSR Recent Activities

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**
** ****Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**
**Improve the identification of significant, innovative and high-impact research**

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**

## Improve Study Section Alignment and Performance a) Biannual IRG Review Schedule

| Scheduled 2007 -- 9 IRGs |
| --- |
| Biology of Development and Aging (BDA) |
| Infectious Diseases and Microbiology (IDM) |
| Biobehavioral and Behavioral Processes (BBBP) |
| Cell Biology (CB) |
| Musculoskeletal, Oral and Skin Sciences (MOSS) |
| Oncological Sciences (ONC) |
| Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) |
| Respiratory Sciences (RES) |
| Renal and Urological Sciences (RUS) |

| Scheduled 2006 --14 IRGs |
| --- |
| Biological Chemistry and Macromolecular Biophysics (BCMB) |
| Cardiovascular Science (CVS) |
| Bioengineering Sciences and Technologies (BST) |
| AIDS and Related Research (AARR) |
| Risk, Prevention, and Health Behavior (RPHB) |
| Genes, Genomes and Genetics (GGG) |
| Digestive Sciences (DIG) |
| Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) |
| Brain Disorders and Clinical Neuroscience (BDCN) |
| Integrative, Functional and Cognitive Neuroscience (IFCN) |
| Molecular, Cellular and Developmental Neuroscience (MDCN) |
| Hematology (HEME) |
| Immunology (IMM) |
| Health of the Population (HOP) |

## 1. Improve Study Section Alignment and Performanceb) Six Open House Workshops in 2007

| Mar. 2, | Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN)  |
| --- | --- |
| April 25 | Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB) |
| Jun. 29 | Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC);  Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB)  |
| Aug.14 | Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) |
| Nov. 9 | Integrated Biological (5): Immunology (IMM); Hematology  (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA)  |
| Dec. 18 | Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG) |

## Don Schneider 

**Clustering of Grant Application for Review**

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**
** ****Shorten the review cycle**

## Shortening the Review Cycle The Goal

**To provide applicants a review and score within 3 months of application submission. This will permit resubmission of applications (when doable and desirable) 4 months earlier than in the past.**
**To permit 3 reviews within one year**

## 2.  Shortening the Review Cycle

**First Step: ****Posting Summary Statements Earlier**
**Post new investigator within ****10 days,**
** ****(within 30 days ****for established investigators****)**
**Second Step: ****Pilot to Shorten the Review Cycle for New R01 Investigators**
**Last year:  About ****2,000**** new investigators in pilot**
**Third Step: ****Shorter Review Cycles to All R01 From New Investigators**
**November 2007. ****All new Investigators**** (about 10,000)**
**June 2007: **** 6,000**** new investigators**
**Nov 2007:    All new investigators (>9000)**

## R01 A1 Resubmissions Reviewed in the Cycle Immediately following the Original Application's Review

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Improve Study Section Alignment and Performance**
** ****Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**

## Reviewers – Past Situation 

**Far too many reviewers on study sections**
**Broader science**
**Decrease in reviewer load **
**Unnecessary**
**Too many ad hoc reviewers**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Remaking peer review a learning experience**
**National registry of volunteer reviewers**
**Moving one third of the meetings in the west coast**
**Rewards for reviewers**
**Require less travel by expanding peer review platforms **
**Shorten applications**

## Colleagues Reviewing for CSR

## Reviewer’s Load

**Reviewer’s Load**

- Applications Per Reviewer

**Oct Council Rounds**

## Academic Rank of CSR Reviewers

**Academic Rank of CSR Reviewers**
** ****Regular, Temp and SEP**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Require less travel by using electronic review modes**

## Expanding Peer Review’s Platforms

**Study Sections**

**Necessity****	****	****    ****●****  **** ****Clinical reviewers**
** **
**Preference     ****              ****●  **** Physicists, computational biologists**
**New Opportunities****      ****● ** **Fogarty, International Reviewers**

**Our Goal:  10% of all reviews to be electronic in 2007**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Require less travel by using electronic review modes**
**Have Shorter Meetings**
**Shorten Applications**

## Shorter Applications

**Trans NIH Committee**
**Communication and Support from Societies, Diseases Groups, Scientific Leadership  **
**2 Retreats of NIH Directors**
**Approval and Support of our Advisory Committee**
**Survey of Scientists**

## Cheryl Kitt

**Trans-NIH Team Short for Application**

## This is CSR