# Presentation: 527421

## Improving Peer Review

**Improving Peer Review**

**CSR Initiatives**

**PRAC,  August 27,  2007 **

**National Institutes of Health****U.S. Department of Health and Human**** ****Services**

## CSR Data

CSR Recent Activities

CSR Organization Initiatives

Realigning CSR Peer Review

Initiatives Under Early Discussion

**CSR Data**
**CSR Recent Activities**
**CSR Organization Initiatives**
**Realigning CSR Peer Review**
**Initiatives Under Early Discussion**

**Peer Review at CSR**

## CSR Data

## Number of Applications Submitted to NIH

**Number of Applications Submitted to NIH**

**Historical Growth**** **

## Changes Impacting CSR Peer Reviews

|  | 5 Years Ago | Now |
| --- | --- | --- |
| Applications Received | 46,000 | 80,000 |
| R01s and R21s Reviewed by CSR | 21,000 | 39,000 |

|  | 10 Years Ago | Now |
| --- | --- | --- |
| Average Number of Applications Reviewed by Reviewers | 
11.6 | 
6 |

**The first problem**

**The second problem**

## 2006: CSR Peer Review Statistics

**80,000 applications received**
**55,000 applications reviewed**
**18,000 reviewers**
**238 Scientific Review Administrators**
**1,800 review meetings**

## Application Reviewed by CSR

## Growth of R01 Applications              Reviewed at CSR vs. Other ICs

**Growth of R01 Applications              Reviewed at CSR vs. Other ICs**

## Growth of R21 Applications              Reviewed at CSR vs. Other ICs

**Growth of R21 Applications              Reviewed at CSR vs. Other ICs**

## CSR Recent Activities

## Changes in CSR Operations

**Increase Communication and Transparency**
**Increase Uniformity**
**Increase Efficiency**
** ****Improve Study Section Alignment and Performance**

**Changes in CSR Operations**

## 1. Increase Communication

**Weekly     Biweekly     Monthly      Monthly      Monthly    Monthly **

## 
2.  Increase Uniformity

Appeal Advisory Committee

      Best Practices Committee Assessments
Appropriate Rosters
Conduct of Meetings
Structure of Summary Statements

**2.****  ****Increase Uniformity**
**     **
**Appeal Advisory Committee**
**      ****Best Practices Committee Assessments**
**Appropriate Rosters**
**Conduct of Meetings**
**Structure of Summary Statements**

**Changes in CSR Operations**

## Changes in CSR Operations

**Increase Communication and Transparency**
**Increase Uniformity**
**Increase Efficiency**

**Changes in CSR Operations**

## Increase Efficiency 

**How we distribute 80,000 applications?**
**Retooled for electronic submission**
**Applications are now submitted electronically**
**Assign applications using text fingerprinting, artificial intelligence software**
**	****	****	****Full Implementation by February 2008**

## Changes in CSR Operations

**Increase Communication and Transparency**
**Increase Uniformity**
**Increase Efficiency**
**Improve Study Section Alignment and Performance**

**Changes in CSR Operations**

## Biannual IRG Review Schedule

| Scheduled 2007 -- 9 IRGs |
| --- |
| Biology of Development and Aging (BDA) |
| Infectious Diseases and Microbiology (IDM) |
| Biobehavioral and Behavioral Processes (BBBP) |
| Cell Biology (CB) |
| Musculoskeletal, Oral and Skin Sciences (MOSS) |
| Oncological Sciences (ONC) |
| Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) |
| Respiratory Sciences (RES) |
| Renal and Urological Sciences (RUS) |

| Scheduled 2006 --14 IRGs |
| --- |
| Biological Chemistry and Macromolecular Biophysics (BCMB) |
| Cardiovascular Science (CVS) |
| Bioengineering Sciences and Technologies (BST) |
| AIDS and Related Research (AARR) |
| Risk, Prevention, and Health Behavior (RPHB) |
| Genes, Genomes and Genetics (GGG) |
| Digestive Sciences (DIG) |
| Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) |
| Brain Disorders and Clinical Neuroscience (BDCN) |
| Integrative, Functional and Cognitive Neuroscience (IFCN) |
| Molecular, Cellular and Developmental Neuroscience (MDCN) |
| Hematology (HEME) |
| Immunology (IMM) |
| Health of the Population (HOP) |

## Six Open House Workshops in 2007

| Mar. 2, | Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN)  |
| --- | --- |
| April 25 | Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB) |
| Jun. 29 | Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC);  Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB)  |
| Aug. 24 | Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) |
| Nov. 9 | Integrated Biological (5): Immunology (IMM); Hematology  (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA)  |
| Dec. 18 | Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG) |

## Neuroscience Open House, March 2007

## Neuroscience Open House Workshop

| Total registrants | 169 |
| --- | --- |
| Study Section Chairs | 26  |
| PRAC Members  | 2 |
| Professional Society Members
    From 26 different societies | 52  |
| NIH staff (non-CSR)
    14 ICs represented | 33 |
| CSR staff members | 50 |
| Public members | 6  |

## CSR Organization Initiatives

## CSR  Organization

## Review Enabling Committees

**Policy**
**Scientific Community Outreach**
**Recruitment of SRAs**
**Recruitment of Reviewers**
**Knowledge Management**
**Electronic Reviews**
**SRA Training**
**Mentoring and staff development**
**Best review practices**

## CSR 4 Review Divisions & 23 IRGs

## Organization of Peer Review at CSR

**Divisions**

**Study Sections**

**I.R.G.s**
**(Integrated Review Groups)**

## New IRG

**ENST/Emerging Neuroscience & Training**

## Organization of Peer Review at CSR

**Divisions**

**Study Sections**

**I.R.G.s**
**(Integrated Review Groups)**

## New Division Neuroscience, Aging and Development

**BDCN/Brain Disorders & Clinical Neuroscience**
**	****(est. 11 SRAs)**
**IFCN/Integrative, Functional, & Cognitive Neuroscience**
**	****(est. 9 SRAs)**
**MDCN/Molecular, Cellular & Developmental Neuroscience**
**	****(est. 10 SRAs)**
**ENST/Emerging Neuroscience & Training**
**	****(est. 8 SRAs)**
**BDA/Biology of Development & Aging  (7 SRAs)**

## Realigning Peer Review 

## CCSR Recent Activities in Peer ReviewSR Recent Activities in Peer Review

**Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**
**Improve the identification of significant, innovative and high-impact research**

## Shortening the Review Cycle

**First Step: ****Posting Summary Statements Earlier**
**Post new investigator within ****10 days,**
** ****(within 30 days ****for established investigators****)**
**Second Step: ****Pilot to Shorten the Review Cycle for New R01 Investigators**
**Last year:  About ****2,000**** new investigators in pilot**
**Third Step: ****Shorter Review Cycles to All R01 From New Investigators**
**Feb 2007:   ****3,000**
**June 2007: **** 6,000**** new investigators**
**Nov 2007:    All new investigators (>9000)**

## R01 A1 Resubmissions Reviewed in the Cycle Immediately following the Original Application's Review

## A Vision for Peer Review

**Shorten the review cycle**
**Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers**

## CSR’s Growing Need for Reviewers

## Reviewers – Current Situation 

**Far too many reviewers on study sections**
**Broader science**
**Decrease in reviewer load **
**Unnecessary**
**Too many ad hoc reviewers**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Remaking peer review a learning experience**
**Massive call for volunteers**
**Rewards for reviewers**
**Require less travel by expanding peer review platforms **
**Shorten applications**

## CSR Recent Activities in Peer Review

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Require less travel by using electronic review modes**

## Expanding Peer Review’s Platforms

**Study Sections**

**Necessity****	****	****    ****●****  **** ****Clinical reviewers**
** **
**Preference     ****              ****●  **** Physicists, computational biologists**
**New Opportunities****      ****● ** **Fogarty, International Reviewers**

**Our Goal:  10% of all reviews to be electronic in 2007**

## A Study Section Chair Talks About Asynchronous Electronic Discussions

**“****Finally, in spite of my initial skepticism regarding this “blog-like” review mechanism, all . . . [reviewers] produced comments, critiques, posed questions in a fashion that I would judge to be the BEST I’ve seen in ~30 years of chairing various review groups.” **

**Craig M. Jackson, Ph.D.            President and Principal Scientist Hemosaga Diagnostics Corp.**

## Near-Term Solutions for Recruiting and Retaining the Best Reviewers

**Require less travel by using electronic review modes**
**Have Shorter Meetings**
**Shorten Applications**

## Shorter Applications

**Trans NIH Committee**
**Communication and Support from Societies, Diseases Groups, Scientific Leadership  **
**2 Retreats of NIH Directors**
**Approval and Support of our Advisory Committee**
**Survey of Scientists**

## Initiatives Under Early Discussion

## Initiatives Under Early Planning/Consideration

**12 Pilots with short applications**
**Continuous receipt of applications for chartered study section members**
**Editorial board reviews**

## Deeper Changes? The NIH ACD Subcommittees

**Locus of review**
**NIH Support for Investigator**
**The Role of Council**
**NIH Portfolio Management**
**Firewall between review and program**
**Scoring**
**Review of deeply innovative, translational and multidisciplinary research **
**Others?**

## This is CSR

## Growth of R01 Applications              Reviewed at CSR vs. Other ICs

**Growth of R01 Applications              Reviewed at CSR vs. Other ICs**

## Growth of R21 Applications              Reviewed at CSR vs. Other ICs

**Growth of R21 Applications              Reviewed at CSR vs. Other ICs**

## Major Factors Affecting Payline

** ****NIH Budget**
** ****NIH Budget  **
** ****NIH Budget**
** ****NIH Budget**

## Major Factors Affecting Payline

** ****Number of Application Submitted**
** ****Targeting of the Budget  **
** ****Recycling of Funds**
** ****NIH Budget**

## CSR Review of Women Applications, Women Scoring 10 and 20 %

## Slide 52

## NIH 20%tile or Better Scoring (R01s, CSR reviews)

| Councils | New Applications New Investigators
(Type 1) | New Applications Established Investigators (Type 1) | Renewals From Established Investigators
(Type 2) |
| --- | --- | --- | --- |
| May 2001 | 15% | 17% | 34% |
| May 2002 | 15% | 17% | 34% |
| May 2003 | 15% | 18% | 33% |
| May 2004 | 14% | 18% | 34% |
| May 2005 | 15% | 18% | 33% |

**CSR New and Established Investigators Score%**

## M. Martin, CSR/NIH/DHHS

## Significant Numbers of Clinical Grantees Are Not Submitting Renewal Applications 

| R01 Applications 2000-2004 | HS+ | HS- |
| --- | --- | --- |
| Type 1 New A0 | 24.59% | 20.20% |
| Type 2 A0 | 11.71% | 17.11% |

- M. Martin, CSR/NIH/DHHS

## CSR Budget

## CSR Budget for 2007

**CSR Budget                         $ 60 M**
**SREA Budget****	****                  $ 40 M**
**Review Cost, including travel for 18,000 reviewers,**
**    ****is less than 0.2% of the budget requested in the applications reviewed**

## Travel

**Purchase non-refundable (restricted) coach tickets**
**instead of unrestricted tickets**

| Saving – June 2006 through July 2007 |  |
| --- | --- |
|  |  |
| # of tickets issued   | 28,493 |
| Changes requested  | 2544 |
| cost of changes + fees   | $550,274 |
|  |  |
| YTD Fare Savings | $14,522,877 |