# Presentation: 425280

## http://enhancing-peer-review.nih.gov/

**Lawrence A. Tabak, D.D.S., Ph.D.**
**Jeremy Berg, Ph.D.**
**On behalf of: **
**ACD WG on Peer Review**
**SC WG on Peer Review**

**http://enhancing-peer-review.nih.gov/**

## A Self-Study by the NIH

**A Self-Study by the NIH**
**in Partnership with the Scientific Community **
**to Strengthen Peer Review in **
**Changing Times**

##   Principles Behind the Study 

**The increasing breadth, complexity, and interdisciplinary nature of biomedical science are creating new challenges for the system used by NIH to support biomedical and behavioral research **
**Peer review is a key component of this system**
**NIH must: **
**Continue to adapt to rapidly-changing fields of science and ever-growing public health challenges**
**Work to ensure that the processes used to support science are as efficient and effective as possible for applicants and reviewers alike**
**Continue to draw the most talented reviewers**

## The Approach to the Study

**NIH will seek input from the scientific community, including: **
**investigators **
**scientific societies**
**grantee institutions **
**voluntary health organizations**
**NIH will also seek input from its own staff**

## Working Groups  

**Keith Yamamoto, Ph.D., UCSF, ****Co-Chair, ACD, Boundaries Report**
**Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****Co-Chair**
**Bruce Alberts, Ph.D., UCSF, ****Chair, Boundaries Report**
**Mary Beckerle, Ph.D., U. Utah, ACD**
**David Botstein, Ph.D. , Princeton, ACD**
**Helen Hobbs, M.D., UTSW, HHMI**
**Erich Jarvis, Ph.D., Duke**
**Alan Leshner, Ph.D., AAAS, ACD**
**Philippa Marrack, Ph.D., Natl. Jewish Med., ****HHMI, Boundaries Report**
**Marjorie Mau, M.S., M.D., U. Hawaii, COPR**
**Edward Pugh, Ph.D., U. Penn., PRAC**
**Tadataka Yamada, M.D., Gates Foundation, ACD**

***Ex officio***
**Norka Ruiz Bravo, Ph.D., OD/OER**
**Toni Scarpa, M.D., Ph.D., CSR**

**External (ACD WG on Peer Review)**

## Working Groups  

**Jeremy Berg, Ph.D.,  NIGMS, ****Co-Chair**
**Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****Co-Chair**
**Marvin Kalt, Ph.D., NIAID**
**Story Landis, Ph.D., NINDS (Co-chair EAWG)**
**Roderic Pettigrew, Ph.D., M.D., NIBIB**
**Norka Ruiz Bravo, Ph.D., OD/OER ****(Co-chair EAWG)**
**Toni Scarpa, M.D., Ph.D., CSR**
**Lana Skirboll, Ph.D., OD/OSP**
**Brent Stanfield, Ph.D., NIDDK**
**Jane Steinberg, Ph.D., NIMH**
**Betty Tai, Ph.D., NIDA**

***Ex officio***
**John Bartrum, OD/OB**
**Jack Jones, Ph.D., ****Acting CIO **
**Catherine Manzi, OGC **
**Jennifer Spaeth, OD**

**Internal (Steering Committee WG on Peer Review)**

## Some Current CSR Initiatives

- Immediate Assignment of Applications to IRGs
- Realignment of Study Sections
- Electronic Reviews
- Shortening the Size of Applications

**Some Current CSR Initiatives**

**    ****The Steering Committee *****Ad Hoc***** WG is coordinating their efforts with CSR’s initiatives**

##  Phases for Review

-  **Diagnostic Phase**
**NIH issues an RFI and creates an interactive web site for soliciting opinion (July-**_**September 7th**_**, 2007) **

##  Phases for Review

-  **Diagnostic Phase**
**NIH issues an RFI and creates an interactive web site for soliciting opinion (July-**_**September 7th**_**, 2007) **

##  Phases for Review

-  **Diagnostic Phase**
**NIH issues an RFI and creates an interactive web site for soliciting opinion (July-September 7th 2007) **
**Dr. Zerhouni and ACD WG co-chairs hold two Dean’s teleconferences- ~ 100 participants**

##  Phases for Review

-  **Diagnostic Phase**
**NIH issues an RFI and creates an interactive web site for soliciting opinion (July-September 7th 2007) **
**Dr. Zerhouni and ACD WG co-chairs hold two Dean’s teleconferences- ~ 100 participants**
**ACD Working Group will hold a series of 5 regional**** ****town meetings (July to October 2007)**
**First meeting held with professional organizations on July 30th**
**Other meetings scheduled:**
**Sep. 12:  Chicago – Fairmont**
**Oct. 8:  New York – Embassy Suites Hotel New York City**
**Oct. 22:  DC – Hilton Washington**
**Oct. 25:  San Francisco – Renaissance Parc 55**
**ACD Working Group selects a series of science liaisons to enhance out-reach to stake-holders****	**
**Common Website created for liaisons and ACD members to submit feedback**

##  Phases for Review

-  **Diagnostic Phase**
**NIH issues an RFI and creates an interactive web site for soliciting opinion (July-September 7th 2007) **
**Dr. Zerhouni and ACD WG co-chairs hold two Dean’s teleconferences- ~ 100 participants**
**ACD Working Group will hold a series of 5 regional town meetings (July to October 2007)**
**ACD Working Group selects a series of science liaisons to enhance out-reach to stake-holders****	**
**SC Working Group Solicits  from IC’s ****	**
**Prior experiments - (July-August 2007)**
**IC-specific statements - (Due August 17th)**
**SC Working Group holds  three consultative meetings within NIH and creates a web-based survey for soliciting opinion (July-August 2007; additional meetings TBD)**

## Additional efforts for the SC WG

**Analysis of peer review literature – summary completed**
**Analysis of other agency approaches (NSF/DOE/DOD/HHMI, other countries)**** **

## Additional efforts for the SC WG

**Analysis of peer review literature – summary completed**
**Analysis of other agency approaches (NSF/DOE/DOD/HHMI, other countries) **
**Psychometric analysis of study section models by experts**

##       Updates to IC Fall Councils      Updates to numerous NIH groups

## Phases: Piloting

**NIH leadership will consider input from the RFI and both working groups and determine next steps, including pilots (February 2008)**
**Design and initiate pilot(s) and associated evaluation(s) (March 2008)**

## Phase: Implementation

**Development of implementation plan**
**Briefings for NIH staff**
**Briefings for scientific societies, NIH Councils, trade press, advocacy organizations**
**Legislative briefings**
**Expansion of successful pilots**
**Development of new NIH Peer Review Policy**

## Some emerging ideas

_**Review Criteria and Focus; Application Structure**_
**Reviewing the project versus funding the person**
**Retrospective versus prospective reviews**
**Separate application modes and review criteria for projects that lack preliminary data or precedent from those that are extensions of current work**

## Some emerging ideas (cont.)

_**Reviewer Mechanisms/Mechanics**_
**New models of review**
**Two stage technical subject matter/Editorial board model**
**Electronic review**
**Virtual electronic applicant/reviewer dialogue to answer questions**
**Different types of review for different types of science**
**Clinical research**
**Can’t mix basic research with clinical expertise on panel**
**Trials tend to be new submissions which historically have low success rates**
**Interdisciplinary research**
**Content expertise vs. big picture; interpreters (editorial board?)**
**SBIR/STTR**
**Academics usually not the “right” person to review small business**

## Some emerging ideas (cont.)

_**Reviewer Mechanisms/Mechanics **_**(cont.) **
**Have investigators designate one application as their “primary” application – use different criteria to review/fund “non-primary” applications **
**Provide more useful feedback to applications from new investigators, including clearer ranking to those who are unscored//eliminate “triage” (for new investigators)**
**Rethink design of an original submission in view of low A0 success rates- “clogs” queue  **
**Pre-applications to provide rapid identification and separation of competitive from non-competitive ideas, and meaningful advice to A0 applicants**

## Some emerging ideas (cont.)

_**Reviewers and Review Culture**_
**Maximization of Review(er) quality**
**How much information to provide reviewers appropriate context (“firewall”)**
**Incentives for reviewers**
**Mandatory service?**
**More flexible service**
**Increased support?**
**Rating the reviewers/SRA’s**
**Don’t publish reviewer ID (e.g. NSF)// Identify reviewers**

## Some emerging ideas (cont.)

_**Scoring**_
**Scoring issues percentiles; binning; triaging; more information from scores desired**
**Consider psychometrics to devise voting scales and to group scores that are not statistically different**
**Consider adding additional dimensions to extract more information from review, e.g., Two scores - application as received/ best potential score**
** **_**Other**_
**Limit % effort that can be recovered on grants for principal investigators/increase % effort to 50+% to be a principal investigator**

## ????????????????????????????????????????????????????????????????????????????????????????????????