# Presentation: 111864

## NIH Director’s Council of 

**Lawrence A. Tabak, D.D.S., Ph.D.**
**On behalf of: **
**ACD WG on Peer Review**
**SC WG on Peer Review**

**http://enhancing-peer-review.nih.gov/**

**NIH Director’s Council of **
**Public Representatives**
**10/26/07**

## A Self-Study by the NIH

**http://enhancing-peer-review.nih.gov/**

**A Self-Study by the NIH**
**in Partnership with the Scientific Community **
**to Strengthen Peer Review in **
**Changing Times**

##   Principles Behind the Study 

**The increasing breadth, complexity, and interdisciplinary nature of biomedical science are creating new challenges for the system used by NIH to support biomedical and behavioral research **
**Peer review is a key component of this system**
**NIH must: **
**Continue to adapt to rapidly changing fields of science and ever-growing public health challenges**
**Work to ensure that the processes used to support science are as efficient and effective as possible for applicants and reviewers alike**

## The Approach to the Study

**NIH is seeking input from the scientific community, including: **
**investigators **
**scientific societies**
**grantee institutions **
**voluntary health organizations**
**NIH is seeking input from its own staff**

## Working Groups  

**Keith Yamamoto, Ph.D., UCSF, ****Co-Chair, ACD, Boundaries Report**
**Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****Co-Chair**
**Bruce Alberts, Ph.D., UCSF, ****Chair, Boundaries Report**
**Mary Beckerle, Ph.D., U. Utah, ACD**
**David Botstein, Ph.D. , Princeton, ACD**
**Helen Hobbs, M.D., UTSW, HHMI**
**Erich Jarvis, Ph.D., Duke**
**Alan Leshner, Ph.D., AAAS, ACD**
**Philippa Marrack, Ph.D., Natl. Jewish Med., ****HHMI, Boundaries Report**
**Marjorie Mau, M.S., M.D., U. Hawaii, COPR**
**Edward Pugh, Ph.D., U. Penn., PRAC**
**Tadataka Yamada, M.D., Gates Foundation, ACD**

***Ex officio***
**Norka Ruiz Bravo, Ph.D., OD/OER**
**Toni Scarpa, M.D., Ph.D., CSR**

**External (ACD WG on Peer Review)**

## Working Groups  

**Jeremy Berg, Ph.D.,  NIGMS, ****Co-Chair**
**Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****Co-Chair**
**Marvin Kalt, Ph.D., NIAID**
**Story Landis, Ph.D., NINDS (Co-chair EAWG)**
**Roderic Pettigrew, Ph.D., M.D., NIBIB**
**Norka Ruiz Bravo, Ph.D., OD/OER ****(Co-chair EAWG)**
**Toni Scarpa, M.D., Ph.D., CSR**
**Lana Skirboll, Ph.D., OD/OSP**
**Brent Stanfield, Ph.D., NIDDK**
**Jane Steinberg, Ph.D., NIMH**
**Betty Tai, Ph.D., NIDA**

***Ex officio***
**John Bartrum, OD/OB**
**Jack Jones, Ph.D., ****Acting CIO **
**Catherine Manzi, OGC **
**Jennifer Spaeth, OD**

**Internal (Steering Committee WG on Peer Review)**

## Slide 7

##  Phases for Review

-  **Diagnostic Phase**
**NIH received strong response to a Request for Information**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences **
**ACD Working Group conducted a series of 5 regional**** ****town meetings **
**July 30:  Washington, DC **
**Sep. 12:  Chicago **
**Oct. 8:  New York City **
**Oct. 22:  Washington, DC **
**Oct. 25:  San Francisco**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences **
**ACD Working Group conducted a series of 5 regional town meetings**** **
**ACD Working Group, Deans, and NIH Institute Directors selected science liaisons to enhance outreach to stake-holders****	**
**	****	**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences **
**ACD Working Group conducted a series of 5 regional town meetings**** **
**ACD Working Group, Deans and NIH Institute Directors selected science liaisons to enhance out-reach to stake-holders****	****	****	**
**SC Working Group solicited broad input from NIH staff**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences **
**ACD Working Group conducted a series of 5 regional town meetings**** **
**ACD Working Group, Deans and NIH Institute Directors selected science liaisons to enhance out-reach to stake-holders****	****	****	**
**SC Working Group Solicited broad input from NIH Staff**
**SC Working Group has analyzed approaches of other agencies both in the United States and abroad**

##  Phases for Review

-  **Diagnostic Phase**
**NIH received a strong response to the Request for Information **
**Dr. Zerhouni and ACD WG co-chairs held two Dean’s teleconferences **
**ACD Working Group conducted a series of 5 regional town meetings**** **
**ACD Working Group, Deans and NIH Institute Directors selected science liaisons to enhance out-reach to stake-holders****	****	****	**
**SC Working Group Solicited broad input from NIH Staff**
**SC Working Group has analyzed approaches of other agencies both in the United States and abroad**

## Phases: Piloting

**NIH leadership, informed by all input, will determine next steps, including pilots (February 2008)**
**Design and initiate pilot(s) and associated evaluation(s) (March 2008)**

## Phase: Implementation

**Development of implementation plan**
**Briefings for NIH staff**
**Briefings for scientific societies, NIH Councils and Committees, trade press, advocacy groups, voluntary organizations**
**Legislative briefings**
**Expansion of successful pilots**
**Development of new NIH Peer Review Policy**

## Some Emerging Ideas of Potential

***Please note: These are not in any priority order. They are presented only to facilitate discussion and feedback and to encourage attendees to think of additional issues**

**http://enhancing-peer-review.nih.gov/**

**Some Emerging Ideas of Potential**
**Interest to COPR**

## Some emerging ideas

_**Review Criteria and Focus**_
**Many suggestions about how to change review criteria to increase risk taking and/or innovation and/or public health focus**
**For example, criteria could be weighted (e.g., public health impact)**
**Use matrix scores to evaluate multiple dimensions of application**
**Reviewing the project versus the person**
**Place less emphasis on methodology and preliminary data**

## Some emerging ideas (cont.)

_**New models of review**_
**The “Wisdom of the Crowd”**
**More rather than fewer people should review applications**
**Establish applicant/reviewer dialogue to correct factual errors during the review in real time**
**Different types of review for different types of science**
**Clinical research**
**Clinical-based research requires involvement of patients or their advocates in the review**
**Community-based research**
**Community-based research requires involvement of community members in the review**

## Some emerging ideas (cont.)

_**Maximization of Review(er) Quality**_
**Provide in-depth training for reviewers **
**Rating the Reviewers/Scientific Review Administrators **
**Place an ombudsperson on each study section**
**How much “context” should Reviewers be provided  (“firewall”)**
**Portfolio analysis of the Institute/NIH current investment in area?**
**NIH/Institute priorities?**
**Should there be a “third” level of review assessing public health and societal impact?**

## Some emerging ideas (cont.)

_**Scoring**_
**Use a scoring system that is consistent with the precision of the process**
**Provide more useful feedback to applicants**
**Tell applicants, unambiguously, if the application is “NR”--not recommended for revision and resubmission**
_**Peer Review Culture**_
**Do “peers” make the best reviewers?**
**How can we re-capture the prestige of being a reviewer?**
**“****Face-to-face” versus “virtual” dynamics **
_**Mechanisms**_
**Should we reduce the number of mechanisms--too confusing and used for different things by different NIH Institutes**
**Do we need different mechanisms for scientists at different stages of their careers?**
**New  investigators--entry into the system**
**“****Established” investigators--stability**
**Senior investigators--giving back **
**Make the NIH system more accessible for “non academic” organizations**

## Some emerging ideas (cont.)

**Other issues related to system used to support research**
**Support for individual investigators vs. “top down”/”big” science **
**How many grants are “enough” for any single investigator?**