# Presentation: 300820

## PRAC

**Lawrence A. Tabak, D.D.S., Ph.D.**
**On behalf of: **
**ACD WG on Peer Review**
**SC WG on Peer Review**

**http://enhancing-peer-review.nih.gov/**

**PRAC**
**December 3, 2007**

## A Self-Study by the NIH

**http://enhancing-peer-review.nih.gov/**

**A Self-Study by the NIH**
**in Partnership with the Scientific Community **
**to Strengthen Peer Review in **
**Changing Times**

##   Principles Behind the Study 

**The increasing breadth, complexity, and interdisciplinary nature of biomedical science are creating new challenges for the **_**system**_** used by NIH to support biomedical and behavioral research **
**Peer review is a key component of this system**
**The Charge:**
** **
***“******Fund the best science, by the best scientists, ***
***	******  ******with the least administrative burden...”***

## Study has been driven by two Working Groups  

**Study has been driven by two Working Groups**

**Keith Yamamoto, Ph.D., UCSF, ****Co-Chair, ACD, Boundaries Report**
**Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****Co-Chair**
**Syed Ahmed, MD, MPH, Dr. PH, MWC COPR**
**Bruce Alberts, Ph.D., UCSF, ****Chair, Boundaries Report**
**Mary Beckerle, Ph.D., U. Utah, ACD**
**David Botstein, Ph.D. , Princeton, ACD**
**Helen Hobbs, M.D., UTSW, HHMI**
**Erich Jarvis, Ph.D., Duke**
**Alan Leshner, Ph.D., AAAS, ACD**
**Philippa Marrack, Ph.D., Natl. Jewish Med., ****HHMI, Boundaries Report**
**Edward Pugh, Ph.D., U. Penn., PRAC**
**Tadataka Yamada, M.D., Gates Foundation, ACD**
**Ex officio**
**Norka Ruiz Bravo, Ph.D., OD/OER**
**Toni Scarpa, M.D., Ph.D., CSR**

**  ****Jeremy Berg, Ph.D.,  NIGMS, ****    Co-Chair**
**  ****Lawrence Tabak, D.D.S., Ph.D., NIDCR, ****    Co-Chair**
**  ****Marvin Kalt, Ph.D., NIAID**
**  ****Story Landis, Ph.D., NINDS (Co-chair EAWG)**
**  ****Roderic Pettigrew, Ph.D., M.D., NIBIB**
**  ****Norka Ruiz Bravo, Ph.D., OD/OER ****    (Co-chair EAWG)**
**  ****Toni Scarpa, M.D., Ph.D., CSR**
**  ****Lana Skirboll, Ph.D., OD/OSP**
**  ****Brent Stanfield, Ph.D., NIDDK**
** ****Jane Steinberg, Ph.D., NIMH**
**  ****Betty Tai, Ph.D., NIDA**
***Ex officio***
**  ****John Bartrum, OD/OB**
**  ****Jack Jones, Ph.D., Acting CIO **
**  ****Catherine Manzi, OGC **
**  ****Jennifer Spaeth, OD**

## Some Current CSR Initiatives

**Immediate Assignment of Applications to IRGs**
**Realignment of Study Sections**
**Electronic Reviews**
**Shortening the Size of Applications**

**Some Current CSR Initiatives**

**            ****The Steering Committee WG is **
**coordinating their efforts with CSR’s initiatives**

## Project Phases

**Diagnostic**

**Implement**
**“****low hanging **
**fruit”**
**Design pilots**

**Implementation**
**   ****and analysis**
**     ****of pilots**

**New NIH   Peer-Review Policies**

**Jul 07 – Jan 08**

**Feb 08 –  **

## Project Phases

**Diagnostic: **
**Data Collection**

**Request For Information: **
**    ****>2600 responses **

**Jul 07 – Jan 08**

##  Project Phases

-  **Diagnostic Phase**
**NIH received strong response to a Request for Information**

## Project Phases

**Diagnostic: **
**Data Collection**

**Request For Information: **
**  ****>2600 responses**

**2 Deans Teleconferences**
**  ****~100 participants**

**5 Regional Meetings:**
**   ****Washington, DC**
**   ****Chicago**
**   ****New York City**
**   ****San Francisco**
**   ****Washington, DC**

**Scientific Liaisons**

**NIH Staff**
**  ****>200 responses via website**

**Jul 07 – Jan 08**

## Project Phases

**Diagnostic:**
**Analysis**

**External Synthesis: **
**Request for Information**

**Jul 07 – Jan 08**

## Slide 11

## Project Phases

**Diagnostic:**
**Analysis**

**External Synthesis: **
**Request for Information**

**Internal Synthesis:**
**Tabular Summary of Input**

**Analysis of Other Agencies:**
**   ****Domestic**
**   ****International**

**Working Group Meetings:**
**   ****Steering Committee WG**
**   ****Advisory Council to the Director WG**

**Jul 07 – Jan 08**

## Project Phases

**Implement**
**“****low hanging **
**fruit”**
**Design pilots**

**Feb 08 –  **

**February 2008:**
**NIH Leadership determines next steps, including pilots**

**March 2008:**
**Design and initiation of pilots and associated evaluations commence**

## Project Phases

**Implementation**

**Implementation plan**
**Briefings**
**Expansion of **
**successful Pilots**

**New NIH Peer-Review Policies**

## Some Emerging Ideas*

***Please note: These are not in any priority order. They are presented only to facilitate discussion and feedback and to encourage attendees to think of additional issues**
**In addition, selection of ideas for follow-up is not predicated on the “Village vote” **

**http://enhancing-peer-review.nih.gov/**

**Some Emerging Ideas***

## Some emerging ideas

_**Review Criteria**_
**Reviewing the project versus the person**
**Retrospective versus prospective reviews for competing applications **
** ****Many suggestions about how to change review criteria to increase risk taking/”innovation”/impact or public health focus**
**Matrix scoring to evaluate multiple dimensions of an application**
**Too much emphasis on methodology and preliminary data**
**R21’s have become “mini R01’s”**
**Kills innovative/risky ideas that typically have minimal precedent**

## Some emerging ideas (cont.)

_**New Models of Review**_
**New models of review**
**The “Wisdom of the Crowd”**
**Even though many lament that there are too many people on study sections - more, rather than fewer people should review applications**
**Electronic review /** **asynchronous discussion / blogs**
**Two stage review: editorial board model**
**Establish applicant/reviewer dialogue to correct factual errors **
**Reviewers should be blinded to applicant identity during first phase of review; “environment” and investigators are rarely an issue **
**Different types of review for different types of science**
**Clinical/Community-based  research**
**requires involvement of patients/community members in the review**

## Some emerging ideas (cont.)

_**Maximization of Review(er) quality**_
**Provide in depth training for reviewers **
**Incentives to attract and retain most qualified reviewers**
**Add time to extant grants**
**Supplement extant grant for p/t administrative help **
**Make service more flexible - 2 rather than 3 times per year**
**Increase honorarium to reflect effort required **
**Rate the Reviewers/Scientific Review Officers**
**Don’t publish reviewer ID (e.g. NSF)// Identify reviewers**
**Place an ombudsperson on each study section**

## Some emerging ideas (cont.)

_**Reviewer Mechanisms/Mechanics **_
**Rethink design of an original submission in view of low A0 success rates- “clogs” queue  **
**Pre-applications to provide rapid identification and separation of competitive from non-competitive ideas, and meaningful advice to A0 applicants**
**Locus of pre-application review: electronic review by Peers  --  NIH staff**
**Administratively consider those applications that can contain minor, easily correctable deficiencies, for funding, rather than placing them back into the queue**
**Provide more useful feedback to applicants**
**Provide scores to applications currently “un-scored”or score and discuss all applications (particularly for new investigators)**
**Tell applicants, unambiguously, if the application is “NR”- not recommended for revision and resubmission**

## Some emerging ideas (cont.)

_**Peer  Review Culture **_
**Do “peers” make the best reviewers?**
**How can we re-capture the prestige of being a reviewer?**
**How much “context” should Reviewers be provided  (“firewall”)**
**Portfolio analysis of the Institute/NIH current investment in area?**
**NIH/Institute priorities?**
**Should there be a “third” level of review assessing public health and societal impact?**
**“****Face-to-face” versus “virtual” dynamics**
**Change evaluation to focus on strengths not weaknesses**
**Focus on the potential impact of the proposal and not the methodology**
**Role is not to “re-write” proposal but to review it’s merit**

## Some emerging ideas (cont.)

_**Scoring**_
**Have study section rank each proposal reviewed at the end of the meeting (and then revisit scores to each to ensure consistency and fairness) **
**All members must remain to have their vote “count”**
**Completely redo scoring system to break old habits**
**Use a 7 point scale rather than the 150 (de facto) point scale**
**Use matrix scores for each criteria or dimension**
**For example, two scores - application as received/best potential score**
**Criteria could be weighted (e.g. Public health impact)**

## Some emerging ideas (cont.)

_**Mechanisms**_
**Reinstitute mechanism for new investigators (R23/29) that is reviewed separately and is appropriately resourced**
**Review different mechanisms in different study sections or at different times of the meeting**
**Sharply reduce the number of mechanisms - too confusing and used for different things by different Ics**
**Do we need different mechanisms for scientists at different stages of their careers?**
**New  investigators - entry into the system**
**“****Established” investigators - stability**
**Senior investigators - giving back **
**Make the NIH system more accessible for “non academic” organization**

## Some emerging ideas (cont.)

_**Other issues related to system used to support research**_
**How many R01’s is enough?**
**Too many overlapping R01’s**
**Require minimum percent effort as PI (20/25/33/40%) with no transfer to junior associates allowed except in extraordinary circumstances (illness; PI is leaving academia and is relinquishing all grants)**
**Should R01s remain the “gold standard” of investigator success? - (Support for R01s vs. “top down”/”big” science)**
**Indirect costs**