# Presentation: 071067

## Initiatives to Enhance Peer Review: Two-Stage Peer Review

**Initiatives to Enhance Peer Review: Two-Stage Peer Review**
**Anita Miller Sostek, Ph.D.**
**Peer Review Advisory Committee**

**National Institutes of Health****U.S. Department of Health and Human Services**

**August 27, 2007**

## Problem to be Solved

**Review of complex, multidisciplinary and translational research that:**
**Emphasizes overall significance and impact**
**Provides specific expertise in a wide range of scientific areas**
**Makes efficient use of reviewer’s time and effort**
**Preserves the dynamic of reviewer discussion**
- The challenge of reviewing complex applications has been consistently noted at each of the CSR Open Houses

## Two-Stage Peer Review Pilot

**A two-stage peer review is similar to a scientific journal editorial board review  **
**The first stage review of methodological and technical aspects of the science**
**The second stage emphasizes impact and significance**
**The second stage could be conducted in a variety of meeting formats (face-to-face, telephone, AED, or VED)**

## What the Two Stages Involve

**methodological and technical merit**
**written critiques similar to ****“****mail****”**** reviews**
***Second stage *****involves a ****small group**** of experienced reviewers with broad scientific expertise**
**focus on impact and significance**
**examine all applications to ensure quality and consistency of assessments**
**Final scoring is done by the second stage reviewers**
**First stage reviewers may provide advice about scoring (perhaps A,B, C, D)**

**What the Two Stages Involve**

## What is the History?

**Examples of complex and non-traditional reviews at the NIH:**
**Pioneer awards**
**SCOR (Women’s Health) Centers Review**
**NHLBI **

## Where Will the Pilot be Applied?

**Research Grants – concentrate on multidisciplinary and translational areas of science**
**Three or four IRGs that focus on disease processes or physiological systems**
**How to define?**
**Status: Under discussion**
**Pilot for Bioengineering Research Partnerships**
**Complex technology applications**
**Status: Currently underway  **
**Pilot For Small Business Applications**
** ****Status: Under discussion**

## Issues to be Addressed	

**Logistics of recruitment and optimal review loads for reviewers at both stages of review**
**Defining conflicts of interest and dyads**
**Scoring procedures**
**Advice from first stage; final scores from secondary stage**
**How will streamlining be conducted?**
**Second stage based on broader issues; e.g., impact **
**Consideration given to scoring advice of the first stage**

## Timing – Will it be Faster?

**Ultimately, it is hoped that Two-Stage Review will be faster than traditional reviews (each reviewer examines the full application in depth)**
**Requirements of the Timeline**
**Adequate time for distribution to two groups of reviewers**
**Turnaround from initial stage of review to the second**
**Reviewers will need to be recruited and assigned before full set of applications are received.**
**What can we learn from other agencies?**

## Potential Advantages		

**Recruitment may be facilitated because first stage reviewers are asked to evaluate specific aspects of the applications  **
**it may be possible to recruit earlier**
**mail reviewers are less limited by dyad considerations**
**Efficiencies will be achieved as the primary reviewer (second stage) does not need to critique approach in detail and may provide a resume**
**Both stages may be well suited to a structured critique formats**
**Greater consistency since scoring will be done by a smaller number of reviewers**

## Process Sample - BRPs

_**Stage One:**_
**At least two mail reviews (first stage critiques) for each application – specialized technology and medical input**
**Critiques at both stages will address all review criteria but will focus mainly on areas within their expertise. **
**Mail reviews will be prepared a month before the second stage deadline to allow time to consider them in preparing the broader critiques.**
**Mail reviewers will provide preliminary scores for the whole application, unless otherwise mentioned.**

## Process Sample - continued

_**Stage Two:**_
**Reviewers “senior level” and/or experienced with BRPs**
**Will have a month to examine the applications and the first stage critiques**
**The second stage reviewers will write critiques that incorporate the first stage reviews**
**Option available to bring first stage reviewers in by phone if their perspective is needed for discussion.**

## Challenges of Pilots

**Adapting the general concept to specific types of reviews**
**Avoid perception of lesser value for the first stage reviewers**
**Defining the main focus of the reviews at each stage**
**Avoid perception of lesser value for the first stage reviewers**
**Logistics**
**timing**
**workloads and recruitment**
**technology adaptations needed to allow broader reading of mail reviews within IAR **

## Models for Two-Stage Peer Reviews	

**Veterans Administration Merit Review ********
**Department of Defense ********
**Private foundations**
******** **** ****Previous presentations discuss review at these agencies**

## Discussion