# Presentation: 772965

## Enhancing Peer Review at the National Institutes of Health

**Enhancing Peer Review at the National Institutes of Health**
***Andrea Kopstein, PhD, MPH***

**AEA Annual Meeting**
**Denver, CO, November, 2008**

**National Institutes of Health****U.S. Department of Health and Human**** ****Services**

## ENHANCING PEER REVIEW PRIORITY 1:  ENGAGE THE BEST REVIEWERS

## Electronic Reviews: Expanding Peer Review Platforms

- Study Sections

- Necessity		    ●   Clinical reviewers
- Preference                   ●   Physicists, computational biologists
- New Opportunities      ●  Fogarty, International Reviewers

- Our Goal:  15% of all reviews to be electronic in 2008

## Expected Benefits for AED

- Greater flexibility for scheduling and running peer review meetings.
- Expand potential reviewer base - eliminate travel and time constraints.
- Allow for more thoughtful and thorough discussions.
- Potentially less confrontational environment
- Improved management of conflicts.
- Website is straightforward for reviewers and requires little training.
- Reduced costs

## AED Progress ....

- Improvements to Interface (Version 3.1 now being used).
- Enhancements to AED:
- Unread comments
- Unscoring
- Identification of conflicts- scoring table
- Assigned reviewer display, etc.
- 170 review meetings done using AED in 2008
- Wider adoption and acceptance – 5 new NIH IC users and 4 agencies educated about AED in FY2008.

## The AED Evaluation

-  Web-based survey to reviewers and SROs.
-  Fixed choice and open-ended questions.
-   Response rates:
-  Reviewers - 56% (786/1460).
-  Scientific Review Officers - 60% (55/92)
-   Both target populations had two follow-up reminders
- Although response rates were somewhat lower than desired, the summary of qualitative comments provide some insight into the AED process and AED is an ongoing initiative

## SOME EVALUATION FINDINGS – SUMMER 2008

## Reviewer Satisfaction with AED Technology

## Scientific Review Officer (SRO) Satisfaction with AED Technology

## Reviewers:  Was the duration of discussion sufficient to allow an appropriate level of consideration of applications? 

## SRO:  Compared to a face-to-face meeting, do you feel the quality of deliberation was:

**SRO:  Compared to a face-to-face meeting, do you feel the quality of deliberation was:**

## REVIEWERS:  Some of the most frequent positive responses to open questions:

- #1 response – AED works well!
- #2 - AED provides a review that is more fair because of deliberation time
- Many times a reviewer may misinterpret certain facts on an application. If one is not a primary reviewer it is impossible to check out these facts.  With this system there is plenty of time to check things out if something does not ring true, or if there is a strong disagreement about a certain point.

## REVIEWERS:  Frequent positive responses......

- This format relies EXCLUSIVELY on the strengths of an opinion, not on the aggressiveness of a reviewer and their ability to speak loud.
- The technology works well
- Not having to travel is a big benefit

## REVIEWERS:  Some of the most frequent negative responses to open questions:

- Complaints about  some reviewers tending to ignore applications for which they were not assigned or going offline for extended periods of time. In a face-to-face meeting, it is possible to absorb the discussion as it occurs.  Passive listening not viable in AED
- More difficult to make your case for an applications than in face-to-face meetings

## REVIEWERS:  Negative responses - continued

- Sometimes questions are asked but then get no response
- A major disadvantage is the an inability to talk to colleagues during the meeting
- Technology was slow, navigation issues, trouble for MACs

## SROs:  Top positive responses to open questions:

- #1- Easier to recruit stellar reviewers
- Easier to get clinical reviewers
- Easier to get international reviewers
- Easier to schedule
- Good for reviewing diverse applications

## SROs:  Top positive responses........

- Allows multitasking while the review meeting is ongoing
- The technology works well
- SROs and Reviewers like not having to travel
- Meeting can be run from any location where there is Internet access

## SROs:  Some of the most frequent negative responses to open questions:

- Loss of direct interaction with reviewers – personal rapport lost
- Concerned that unassigned reviewers don’t review or score carefully
- Spotty reviewer participation in discussion, sometimes difficult to engage
- More Labor intensive than face-to-face meeting for smaller numbers of applications
- Sometimes applications are nitpicked because there is so much time