# Presentation: 438967

## Peer Review in a Rapidly Changing Biomedical Environment

**National Institutes of Health**
**Department of Health and Human Services**

**Toni Scarpa**

**Peer Review in a Rapidly Changing Biomedical Environment**

**Peer Review Advisory Committee Meeting 9-26-05**

## July 1, 2005

**My First Day at Work**

## Applications received for all of NIH and applications referred for CSR review, FY 1998-2004

- 0

**20,000**

**40,000**

**60,000**

**80,000**

**1998**

**2000**

**2002**

**2004**

**Fiscal year**

**Number of applications**

**Applications received for all of NIH**

**Applications assigned for review by CSR**

## CSR Mission Statement

- To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

## Possible Changes in CSR Peer Review Operations

-                        **Complexity and Impact**

**CSR Operations          Current Systems            New Systems?**

**Possible Changes in CSR Peer Review Operations**

## Possible Changes in CSR Operations

**Increase communications between CSR, the ICs, our reviewers and applicants**
**Increase uniformity**

**Possible Changes in CSR Operations**

## Consistency and Transparency of CSR Operations Current Efforts

**Surveyed SRAs**
**Standardized Resumes**
**Developed Uniform Policies for--**
**Posting critiques in IAR before meetings**
**Posting summary statements**
**Streamlining 50 percent of the applications**

## Increased CSR CommunicationsCurrent Efforts

**Internal CSR/NIH Communications**
**Met with IC Directors and program staff**
**Integrated IC program staff into CSR committees**
**Established an open-door policy for staff**
**External Communications**
**Developed NIH Review Connect ListSERV**
**Expanded the CSR Peer Review Notes Newsletter**

## Possible Changes in CSR Operations

**Increase communications between CSR, the ICs, our reviewers and applicants**
**Increase uniformity**
**Increase efficiency**

**Possible Changes in CSR Operations**

## Potential of Knowledge Management Tools for Peer Review

**Collexis Software**
**Knowledge management solutions**
**Fingerprinting and text retrieving**
**Disease coding**
**Benefits for Peer Review**
**Assigning applications to Integrated Review Groups**
**Selecting reviewers (one application, multiple applications)**
**Four pilots are underway to begin to assess these benefits**
**Implementation will give NIH the ability to assess the research content of all unfunded applications**

## Possible Changes in CSR Operations

**Increase communications between CSR, the ICs, our reviewers and applicants**
**Increase uniformity**
**Increase efficiency**
**Facilitate work of IC program staff**

**Possible Changes in CSR Operations**

## New Secure Speakerphones to Give NIH Program Officers Greater Access to CSR Review Meetings

- VTX 1000 Sound Station from Polycom

## Possible Changes in Current Systems

**Shorten the review cycle**

## NIH Committee to Shorten the Review Cycle: Current Efforts

- Trans-NIH Staff
- Olivia Bartlett, Ph.D.
**National Cancer Institute **
- Bettie J. Graham, Ph.D.
**National Human Genome Research Institute**
- Marcia Hahn
**Office of Extramural Research**
- Robert Hammond, Ph.D.
**Ex National Institute of Diabetes & Digestive &**
**Kidney Diseases**
- Jack Jones, Ph.D.
**Center for Information Technology**
- John J. McGowan, Ph.D.
**National Institute of Allergy and Infectious **
**Diseases**
- Belinda Seto, Ph.D.
**National Institute of Biomedical Imaging and**
**Bioengineering**

- Sophonia Simms, M.S.
**Office of Extramural Research**
- CSR Staff_ _
- Eileen Bradley, D.Sc.  (Chairperson)
**Surgical Sciences, Biomedical Imaging and**
**Bioengineering Integrated Review Group**
- Suzanne Fisher, Ph.D.
**Division of Receipt and Referral**
- Richard Panniers, Ph.D.
**Genes, Genomes and Genetics Integrated**
**Review Group**
- Donald L. Schneider, Ph.D.
**Division of Molecular and Cellular Mechanisms**
- Thomas Tatham, Ph.D.
**Small Business Developmental Disabilities,**
**Communication, and Science Education Panel**

## This is Not a Ford Assembly Line

**Receipt**

**Refer**

**Evaluate Scientific**
**Merit of Applications**

**EnterpriseArchitecture@mail.nih.gov**

## Possible Changes in Current Systems

**Possible Changes in Current Systems**

**Shorten the review cycle**
**Address concern that clinical research is not properly evaluated**

## R01s Reviewed by ICs

|  | N | Median |
| --- | --- | --- |
| HS+ | 496 | 234.0 |
| HS- | 152 | 212.0 |

**R01s Reviewed by ICs**

|  | N | Median |
| --- | --- | --- |
| HS+ | 1,890 | 210.0 |
| HS- | 3,169 | 198.0 |

**R01s Reviewed by CSR**

**Review Outcomes for HS+ vs. HS- R01 Applications Reviewed by Panels Organized by CSR or Other NIH ICs**
**October 2002 Council Data**

## Possible Changes in Current Systems

**Possible Changes in Current Systems**

**Shorten the review cycle**
**Address concern that clinical research is not properly evaluated**
**Improve the assessment of innovative, high- risk/high-reward research**

## “The judging of grants has become a charade.”

**The American Society for Cell Biology**

**“****The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.” **
**Zena Werb**
**President, ASCB**

**Newsletter August 2005**

## Possible Changes in Current Systems

**Possible Changes in Current Systems**

**Shorten the review cycle**
**Address concern that clinical research is not properly evaluated**
**Improve the assessment of innovative, high- risk/high-reward research**
**Do more to recruit and retain more high quality reviewers**

## Distribution of Reviewer Academic Titles in May 1998 and May 2005

**Reviewers in Chartered CSR Study Section Meetings**

## Expanding Peer Review’s Platforms

**Electronic Reviews**
**Telephone Enhanced Discussions**
**Video Enhanced Discussions**
**Asynchronous Electronic Discussions**

**Study Sections**

**Necessity    ****              **** ****●****   **** ****Clinical reviewer (especially inter-****	****	****	****                                 ventional medicine)**
**Preference****                 ****●  **** Physicists, computational biochemists**
**New Opportunities****    ****●**    **Fogarty, International Reviewers**

## Possible New Systems

**If we didn’t have any peer-review system and we had to design one from scratch, what would it look like? **

**Possible New Systems**

## Slide 24

## This is CSR

## The Scientific Review Administrators Our Unsung Heroes

**The Scientific Review Administrators ****Our Unsung Heroes**

**They give to peer review “a special, precious character.  Their contributions tend to be overshadowed by the forces and discoveries they sustained.  Yet they are the pre-eminent pillars of the civilization that strives to leave a legacy of knowledge through an endless process of discovery and utilization of knowledge.”**
***Donald S. Fredrickson NIH Director, 1975-1981***