**National Institutes of Health**

**Center for Scientific Review**

**Open House Workshop: Biomolecular**

**December 18, 2007**

**Natcher Conference Center, Bethesda, MD**

**Meeting Summary**

# Welcome and Introduction

Dr. Toni Scarpa (Director, CSR, NIH) welcomed participants and reminded
them that NIH first adopted peer review 50 years ago, and it has been a
major factor in NIH's subsequent success. CSR undertook a major
realignment of its study sections five years ago, and the time has come
to reevaluate that alignment. This will be the sixth and final open
house designed to determine how well science is being reviewed under the
current alignment. It is also the smallest meeting, with only 100
participants instead of 150-200, in part because of the holidays,
university exam schedules, and because the science is so concentrated in
certain areas. By the end of this process, however, CSR will have heard
from nearly 1,000 scientific leaders, including more than 250 study
section chairs and 500 leaders of scientific and professional societies.
These open houses have focused on two fundamental questions:

1\. What will be the most important questions and/or enabling
technologies you see forthcoming within the science of your discipline
in the next 10 years?

2\. Is the science of your discipline, in its present state,
appropriately evaluated within the current study section alignment?
Suggestions?

Dr. Scarpa added that time will be reserved at the end of this meeting
to discuss process, but for the time being participants should focus on
the science of their disciplines. The Director of NIH has recently
launched two high-level advisory committees -- one internal and the
other external --to gather additional input on the peer review process,
including funding levels and operational procedures. CSR, for its part,
has already begun to implement some of the recommendations from the
earlier open houses, and it continues to test new operational
initiatives that will increase the efficiency of the peer review
process, while reducing the burden on individual reviewers. Finally, Dr.
Scarpa recognized the contributions of Dr. Mary Ann Guadagno (Scientific
Review Officer, CSR), who has labored long and hard to organize these
open house meetings.

# Overview of Changes in the Biomolecular Study Sections

Dr. Donald Schneider (Director, Division of Molecular and Cellular
Mechanisms, CSR) reviewed the findings of the Panel on Scientific
Boundaries of Review (PSBR), which in January 2000 recommended that CSR
study sections be organized by organ or disease. In the biomolecular
sciences, this resulted in four clusters:

1\. Biological Chemistry and Macromolecular Biophysics (BCMB), 7 study
sections;

2\. Bioengineering Sciences and Technologies (BST), 6 study sections;

3\. Cell Biology (CB), 6 study sections; and

4\. Genes, Genomes and Genetics (GGG), 7 study sections.

These chartered study sections are supplemented by fellowship study
sections, small business/technology transfer study sections, and Special
Emphasis Panels (SEPs). New study sections can be created in response to
changes in the underlying science, as well as changes in workload, and
indeed all four of these clusters have seen the addition of at least one
study section or SEP since 2005. The process for creating new study
sections involves the identification of an issue (changes in science,
growth in workload), the convening of an internal working group,
interaction with the scientific community, review by the Peer Review
Advisory Committee, and final approval by the Director, OSR. This takes
about 9 months from the identification of a need to the first meeting of
the new study section.

# Explanation of and Charge to Breakout Groups

Dr. Cheryl Kitt (Deputy Director, CSR) explained that the breakout
sessions are designed to facilitate a more detailed discussion of the
two central questions posed by Dr. Scarpa. Participants have been
preassigned to a specific group but are free to join another group as
they feel appropriate. She cautioned them to focus on questions of
science. The four breakout groups were:

1\. Biophysics and Cell Biology (1);

2\. Biophysics and Cell Biology (2);

3\. Biochemistry, Molecular Genetics, Genetics, and Evolution; and

4\. Computation, Modeling, Large Data Sets, Theory, Genomics, and
Proteomics.

Each breakout group was co chaired by a study section chair and a
professional or scientific society president, assisted by two scientific
review officers prepared slides of their findings to report back to the
larger group.

# Report Out on Question 1

What will be the most important questions and/or enabling technologies
you see forthcoming within the science of your discipline in the next 10
years?

The overwhelming theme for all of the breakout groups was integration
--the integration of physics and chemistry, the integration of modeling
and experimentation, the integration and understanding of different
scales, the integration of multidisciplinary approaches, the integration
of structure/composition with function/dynamics, the integration of
signal transduction and gene regulation with cellular pathways and
biological systems. The most important research questions will deal with
how a single protein can execute multiple functions, how physiology
arises from assemblies of molecules and functions, and which processes
drive diversification.

Not surprisingly, the most important enabling technologies will be those
that support this kind of integration -- quantitative proteomics,
high-performance computing, high-throughput/low-cost sequencing (the
"thousand-dollar genome"), new model organisms, synthetic biology,
bioinformatics to mine and integrate existing knowledge from diverse
sources, and the interdisciplinary training that will be needed to
exploit these technologies.

# Discussion

In the discussion that followed, several audience members seconded the
importance of gaining a better understanding of the information we
already have, e.g., what proteins have been discovered, what functions
defined, what systems described. One way to achieve this is through
data-mining of existing protein databases (i.e., mouse, rat and human),
but it would be a logical extension to support additional protein
databases (e.g., rabbit). More generally, there needs to be a more
broadminded approach to informatics, not just as databases and models
but also as tools for hypothesis-driven research. The current approach,
like the publishing model, creates data silos that work against
integration. The databases may be built by computer scientists, but the
questions are still biological, and it will take a special breed of
researcher to bridge this gap. Of paramount importance is to make
biophysics relevant to human disease, but some participants see a gap
between the computational and disease-specific study sections that will
have to be bridged to achieve this relevance.

# Overview of Changes to the Peer Review Process

Dr. Scarpa suggested that, since the discussion was starting to move
from science to the process of peer review, this would be a good time to
review the steps that CSR has already taken to reform that process.
CSR's mandate is to review applications solely on the basis of
scientific merit; funding decisions are made by other Institutes and
Centers based on other criteria. But several important trends are
currently driving change at CSR.

The first driver is the number of applications, which doubled between
2000 and 2005 in response to the doubling of the NIH budget. CSR now
receives 80,000 applications per year and reviews over 52,000 of them,
using 18,000 reviewers in 1,800 separate review meetings.

The second driver is the workload of individual reviewers -- federal law
requires that there be three reviewers for each application, and even
with an increase in the number of reviewers this means that some of them
were being asked to review as many as 12 applications per cycle, a
crushing workload for volunteers. CSR has managed to reduce this to
seven applications per reviewer and would like to reduce it further, but
this also means that some review meetings include as many as 50
chartered and ad hoc members, with a consequent dilution of the quality
of the review experience.

Other drivers for change include CSR's budget, which has grown faster
than inflation but not as fast as its workload, and changes in the
nature of biomedical research itself, with a new emphasis on
interdisciplinary team approaches and a focus on chronic rather than
acute conditions. These changes have brought complaints that the system
has become a burden to reviewers and applicants alike -- it's too slow,
it favors predictable research over innovation, and there are too few
experienced researchers.

CSR has improved its own efficiency by shifting to electronic submission
and changing its travel policies. It has made major efforts to recruit
additional scientific review officers, increase transparency and
increase the consistency of review across study sections. CSR is pilot
testing other changes, such as shorter application forms (especially for
new investigators), electronic assignment of applications to study
sections, and a number of new review formats, including teleconference,
videoconference, and asynchronous web-based review. One of the next
tests will be with continuous submissions, starting with applications
from chartered members of study sections. Other changes in the future
might include changes in NIH portfolio management, changes in the
scoring system, and a separate review mechanism for deeply innovative,
translational and/or multidisciplinary research applications. CSR is
currently pilot testing two review models that would use an "editorial
board" to evaluate specific science or impact.

The need for experienced reviewers will not be solved easily. Recent
trends show an increase in ad hoc reviewers and a decline in the total
number of reviewers. CSR has done what it can to make peer review a
learning experience, and it is willing to at least experiment changes in
the location and format of its meetings, for example holding more
meetings on the West Coast. CSR has assembled a national registry of
potential peer reviewers, and they urge the presidents of professional
and scientific societies to suggest names and urge their members to
participate. And there are suggestions that CSR consider more tangible
rewards to reviewers, such as expedited review of their own grants or
preferential renewal of multiyear grants.

Members of the audience suggested that there may just be too many
Principal Investigators (PIs) chasing too few dollars, putting stress on
academic research as well as the peer review system. University
administrators can no longer assume that research funding will grow at
15 percent per year. Others suggested that CSR put more emphasis on the
potential impact of the research, rather than preliminary data, or find
a way to "triage" applications in order to reduce reviewer workload.

# Report Out on Question 2

Is the science of your discipline, in its present state, appropriately
evaluated within the current study section alignment? Suggestions?

The breakout groups reported that, while most study sections are
properly aligned, many basic science applications are "orphans" that
cannot find a home in the current disease-and-organ alignment. If the
grand challenge is integration, better integration is needed on the
study section if it is to evaluate an integrative application. One
example is multidisciplinary applications; other examples are
computational-experimental applications and those that focus on the
development of tools and technologies, rather than a specific biological
question.

Despite instructions to focus on the science, the breakout groups spent
some time making suggestions to improve the process of peer review.
Specific suggestions included:

-   To deal with interdisciplinary applications, create a floating pool
    of ad hoc members who move from one study section to another as
    their specialized expertise is required.

-   Hold several review meetings at the same time and place, so study
    sections can share reviewers.

-   Provide greater clarity and transparence on how applications are
    assigned to specific study sections.

-   Encourage reviewers to submit comments on meetings to senior CSR
    staff.

-   Limit reviewer workload to ten applications, even if shorter
    applications come into use.

-   Use a two-stage review, first an editorial board (with technical
    expertise) to assess technical merit, then a broader panel to assess
    potential impact.

-   Make sure that study sections include members with both theoretical
    and experimental expertise.

-   Where necessary, assign a particularly complex application to more
    than three reviewers.

-   If applications become shorter, allow applicants to use alternative
    means to attach supporting materials.

# Discussion

In the discussion that followed, participants suggested that the
community (not just CSR) needs to find a way to encourage deep
innovation, true discovery and serendipity. They recognized the
difficulty of finding quality reviewers, especially because of real or
perceived conflicts of interest, and they suggested that CSR pursue both
senior and junior investigators to serve in this capacity. They also
suggested that there is a natural evolution and lifetime for a study
section -- CSR is right to seek out an evaluation of their performance
and relevance. They were glad to learn that it is the IRGs that are
chartered and (semi-) permanent; the Special Emphasis Panels (SEPs)
operate as subcommittees to the IRGs. One way to deal with "orphan"
applications would be to create an "orphan" study section, with a
constantly shifting membership, specifically to deal with applications
that otherwise would have no home. Another possibility would be to allow
self-assignment -- provide a checkoff on the application so the
investigator can indicate that his or her application is an integrative
or interdisciplinary application that should go to an integrative or
interdisciplinary study section. Dr. Scarpa reported that there is a
trans-NIH committee working on the format for a shorter application and
the criteria for judging it. He promised to educate applicants on how
best to use that format.

The open house adjourned at 2:20 p.m.