**Minutes of the Center for Scientific Review\
Advisory Committee Meeting\
January 10 and 11, 2000** 

The Center for Scientific Review (CSR) convened the 23^rd^ meeting of
its Advisory Committee at 8:30 a.m. on Monday, January 10, 2000, in
Conference Room 9100, Rockledge II Building. The entire meeting was in
open session. Dr. Keith Yamamoto presided as Chairperson.

 **Members:**

-   Keith Yamamoto, Ph.D., Chairperson

-   Michael Berns, Ph.D.

-   Shu Chien, M.D., Ph.D.

-   Michael Colvin, M.D.

-   Karen Matthews, Ph.D.

-   Raphael Pollock, M.D., Ph.D. \*

-   Marvin Wickens, Ph.D.

-   Tadataka Yamada, M.D.

**\*** Was not present at the meeting. 

**Ad Hoc Advisors:**

-   Leonard Epstein, Ph.D.

-   James Kushner, M.D.

-   Richard Lifton, M.D., Ph.D.

-   Gabriel Navar, Ph.D.

**I. Welcome and Opening Comments........................Dr. Keith
Yamamoto**

Dr. Yamamoto called the meeting to order, welcomed the participants to
the meeting, and immediately turned the meeting over to Dr. Ehrenfeld.
Dr. Ehrenfeld likewise welcomed the participants, and then briefly
mentioned Dr. Harold Varmus, Director of the National Institutes of
Health (NIH) for the past six years, left NIH as of the first of January
to head Sloan Kettering Memorial Cancer Institute in New York. A search
is underway for a new Director, which is a Presidential appointment.

Dr. Ehrenfeld then introduced Dr. Ruth Kirschstein, who is now Acting
Director of NIH. Dr. Kirschstein has been Deputy Director of NIH for the
past six years, and was Acting Director of NIH in 1993 between Dr.
Bernadine Healey and Dr. Varmus. From 1974 to 1993, she was Director of
the National Institute of General Medical Sciences (NIGMS). ** **

## II. Comments......................................................Dr. Ruth Kirschstein  

Dr. Kirschstein explained her long-time interest in peer review at NIH,
which began when she became Director of NIGMS. Early in 1975, she was
appointed to head up a Grants Peer Review Study Team to perform a broad
evaluation of the peer review system. This Team came up with a number of
recommendations, many of which were radical for that time, but have
since been adopted by NIH. Examples of this Study Team\'s
recommendations include use of an ombudsman, which took 20 years to
implement, release of priority scores and summary statements to
applicants, and implementation of an appeal system.

Dr. Kirschstein emphasized her profound respect for the peer review
system. However, the price of excellence in peer review is eternal
vigilance. She emphasized that the CSR Advisory Committee is part of
this eternal vigilance, by making sure that the peer review system is as
excellent as possible. The American public, through Congress, has
provided NIH with remarkable largess, which NIH must use wisely. The
work of the Panel on Scientific Boundaries for Review and the advice of
the CSR Advisory Committee are landmark activities that will bring peer
review into the 21^st^ Century. Dr. Kirschstein believes that the
foundation of peer review is and should be CSR, and she has committed
herself to providing CSR with the funds necessary to accomplish its
task.

The President's budget for FY 2001 will be released on February 7^th^.
The NIH Appropriations hearings will start a week later, and Congress is
anxious to move as fast as possible on the hearings. NIH has had two
remarkably successful years in terms of the budget, and must be prepared
to explain to Congress what has been done with the increases of the past
two years and why increases should continue. NIH must be clear about its
goals for spending funds appropriately for medically important projects.
She mentioned the Institute of Medicine (IOM) report on setting
priorities and the new Council of Public Representatives, composed of
members of the public, to advise the NIH Director on goals and needs.
Dr. Kirschstein ended by expressing her gratitude to the CSR Advisory
Committee members for their dedicated service to CSR and to NIH.

**III. Opening Comments by the Chairperson........................Dr.
Keith Yamamoto\
** **Approval of Minutes**

Dr. Yamamoto thanked Dr. Kirschstein for her comments and reminded the
members that the role of the CSR Advisory Committee is to advise the CSR
Director on all aspects of peer review, namely the basic organization of
review, the mechanics of review, and the culture of peer review, and to
make recommendations and proposals based on information gathered from a
variety of sources. Dr.Yamamoto noted that these are critical times for
CSR. The CSR professional staff members have been overworked, but as
always, have performed beautifully. The CSR Advisory Committee is at the
brink of making important decisions about previous recommendations, and
implementing changes recommended in the past. All this comes at a time
of uncertainty over who will be the next NIH Director and how things
will operate over the coming years. Dr. Yamamoto mentioned the
importance of asserting the key role of CSR peer review in maintaining
the excellence of the NIH. He also recognized the efforts of the
Division Directors and the staff of the CSR Office of the Director in
preparing and organizing this meeting. He specifically thanked Ms. Linda
Engel, who recently left CSR to become Special Assistant for Program
Coordination to the Director of the National Center for Complementary
and Alternative Medicine.

The Minutes to the Meeting of September 1999 were then unanimously
approved as submitted.**\
\
IV. Comments of the Director.......................................Dr.
Ellie Ehrenfeld**

Dr. Ehrenfeld called attention to a list of names and biographical
sketches of scientific staff new to CSR during the past year. Of these
23 new Scientific Review Administrators (SRAs) hired last year, two are
also Integrated Review Group (IRG) chiefs; in addition, a new Division
Director was hired. The Center is proud of these recent recruitments, as
it is of all the SRAs. Dr. Ehrenfeld also noted Ms. Linda Engel's
leaving, stating that she is already missed.

Dr. Ehrenfeld then provided an update on the CSR budget. Although the
NIH budget has increased roughly 15 percent for each of the past two
years, the CSR budget has remained constrained because the CSR budget is
classified as a management and services allocation, which has been
capped by Congress. As a result, CSR has experienced workload increases
in both the number and complexity of applications reviewed, without
comparable increases in staff. Therefore, the proposed CSR budget
submitted last August for FY2000 included a plan for an increase in
staff to accommodate the increased workload as well as to coordinate,
manage, and evaluate the many changes in organization and procedures
that are occurring at CSR. Although\
Dr. Ehrenfeld does not yet have a formal budget allocation for FY2000,
she feels that we have good support from the central NIH administration
and from the other Institutes and Centers. Dr. Ehrenfeld is concerned
about developing a long-term process that will link further increases in
funds for research initiatives to an increase in the CSR budget for
additional review resources. Dr. Kirschstein expressed support for this
concept.\
\
Dr. Ehrenfeld next noted that the data prepared by Drs. Karen Matthews
and Christine Melchior regarding the review of study section member
applications are now posted on the CSR website. She noted that the data
show, in general, that study section members are not disadvantaged by
serving on study sections. If perceptions continue that this is a
problem, we may have to develop additional ways to educate the
community.\
\
Dr. Ehrenfeld then mentioned Dr. Nancy Pearson's efforts to develop and
pilot test the recommendation to cluster the review of fellowship
applications into larger, broader panels. Fellowship applications for
the May Council round are now being sorted into five broad scientific
categories. Once the initial rough sort is complete, input will be
sought from Institute staff and from Advisory Committee members, staff
members will be identified for these pilots, and the pilot trial will
begin. A more detailed progress report will be available for the next
Advisory Committee meeting.\
\
Lastly, Dr. Ehrenfeld reported on efforts to develop an evaluation
process for the 21 new neuroscience study sections comprising three IRGs
that were first convened in June 1998. The evaluation process involves
two committees that were described at a previous meeting. The first
committee, consisting of both external and internal neuroscientists, has
met and developed a set of questions to collect data from NIH staff and
from applicants whose applications have been reviewed in these study
sections. In addition, external IRG working groups will site visit the
study sections in June 2000 to assess the scientific scope and
operations of these study sections. The second committee provides
biometrics expertise. This committee met in December to design survey
instruments that will provide statistically sound responses from those
people polled, and will analyze and interpret the data collected. The
evaluation system is not yet fully in place, but considerable progress
has been made.\
\
**Discussion\
\
**Dr. Chien felt that a separate line item in the budget for CSR would
help accommodate CSR's workload, as well as evaluation activities and
other initiatives. Dr. Kirschstein responded that, as Chairperson of the
Central Services Review Committee, she has struggled with the problem of
supporting central services at a level that keeps NIH functioning well.
The Federal government has characteristically tried to keep its
administrative activities at the lowest possible level. However, Dr.
Kirschstein is not sure that a \"line item\" in the NIH budget would be
in CSR's best interest. As the system is now, there is the possibility
of correcting things late in the year, which might be precluded if there
were a set budget. Dr. Kirschstein has been able to provide some of what
CSR needed in the past, but suggested that a brainstorming session with
the relevant NIH staff might be needed to figure out the best solution.\
\
Regarding the report on success rates of study section members, Dr.
Navar was concerned whether a 59 percent success rate for study section
member applications was really a success. Since study section members
have a history of success, it might not be appropriate to compare them
to the total applicant population. Dr. Ehrenfeld responded that this had
been factored into the analysis, which compared success rates of
individuals during their study section service with their success rates
pre- and post-service. Dr. Ehrenfeld did not know what a "reasonable
success rate" should be for study section members, but clearly it should
be better than that for the total applicant population. However, unless
the success rate is 100 percent, some members will perceive themselves
to be at a disadvantage by serving on a study section. To compensate for
the perceived disadvantage of study section service, suggestions have
been made that funding for members be extended for an additional year;
however, others have pointed out drawbacks of this scheme.\
\
Dr. Berns next asked how much of the review is performed by Institute
review committees, and whether the review of these applications may
eventually move to CSR. Dr. Ehrenfeld stated that CSR reviews about 75
percent of the grant applications submitted to NIH and the Institute
committees review about 25 percent. The Institutes review mission
specific applications, such as those solicited through Requests for
Applications (RFAs), and multi-center clinical trials, and it is
unlikely that these reviews will come to CSR. However, in the past year,
because Institute review workloads have also increased, the Institutes
have requested that CSR handle more RFA reviews. Also, when multiple
Institutes are sponsoring a joint RFA, CSR has been asked to perform the
review because CSR represents a neutral venue for review. In the last
few years, CSR's review of RFAs has increased from approximately 8
percent to 35 percent, at the Institutes\' request, without a
corresponding budget increase. CSR has had to refuse some Institutes,
and has developed a process for prioritizing requests. Dr. Ehrenfeld
again emphasized the need to link funding for new initiatives with
funding for review.\
\
 Dr. Yamamoto next asked whether Dr. Ehrenfeld felt that she was moving
ahead in the workload problem, and whether different kinds of positions
were needed at CSR to accomplish the evaluation activities as well as
other missions. Dr. Ehrenfeld felt that the Center was moving ahead, but
because of retirements, it was often two steps forward and one step
backward. The current Fiscal Year is the first time in at least a decade
that CSR has utilized its full FTE (Full-Time Equivalent) ceiling, and
proposed future budget requests include an increase in CSR's FTE
ceiling. As to the types of staff needed in CSR, there are many new
activities and new types may be needed. CSR is currently recruiting for
a Deputy Director, and considering a Planning, Analysis, and Evaluation
position, as well as additional managerial staff, statistical staff, and
analytical staff. There is a need to provide statistical reports such
as, for example, what has happened to new investigators since the
abolishment of the R29 mechanism.**\
\
V. Phase 1 Report of Panel on Scientific Boundaries for
Review...............Dr. Keith Yamamoto\
\
**Dr. Yamamoto noted that the Panel on Scientific Boundaries for Review,
which is a working group of the CSR Advisory Committee, was established
in April 1998 and charged with recommending how peer review should be
organized in order to review research appropriately now, and also to be
able to anticipate future changes. The Panel, chaired by Dr. Bruce
Alberts, has now submitted its Phase I Report to the CSR Advisory
Committee. In their Phase 1 Report, the Panel affirmed that an
outstanding grant review process must set high standards, contribute to
the advancement of the health-related science, encourage innovation and
risk taking, exercise fairness, be transparent to all participants; and
undergo periodic review.\
\
To accomplish this, the Phase 1 report recommended a new organizational
structure of IRGs. Phase 2 will determine the study sections that will
populate these IRGs. The proposed IRG structure is based on several
guiding principles: (1) there should be at least one appropriate venue
for the review of all applications relevant to health-related research;
(2) research topics encompassed by each IRG should be sufficiently
cohesive to allow an external advisory group to judge the contents of
the entire portfolio; and (3) the organization should be flexible enough
to adjust to rapid changes in scientific opportunity.\
\
In developing the list of proposed IRGs, there were two key principles.
The first was an attempt to cluster all types of research whose major
emphasis is focused on a given organ system or disease to an IRG devoted
to that system or disease. The second key principle was to create IRGs
for the review of basic scientific discovery and the development of
methods not explicitly focused on a specific system or disease. The
Panel also emphasized the notion of a continuum of research from basic
to applied, rather than one type versus another.  \
\
There were multiple rationales for adopting these principles. First, the
Nation's health will best be served by reviewing clinically relevant
science in the context of the basic knowledge in which it was founded,
and by reviewing basic science in the context of the human condition
that it is designed to improve. Second, the new knowledge from basic
science should be applied to the disease problems in a timely and
appropriate way. The best way to accomplish that would be to mix the two
together in appropriate ways so as to translate progress from basic
science into improving human health. By the same token, human disease
research can help direct new efforts aimed at understanding fundamental
processes. Third, the exchange of ideas and perspectives among reviewers
who work in the same biological systems but with different points of
view may promote identification of ambitious and creative applications,
and encourage submission of broader and more innovative applications.
Fourth, the review of basic and applied science in the same IRG can
provide an overview of the quality of work across all its study
sections. Finally, applications using particularly powerful
methodologies will be distributed across different IRGs by these
principles, thus broadening the application of these new methods to many
disease and science system oriented systems.  \
\
The third part of the Phase 1 report focused on cultural norms for IRGs
and study sections. The purpose was to point out the importance of how
the review process actually operates within a study section. The
following issues were addressed:

-   Who is a peer? (*An active, experienced researcher)*

-   What is a peer reviewer's role? *(To judge the research proposed,
    providing funding Institutes with advice about its scientific
    merit)*

-   What are the roles of study section members, study section
    chairpersons, and SRAs? *(Emphasis on partnership of chairpersons
    and SRAs and complementary roles of members and chairpersons)*

-   What is the appropriate relationship between study sections and
    disciplines? *(Study section evaluates and Institute supports
    research, not disciplines or fields)*

-   What types of research will help NIH achieve its mission? *(Broad
    portfolio of grants, including high risk, innovative projects;
    avoidance of current overemphasis on hypothesis-driven research)*

-   How should the results of the review be communicated to applicants?
    *(Summary statements should convey the rationale for the score, but
    should not provide a detailed recounting of errors or a compendium
    of redesigned experiments for future amended applications)*

-   What is the role of preliminary data? *(An obsession with
    preliminary data discriminates against young scientists, against
    bold new ideas, and against risk-taking)*

-   What procedures can be introduced to improve the operation of study
    sections? *(Increase the number of reviewers per proposal,
    facilitate self-referral, and explore ways to review
    multi-investigator projects)*

Phase 2 will begin in 2000 and continue for the next two years. In Phase
2, study sections will be designed to populate the IRGs identified in
Phase 1. This will be a highly interactive process in which we will call
upon the extramural community, professional societies, and others, not
just to provide feedback, but to actually participate in developing the
study sections. The study sections should be reviewed approximately
every five years. They should be neither too broad nor too narrow. The
range of science should be sufficiently coherent to allow members to
reach independent evaluations of scientific merit. Overlap is desirable.
The study section should be connected to specific diseases or organs
and, whenever appropriate, basic research should be reviewed in the
context of the biological question to which it relates.

With respect to density of expertise, at least 30 percent of the
reviewers (including ad hoc reviewers and outside opinions) should be
experts in the field of any given application. Dr. Yamamoto noted that
the balance of breadth and depth in study sections and reviewers is easy
to say, but hard to accomplish. One possibility is to create an IRG-wide
pool of reviewers who could serve as broadly based advisors to related
study sections. Finally, study sections should serve multiple Institutes
rather than be captive to one Institute.

**Discussion**

In response to a question about when the new organizational structure
will go into effect, Dr. Yamamoto noted that the Committee is currently
discussing whether it should be phased in, or whether to wait until the
reorganization is complete before taking effect. Dr. Yamamoto felt that
there is a psychological advantage to a phase in, but that it may be
more complicated logistically. Dr. Yamamoto also pointed out that during
Phase 2, as applications are sorted into potential study sections, it
may be necessary to make adjustments to the Phase 1 organization. Dr.
Lifton stated that, no matter which end of the spectrum a particular
discipline is on, the gradual approach may leave the extramural
community feeling that the process is unfair. Dr. Colvin preferred the
phased-in process, for it would enable the Committee to make any needed
adjustments easily. Dr. Chien also preferred the gradual approach as
less chaotic for the scientific community and for CSR staff. Dr.
Ehrenfeld added that a phased in approach would communicate to the
community that NIH recognized that these events deeply affected peoples'
lives, and indicated that there is good reason to appear to be moving
slowly and cautiously rather than making a major change.

Dr. Yamamoto noted that, during the public comment period for the draft
Phase 1 report, CSR had received approximately 800 comments from
scientists, professional societies, academic institutions, and other
organizations. These comments, plus those from the previous meeting of
this Advisory Committee, were seriously considered by the Panel, and
extensive changes were made into the final Phase 1 Report. One of the
most substantial changes was the addition of three more IRGs.

Dr. Navar questioned whether the Panel had overemphasized that the role
of peer reviewers is to evaluate applications for scientific merit,
rather than act in a mentoring role. Summary statement critiques have
traditionally had tremendous instructional value for all investigators,
not just for young investigators. Dr. Yamamoto responded that one
difficulty with assuming a mentoring role is that the membership of a
study section will have changed somewhat when a revised application
comes back to that study section. Applicants become frustrated if they
had considered the summary statement to be a recipe for funding.

Dr. Berns questioned the separation of bioengineering into both the
Fundamental Bioengineering IRG and the Applied Bioengineering IRG,
especially since one guiding principle was to have both applied and
basic research in the same IRG in order to achieve the continuum
concept. Dr. Ehrenfeld responded that this division was an effort to
separate development of instrumentation or new methodologies from the
application of methodologies or instrumentation to an organ system or a
set of biological questions. But this organization could change in Phase
2 when we actually deal with sets of applications.

Dr. Navar felt that the reorganization should not be a radical
transformation but, rather, a fine tuning to be more current and more
responsive to emerging opportunities. As long as the reorganization is
done rationally and gradually, there should be few problems except in
certain areas where some study sections are eliminated and repositioned
to another IRG. Dr. Yamamoto remarked that although this was not a
revolution, for the individual investigators and for CSR staff, the
changes are substantial. Dr. Schachman added that he has found
considerable paranoia in the scientific community and concern about the
elimination of study sections. The phase-in of certain areas can be
accomplished relatively soon, but other areas are more complicated. He
was concerned that, rather than complain about the process, the
scientific community should get more involved in defining study
sections. For Dr. Ehrenfeld, the most important message for the
scientific community was that, as CSR starts populating the IRGs with
study sections, the external community will be called upon to
participate in the design of appropriate study sections.

Dr. Chien then moved that the CSR Advisory Committee accept the Panel's
excellent Phase I Report, and commended the Panel for taking into
account the many responses from the scientific community as well as
those from this Committee. He noted that the Phase 1 organization will
need continuous adjustment as new needs arise and in response to further
feedback from the scientific community. This motion was seconded and
unanimously approved. Dr. Yamamoto noted that the final Phase 1 Report
will be posted on the CSR website now that it has been accepted.

**VI. Periodic Review of Study Sections...............................
Dr. Yamamoto\
Dr. Arnold Revzin  **

Dr. Yamamoto explained that, since the Boundaries Panel emphasizes the
fact that biological science is a dynamic process, the review process
itself should be commensurately dynamic and continuously renewed.
Therefore, the Committee is developing a mechanism for periodic review
of individual study sections. The goals of the periodic review are: 1)
to annually collect information, using self-assessment and
questionnaires, that tracks study section performance; and 2) to assess,
at five-year intervals, the organizing principles and operating
procedures for each study section. Dr. Yamamoto noted that, in an effort
to find ways to gather information from the various practitioners and
customers of the review process, survey instruments have been developed
in collaboration with consultant Gail Herzenberg (Health Research
Sciences, Inc). CSR staff members have also participated in the
development of this periodic review process.

Dr. Yamamoto then called on Dr. Arnold Revzin, SRA of Special Reviews in
the Biophysical and Chemical Sciences IRG, to further elaborate progress
in this area. Dr. Revzin stated that he and Dr. Rona Hirschberg, Chief
of the Infectious Diseases and Microbiology IRG, were the CSR
representatives for this project. He noted that the periodic review
process, as recommended by the Advisory Committee, has two components.
One of these, which could be termed the \"scientific boundaries
component\", would involve annual analyses by the SRA and study section
chairperson of applications reviewed by the panel, classifying the
applications into perhaps 10 to 20 topic areas. He emphasized the
importance of involving the chairperson in this process, although this
activity is not currently listed as an item in the \"Guidelines for
Study Section Chairs.\" It is not really additional work for the SRA
since a similar accounting is needed as part of the "State of the Study
Section" portion of the nomination of new members. 

The second aspect of periodic review is the use of surveys to obtain
some measure of study section performance. One survey, currently under
development, will focus on the grant applicants. The second survey is
focused on study section members and seeks their opinion on how the
study section is functioning. This survey has been shared with the SRA
Council, which had no major concerns with the content, but formed a
subcommittee to make recommendations on revising the wording of specific
items. The questionnaire will be distributed at the February meetings to
three study sections, one from each Division. Following this pilot, the
plans are to distribute the questionnaire to all study section members
during the June round of meetings. Responses will be collected in sealed
envelopes and distributed to Gail Herzenberg for analysis. 

**Discussion  **

Dr. Ehrenfeld explained that it was necessary to obtain clearance from
the Office of Management and Budget (OMB) to distribute the survey,
which should take about two weeks. Some modifications may be necessary
depending on OMB's comments.  

Dr. Matthews noted that the questionnaire now has about 52 items and was
concerned that this would be burdensome. Ms. Herzenberg, the consultant
developing the questionnaire, stated that the number of questions had
been reduced from a previous version, and that the estimated time for
completion of the survey was now 7 to15 minutes (depending on whether a
response is given to narrative questions), which should be acceptable to
OMB. Dr. Chien felt that the survey could be further simplified, and the
questions should be phrased and formatted uniformly, using bold-faced
type and underlining when appropriate. 

Dr. Yamamoto explained that information will be collected annually and
used by the SRA and the IRG Chief for annual evaluations of study
section performance. In addition, the information from each annual
survey will be available to the IRG working groups for their five-year
periodic evaluations. 

Discussion then centered on review of study section performance by
groups external to the study section. Dr. Lifton suggested that ad hoc
members would be a valuable source for independent evaluation. Dr.
Epstein agreed, noting that reviewers who had seen other study sections
operate could provide validation of the evaluation instruments. Dr.
Navar envisioned an external review committee that would observe and
evaluate many study sections, and Dr. Ehrenfeld explained that that was
the function of the external IRG working groups.  

In response to the lack of survey questions pertaining to the
performance of study section chairpersons, Ms. Herzenberg explained that
the questions are there, but deliberately disguised with euphemisms, so
that the questions would be less threatening and more acceptable. Dr.
Ehrenfeld suggested that the SRA Council discuss whether these questions
should be phrased more directly. 

Dr. Ehrenfeld was concerned about the expense of an annual survey of
study section members, and asked whether the survey should be done on an
annual or biennial basis. SRA self-assessments would still be annual,
since SRAs annually go through a self-assessment process in conjunction
with the nomination process. Dr. Yamamoto preferred annual surveys for
study section members, particularly during these times of change, and
suggested pressing for added budgetary support to perform the survey.
Dr. Epstein wondered whether electronic surveys could save money, but
concerns were raised about the confidentiality, and about the timeliness
of obtaining responses.

Dr. Lifton asked for clarification of appropriateness of research
topics. Dr. Yamamoto responded that appropriateness meant whether the
range of research covered by a study section fit the charge of the
group, and, reciprocally, whether the study section expertise fit the
topic assignments. Such questions are aimed at assessing the dynamics of
the field. Dr. Lifton suggested having a separate survey question on the
impact of research emerging from the study section. Dr. Yamamoto
concurred, but Dr. Ehrenfeld noted that such analyses are normally
performed by program staff within the Institutes.

**VII. Promoting Best Practices. .....................................
Dr. Richard Lifton\
Dr. Alex Politis**

Dr. Lifton described their goal of developing guidelines for new
reviewers joining study sections to promote best practices. They
recognized that most of what they propose is already in practice in the
best study sections and is familiar to those experienced in the review
process. However, such guidelines should prove useful for new reviewers,
and might be valuable reminders for experienced reviewers.

Dr. Alex Politis, SRA of the Immunological Sciences Study Section,
explained that the guidelines for reviewers were developed to complement
similar guidelines for SRAs and chairpersons. The current draft document
is based on discussions involving Drs. Lifton, Politis, and Wickens, and
is primarily a list of issues and a suggested format. The introduction
is followed by a chronological description of the duties and
responsibilities of reviewers. The final section contains reference
materials in appendices.

The introduction will discuss the responsibility of the reviewers to
evaluate the science, as related to the mission of NIH. In addition, the
role of the reviewer will be put in perspective, namely, that the study
section evaluates the science and the Institute makes funding decisions.
There will also be discussion of the roles of the Chair, SRA, and GTA.
Thus, there will be some redundancy with the documents on Chairperson
Guidelines and on the role of the SRA, although an effort will be made
to integrate the three documents.

The next section deals with pre-meeting activities, such as when
reviewers can expect to receive applications, appropriateness of
application assignments, conflict-of-interest forms, and criteria
(appropriate and inappropriate) for evaluation of applications. A
recommended length for written critiques would help keep reviews
informative but free of excessive issues and details. Other issues, such
as budget, human subjects and vertebrate animals will also be discussed
briefly. Additional topics include streamlining and priority score
determination.

The next section of the reviewer guidelines would contain a description
of what happens at a study section meeting. This would include a
discussion of opening remarks followed by the streamlining process. For
applications in the top half, the importance of listening carefully and
participating in discussions will be emphasized. New reviewers would not
have to give their reviews first. It will be emphasized that reviewers
need to have critiques ready before the meeting, and that they may need
to modify them based on the discussion.

After the meeting, it will be noted that reviewer responsibilities are
minimal, although on occasion they may be asked to provide clarification
to the SRA writing the summary statements. On rare occasion, they may
also be asked to respond to an appeal letter.

Finally, the document will include several appendices addressing, for
example, what constitutes a conflict of interest. Also included will be
a work flow chart, a brief discussion of the various grant mechanisms,
travel and reimbursement information, and reference material.

Dr. Politis next spoke about developing a video to complement the
reviewer guidelines that would illustrate peer review. The goals would
be to encourage new reviewers, and to teach and orient them in the peer
review process. The general approach would be to have an introduction by
the CSR Director or by the NIH Director, to use a professional narrator,
and to demonstrate peer review via a mock study section. It would
include examples of proper procedures and best practices as well as
inappropriate reviewer comments or behavior.

**Discussion**

Dr. Yamada felt that new reviewers would benefit from seeing the
distribution of priority scores for their study section for the past
three meetings, along with corresponding percentiles. Dr. Politis
agreed, noting that the scoring procedure is difficult to teach to new
reviewers. However, Dr. Yamamoto saw a problem with using past data as a
guideline for scoring because there is a current effort to normalize
scoring among study sections as much as possible, and some study
sections are trying to change past history. Dr. Ehrenfeld thought it
would be helpful for new reviewers to learn and understand the concept
of spreading scores within the range that is used.

In response to a question on handling outside opinions and mail reviews,
Dr. Politis said that in his experiences, most outside opinions are
solicited after a reviewer expresses a need for them to an SRA. Dr.
Politis felt it would be useful to include a brief discussion about how
reviewers should evaluate or incorporate outside opinions. To ensure
that an outside opinion is not colored by the reviewer's opinions, Dr.
Lifton felt that outside opinions should be presented to the study
section either by the chairperson or by the SRA.

Dr. Yamamoto felt that the guidelines should include a recommendation
for the appropriate level of detail or length of the critique. The
written reviews should be shorter than they have been in the past,
succinct with just the main points. He also felt that the critiques
should not be read at the meeting. An additional point to include in the
guidelines is that reviewers should not interact with applicants
following the meeting. With respect to the length of critiques, Dr.
Epstein suggested providing three sample summary statements in an
appendix, deliberately made short. Dr. Politis had done this in the
past, and found the practice helpful. As for the actual length, Dr.
Yamamoto suggested approximately a page and a half. Dr. Chien also
preferred this shorter length as easier for reviewers and clearer for
applicants. However, Dr. Yamada was concerned about omitting secondary
criticisms that may then come up in the revised application. Dr.
Matthews suggested making the language more flexible, explaining that a
longer review does not necessarily mean a better review, but that, in
general, a critique should be under two pages.

The discussion then moved to the problem of reviewers not explicitly
stating their lack of enthusiasm for a proposed project, and instead,
concentrating on minor details. Applicants then address each point, but
still don\'t get a better score. It should be emphasized that reviewers
must comment on each of the five criteria for rating applications, and,
in particular, should make explicit statements about the impact and
significance of a project.

Dr. Yamamoto commented that the next step with regards to the
development of reviewer guidelines would be to refine the document based
on comments from this meeting, and present the revised version at the
next meeting for final modification and approval.

With regards to the development of a video, the consensus was that Dr.
Ehrenfeld should provide the introduction. Dr. Navar commented on the
expense of developing a high-quality video, and questioned the
cost/benefit ratio of the project. Dr. Ehrenfeld estimated a one-time
cost of \$50,000 for a 10 - 15 minute tape. Dr. Lifton was skeptical
whether the video would add much to a well-written document, but Drs.
Berns and Chien thought it could be useful if done well. It was
generally agreed to proceed with the project, and then evaluate its
usefulness. The next step is to write a script.

**VIII. Logistics of Study Section Service.................... Dr.
Joanne Fujii**

Dr. Joanne Fujii, SRA of the Molecular, Cellular, and Developmental
Neurosciences -- 7 Study Section, first reviewed the mandates of this
subcommittee, which were to consider how to redefine the concept and
logistics of study section service to better capture and maintain the
participation of all qualified reviewers. In addition to herself,
members of the subcommittee are Drs. Matthews and Yamada from the
Advisory Committee, and\
Dr. Dan McDonald, Chief of the Musculoskeletal and Dental Sciences IRG
and Chair of the SRA Council.

Dr. Fujii first considered the desirable characteristics of study
section members. Of paramount importance is obtaining appropriate
expertise and a fair representation of the different scientific fields
addressed in the applications. In addition, it is important to balance
scientific viewpoints regarding different approaches. It is also
desirable to balance the experience of reviewers, as in the broad
experience of senior scientists, the expertise of leading edge
scientists, and the in-trenches viewpoint of junior scientists. The
group must be of workable size, in the range of 20 to 35 reviewers, and
the reviewers should work well in a group context. Finally, geographical
and institutional diversity, as well as gender and minority
participation, are required.

She next noted that the pool of potential reviewers is limited, which
can be an obstacle to assembling a high-quality study section. The
limitation is caused, in part, by reluctance of potential reviewers to
serve, by the fact that some potential reviewers are already serving on
advisory committees outside of CSR, and by conflict of interest
policies, institutional balance, and gender or racial considerations.
Finally, there is a cultural attitude that study section service is a
once in a lifetime event.

A standard mechanism for managing the limited pool of reviewers and for
testing potential members is the use of ad hoc reviewers. This is also a
way to attract senior investigators who may not be willing to commit to
full service, as well as a way to supplement the expertise of a study
section.

To analyze workloads, an informal e-mail survey was distributed to SRAs
of CSR study sections. Based on 55 responses, the average meeting had 26
reviewers (range of 18 to 45) including telephone reviewers; 8 ad hoc
reviewers (range of 1 to 20); 75 applications (range of 30 to 120); 3
reviewers per application (range of 3 to 5); and 11 applications
assigned for in-depth review to each reviewer (range of 4 to 12), of
which 7 to 8 required a written critique. The average study section
meeting lasts a little under two days. The SRAs who responded to the
survey reported that many members missed at least one meeting during
their four-year terms.

In considering ways to rethink study section service, flexibility is
essential. The following suggestions were made:

-   Change the definition of study section service. Instead of defining
    service as four years where members are encouraged to attend three
    meetings per year, change the definition of service to a total
    number of review meetings.

-   Share reviewers among study sections of an IRG. This would be
    especially useful for a reviewer with a highly specialized area of
    expertise that may be needed only for the occasional application.

-    Develop a pool of experienced, high-quality reviewers available to
    all of CSR, similar to the Reviewer Reserve.

-   Control the workload demanded of reviewers.

-   Share slate positions whereby more than one person would share the
    position, but only one person at a time could attend a given study
    section meeting.

-   Reconsider and redefine the conflict-of-interest rules, in
    particular, institutional clustering rules.

-   Increase the use of technological options. Teleconferences are
    widely used, but other options include cyberspace review on a web
    site, PC based videoconferencing, and chat rooms for study section
    members.

-   Develop an incentive program in conjunction with the Institutes. For
    example, provide an extra year of funding for four years of study
    section membership, or identify review service as a
    responsibility/obligation associated with funding.

**Discussion  **

The first topic discussed was Dr. Fujii\'s general principle that study
sections work better when reviewers feel that they belong and identify
with a particular review group, and when they develop a working
relationship with each other. Dr. Lifton pointed out that journals
operate without reviewers having a working relationship, yet come to a
reasonable consensus. Dr. Fujii pointed out that there are more concerns
with funding decisions than with publications decisions, and that
fairness in review is enhanced by the interactions of a large number of
reviewers. She felt it is better all reviewers are present, although
SRAs often bring in outside opinions and telephone reviewers. Dr. Chien
agreed that group dynamics are essential, and can often change the
impact of the initial primary reviewer. Some agencies and journals use
mail reviews, although Drs. Chien and Lifton felt that the NIH system
worked better for grant applications. Dr. Yamada had used video
teleconferencing for reviews, but while this technique was fine for
sharing information and discussions, it did not work well for arriving
at priority decisions or achieving consensus.\
\
Dr. Fujii then discussed changing the definition of study section
service from a time commitment to a commitment for number of meetings.
For example, reviewers could have a longer time frame than 4 years
(e.g., 6 years) in which to complete the 12 meetings. One advantage is
that it would maximize flexibility for reviewers to meet commitments.
Another is less risk of reviewer burnout, thereby improving the quality
of review. It would also be a fairer recognition of the service of all
members, including ad hoc reviewers. Disadvantages are that it is
administratively more complex, that there could be a lack of continuity
from meeting to meeting, and that more reviewers would be needed per
study section.

One possible scenario would be for members to attend all meetings in the
first year, then reduce the number of meetings in the second, third, and
fourth years. Dr. Yamamoto thought that all study section members should
start the same way, attending three meetings in a row in order to
observe the culture of the study section and understand how it operates.
After this first year, there could be a flexible menu of options. This
might entice people who otherwise could not or would not serve. Dr.
Fujii suggested that new reviewers could serve a traditional type of
service, whereas more experienced reviewers would have other options,
including participation in a shared slate, attending fewer meetings per
year, and becoming an IRG-wide reviewer. Dr. Matthews suggested that all
members attend three meetings in the first year, and then negotiate how
to meet their obligation for nine more meetings over the next four or
five years. Dr. Berns, on the other hand, was concerned about the
administrative burden of handling so many different options, and thought
that there may need to be some limits on the options. Dr. Navar also
thought there would be logistical problems in implementing this idea.
Reviewers could no longer count on interacting with other specific
reviewers on the study section, and applicants may feel misled in not
dealing with the posted composition of the study section.

Dr. Kushner noted that redefined study section service, shared slate
positions, and incentive programs were all designed to encourage
individuals to serve on study sections. However, he felt that the shared
slate position might be most useful for trying out ad hoc members, and
that spreading out the 12 review meetings over 5 or 6 years was not
likely to be successful. He was also opposed to the incentive approach,
and felt that the obligation approach was better. Drs.Yamamoto, Chien,
and Epstein likewise saw problems with any type of reward program, and
Dr. Ehrenfeld noted that the use of funds as an incentive for study
section membership is an Institute decision, not a CSR decision. Dr.
Kushner preferred incentive via social pressure; thus, an individual who
has been served by the system (i.e., funded) should feel a
responsibility for serving the system.\
Dr. McDonald indicated that he has been successful at bringing
clinicians to his study section by going to the communities and
impressing them with the need to serve. Dr. Kushner added that the
educational value of study section membership should be emphasized, and
that the experience is beneficial to an individual\'s ability to write
grant applications. There was also discussion that the study section
Chairperson might be helpful in recruiting new members.

Dr. Matthews noted that workload control was intertwined with length and
frequency of service. The data collected by e-mail show a large range in
workloads across study sections. She thought it would be useful to
define an optimal functioning study section in terms of number of
members and number of applications. Dr. Ehrenfeld responded that this
was an ongoing activity. The average number of applications in CSR is
around 75 per meeting but the range is large, and too many study
sections review more then 90 applications per meeting. The number of
applications assigned to individual reviewers also ranges considerably;
it can be as low as 4 for new reviewers and 12 or more for experienced
reviewers. The optimal number of reviewers assigned to an application
would be difficult to determine; it also depends on the size and
complexity of the applications, as well as on the capabilities of the
individual reviewer.

Dr. Yamada was concerned about the shared slate concept because it set
up a different dynamic. First, it is not clear that each of the
individuals in the group will have equal capabilities for serving as a
reviewer, and, second, there are problems with the concept that a seat
on a study section belongs to an institution. This creates an
institutional power and dynamic that is antithetical to the process of
individual peer review.

Regarding shared slates, Dr. Colvin suggested that the reviewers could
come from different places, such that the position was focused on a
discipline or area of expertise rather than institutionally focused. Dr.
Yamada added that this would be created by flexible membership, as when
members were committed to a fixed number of meetings such as one meeting
a year for 10 years. This could be appropriate for experienced senior
reviewers, but not appropriate for new reviewers. However, Dr. Fujii had
concerns about creating different classes of reviewers within a study
section. Dr.Yamada suggested that this type of flexibility could be
limited to those members who had already paid their dues by past service
on a study section. Dr. McDonald noted that this was similar to the
Reviewers' Reserve, which was a pool of reviewers composed of former
members of study sections.

Dr. Epstein, however, while recognizing the advantages of flexibility,
also noted the advantages to study section stability, especially from
the perspective of the applicant. While reviewers do not necessarily
agree completely with the previous review, it is useful for some
reviewers at least to have some history and knowledge about the previous
discussion. Dr. Yamamoto agreed that stability was important, and
suggested limiting the number of flexible slots. 

Dr. Ehrenfeld noted that almost all of the suggestions proposed have
been piloted formally or tested informally. For example, the shared
slate proposal was piloted in the Diagnostic Radiology Study Section, in
a clinical field where there are only a few institutions available with
good active research programs. In that case, the experiment worked out
well. It has been suggested that this particular solution be used for
other study sections with similar problems, especially in the clinical
area. Dr. Yamada, however, was concerned that through multiple
memberships, a few institutions could control what happens in their
fields. Dr. Eileen Bradley, the SRA of the Diagnostic Radiology Study
Section, remarked that shared slates also build in the flexibility of
having different expertise on the Study Section, since some large
departments have multiple areas of expertise.

The discussion continued into the second day. The following morning, Dr.
Yamamoto expanded on his idea of two tracks for study section
membership. First-time members would serve for three consecutive
meetings, and then there would be a menu of possibilities for continued
service that would be negotiated with the SRA. The second track would be
for experienced reviewers, and terms of service would be negotiated with
the SRA. Some would serve once a year, some would become IRG-wide
reviewers, and others might serve more frequently. Such a scheme would
offer considerable flexibility.

Dr. Ehrenfeld questioned the advantage of the second track for
experienced reviewers versus simply having them serve on an ad hoc
basis. Dr. Yamamoto felt that it would be important for the community to
know that there were two types of reviewers, and that there was a track
for each type. Dr. Berns pointed out that keeping track of all the
different arrangements for all the study section members could be
extremely complex, and Dr. Navar was concerned that the proposal could
be too flexible, and encourage excessive absences. Dr. Matthews
preferred having a flexible system with constraints, with three meetings
in the first year of membership, and nine more meetings within a total
of five (or perhaps six) years. Dr. Matthews also thought that it would
be helpful to obtain more data on the perceived workload and whether
perceived workload influenced decisions about study section membership. 

**IX. IRG Working Group: Status Report.................... Dr. Donald
Schneider  \
\
**Dr. Donald Schneider, Acting Director of the Division of Molecular and
Cellular Mechanisms, reported on progress with the IRG Working Groups.
These groups consist of outside scientists brought in periodically to
assess the IRG function and study section performance.

The burden for identifying members of an IRG Working Group is on the IRG
Chief because that person is in the best position to know the science. A
list of candidates for the working group is prepared based on
consultation with SRAs, with Institute program staff, and with the
scientific community. Current study section members and chairpersons are
not eligible. The list is given to the Division Director who, with
additional input from stakeholders, appoints 6 to 10 members from that
list to serve as a Working Group for evaluation of the IRG. One
individual on the Working Group is identified as Chairperson.

The overall process should take place in one or more rounds, but
certainly within a year. The process begins with a conference call
involving the CSR Director, the Chair of the CSR Advisory Committee, and
the Chair of the Working Group in which the Working Group is charged
with its mission to assess the IRG. The working group members are then
given assignments to attend the various study section meetings of that
IRG, not just as observers, but as participating members. While their
review workloads are reduced, they do have some written assignments and
are expected to participate in discussions. They try not to participate
in their former 'home' study sections, although this cannot always be
avoided.

After the last study section meeting of that round, the group has a
telephone conference call for a first level assessment that is mainly
anecdotal. This is followed by a meeting in Bethesda that is attended
not only by members of the working group, but also by the Division
Director, the IRG Chief, and many of the SRAs. Shortly after the in-town
meeting, the Chairperson prepares a draft report, which is circulated to
other members of the working group for comment. The final report is then
prepared that addresses the scientific scope and peer review reports
from each member. The working group also looks for scientific overlap
among the study sections, one of the principles of the Boundaries Panel
Report, for consistency of review among the study sections of the IRG,
and for effectiveness of the SRA and Chairperson in ensuring quality
peer review. They also look for ways in which peer review can be
improved, and are asked to consider the Scientific Boundaries Panel
report and its implications.

CSR currently has four active working group rosters. Dr. Schneider
discussed in detail the Working Group for the Biophysical and Chemical
Sciences (BPC) IRG where Dr. Schneider is Acting Division Director and
Dr. John Bowers is Acting IRG Chief. The IRG consists of nine study
sections. The other three IRG Working Groups with active rosters are for
the Cardiovascular Sciences IRG (CVS), the Musculoskeletal and Dental
Sciences IRG (MSD), and the Oncological Sciences IRG (ONC).  

The BPC Working Group attended the October/November 1999 round of study
section meetings. The Working Group concluded that the IRG seemed to be
functioning well and should not be compromised. They did, however, have
some specific comments. For example, the Medicinal Chemistry Study
Section (MCHA), which has traditionally focused on organic synthesis,
should continue the recent broadening of its scientific base by
continuing to move into such areas as host guest chemistry. They also
applauded the fact that the IRG was attentive to new emerging areas,
such as the creation of a Special Emphasis Panel (SEP) that focuses on
methods applications. They also felt that the SRAs need time to attend
scientific meetings, both in order to recruit members, but also to
recognize emerging areas.

In terms of reviewers, they felt that chairpersons require careful
training, and applauded the document prepared on \"Guidelines for Study
Section Chairs\". October is when new chairpersons come on board, and
there were three new chairs within the IRG. The Working Group
recommended that the individual who will become chair in October serve
as vice-chair in the previous June meeting, chairing the review for
perhaps half a day. At the October meeting when new members are on
board, it is important to have senior members present to promote best
practices and consistency. The Working Group also felt that senior
members are an underutilized force in developing study section culture
and consistency, and that senior members could move from one study
section in one round to another study section in another round to spread
the word on best practices.

In terms of fairness of review, the Working Group members found that the
reviewers were serious about the task at hand, and generally very fair.
In general the comments were laudatory, and the Working Group was
impressed at how well the revised review criteria worked, especially
with the emphasis on innovation and impact on the field. They found no
problems with the review of fellowship applications despite the fact
that these applications were reviewed in clusters of only about 12
applications. Similarly, they were pleased with the review of Small
Business Innovation Research (SBIR) applications, and felt that the SRAs
had selected just the right experts.

On the other hand, the members of the Working Group were quite critical
of the streamlining procedure, and felt that placing 50 percent of the
applications in the lower half, without discussion, was artificial and
unreasonable. In fact, they recommended eliminating the streamlining
procedure, although it was made clear to them that this was not a CSR
decision. The Working Group also suggested that each study section
meeting start by reviewing one strong application and one weak
application to calibrate scoring, and recommended the use of adjectives
to promote the spreading of scores.

Dr. Schneider also reflected that the Cell Development and Function IRG
(CDF) had pioneered the idea of IRG oversight groups. However, the
original CDF paradigm was a conflict of interest, with members assessing
themselves. The new IRG Chief, Dr. Marcia Steinberg, has recently
disbanded that group in a \"community discussion meeting\" held in
conjunction with the annual Cell Biology meeting. The CDF community
endorses the creation of IRG Working Groups, but recommends that they
meet more frequently than once every five years. Finally, this group
felt that modular budgets do not provide sufficient information for
review and recommended that budgets should at least have broad
categories (e.g., personnel).

As for the timetable, Dr. Schneider noted that if the current schedule
is kept, all working groups will have met by the summer of 2001 with
reports completed by September 2001.

**Discussion**

Dr. Matthews asked whether there was variability among study sections in
the use of streamlining procedures. Dr. Ehrenfeld acknowledged that
there is considerable variability. In some study sections, streamlined
applications receive a brief discussion. In other instances, if there is
a significant discussion, the SRA often writes a resume and summary of
discussion, but there is no score. CSR encourages the SRAs to work
toward triaging 50 percent, but in reality, it is closer to 37 or 40
percent.

Dr. Ehrenfeld went on to explain that streamlining the bottom half of
applications was put into effect under Dr. Varmus and seemed reasonable
when Institutes were paying only to the top 10^th^ or 12^th^ percentile.
Now that pay lines have improved, and some Institutes are funding to the
30^th^ percentile, a 50 percent cutoff may not be reasonable. Many
letters of complaint have been received, and, although CSR cannot change
this trans-NIH rule, if the CSR Advisory Committee raises the issue for
discussion in other venues, that could lead to a reevaluation of whether
streamlining is still appropriate.

Dr. Wickens was concerned about the inconsistency that might result from
the Working Group's suggestion that the vice chairperson handle the
reviews for one morning during the June meeting. Dr. Schneider agreed,
but felt that the inconsistency would be greater during the October
meeting without it.

Dr. Matthews asked about the dialogue between CSR and Institute staff to
identify emerging areas of science. Dr. Schneider noted the change in
culture since Dr. Ehrenfeld became Director, with a greater emphasis on
science and more sharing of information with the Institutes.
Interactions are informal and much more frequent. Dr. Ehrenfeld noted
the importance of improving interactions and relationships with the
Institutes, and added that CSR is in the process of trying to develop
more structured mechanisms for this.

**X. Number of Reviewers per Application.................. Dr. Michael
Berns**

Dr. Berns presented three assumptions related to the number of reviewers
per application; to stimulate discussion, he challenged the validity of
these assumptions. The first assumption is that the number and
complexity of approaches envisioned within applications is steadily
expanding. If more applications are multidisciplinary, the complexity
must be increasing, although this may not be true for all applications.
The second assumption is that, with increasing complexity, adequate
review by two reviewers is often not possible. Most applications are
actually reviewed by three reviewers although there are only two written
critiques; perhaps even three reviewers is not adequate. The third
assumption is that, in general, the quality of review increases with the
number of reviewers that examine each application in detail because the
more discussion, the better the quality of review.

**Discussion**

Dr. Navar felt that, in general, after two critiques, the destiny of the
review for a given application is set. The real advantage of additional
reviewers is when there are mixed opinions. It would be hard to make the
case that the quality of the review will increase substantially as more
reviewers are added.

Dr. Berns suggested that there are two categories of applications, those
well-defined projects that can be handled easily by two reviewers, and
multidisciplinary projects that may require more reviewers. SRAs may
already be assigning more reviewers to the complex applications.
However, having more reviewers routinely assigned to applications will
increase the workload, which is not desirable.

For Dr. Lifton, the quality of reviews increases with the quality of the
reviewers, not with the number of reviewers. For many applications, SRAs
may have difficulty finding two reviewers who can provide a high-quality
review; and adding a third discussant may not increase the overall
quality of the review. The key is in providing expert review.

Dr. Epstein thought that adding reviewers would increase the number of
priority scores in the middle range. In his experience, applications
with outstanding scores had a strong advocate at the review meeting.
With more reviewers, there is a greater chance of having reviewers who
are not strong advocates, and who will keep the score from the
outstanding range.

In response to Dr. Yamamoto\'s request for input from SRAs, Dr. McDonald
agreed with the first assumption that the complexity of applications is
steadily expanding. With regard to the second assumption, standard
practice is to use three designated reviewers per application, although,
depending on circumstances, more are often used. As for the third
assumption, that the quality of review increases with the number of
reviewers, he felt that there is a limit on the number of reviewers that
can be effectively utilized. Further, the quality of review does not
necessarily increase just by increasing the number of reviewers; the
number should be driven by the complexity and the need to obtain breadth
of expertise for a given evaluation. Dr. Bradley added that with many of
the recent initiatives, such as the bioengineering partnership grants,
there is need to utilize additional reviewers because of the
multidisciplinary aspects.

Dr. Berns asked whether there was some way to encourage more reviewers
to read more applications. He felt that the only way to do that was to
reduce the number of applications each reviewer must critically analyze.

Dr. Ehrenfeld suggested collecting data to determine whether the SRAs
have already recognized this issue, and are routinely increasing the
number of reviewers for the more complex applications. If the process is
evolving naturally, there may be no action needed.

Dr. Navar commented that as the number of reviewers increases,
vulnerability may increase because the score may be a direct function of
the number of negative comments. He was also concerned that critiques
are often conflicting, and that as the number of critiques increase, the
applicant will have an even greater difficulty responding to them.

Commenting about the review of multidisciplinary applications, Dr.
Epstein felt that the ideal reviewer is someone who engages in
multidisciplinary research because they understand the integration of
different perspectives. It is better to have a few people who understand
multidisciplinary research than a large number of researchers to cover
each technical area. In a mature field, this may be easy to find, but in
a field where integration is just beginning, it may be harder to find.

Dr. Fujii felt that, while there are advantages to having more
reviewers, there is a point of diminishing returns. After three or four
reviewers for an application, additional reviewers often just agree with
what has already been said.

Dr. Wickens noted that they were co-mingling two different issues: (1)
multidisciplinary applications, where an increasing number of reviewers
may be particularly useful; and (2) traditional applications, where
additional reviewers might lead to a more vigorous and informed
discussion. He felt that the more informed the discussion, the better
the quality of review. Even reviewers who just looked the applications
over could contribute significantly to this informed discussion; perhaps
if the applications were shorter, more reviewers would read more
applications. Dr. Berns, however, questioned whether reviewers would use
the additional time from having fewer or shorter applications to read
more applications. He commented that he had observed considerable
discussion, even by members not assigned to applications, when he
visited Dr. Bradley\'s Diagnostic Radiology Study Section. Dr. Bradley
commented that she utilized the electronic review process whereby
reviewers could see the critiques prior to review, and felt that this
led to better discussions at the meetings.

Dr. Matthews felt that the amount of discussion might reflect the
initial assessment of the application; thus, those in mid-range might
generate the most discussion, whereas those at the bottom would not
generate much discussion. Also, the review of multidisciplinary
applications does not simply require more individuals with expertise in
specific areas, but individuals who can evaluate broad-based goals.

Dr. Berns felt that there were two major issues: 1) how to get more
individuals to read more of the applications, and 2) how to identify the
more interdisciplinary applications that not only need more reviewers,
but special reviewers with interdisciplinary understanding. Dr. Yamamoto
suggested gathering data on SRA mechanisms to achieve broad-based review
as well as mechanisms for workload reduction.

**XI. Qualifications for Membership on Study Sections............. Dr.
James Kushner\
          Dr. Michael Colvin**

Dr. Yamamoto noted that this topic stemmed from the practice of having
public representatives, or non-professional scientists, serve as members
on Institute review committees. It seems important for CSR to develop a
position on this practice in the event that it becomes necessary to face
this issue in the future.

Dr. Kushner referred to a draft document entitled \"The Role of Lay
Persons/Consumers in Peer Review\". Dr. Kushner preferred the words
"public representative" over "lay member\", with its religious
connotations. He also criticized the use of the word "consumer," which
is more appropriate for a business report. The crux of this issue,
however, is the definition of the word "peer" and what is meant by
\"peer review.\" The general meaning of peer is "of the same or equal
standing.\" In terms of the peer review process, peers are, first of
all, defined as scientists who are active and experienced researchers
and who have been successful in receiving support from some foundation.
On the other hand, public representatives are, in general, advocates for
a particular group. The definition of an advocate is one that argues
for, defends, maintains, or recommends a cause of a proposal and this
immediately raises a flag for a potential conflict of interest. The
Boundaries Panel Report specifically states that advocacy or gatekeeping
for a field, discipline or style of research is not the function of a
peer reviewer.

Public representatives can bring important strengths to the overall
process of review and funding. First, they represent the constituency of
NIH. Second, they bring a different perspective, and knowledge on
practical issues, especially on ethical issues. The key is that their
strengths are best applied after the initial review and not as part of
the initial review. It is at the institute council level where the
public representative can have the greatest impact on decisions about
the funding of applications deemed scientifically meritorious by the
initial review group. The most important problem that could arise from
the use of public representatives on initial review groups is that
priority scores might not uniquely reflect the scientific merit of the
application.

In summary, a public statement from this Committee about the use of
public representatives on study sections should simply state that public
representatives are not peers by our definition, and that the advocacy
component of a public representative raises too strong a potential for
conflict.

**Discussion**

Dr. Colvin thought it was increasingly important for the American public
to strongly support NIH. The best guarantee of increasing support for
health research is for the people to believe that it is important. Our
people are increasingly sophisticated and informed, and having them as
advocates for our cause is the best situation. If these strong groups
are interested in study sections, perhaps they could participate as
observers.

Dr. David Maslow, SRA in the Review Branch of the National Cancer
Institute (NCI), said that his Institute had been using public advocates
for patient-related and population-related research for over a year.
They found that the public representatives tended to defer to the
scientific reviewers on scientific matters. However, they provided a
degree of honesty and practicality. They also tended to be active in
their constituent groups, and took back an awareness of the diligence of
the review committees.

Dr. Yamamoto pointed out that study section meetings are closed to the
public and that bringing public representatives to the meetings as
observers is not permissible. In response to several comments that it
could be desirable to have observers, Dr. Postow pointed out that
meetings were legally closed to the public under the Federal Advisory
Committee Act. Dr. Ehrenfeld added that confidentiality issues were
involved.\
Dr. Bradley suggested that a videotape of a mock study section might be
valuable in informing the public as to the workings of a study section.
This could be provided on the CSR Web site.

**XII. Concluding
Remarks.............................................Dr. Keith Yamamoto**

Dr. Yamamoto noted possible action items for the next meeting, which is
scheduled for May 8 & 9, 2000:

-   Develop a process for providing feedback from the Advisory Committee
    to the Panel on Scientific Boundaries for Review in its Phase 2
    activities;

-   Review findings of the pilot survey of reviewers, prepare a pilot
    for an applicant survey, and propose a design for a pilot test of
    the periodic review system;

-   Finalize the Guidelines for Reviewers and develop an action plan for
    a videotape;

-   Consider the issue of vice-chairs for study sections;

-   Continue discussions on redefining study section service,
    incorporating SRA input and experiences;

-   Examine mechanisms for ensuring that study sections have the
    appropriate breadth of scope in reviewing broad applications;

-   Continue to consider the workload issue for staff and for reviewers;

-   Re-work the document on qualifications for study section membership;

-   Define issues and mechanics for review of applications with complex
    structure (multi-investigator, multi-discipline, multiple
    institutions);

-   Reconsider the appropriateness of streamlining.

Dr. Yamamoto also suggested that after each meeting, or between
meetings, the Chairperson of the Advisory Committee interact with a
representative of the IRG Chiefs to obtain input about issues the
Advisory Committee is considering. He will interact with Dr. Ehrenfeld
on ways to accomplish this.    \
\
Drs. Ehrenfeld and Yamamoto thanked the participants for providing
interesting and informative discussions. There being no further
business, the meeting was adjourned at 11:47 a.m. on Tuesday, January
11, 2000.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes are accurate and complete. The minutes will be considered at its
May 2000 meeting of the Advisory Committee, and any corrections or
comments will be made at that meeting.

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Samuel H. Joseloff, Ph.D.\
Executive Secretary\
Center for Scientific Review Advisory Committee

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Ellie Ehrenfeld, Ph.D.\
Director\
Center for Scientific Review

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Keith Yamamoto, Ph. D.\
Chairperson\
Center for Scientific Review Advisory Committee