**Minutes of the Center for Scientific Review\
Advisory Committee Meeting\
February 17 & 18, 1998**

The Center for Scientific Review (CSR) Advisory Committee convened its
18th meeting at 9:00 a.m., on Tuesday, February 17, 1998, in Conference
Room 6, Building 31C. The entire meeting was in open session. Dr. Keith
Yamamoto presided as Chairperson.

**Members Present:**

 Olga Jonasson, M.D., Director, Education and Surgical Services,
American College of Surgeons, Chicago

 Keith Yamamoto, Ph.D., Professor and Chairperson, Department of
Cellular and Molecular Pharmacology, University of California, San
Francisco

**Temporary Members Present:**

 Michael Colvin, M.D., Director, Duke University Comprehensive Cancer
Center

 Enrico Gratton, Ph.D., Professor and Principal Investigator,
Laboratory of Fluorescence Dynamics, University of Illinois, Urbana

 James Kushner, M.D., Wintrobe Professor and Chief, Hematology/Oncology
Division, University of Utah School of Medicine

 James Liu, M.D., Professor, Department of Obstetrics and Gynecology,
University of Cincinnati College of Medicine

 Karen Matthews, Ph.D., Professor of Psychiatry, Epidemiology, and
Psychology, Department of Psychiatry, University of Pittsburgh

 Suzanne Pfeffer, Ph.D., Associate Professor, Department of
Biochemistry, Stanford University School of Medicine

 Raphael Pollock, M.D., Ph.D., Head, Division of Surgery, Professor and
Chairperson, Department of Surgical Oncology, University of Texas M.D.
Anderson Cancer Center

 Marvin Wickens, Ph.D., Professor, Department of Biochemistry,
University of Wisconsin, Madison

 Tachi Yamada, M.D., President, Health Care Services, SmithKline
Beecham, Philadelphia

**I. Call to Order and Opening Remarks**
.............................................. **Dr. Keith Yamamoto**\
\
Dr. Yamamoto, Chairperson, called the Advisory Committee meeting to
order, thanked the participants for joining the meeting, and invited the
individuals around the table to introduce themselves.

As a service to the new temporary members, Dr. Yamamoto briefly
described the composition and functions of the Advisory Committee.
Although he and Dr. Jonasson were the only regular members, several
other temporary members had been nominated for full membership. In part,
the low number of permanent members was intentional, as it enabled
discussions to involve a broader range of topics. In the future, the
Advisory Committee will probably continue to have a core of permanent
members for continuity as well as temporary members who will help focus
commentary and enliven debates.

The primary function of the Advisory Committee is to advise the Director
of CSR on all matters pertaining to CSR operations, such as the
structure and mechanics of review and the impact of changes in science
on review. The Advisory Committee does not set policy and does not make
rules. In addition to participating in discussions during meetings, some
members participate in committees and help produce various reports. The
Advisory Committee can also itself recommend the formation of committees
to examine specific topics. The Director often acts on these
recommendations, as seen in the recent establishment of the Panel on
Scientific Boundaries for Review.

Dr. Yamamoto then asked for consideration of the minutes to the previous
meeting. After noting a few minor typographical and other errors, which
will be corrected in the official files, the minutes were unanimously
approved.

**II. Director\'s
Remarks............\..................................Dr. Ellie
Ehrenfeld**

Dr. Ehrenfeld provided an overview of the various CSR activities and
initiatives, beginning with the **Advisory Committee**. In the Advisory
Committee, Dr. Ehrenfeld was striving to develop a balance of
viewpoints, including representation from the basic sciences, clinical
sciences, behavioral sciences, academia, industry, and other
communities. She expressed interest in the Advisory Committee\'s
opinions on various review issues affecting the overall biomedical
enterprise in the Nation as well as on other specific matters. The CSR
Advisory committee is in a transition period, as they try to identify
new Committee members with different viewpoints, while building a stable
group that can work well together.

Dr. Ehrenfeld then described progress in six priority areas, almost all
of which were discussed in greater depth later in the meeting. These
priority areas resulted from last year\'s outreach efforts in
identifying concerns of the scientific community.

The first priority area is to have the **receipt, referral,** and
**review** process work **faster** and **more efficiently**. CSR plans
to take advantage of automated electronic information technology and
reduce the paperwork burden. The most effective way to accomplish this
objective is first to streamline, than to automate the process. Many
pilot projects and experiments are in process at the NIH, coordinated
mainly through the Office of Extramural Research, under the leadership
of Dr. Wendy Baldwin, NIH Deputy Director for Extramural Research. CSR
has considerable interest, and participates actively, in this effort.

The second priority area is the **organization of the CSR initial review
groups (IRGs) and study sections**. While IRGs and study sections have
changed in response to changes in science, no one has thoroughly
examined the principles that guide study section organization. At
present, some study sections are organized vertically, according to a
biological or clinical theme; others are organized horizontally,
centered on a common biological process. Which way is better ? Perhaps
both are necessary. Some community concerns have already been discussed
by the Advisory Committee, such as whether some study sections have too
many outstanding applications competing against each other, whether some
study sections are reviewing scientific areas that are not very active
anymore, or whether some newly-emerging scientific fields or
methodologies do not fit appropriately into a current study section.

To address some of the broader issues of review group organization, Dr.
Ehrenfeld recently appointed the Panel on the Scientific Boundaries of
Review. The Panel is composed of distinguished scientists with a broad
vision, who can help establish guidelines about scientific boundaries
for review that both reflect today\'s science, and also allow for
continued flexibility to adapt to future changes.

CSR has already begun reorganizing some study sections. Working with the
Institutes, CSR has completed the integration of neuroscience review
activities from the National Institute of Mental Health (NIMH) and the
National Institute on Drug Abuse (NIDA). That effort has resulted in 21
new study sections, which will begin operation in June. Eight AIDS study
sections, designed via a similar process, also will meet in June to
accommodate integration of the AIDS review activities from NIMH and
NIDA. Integration efforts involving the behavioral sciences are
underway, and newly configured study sections are expected to be
operational in June 1999.

CSR also is establishing **IRG Oversight Groups**, designed to help
evaluate the functions and operation of the current IRGs and study
sections. The first of those IRG oversight groups, for the Cell and
Developmental Function IRG, was established before Dr. Ehrenfeld\'s
tenure. The establishment of two more IRG oversight groups is under way,
for the Health Promotion and Disease Prevention IRG and the
Musculoskeletal and Dental Disease IRG. Dr. Ehrenfeld noted that this
practice is similar to universities periodically bringing in outside
experts to review the function and direction of individual departments.

The third priority area is **recruitment of reviewers**. In addition to
internal CSR efforts to increase flexibility in service terms and
conditions, CSR anticipates that the IRG Oversight Groups can provide
considerable assistance in identifying and recruiting study section
members.

A fourth priority area includes developing more **structured training
activities for new SRAs and study section chairpersons** as well as
identifying **resources** for the **continued professional development
of the SRAs**. Two recent examples of trans-NIH changes requiring
additional training for staff and reviewers are the new research grant
review criteria and the termination of the FIRST (R29) grant program.
CSR staff has met the challenge of these and other changes
professionally, often developing guidelines on very short deadlines. Dr.
Ehrenfeld recently set up an SRA training committee.

The fifth priority is to **improve communications and interactions with
the NIH Institutes.** Dr. Anne Sassaman, from the National Institute of
Environmental Health Sciences, has been detailed to CSR to assist in
this effort.

The final priority area involves addressing concerns of **research
groups who feel ill served by the current system**. Included here are
the clinical research community, behavioral scientists, and researchers
working with bioengineering and instrumentation and technology
development. Dr. Ehrenfeld began working with the clinical research
group, assisted by Dr. Michael Simmons, a clinician researcher whom she
hired as a consultant to serve as a liaison with the clinical research
community. Working with CSR staff, Dr. Simmons proposed solutions to
problems identified in previous studies. Among his recommendations is
the establishment of two special emphasis panels (SEPs) in which
clinical cardiovascular and oncology applications each would be
clustered. CSR is in the process of identifying an individual to serve
as liaison with the behavioral sciences community, and is developing
plans to address the needs of the bioengineering constituency.

**Discussion**\
In response to a question about CSR handling reviews normally done in
the Institutes, Dr. Ehrenfeld explained that CSR routinely responds to
Institute requests to take on such reviews. With some new programs, for
example, the Institutes do not have the capability for handling the
reviews themselves, especially when these new programs are
multi-Institute efforts. There will always be a need for
mission-oriented reviews that the Institutes do best, but despite
strained resources, CSR wishes to respond to calls for help that come
from the Institutes.

**III. Reorganization of the Review of Behavioral Science
................Dr. Virginia Cain**

Dr. Cain, Chairperson of the Behavioral and Social Science
Implementation Working Group, spoke about the Working Group\'s
activities. Building on a process developed by the Neuroscience Working
Group, they examined the array of behavioral and social science research
reviewed by the NIH, and then developed clusters of research categories
that encompass that breadth. From those clusters, they will develop
study sections.

At present, the Group has identified four major clusters. Two are
disease oriented - (1) Behavioral, Communication, Mental Disorders, and
Substance Abuse, and (2) Behavioral Factors Relating to Physical
Diseases; while the other two are more basic - (3) Social Processes, and
(4) Biobehavior, Behavior, and Cognition. At a February 24th meeting of
the Working Group, these clusters will be refined. During the meeting,
to which a number of extramural, funded scientists with review
experience have been invited, the participants will also examine
cross-cutting issues relevant to all four clusters, such as life span,
ethics, gender, and ethnicity.

Following this activity, at a May 5th meeting and over the summer, the
Group plans to complete the design of the study sections, so that the
implementation process can begin in the fall. Hopefully, the new
structure will be in place for the June 1999 round of study section
meetings.

Throughout the process, the Group has been concerned with outreach to
the scientific community. They established an outreach committee, whose
activities have included: soliciting names of potential reviewers from
scientific societies; meeting with various NIH committees, such as study
sections and Councils; giving presentations at annual meetings of
scientific societies; and using electronic communications, such as
putting information on the CSR Web Site, and having a dedicated E-mail
address for receiving comments from the scientific community.

**Discussion**\
Dr. Karen Matthews, the assigned discussant, approved of the general
strategy. However, she noted the importance of reviewing behavioral
science that is well integrated into clinical sciences in the context of
its clinical setting, and the need to better integrate the behavioral
sciences with other disciplines.

In the general discussion, the point was made that behavioral scientists
tend to be highly self-critical, excessively focusing on problems. Then
when behavioral applications are reviewed in study sections that review
mostly basic biomedical applications, they fare poorly because reviewers
are unfamiliar with this practice. Another concern is that the recent
explosion of behavioral research has not been matched by an increase in
the number of behavioral study sections.

In determining the number and type of study sections needed to manage
the rapidly expanding field of behavioral research, Dr. Cain explained
that CSR did not plan to add study sections arbitrarily, but rather,
whenever possible, to modify study sections to reflect contemporary
science. The reorganization will be driven by science considerations not
by the existence of study sections.

Dr. Pollock felt that the Working Group needed to articulate clearly
when and how changes occur, so that they will not be only reacting to
pressures from internal and external pressure groups, but rather acting
in response to the current and anticipated directions of science.

In response to a question about the numbers of behavioral applications,
Dr. Cain estimated that approximately 890 applications would be reviewed
in the upcoming receipt, review, and award cycle. Dr. Cain also noted
that the system is somewhat in flux. For example, some health services
research could be reviewed by either NIH or the Agency for Health Care
Policy and Research.

While the various review problems were important and needed discussion,
Dr. Ehrenfeld emphasized that the impetus for the reorganization
activity was not the perceived problems with the current reviews, but
rather the integration of the ADAMHA Institutes into the NIH peer review
process. The Panel on Scientific Boundaries of Review would be a useful
group to examine whether behavioral science research should be reviewed
separately or integrated into clinical research study sections.

**IV. Reorganization of the Review of AIDS
Research.......................Dr. Polly Sager**

Dr. Sager noted that the reorganization of the AIDS study sections was
also based on the integration of the former ADAMHA applications into the
CSR system. Basically, their Working Group was trying to integrate
applications from seven CSR study sections, two study sections at NIMH,
and two study sections at NIDA. The guiding principles were that: (1)
study section reorganization should be driven by science considerations;
(2) study section membership should balance breadth of perspective with
depth of scientific expertise, e.g. covering molecular to clinical
research; (3) flexibility and overlap should be built in to handle
potential conflicts of interest; and (4) reorganization should be
responsive to scientific trends.

The Working Group started with 430 abstracts from AIDS applications,
which were divided into four scientific clusters: (1) basic biomedical
research, including molecular research, virology, and immunology; (2)
epidemiological and clinical research; (3) behavioral and social
sciences research; and (4) vaccine development. Vaccine development was
handled separately (see below).

The validity of these clusters was then tested by a focus group of nine
former study section chairpersons. The chairpersons added some
scientific areas, but basically agreed with and validated the cluster
organization. The results were then presented to the members of the
current 11 AIDS study sections, after which a new set of 450 abstracts
were evaluated by a new committee. The Working Group then drafted eight
new study sections.

The Institutes are now reviewing the guidelines, which should appear
soon on the CSR Web page for comments from the community. CSR plans to
encourage self-referral to study sections by investigators beginning
with the May, 1998, AIDS application receipt dates. Still to be done is
the assignment of current members to new study sections.

**Note:** Dr. Ehrenfeld remarked that the neuroscience study sections
had used a different method, which also seemed to work well. Reviewers
were asked to self-refer, i.e., to choose the study sections where they
felt their expertise would best fit.

**Vaccine Research**\
Dr. Sager next addressed the topic of vaccine research. The vaccine
research community and the Levine Panel, which was established by the
NIH Office of AIDS Research to review NIH AIDS research, strongly
recommended that CSR form a new vaccine development study section. It
was recommended that review of vaccine development of both AIDS and
other infectious diseases be merged together. Historically, these
applications have not done well, perhaps because of reviewer bias
against applications that involve applied research and are not
hypothesis driven. It was recommended that a two-year pilot study be
developed with the goal of providing a more focused review of vaccine
applications.

The same process was used for vaccine development as for the AIDS
integration activities. The guidelines should be on the CSR Web page
soon, with self-referral to begin with the June-July receipt dates for
non-AIDS and the September receipt date for AIDS applications.

**Discussion**\
Dr. Michael Colvin, the assigned discussant, praised the process, but
expressed concern that some of the study sections, such as the one for
AIDS epidemiological and clinical studies, covered too broad an area. He
also did not see much difference between the two behavioral and social
science study sections.

Dr. Sager noted that a current study section already successfully
manages epidemiology and clinical trials, and she expected only about 30
applications each review round, because most therapeutic trials are
reviewed at the Institute level. Dr. Sager admitted that the division
into two AIDS behavioral study sections was artificial, done primarily
in response to the workload. Dr. Matthews suggested revising the mission
statements of these two behavioral study sections to distinguish between
them as much as possible.

The Advisory Committee then discussed the value of study sections
evaluating their own structure, voting behavior, and response to changes
in science. Study sections did this self-evaluation more frequently in
the past when the workloads were not so consistently heavy. Nonetheless,
such introspective discussions still occur, often informally, and have
proven to be useful.

There was also discussion of the potential benefits of overlap among
reviewers and study sections. Dr. Yamamoto found the overlap in the AIDS
study sections extremely well thought-out and designed. The study
sections were balanced, not completely distinct, yet not completely
overlapping. Dr. Sager noted the value of overlap in many areas; to
handle overlap efficiently, they proposed having core members assigned
to the study sections, while other reviewers, such as those using animal
models, or dealing with biostatistics or ethical issues, would move from
study section to study section depending on the expertise needed for a
specific meeting.

There was concern about the small number of applications in some new
study sections. For example, the proposed AIDS vaccine study section
would have only about 20 applications. Dr. Sager responded that the
number would probably increase with the addition of Small Business
Innovation Research applications and some proposals that had been
managed by the Institutes as contracts or with other mechanisms. Private
industry might also be especially interested in vaccine research and
provide support Dr. Yamamoto emphasized that this was an experiment,
which the Committee will evaluate in the future based on the data
produced.

**V. SRA Training Committee..........................................Ms.
Jo Pelham**

Ms. Pelham explained that the genesis of the SRA Training Committee was
a recommendation from the NIH Peer Review Oversight Group (PROG) for
additional training for SRAs and study section chairpersons. The SRA
Council had also recommended this training. In response Dr. Ehrenfeld
established the SRA Training Committee.

The committee has met once, and is still in the data-collecting phase.
The Committee felt that training for SRAs was especially critical, since
the Center will probably hire 10 to 12 new SRAs in the near future. They
also felt it was important to train SRAs and chairpersons at the same
time. In its efforts, the Training Committee was careful not to
duplicate existing training opportunities for SRAs, such as the Grants
Associate Seminar Series or the STEP (Staff Training in Extramural
Programs) training modules.

Currently no formal training program exists for chairpersons. The
Training Committee will involve chairpersons in determining what types
of training would be useful. The Committee also welcomed the advice of
the CSR Advisory Committee in this respect.

**Discussion**\
Dr. James Liu, the assigned discussant, suggested setting up specific
goals with deadlines to speed up the process. He was anxious to evaluate
the results of the initial survey with the questionnaire.

The discussion then dealt with the selection process for chairpersons,
which some participants felt was too informal and \"localized\" to the
study section members and certain CSR staff. Selection should be linked
to specific criteria. Dr. Yamada suggested broadening the possible
choices by including individuals outside the study section. Dr. Yamamoto
thought SRAs were unlikely to choose a chairperson outside the study
section, but they could expand the base to consider members of other
study sections within the IRG.

As to the type of training that would be useful, Dr. Matthews suggested
some exposure to group process. Group process would include: ensuring
that all differences of opinion are aired if not resolved; drawing the
quieter individuals into the discussions; and explaining explicitly any
inconsistencies among reviewers. Dr. Pfeffer added that training should
assist chairpersons not just in bringing out general discussion, but
also in keeping discussions focused on important scientific issues.

**VI. IRG Oversight Groups ..........................................Dr.
Elliot Postow**

The establishment of IRGs included a plan to create ad hoc groups to
provide advice and oversight regarding their function. Drawing on the
experience of the first IRG Oversight Group, for the Cell Development
and Function IRG, guidelines to be followed by this and other such
groups are being developed.

Oversight Groups were to include 5 to 10 members, who were senior
researchers with a broad view of science and the review process and
whose expertise matched the referral guidelines of the IRG. One of their
main responsibilities would be to evaluate the scientific review
responsibilities of the study sections that comprise the IRG. The groups
were to help ensure that cutting-edge science is fairly uniformly
distributed among the study sections in an IRG, that the distribution of
applications is consistent among the IRGs, and that the Referral
Guidelines have a degree of overlap, so that any application could be
reviewed in at least two study sections.

Other possible responsibilities of oversight groups included: (1)
helping SRAs identify possible members of study sections; (2) validating
the appropriateness of prospective members nominated by the SRA; (3)
advising CSR about the functioning of the study sections; and (4) having
members serve as ombudsmen, bringing concerns from the community to the
study sections and IRGs and explaining to the community how peer review
works.

Although members may currently serve on study sections during their
tenure on oversight groups, this may create a conflict-of-interest
issue, which would have to be worked out.

Dr. Postow stressed that the oversight groups, with no externally
mandated time schedule, would be looking for long-range solutions to
problems.

**VII. Report from the Cell Development and Function Oversight
Group..................Dr. Marvin Wickens**

Dr. Wickens explained the organization of the Cell Development and
Function (CDF). The guiding principles of the CDF Oversight Group were:
(1) to strive to ensure high-quality, fair reviews, with the best
science so recognized across the IRG; and (2) to view the IRG as a unit,
exploiting the advantages of such a classification. The Group met for
the second time in December 1997, and Dr. Wickens summarized the agenda
for that meeting. The major discussion items were the duties and
interactions of the chairpersons and SRAs. The chairpersons should work
actively with the SRAs to identify and invite members, with the actual
invitation coming from the SRA, as well as to invite two senior,
well-respected scientists with peer review experience to attend one
meeting per year to provide a broad perspective on the functioning of
the study sections.

The Oversight Group also proposed that reviewers should have a shorter
time to review applications, so that the review meeting could occur
earlier, and investigators could receive their summary statements in
time to revise the application, if necessary, before the next receipt
deadline. The consensus was that three weeks were adequate to prepare
reviews, although it was recognized that conflicts might exist with some
academic calendars.

Throughout the meeting, the SRA and chairperson ideally form a
partnership, ensuring that the meeting runs as smoothly as possible.
Chairpersons and SRAs together should train new study section members,
involving the next chairperson whenever possible.

After the meeting, the chairperson, SRA, and possibly senior study
section members should informally evaluate other study section members,
especially the ad hoc reviewers. In addition, the chairperson should
encourage study section members to submit to the SRA any evaluations of
the study section, including the chairperson.

The Oversight Group discussed how to balance the portfolios of the study
sections within an IRG, and the Cell Development and Function IRG Chief
was asked to develop a plan for more balanced portfolios for the various
study sections within the IRG. The IRG Oversight Group, SRAs, and
chairpersons would then evaluate the plan and make recommendations. The
final recommendations were that members of one study section should
visit other study sections within their cluster, and that everyone
involved in the IRGs should be kept informed of the activities of the
IRG Oversight Group. The Group, however, was not clear about the extent
of their authority and the degree of separation from the study sections.
These topics need further examination.

**Discussion**\
Dr. Suzanne Pfeffer, the first discussant, found the IRG concept
refreshing and felt that the flexibility of movement of study section
members of different study sections within an IRG could improve the
competency and fairness of the reviews. For the Oversight Group to be
most effective, Dr. Pfeffer felt that it should be independent of any
IRG or study section. But the key to success rests with the SRAs, for
they set up the groups and work in the \"trenches\" of peer review. She
wondered whether some SRAs, justifiably proud of the study section they
had assembled, would resent any changes as an invasion of their turf.
Faced with possible resistance, how could the Oversight Group best help
the SRAs?

Dr. Yamamoto, the other discussant, saw the advisory process as helping
to broaden the way CSR looks at review, from the study section to the
IRG level. The oversight groups would evaluate whether the IRG research
portfolios are balanced, fair, and inclusive of the relevant scientific
advances. This information could then be funneled into the Panel on
Scientific Boundaries for Review, to be part of their broader, more
visionary recommendations involving the entire CSR, IRG, and study
section organization. Dr. Yamamoto also noted that an important
responsibility, on which the SRA and chairperson can collaborate, is
integrating the discussions during the meetings, i.e., being able to
reach a consensus even when the primary and secondary reviewers do not
agree, and bringing discussion to closure when further discussion would
prove fruitless.

Dr. Yamamoto then requested more details about balancing IRG and study
section portfolios. Dr. Wickens explained that the process was similar
to that used for reorganization discussions. They began by identifying
12 to 24 broad scientific areas encompassing the review areas of the
IRG. They then determined how many applications were in each broad area,
and to what extent reviewers\' expertise overlapped. The IRG chief could
then divide the applications into the various study sections. Potential
difficulties in balancing study section portfolios include: (1) some
study sections being organized horizontally, others organized
vertically; and (2) some study sections having considerable clinical
research to review, while others have mainly basic molecular research.
The applications are not always easily comparable.

In response to Dr. Yamamoto\'s question about potential tension points,
Dr. Wickens felt that the main point of tension arose from not knowing
their exact responsibilities and limitations.

There was some concern that the process might develop into an
adversarial situation, and the Advisory Committee felt that
communication and dialogue should be emphasized. There was considerable
discussion about whether a report should be developed and to whom the
report should be addressed, but Dr. Ehrenfeld refocused the discussion
from specific procedures to a broader consideration of membership and
purpose.

Dr. Yamamoto then invited further discussion on the practice of allowing
current study section members to be on oversight groups. After some
discussion, the Advisory Committee recommended that future oversight
groups should **not** include current study section members. Eliminating
current members would: (1) help the committee give a more independent
evaluation of the system; (2) heighten the credibility of the oversight
function with the public; and (3) avoid the inherent myopia of
individuals trying to get an overall, objective perspective of a system
in which they are currently invested. Dr. Pollock suggested that to
ensure that the group is familiar with peer review changes, at least a
fixed percentage of the group should have recent (within three to four
years) study section experience.\
\
Other proposed membership suggestions included: previous chairpersons; a
balance between researchers with recent experience, who know current
changes and concerns, and experienced researchers, who have the broader
perspective that comes with maturity; individuals suggested by
professional societies; editors of professional peer-reviewed journals;
well-known academic departmental administrators, though these
administrators may not be aware of the reality of, and recent changes
in, peer review; and program directors of clinical research centers.

Dr. Yamamoto asked whom the process was supposed to serve and to whom
the oversight groups were to report. In the broadest terms, Dr. Pollock
said, the process was supposed to serve the external research community.
However, if done properly, it is actually helping the CSR staff,
specifically the SRAs, provide a high level of service to the outside
community. Dr. Pollock felt that pluralism and overlap benefit oversight
groups by giving them strength and resiliency.

Dr. Colvin wondered what objective measures would be used to evaluate
whether the changes will be beneficial? Dr. Ehrenfeld admitted that
because of the many variables, their experiments often could not be
easily controlled scientifically. Some issues could not be measured
quantitatively. One example might be \"customer satisfaction\" or
researchers\' happiness with the system. CSR is using advice from
outside consultants, and also setting up some infrastructure within CSR,
through the establishment of an evaluation office, to assist CSR staff
in this difficult problem area. Dr. Matthews suggested that Dr. Samuel
Rawlings, CSR Evaluation Officer, return to the Advisory Committee in
the future with a more formal presentation.

The discussion ended with Dr. Yamamoto proposing, and the Advisory
Committee unanimously accepting, that the SRA Training Committee address
first the training of study section chairpersons and members. Drs.
Postow and Wickens were asked to develop written orientation guidelines,
which would be discussed at the next meeting. With clarification of the
responsibilities of SRAS, chairpersons, and members of IRG Oversight
Groups, these draft guidelines and could serve as the framework for a
document that would be valuable for all IRGs.

**VIII. Nomination and Recruitment of Reviewers.................... Dr.
Jean Paddock**

Dr. Paddock noted the increasing difficulty in recruiting reviewers for
study sections. Potential members felt that because of the extreme
competition, they needed more time to prepare their own applications.
Some researchers were so discouraged, they were leaving academia for the
financial security of private industry or elsewhere. Whether these
complaints are real or perceived, CSR, hopefully with the assistance of
the Advisory Committee, will work to improve reviewer recruitment.

Dr. Paddock then briefly described the current process for selecting
reviewers. The SRAs begin recruitment about a year ahead of time, by
networking with NIH Institute program staff, members of scientific
societies, and other extramural researchers. The SRAs would certainly
welcome reviewer suggestions from the Advisory Committee.

In the nomination process, the CSR Committee Management Office first
examines the proposed nomination for compliance with Federal, NIH, and
CSR policies and procedures. The package then goes to the division
director and the CSR Director for approval, and then to the NIH Director
for final approval.

Changes in committee management policies and procedures occur
frequently. For example, in a change based on the IRG structure, SRAs
will no longer have to submit separate justification statements on
gender and minority representation and geographical balance for each
study section. Now one statement for each IRG also covers all the study
sections in that IRG. An additional change is a cover memo attached to
the nomination package to Dr. Ehrenfeld. This memo explains in detail
the rationale for nominating a particular reviewer; any changes in the
science reviewed by the study section; and any unusual situations that
require special attention by Dr. Ehrenfeld, such as the inclusion of an
assistant professor.

Other areas for discussion with the Advisory Committee include using
more flexible appointments for members. Thus appointments could be for
two instead of four years. SRAs already have this option, but perhaps
the process could be expedited.

**Discussion**\
Dr. Pollock, the assigned discussant, was impressed by the database of
former reviewers, which presumably is organized by geography as well as
by areas of interest. This database could be a valuable resource for
polling people with specific interests. The issue of more flexible
appointments for members needs to be evaluated carefully. The four-year
term has value because of the dynamics and interactions among reviewers
as well as networking. In the balance of core and ad hoc members, the
atmosphere of a study section would probably change adversely if there
were too many ad hoc people.

After some discussion about the variability of SRA efforts to obtain
input for potential study section members from outside sources, it was
suggested that these efforts be more formalized, especially those with
professional societies. The nomination package should include the source
of recommendations of reviewers, which along with a roster, should be
provided to Institute staff.

With respect to the composition of study sections, Dr. Paddock responded
to a question about written guidelines for members of review groups by
summarizing information from the Code of Federal Regulations (CFR 442).
These general guidelines stressed education, research experience, and
success in garnering awards. Dr. Matthews added that potential members
should have been principal investigators on grants, and should be high
on the citation index list. The common practice is to invite potential
members to attend a meeting as ad hoc reviewers before nominating them
as members.

When considering study section composition, SRAs faced certain
restrictions, such as the need to achieve wide geographical distribution
(though this requirement has eased recently), the need for adequate
representation of women and minority scientists, and the need to avoid
overrepresentation from one institution and excessive overall service.
Researchers at the assistant professor level are generally not
appropriate, because it is thought that they should be free to develop
as researchers before committing to study section membership.

Dr. Yamada asked about clinical expertise, since apparently quite a few
study sections had no clinical research members. Dr. Paddock noted that
clinical representation varied depending on the nature of the
applications reviewed in the study section and that some study sections
reviewed only basic research.

Within the guidelines and restrictions, SRAs have been encouraged to
strive for flexibility. Dr. Pfeffer, after agreeing with an earlier
comment by Dr. Pollock that four years of service were important to
build a rapport and trust among reviewers as well as a deepened
understanding of the peer review process, suggested that some reviewers
should be permitted to come twice instead of the normal three times a
year for meetings.

In response to a question about how CSR would handle a sudden influx of
applications in a new field, Dr. Paddock explained that the CSR staff
use temporary reviewers judiciously. Special emphasis panels can also be
established to manage this situation. CSR tries to be proactive and plan
ahead for possible scientific advances in areas reviewed by existing
study sections.\
\
**IX. CSR Interactions with the Institutes .......................Dr.
Anne Sassaman**

In the past, Dr. Sassaman noted, interactions between CSR and Institute
program staff have been limited. Today, the emphasis is on cooperation
and communication.

One major part of her assignment has to do with the referral of
applications to the Institutes. This topic grew out of Dr. Varmus\'s
Leadership Forum last September, where it was discussed in response to
concerns by some members of the biomedical research community that in
some broad areas of science, researchers felt disadvantaged by their
perception that they were constrained to only one Institute for
potential funding and grants management.

Applications are referred to Institutes based on written guidelines
developed by the Institutes.. The guidelines are normally updated every
two years; the latest version is 1994. While some scientific areas in
the guidelines are clearly attributed to one Institute, others have
considerable overlap among the Institutes. In discussions involving
overlaps, CSR tries to encourage collaboration rather than
confrontation.

Dr. Sassaman noted that both process and personalities are components of
this issue. She has interviewed Institute extramural program directors,
spoken with about one-third of the Institute directors, and put together
a working group of Institute and CSR staff to discuss the issues. By
early spring, the working group plans to submit a report, with their
recommendations, to Dr. Ehrenfeld and the Institute directors.

Charges to the working group include: developing better processes for
updating the referral guidelines; finding appropriate ways to have
Institute staff more involved with CSR referral officers in referring
applications; considering a process for self referral to the IRGs and
also to the Institutes; and evaluating the practice of dual or multiple
assignments of applications to Institutes. For the past five years,
about one-third of the applications have been assigned to more than one
Institute. Under NIH policy, one Institute can not indefinitely hold an
application for potential funding if another Institute is willing to
fund it.

**Discussion**\
Dr. Gratton, the assigned discussant, noted the outside community often
perceives that some Institutes have a better funding record than others
do. Also some topics, such as biotechnology, can be assigned to
different Institutes. The applicant community needs more specific
information about, for example, how a referral officer determines, with
multiple assignments, which is to be the primary Institute.

Dr. Pfeffer noted that dual assignment is popular in the scientific
community in this time of tight funding, and it has rescued some
investigators. Also, the number of multi-Institute program announcements
have been increasing, which reflects the crosscutting nature of
contemporary science. She wondered whether the Institutes could be
encouraged to have co-funding. Dr. Sassaman noted that while such a
practice may complicate the bookkeeping for grants management staff,
there are benefits to the practice, which is being used more frequently
than in previous years.

Dr. Yamamoto asked about interactions between CSR and program staff. Dr.
Sassaman noted that the practice varied among SRAs. One SRA related an
incident where the various SRAs within an IRG invited Institute staff to
a meeting to discuss how they could work together more productively. Dr.
Postow also explained that several IRGs have set up systems for informal
meetings on a regular basis with Institute staff. Dr. Yamamoto suggested
that such IRGs could serve as a model for other study sections and IRGs
interested in improving communications. Dr Yamada added that the real
challenge was to convert these best practices within CSR to standard
operating practice.

Dr. Sassaman asked the Advisory Committee for its thoughts on applicant
self-referral to Institutes for potential funding and management of the
grant. (Currently applicants can request Institute assignments, which
are considered by referral officers, but the practice is not
encouraged.) Dr. Sassaman stated that the existing Referral Guidelines
for Institute Assignments are for internal use and are not in a user
friendly format for applicants to make informed decisions about the most
appropriate Institute(s). Dr. Ehrenfeld noted that the current sources
for information on the Institutes\' research interests and priorities
are the home pages of the various Institutes on the Web.

**X. Speeding Up the Referral and Review Process...................Ms.
Linda Engel**

Ms. Engel noted that at the last meeting, the Advisory Committee
recommended that CSR speed up the referral and review process so that
unsuccessful investigators could submit their revised applications for
the next receipt date. In response, Dr. Ehrenfeld established a small ad
hoc-working group of NIH staff and individuals from the outside
community. The charge to this group is to outline the changes in the
receipt, referral and review processes and the additional resources
needed not only to allow applicants to submit applications in time for
the next receipt date but also to shorten the receipt to award cycle
from 10 months to 5 months. This quick turn-around project is limited to
outlining, broad concepts, rather than specifying details and
implementation plans.

This CSR activity is complementary to the projects aimed at expediting
the receipt to award process that are among the reinvention activities
underway at NIH under the leadership of Dr. Wendy Baldwin, the Deputy
Director for Extramural Research. Ms. Engel then reported on the
progress made in implementing the recommendations from these reinvention
activities, some of which the Advisory Committee had heard about at its
last two meetings. One ongoing pilot study, involving the Tropical
Medicine and Parasitology Study Section, is testing on-time
Institutional Review Board approval; self-referral of applications;
electronic uploading of information on the Internet for reviewers prior
to the meeting; access by Councils to Internet materials; and expedited
review of selected applications.

For receipt and referral, the greatest gains will come with the fully
developed capacity for electronic research grants administration.
Initially NIH will receive core elements of the application, and then
the entire application. However, it will be a couple of years before the
system is fully operational.

Meanwhile, NIH is seeking ways to speed up the current paper-based
process and to develop the capacity to operate in a dual mode. NIH will
have to work with electronics and paper for some time. A receipt and
referral laboratory has been established, under the auspices of the
Office of Extramural Research, to test the efficacy of digitizing
applications and managing the workflow electronically.

With respect to referral, CSR is following the recommendation of the
reinvention team to make referrals to the IRG rather than to the study
section level to capitalize on the potential flexibility inherent in
this organizational unit. CSR is also encouraging applicants to use the
current option of self-referral to IRGs. In addition, CSR will automate
the referral of competing renewal and supplemental applications.

The ad hoc Working Group plans to bring its findings and recommendations
to the next Advisory Committee meeting. Additional resources will likely
be required to accomplish the goals.

**Discussion**\
Dr. Yamada, the assigned discussant, noted that industry always faces
the fact that time is money. For example, in the pharmaceutical
industry, research and development takes 10 years between the idea and
the product. His company has undertaken a major effort to cut this time
to 7 years. Dr. Yamada also emphasized that although speed is important,
CSR must not sacrifice but rather improve quality. Outside consultants
can often be valuable as dispassionate observers. Dr Yamada felt that
CSR NIH had made an impressive start, and it would be exciting to watch
how the process rolls along.

**XI. Review of Clinical Research.......................Dr. Raphael
Pollock**

Dr. Pollock summarized a draft report prepared by Dr. Michael Simmons, a
consultant to CSR, entitled ***The Review of Clinical Research in the
Center for Scientific Review***. Data of previous studies indicate that
clinical applications fare as well as laboratory-oriented research in
study sections in which they represent at least 30 percent of the
applications being reviewed, i.e., high-density study sections. Dr.
Pollock questioned whether there are enough of these \"good\" study
sections to accommodate the workload and whether the problem is
essentially one of moving clinical applications to high-density study
sections that have between 30 and 100 percent clinical applications

To address the low-density study section problem, Dr. Simmons proposes
creating special emphasis panels (SEPs) to review patient-oriented,
translational research and small clinical trials. Two areas are singled
out - cardiovascular science and clinical oncology, which constitute
almost half of the applications reviewed in low-density study sections.
To deal with the remaining unusual applications, or \"odd ducks,\" he
proposes four possible solutions: (1) creating an ombudsman for clinical
research in the low-density study sections; (2), reviewing the
application simultaneously in low-density and high-density study
sections, which could also show the potential impact of creativity as a
separate review criterion; (3) using outside opinions and ad hoc
reviewers; and (4) establishing a SEP for the review of patient-oriented
translational research and small clinical trials. Each of these
approaches has recognized limitations. The last recommendation, which
Dr. Pollock enthusiastically endorsed, is to establish a study section
capable of reviewing large, population-based, multi-centered clinical
trials composed of a stable panel of experts who have expertise in
review of clinical trials and supplemented when appropriate with ad hoc
reviewers.

Based on his expertise and current positions in cancer research and
clinical care, Dr. Pollock examined clinical oncology. The study section
on which he served, Experimental Therapeutics-2 (ET-2), included more
than one surgical oncologist among its members. If CSR established a
clinical oncology SEP, Dr. Pollock felt that CSR should ensure than the
applications could not be reviewed competently by existing, high-density
study sections such as ET-2.

**Discussion**\
The Advisory Committee wanted to see more data on which study sections
were high density, which were low density, how clinical research has
fared in these study sections, and the composition of those study
sections. Dr. Ehrenfeld noted that such data have been available for
some time in earlier reports on clinical research and peer review. While
one could argue over the validity of the data, e.g., whether the
high-density versus low-density cutoff should be 49 percent or 30
percent, this would probably not be too useful. The data do, however,
show that the success rates for clinical research in high-density study
sections are the same as for laboratory-oriented research in those study
sections.

Discussion turned to the establishment of SEPs limited to clinical
applications and which types of applications do best with SEP review.
Dr. Colvin felt that SEPs would be very useful for patient-oriented,
translational research and small clinical trials, but that large
epidemiology and outcome trials already have good review group homes.
Dr. Pollock felt that at least in the oncology field, current,
high-density study sections adequately cover human translational
research. Small and even large clinical trials, however, might well
benefit from review in SEPs devoted to clinical trials. Dr. Yamada felt
that integrative biology was another area that would benefit from SEP
review.

Committee members noted that clinical research is only sometimes rated
low due to a flawed review, for example, a low-density study section
focusing on methodology instead of on the potential impact. In other
cases, poor reviews reflect a badly prepared application by a clinician
inexperienced in research. Dr. Colvin recounted being a member of a
newly established clinical study section. The first group of clinical
applications were terrible and were virtually all rejected. The second
time, however, after the principal investigators resubmitted modified
applications that responded to reviewers\' critiques, the applications
were much improved, and a high percentage were funded. Dr. Kushner
thought that the main problem in clinical research grants was an
applicant\'s lack of training in the methodology necessary to do
clinical research. Most of this training should take place at their home
universities, but the SRAs and the study sections, through their
reviews, can also provide a useful education to clinicians.

If part of the problem with clinical applications not being rated highly
is their possible review by a low-density study section, how did this
happen? Based on referral guidelines, referral officers chose the study
section with the most appropriate expertise to evaluate the subject of
the application. To increase flexibility, CSR is promoting more overlap
among study sections, so that clinical applications can be more easily
clustered either in current study sections or SEPs. Still, even with the
proposed clinical clusters, Dr. Ehrenfeld remarked that about one-half
to one-third of applications currently reviewed in low-density study
sections remain as odd ducks.

Dr. Pollock recognized the problems and the need for change, but advised
that before introducing broad changes, CSR should take care to ensure
that the positive elements of the current system are retained.

**XII. Closing Remarks**

Dr. Yamamoto and Dr. Ehrenfeld thanked the members and ad hoc advisors
for participating so competently and enthusiastically in the meeting.
There being no further business, the meeting was adjourned at 1:30 p.m.
on Wednesday, February 18, 1998.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes and attachments are accurate and complete.

/s/\
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Samuel H. Joseloff, Ph.D.\
Executive Secretary\
Center for Scientific Review\
Advisory Committee

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Ellie Ehrenfeld, Ph.D.\
Director\
Center for Scientific Review

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
Keith Yamamoto, Ph.D.\
Chairperson\
Center for Scientific Review\
Advisory Committee

NOTE: These minutes will be formally considered by the Advisory
Committee at its next meeting on September 14-15, 1998. Any corrections
or notations will be incorporated in the minutes at that meeting.

*Last update: June 29, 1998*