## DRG Advisory Committee Minutes November 13 & 14, 1995

[I. Call to Order and Approval of
Minutes](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#I%23I)\
Dr. Thomas Braciale

[II. Report of the Acting
Director](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#II%23II)\
Dr. Donald Luecke

[III. View from the Office of Extramural
Research](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#III%23III)\
Dr. Wendy Baldwin

[IV. NIH and DRG
Budgets](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#IV%23IV)\
Mr. Fred Wong

[V. NIH Reinvention
Update](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#V%23V)\
Dr. Anthony Demsey

[VI. The Initial Review Group
Concept](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#VI%23VI)\
Dr. Donald Luecke

[VII. Overview of the Pilot Study of Initial Review
Groups](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#VII%23VII)\
Dr. Donna Dean\
Dr. Raymond Bahor

[VIII. Fellowship Reviews as a Model of Initial Review Group
Flexibility](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#VIII%23VIII)\
Dr. Jerry Roberts\
Dr. Everett Sinnett

[IX. Joint Workshops and Study Section
Meetings](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#IX%23IX)\
Dr. Sooja Kim\
Dr. Fayez Gishan\
Dr. Paul Strudler

[X. Anticipating and Responding to Change in
Science](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#X%23X)\
Dr. Michael Knecht\
Dr. Gerald Greenhouse

[XI. Outreach to the Scientific
Community](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XI%23XI)\
Dr. Gerald Greenhouse\
Dr. Anita Weinblatt

[XII. Developing Closer Links with NIH Institutes and
Centers](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XII%23XII)\
Dr. Anita Weinblatt\
Dr. Paul Strudler\
Dr. Stephen Rose

[XIII. Report of the Subcommittee on Rating Grant
Applications](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XIII%23XIII)\
Dr. Walter Stolz\
Dr. Hugh Stamper

Assigned Discussants\
Dr. Nesselroade\
Dr. Roger Bakeman

[XIV. Cooperative Review Effort between the National Institute on
Alcohol Abuse and Alcoholism (NIAAA) and
DRG](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XIV%23XIV)\
Dr. Jeanne Ketley\
Dr. Faye Calhoun\
Dr. Kenneth Warren

[XV. General
Discussion](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XV%23XV)

[XVI. 1996: 50 Years of NIH Peer
Review](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XVI%23XVI)\
Dr. Donald Luecke\
Dr. Suzanne Fisher\
Dr. John Sherman

[XVII. The Next Step for DRG and the DRG Advisory
Committee](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XVII%23XVII)\
Dr. Donald Luecke

[XVIII. Closing
Remarks](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/nov95min.html#XVIII%23XVIII)\
Dr. Thomas Braciale

[DRG Advisory Committee Roster (November
1995)](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/ROSTER.HTM)

[List of
Attendees](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/8579/ATENDN95.HTM)

**I. Call to Order and Approval of Minutes**

> **Dr. Thomas Braciale**

Dr. Thomas Braciale, Chairperson of the DRG Advisory Committee, called
to order the 13th meeting of the Committee, thanking the ad hoc advisors
for joining the Committee for the meeting. The first item of business
was to approve the minutes of the May meeting. With the exception of a
correction in the number of applications received by DRG in the last
year, the minutes of the May meeting were unanimously approved.

**II. Report of the Acting Director**

> **Dr. Donald Luecke**

Dr. Luecke explained the procedures for the impending Government
shutdown. He then noted that the meeting was a historic event, the first
opportunity to meet in the new Rockledge Building.

Next, Dr. Luecke welcomed eight ad hoc advisors: (1) Dr. Keith Yamamoto,
Chairperson of the Department of Pharmacology at the University of
California at San Francisco; (2) Dr. John Nesselroade, Acting
Chairperson for the Department of Psychology at the University of
Virginia; (3) Dr. Bruce Stillman, Director of the Cold Spring Harbor
Laboratory; (4) Dr. David Center, Professor and Chief of Pulmonary and
Critical Care Medicine at the Boston University School of Medicine; (5)
Dr. Howard Schachman, ombudsman at NIH working in the Office of the
Director; (6) Dr. Richard Hynes, Professor of Biology and Director of
the Center for Cancer Research at MIT; (7) Dr. Eveline Schneeberger,
Professor in the Department of Pathology at the Mass General Hospital in
Boston; and (8) Dr. Fayez Ghishan, Professor and Chairperson of the
Department of Pediatrics at the University of Arizona Medical Center in
Tucson.

With respect to the search for a new Director for DRG, the application
period closed November 1st. The selection will probably be some time in
the spring. The Co-chairs for the search committee are Dr. Wendy
Baldwin, Deputy Director for Extramural Research, Office of the
Director, and Dr. Claude Lenfant, Director of the National Heart, Lung,
and Blood Institute.

Dr. Luecke then summarized the developments related to the Cassman
Committee report, which had been presented at the May meeting of the DRG
Advisory Committee. The Peer Review Oversight Group (PROG) has a charter
signed by Dr. Varmus. PROG will have 16 members, 10 representatives from
the extramural scientific community, 6 from within NIH. One of those 6
will be the Director of DRG. Dr. Baldwin will serve as the Chairperson
for PROG.

Drs. Luecke and Baldwin agreed on the staff who would be leaving DRG to
join the Office of Extramural Research. DRG will be losing 89 people, 79
from the Information Systems Branch and the remaining 10 from various
other offices. All of these people will remain in the new Rockledge
building, either in vacant space on the sixth floor or in former DRG
quarters.

**III. View from the Office of Extramural Research**

> **Dr. Wendy Baldwin**

Dr. Baldwin explained that NIH is working under a continuing resolution,
which is \$200 million below the NIH FY 94 level of funding; so it has
been a very difficult time. If there is a new continuing resolution, it
would likely be at the FY 95 level, which would be a help. NIH
continuing grantees have received a letter from Dr. Baldwin advising
them of the budget situation and explaining that, unless there were some
dramatic change, NIH would not be able to support the normal four
percent escalation in future years. For new grantees, the plan is to
take reductions at the outset, so that investigators will have a sense
of stability and know funding levels over the lifetime of the grant.
However, every Institute is going to have to look at its own available
funds, and the specifics will vary from Institute to Institute.

Dr. Baldwin then commented on a recent report by a committee looking at
the relationship of the NIH with the Howard Hughes Medical Institute
(HHMI). First, the policy of excluding HHMI investigators from the room
when other HHMI investigators are being reviewed, done initially at the
request of HHMI, has been eliminated. As long as there is not a conflict
of interest, they can now remain in the meeting room. This will
effectively give NIH a larger pool of reviewers. The second feature is
the need for NIH reviewers to focus continually on scientific issues.
Other issues that reviewers may have knowledge or opinions about should
be placed in an administrative note, and not be part of the scientific
critique. Finally, NIH should return to the grant-in-aid approach to
partially support research that may also be partly supported by other
sources.

In response to a question about whether the features were unique to the
Howard Hughes Institute, Dr. Baldwin noted that such features were not
specific to, but highlighted in, NIH\'s experiences with Howard Hughes.
These are general issues and would have general solutions.

What about the issue of overlap between Howard Hughes and NIH support?
For some investigators it is quite easy to see the distinction between
the Hughes-supported and NIH-supported research, but in other
circumstances it blurs dramatically. Dr. Baldwin noted that NIH, in the
past, always had situations where an overall activity is not a severable
NIH activity. Basically, this is a co-funding strategy, and Dr. Baldwin
noted that it must be handled carefully or NIH will create a new set of
problems.

With respect to the PROG Committee, Dr. Baldwin noted that the majority
of the membership will come from the scientific community. PROG should
be in place by about March 1996.

The decision on the new DRG Director could be made as early as January,
although it will likely be in the spring. When the individual could
assume the position is uncertain.

**IV. NIH and DRG Budgets**

> **Mr. Fred Wong**

Mr. Fred Wong, DRG Budget Officer, noted that the current budget
disagreement and potential shutdown impact on the annual appropriations.
The FY 1995 estimate was \$11.3 billion. The FY 1996 President\'s Budget
was \$11.8 billion, a four percent increase over the FY 95 estimate.
After Congress reviewed the President\'s Budget, the House report was
\$11.9 billion, a 5.7 percent increase over FY 95. The Senate\'s
proposal, \$11.6 billion, represented a 2.7 percent increase over the FY
95 estimate. The DRG budget has been about \$40 million.

**V. NIH Reinvention Update: Present and Future Activities**

> **Dr. Anthony Demsey**

Dr. Demsey, Acting Deputy Director, noted that reinvention information
is available to the public on the Worldwide Web. Unfortunately it has
been revised infrequently. A number of activities underway at NIH can be
classified under the broad umbrella of reinvention. The transfer of DRG
staff from the Information Systems Branch to the Office of Extramural
Research is related to the overall electronic research administration
effort at NIH. The project has received a much greater degree of
prominence because of the move to centralize activities within the
Office of the Director of NIH.

Less global, but of equal interest, are endeavors such as the \"Just in
Time\" submission of application data. The concept is fairly
straightforward. Some of the information that applicants submit with
their applications is not necessary for the review of an application but
only for the award of a grant. So why not ask for this information at
the time that NIH plans to fund the approximately 25 percent of the
reviewed applications?

Another concept is called the modular grant, asking for a block of
money, a set amount of dollars. Mechanisms that have restrictive budgets
and so lend themselves to this idea are Small Grants (RO3s), FIRST
awards (R29s), Phase 1 Small Business grants (STTRs and SBIRs, R41,
R43), and some career development awards (K series).

A number of other activities are generally related to study section
operations. For example, in the rating of grant applications, the
current single score could become multiple scores based on various
criteria. A second interrelated activity is the National Institute of
General Medical Sciences experiment on rounding off priority scores and
percentiles. Instead of a 187, the score would be 190. Then the
calculation of percentiles is further rounded off. Essentially the
Institute is trying to create a large number of tie scores in an attempt
to reflect the fact that there are not necessarily significant
differences between a 187 and a 192 priority score or a 23rd and a 21.7
percentile.

Another committee is working on the consolidation of granting
mechanisms. The proliferation of mechanisms is confusing to the
scientific community. Another issue is who can vote at study section
meetings. The NIH Reviewers Reserve currently allows ad hoc reviewers to
vote. A fresh look is being given to whether that practice should be to
continued or whether all individuals, ad hoc or not, could vote. Also,
as Dr. Baldwin explained, Howard Hughes investigators no longer will be
considered automatically in conflict during the review of another Hughes
investigator simply on the basis of their employment with the Howard
Hughes Medical Institute.

A final issue is limiting the number of amendments. Possibly that would
provide less ownership by the study section, as well as more flexibility
in scoring the application. An additional concern is for a more rapid
turnaround of amended applications.

**[Discussion]{.underline}**

In response to a question about whether study sections should suspend
their mentoring role in the grant application process (for example,
amended applications being continually fine tuned by reviewers), Dr.
Demsey noted this philosophical debate has been going on for some time.
That reflects the concern about study sections continuing to provide
input round after round, which then turns the process more into a
tutorial than it should be.

One concern was raised that eliminating or reducing mentoring might be
acceptable to a seasoned, experienced individual, but it may be a
hardship with young investigators. However, it might improve some
inappropriate practices of universities and departments where grant
applications are tied to salary increments.

One discussant agreed that NIH cannot write rules for such a diverse
population, but noted that something has to be done to change the
concept that summary statements are supposed to be tutorials for
everybody. Summary Statements need to be concise, to deal with the
important issues, instead of including all the strong and weak points on
every grant application. Such excruciating details on summary statements
of excellent applications are a burden on the study sections. Certain
amendments ought to be handled at the Institute level without a
resubmission, because the application is clearly fundable; others should
have been turned down and the applicant free from the illusion that
amending the application may lead to an award.

One person noted that with senior members on study sections, there is a
mentoring role for young investigators. Often a study section can, in
the first round, help good investigators focus their research. The
investigator then comes back with a superb application, and most
importantly, goes on to do research that is universally rated as
outstanding.

**VI. The Initial Review Group Concept**

> **Dr. Donald Luecke**

Dr. Luecke reminded the Committee that reorganization of study sections
started as a result of an executive order in 1993 by President Clinton
to reduce the number of public advisory groups. In response, DRG went
from 85 chartered groups or study sections, with 105 different groups,
to 18 initial review groups (IRGs). Each IRG has a charter. The study
sections are not chartered, but exist as subcommittees within an IRG.
Dr. Luecke then noted two important topics: (1) occasionally DRG has not
been as interactive with Institute staff as they should be; and (2) DRG
needs to engage the community in a continuous process of looking at the
way the Division is constituted, the way in which the advances in
science may be reflected in the different kinds of applications, and how
to better accommodate those changes in the study sections.

**VII. Overview of the Pilot Study of Initial Review Groups**

> **Dr. Donna Dean\
> Dr. Raymond Bahor**

Dr. Donna Dean, Acting chief of the DRG Referral and Review Branch,
explained that the goal of the Pilot Independent IRG was to work towards
a more flexible peer review structure and operative environment in DRG.
Thus, the Division could be more immediately responsive to changes in
emerging new areas of science, while still keeping the primary focus on
the intrinsic high quality peer review that has been the Division\'s
strength over the last 50 years.

Since the first of June, 8 of the 18 IRGs have been operating in this
rather independent capacity; the other 10 IRGs have been operating in
the traditional review section structure within the Division. The
Division has looked at restructuring not only structurally but also
functionally, examining issues of breadth and focus. How does one
periodically reevaluate current practices? (Certainly this also is a
valuable function of the Advisory Committee.) How does the Division form
new committees to cover areas? How does DRG handle cross-fertilization
across fields of science? How can members be used where their expertise
is most needed? Throughout, the Division should create an environment of
increased communication among the IRG, NIH staff, and the scientific
community.

After briefly summarizing the organization and staff of the Pilot
Independent IRG structure ([Figure
1a](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/images/k164ppa.gif)
and
[Figure1b](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/images/k164ppb.gif)),
Dr. Dean ended with four quotes, which summarized the unavoidable
effects of organizational changes: \"I was gratified to be able to
answer promptly. I said I don\'t know.\" (Mark Twain); \"There \'s no
off season any more.\" (Nolan Ryan); \"If you come to a fork in the
road, take it.\" (Yogi Bera); and \"You miss 100 percent of the shots
you never take.\" (Wayne Gretsky).

Dr. Raymond Bahor, Chief of the Pilot Independent IRG, indicated that
this project was really an empowerment of the SRAs. They know the
available resources and are best able to make the decisions to use those
resources most effectively. He regarded his position as making
administrative decisions or facilitating the work of the SRAs who are
involved in review.

**VIII. Fellowship Reviews as a Model of Initial Review Group
Flexibility**

> **Dr. Jerry Roberts\
> Dr. Everett Sinnett**

Dr. Roberts, SRA of the Mammalian Genetics Study Section, spoke about
the review of fellowship applications in the Genetic Sciences Initial
Review Group. Their Biological Sciences 1 Study Section had reviewed
only fellowship applications, covering a broad area of science, from
clinical studies to Drosophila and yeast genetics. With the move to
disband fellowship study sections and move fellowship applications into
RO1 committees, this was a chance to move some RO1s into Biological
Sciences 1 and experiment with the review of these applications using a
broader committee.

To equalize the workload, fellowship applications were moved from the
Biological Sciences 1 Study Section to three RO1 study sections. A
problem with the mix of RO1 and fellowship applications arose because
fellowships are reviewed together since the review criteria are
different. However, some Institute program staff, who needed to be
present during the review of their RO1 applications, had to return for
the review of fellowship applications. This was a logistical problem,
which pointed to the need for understanding and accommodation on both
sides.

**[Discussion]{.underline}**

In the study sections where fellowships were included with research
grants, were the fellowships counted in the percentile of the total? Dr.
Luecke responded that only RO1s and R29s (FIRST applications) are
included in the base for percentiling. RO3s (Small Grants), and R15s
(Academic Research Enhancement Awards-AREAs) are also not currently
included in the percentiling base. A second comment pertained to the
potential advantage with this new structure in that the applications
with broad-based approaches could more easily find \"outside reviewers\"
from within other parts of the same IRG rather than having to get a mail
review from someone outside the process.

Some SRAs noted that when related study sections met at the same time
and in nearby locations, the sharing of reviewers worked out pretty
well. Several members cautioned that while it is desirable to broaden
the scope of review groups, there is also value in having a stable
membership component. Both functions must be considered in the initial
review process.

Dr. Sinnett, SRA of the Respiratory and Applied Physiology Study
Section, who was the next assigned discussant, spoke about their recent
experience reviewing fellowship applications. Altogether there were only
14 fellowships for the whole IRG. As to the fairness and the
appropriateness of the review, several issues deserve comment. One issue
was the lack of experience of their study sections with fellowship.
Therefore, information on fellowships was sent to the reviewers in the
pre-meeting mailings. In addition, they made sure that reviewers with
previous fellowship review experience attended the meeting.

There seemed to be little problem for the reviewers in terms of
incorporating additional criteria in the reviews and discussion. In
terms of scheduling, they were requested to review all the fellowships
at the start of the second day, and with the small numbers, they were
able to comply with that request. But Dr. Sinnett could foresee future
problems of not only Institute program staff availability but also
reviewer availability in reviewing fellowships as a group.

In the discussion, one SRA explained that their IRG decided to cluster
their 40 fellowship applications into a special emphasis panel. The
special emphasis panel worked very well, and the Institute program
representative wrote commendatory letters to the Division.

**IX. Joint Workshops and Study Section Meetings**

> **Dr. Sooja Kim\
> Dr. Fayez Ghishan\
> Dr. Paul Strudler**

Dr. Kim, SRA of the Nutrition Study Section, reported on a Nutritional
and Metabolic Sciences IRG Workshop on Molecular Biology, which was held
on October 23, 1995. The workshop, which involved the membership of the
three IRG study sections, included both scientific sessions and also a
discussion about the NIH peer review process.

Dr. Ghishan, a participant in this workshop, then summarized the
proceedings of this successful workshop on the new tools of molecular
biology. The peer review system discussion covered several areas. With
respect to scoring, the consensus was that a system using scores for
different criteria would not be ideal, and that the application should
continue to be reviewed and scored as a whole.

Concerning R29 (FIRST) and RO1 applications, the attendees believed that
no need existed for separate review systems, and these applications
should be reviewed together in the same study section. They also felt
that new and continuation grant applications should not be reviewed
separately by different review groups. Clinical and basic research
should also continue to be reviewed together; however, reviewers who are
more familiar with clinical studies should be members of the study
section or ad hoc reviewers.

Given the continuous changes in science and new and emerging scientific
areas, how can a study section deal with these areas most effectively?
The attendees recommended periodic workshops to bring these newer areas
of science to the members of the study section. The addition of ad hoc
members with specific expertise would also be useful.

The workshop attendees were concerned about the triage system. In some
universities, if an investigator receives a non-competitive score, then
their funding is cut. However, if the investigators receive a score
better than the 20th percentile, the universities can provide them with
bridge funding through the next cycle. In their opinion, the triage
system should be modified to identify non-competitive but very good
applications, so that investigators who may just miss the cut off will
not have their funding cut.

Can the review process be further streamlined? For example, should a
limit be placed on the number of times an application can be
resubmitted? The consensus was that there should be a limit. Should an
application that received an outstanding score but just missed being
funded have to be rereviewed during the next grant review cycle? They
believed it should not be rereviewed.

Dr. Paul Strudler, SRA of the Radiation Study Section, who was the next
presenter, noted that a central theme in recent workshops and peer
review studies is isolationism\--the notion that DRG is isolated from
the Institutes, and that study sections are isolated from the science.
Dr. Strudler discussed study section interactions and the opportunities
for SRAs to be current in their scientific areas.

This winter, the Radiation and Chemical Pathology Study Sections\* were
scheduled to meet jointly at the Keystone Resort for one day in
conjunction with the AACR Symposium on Cancer Susceptibility Genes and
Molecular Carcinogenesis. The Chemical Pathology Study Section reviews
applications where the emphasis is on the various chemical and physical
aspects of pathology; the Radiation Study Section is more involved with
the physical aspects of pathology. Pragmatically, the two study sections
overlap in the areas of free radical chemistry, DNA repair, and UV
induced mutagenesis. The workshop will be largely devoted to free
radicals and oxidative damage, matters of considerable concern to both
study sections. Both study sections have a history of publishing their
workshop reports, and this one will undoubtedly also be published. \*
Note: Due to the subsequent Government furlough, it was necessary to
reschedule these meetings.

The DRG management is working very hard under budgetary constraints to
meet the desires of the community that the SRAs attend more professional
meetings to remain current with the science and to interact with
scientists. But there is only so much that can be done, and Dr. Strudler
and other SRAs volunteer their time and monies to attend scientific
meetings. For example, in October 1995, with DRG paying only the
registration fees, Dr. Strudler attended scientific meetings in Houston
and in Baltimore of prime interest to his scientific work, and gave
presentations on peer review. In this issue of money versus travel, the
difficulties of dealing with the desires of the community are felt by
SRAs trying to meet these obligations.

**[Discussion]{.underline}**

Was Dr. Strudler implying that the SRAs would like to become more
involved in giving both positive and negative feedback to applicants
after they receive their reviews? Dr. Strudler responded that he was
referring to attending meetings and conferences and making himself
available so that people wouldn\'t feel frightened to pick up the phone
and call. He indicated his personal belief that it would be expeditious
for an SRA to provide direct feedback to applicants as quickly and as
early post-review as possible. However, at present, SRAs are not to have
contact with applicants post-review; such contact is to be done by the
Institute program administrators. This works well when there is good
communication between review and program staff.

**X. Anticipating and Responding to Changes in Science**

**Dr. Michael Knecht\
Dr. Gerald Greenhouse**

Dr. Knecht, SRA of the Biochemical Endocrinology Study Section and
Coordinator of the Endocrinology and Reproductive Sciences IRG,
explained that a major future consideration is to meet with scientists,
the heads of various endocrine-related societies, Institute program
staff, and SRAs to review the field of endocrinology to see where the
field is headed and how the IRG should evolve. For example, the numbers
of applications in thyroid, ovarian, and neuroendocrine research areas
have decreased recently, while applications in other areas, including
insulin, GI hormones, and endocrine cancers, have risen. Study section
membership and guidelines need to meet these changing challenges.

Dr. Gerald Greenhouse, SRA of the Cell Biology-1 Study Section and
Coordinator of the Cell Development and Function IRG, noted that the
Acting Director of DRG is allowing and indeed encouraging the SRAs and
coordinators to make substantial decisions at the basic level of the
study section.

The Cell Development and Function IRG, has reviewed cell and
developmental biology for some time. Dr. Greenhouse has been Referral
Officer for the group since 1982. The component study sections of the
IRG have a long history of interacting together with a certain amount of
continuity. Because of a reasonable amount of overlap among study
sections, every application assigned to the IRG could be assigned to at
least two study sections. They have also exchanged members, in the past,
although not every round and not too often. When they exchanged members
the members generally stayed for the entire meeting. For example, with
the addition of fellowship applications, the CTY study section became
overloaded, so they moved a subset of applications involving
cytoskeleton studies to another study section within the IRG. Another
example was at a recent Gordon Conference on biological Structure and
Gene Expression, where he spoke with researchers interested in how
structure affects gene regulation. They did not feel represented well
enough on the study sections. After talking more to the community, the
Division recruited some additional members to existing study sections.

**[Discussion]{.underline}**

Comments were made by several Advisory Committee members that
applications reviewed by study sections are what science is going to be.
In contrast, at major meetings, the lectures often are about where
science has been. The SRA should certainly go to such meetings, but
should also have broader interactions with members of the scientific
community. Institutes also push their agenda in certain ways by issuing
Requests for Applications (RFAs), but some of those are not necessarily
the best way to do science. Dr. Luecke responded that for many reasons,
some of which relate to science, the number of RFAs issued by many
Institutes will be reduced. One of the real problems that resulted from
the widespread, indiscriminate use of RFAs, many of them searching for
investigator-initiated ideas, is that applicants are not successful in
their first review in an ad hoc setting, and DRG is then faced with
reviewing the amended applications in study sections.

**XI. Outreach to the Scientific Community**

> **Dr. Gerald Greenhouse\
> Dr. Anita Weinblatt**

Dr. Greenhouse noted that in conjunction with the American Society of
Cell Biology, they have organized sessions for attendees to meet the
SRAs in an informal setting. The main objective is for people to
understand that SRAs are approachable and can be contacted. SRAs
consider such discussions to be an enjoyable part of their job.

Dr. Weinblatt, SRA of the Immunological Sciences Study Section and
Coordinator of the Immunological Sciences IRG, then spoke about her
interactions with the American Association of Immunologists. Dr.
Weinblatt had two face-to-face meetings with the Executive Director on
issues of mutual concern, and in addition, she was invited to address
the Executive Committee and hear their concerns. Recently, she used the
study section as a mini-advisory committee. At the end of the study
section meeting, the floor was open to hear any concerns of the study
section members. That was a very useful exercise, for it gave them an
opportunity to give Dr. Weinblatt some feedback about various issues.

**[Discussion]{.underline}**

In response to a question about whether these groups included colleagues
from the Institutes, Dr. Weinblatt stated that they have not yet been
included, but felt it would be a good idea.

The next question was that having had a high level of success, does DRG
plan to make more explicit and directed contact with the scientific
societies? Dr. Weinblatt noted that for a combined meeting between the
American Association of Immunologists and the American Association of
Asthma and Allergic Diseases, several study sections are discussing a
joint workshop.

The request was made for data on the number of individuals who have had
continuous NIH funding for a period of years but who have not sat on DRG
study section, as well as data on those individuals who have and in what
capacity for the last 10 or 15 years. It was further suggested that the
database include serving on review panels outside the NIH. Such service
is becoming an increasing burden for many investigators, and should be
taken into account.

**XII. Developing Closer Links with NIH Institutes and Centers**

> **Dr. Anita Weinblatt\
> Dr. Paul Strudler\
> Dr. Stephen Rose**

Dr. Weinblatt spoke about recent activities in the Immunological
Sciences IRG that started at the beginning of June 1995. The SRAs in
their IRG met with the program staff from the National Institute of
Allergy and Infectious Diseases that they interact with\--the Division
of Allergy, Immunology and Transplantation. They invited the program
staff to Rockledge II to see DRG\'s new quarters and also to discuss
some issues. They discussed the Pilot Independent IRG experiment and the
composition of their IRG, especially the possible restructuring of the
former fellowship study sections.

Their goals were to use Institute program staff as a source of advice
and also to explore ways to work better together. From that meeting came
their plan to do an in-depth analysis of each study section to see if
there are ways of restructuring that would help them review the science
better. The flexibility of the IRG will, she hoped, allow SRAs to be
more flexible in their thinking about program and review interactions.

Dr. Strudler noted that the stakes are high, for this involves careers
and people\'s lives. The level of shrillness and paranoia will perhaps
continue increasing despite the increasing efforts of DRG to interface
with the community. The SRAs in Dr. Strudler\'s IRG met separately both
with the program administrators from the National Cancer Institute (NCI)
and also with the SRAs in the NCI.

The main accomplishment from the meetings was dealing with the mistrust;
but they also discussed that especially for program announcements, it
would be valuable to involve the Division\'s SRAs in the proposed
subject matter. Also, traditionally, the Institute/Center review branch
selection of site visitors occurs without interaction with DRG, and the
Division\'s SRA\'s offered their assistance. In all, the prime result of
these meetings with NCI was to break down the barriers, so that they
will use each other as cooperative resources.

Dr. Stephen Rose, Chief, Genetics and Transplantation Branch, National
Institute of Allergy and Infectious Diseases, noted their interactions
with the SRAs of the DRG immunology study sections have always been
phenomenal. The SRAs have been professional and highly competent and
have treated program staff with respect, and vice versa. Their Institute
has held several focus groups with the scientific community, and a major
issue is that feedback from Institute program staff on review must be
timely and blunt. Investigators are getting considerable pressure from
their organizations; they can not go unfunded as long as they could in
the past. The safety nets are basically disappearing.

The focus groups also unfortunately reinforced something that they had
heard frequently: people still view NIH as a black box. They send in an
application and then something comes back. For example, even senior
grantees complain that \"DRG\" did not fund their grant. One of the
topics they have been trying to deal with internally is how Institute
program staff attending a study section meeting should react if they
believe that something is missed in the review. Granted, their role is
not to participate in the review. But if something is missed by the
reviewers and the program staff notice it, how can this be brought to
the attention of the study section? One possibility, suggested by an SRA
was to write a note, and slip it to the SRA. If an application page
number is available, include it in the note.

**[Discussion]{.underline}**

Does electronic transmission technology help the process of getting the
critiques out to the investigators? Dr. Weinblatt responded that at the
present time, this does not speed up the procedures. Streamlining has
been more helpful in expediting the preparation of summary statements.

Another question was whether Institute program staff not attending study
section meetings was a major problem. Are program staff required or just
encouraged to show up? Dr. Rose responded that in his Institute, their
job is to be there. Attendance is important for performance of their
responsibilities. There is, however, a scheduling problem, with too few
staff to cover too many meetings. One problem is that the schedule for a
particular study section may not be in the computer system.

The next topic for discussion was clearances needed for appointing
members. Currently, the SRA needs to get clearance from the NIH
Director, and the nominations are at the level of a specific study
section, rather than at the level of the entire IRG. For example, an IRG
may have one charter for seven study sections. They can be flexible
about the way to use these members; but when they appoint members, they
are still held to produce seven separate slates; it would be to their
advantage to be able to combine all the nominations into one big
package. They could do a much better job of balancing all the
administrative constraints while still trying to find the best
scientists. Dr. Luecke responded that some Institutes have already begun
to implement this practice and the Division may also be able to
accomplish this. Progress on this matter will be reported to the
Advisory Committee.

The next topic of discussion was whether all the study sections within
an IRG should meet at a specific time and location, so that the
possibility of movement of members within an IRG could be facilitated.
However, it was pointed out that the usefulness depends on the overlap
in science of the individual study sections. Switching members is a new
process, and the SRAs are working out some issues, including the
logistical difficulties and the different voting behaviors within study
sections. Another SRA explained that based on overlap and workload, they
have recently begun shifting blocks of applications and reviewers rather
than one at a time. It also was a way to level workloads in the current
fiscal climate and downsizing. If SRAs take a reviewer out of a study
section, they must realize the impact on the study section, and it may
be necessary to get another ad hoc reviewer. It might be better to find
an outstanding ad hoc reviewer to go to the first study section. Given
the obvious logistical problems of trying to get all the scientists and
the Institute program people together, one discussant questioned the
advantage of shifting reviewers as opposed to applications. One
discussant noted, however, that the best science in the future will
probably be nonhomogeneous, and that individual projects will require a
large range of technological and scientific expertise for review.

One SRA, who was also a Referral Officer, noted that when an application
does not fit anywhere, they negotiate with the SRA of the study section
closest to it, often discussing possible ad hoc reviewers. This may
involve senior investigators who are willing to come for one meeting.

One person suggested encouraging SRAs to appoint individuals for shorter
terms, for 2 years, to be renewed for a 3rd year and 4th year if their
particular areas of expertise still match the numbers of relevant
applications received.

One SRA explained that a possible drawback to cross-fertilization and
trading members for one meeting is that the individual would not feel a
part of the camaraderie and collegiality that develops among members who
get to know each other well.

Dr. Luecke responded that with regard to the tradeoffs of short-term
versus long-term membership, there is no \"right\" answer. It depends on
the judgment of the SRA and willingness of the reviewer. DRG management
is not comfortable with creating rules that must be strictly adhered to;
rather the process should enable SRAs to recognize and hopefully better
utilize available resources. The final integration is going to have to
occur at the SRA level.

Another discussant stressed the importance of flexibility, complimenting
DRG for its various experiments in peer review. Although many study
section members greatly value the particular culture of their study
section, cross-cultural education is also useful at times, and they can
learn from seeing how different groups operate.

The next topic was the role of the SRA in interacting with the
extramural community. How should that be implemented? What role should
DRG play in that process? Should DRG have a model, wherever practical,
that could be used across all components? If so, how could the Advisory
Committee help implement this?

One discussant noted that when SRAs attend scientific meetings, it
really makes a difference. Could this become part of the requirements of
an SRA? Dr. Luecke responded that it is not so much a problem of the
function of the SRA, but more a problem of resources. Thanks in part to
the DRG Advisory Committee, there has been some recent progress in this
area; DRG needs to explore other innovative ways to facilitate the
travel of SRAs, possibly in a way that would not tie a particular SRA to
any group of applicants because of conflict-of-interest regulations.

There was considerable discussion about having societies create funds
for travel for NIH staff to go to meetings. That problem has to be
resolved in terms of even the appearance of conflict of interest. One
person wondered whether a way to avoid conflicts of interest might be to
have a group or consortium of societies put money into one large account
as a type of blind trust. Dr. Luecke noted the DRG has begun to explore
various possibilities. One of the members of the Advisory Committee
suggested that the Committee draft a recommendation about the need of
the SRAs to get out to the community by attending meetings, preparing
workshops, or doing other outreach activities for the communities\'
benefits. If the Division and the Advisory Committee can work to improve
the information flow, this is a useful goal.

**XIII. Report of the Subcommittee on Rating Grant Applications**

> **Dr. Walter Stolz\
> Dr. Hugh Stamper**
>
> **Assigned Discussants\
> Dr. Nesselroade\
> Dr. Bakeman**

Dr. Stolz, Director of the Division of Extramural Activities, National
Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), and
Co-chairperson of the Committee, explained its establishment as a
Subcommittee of the NIH Committee for the Improvement of Peer Review.
The Subcommittee met regularly for about a year to evaluate the system
currently used by NIH to rate grant applications and to address the
means to report scores of scientific merit. The Subcommittee considered
the areas of psychometrics and decision science, and enlisted the aid of
prominent experts in the behavioral sciences. Recommendations presented
in the report are in four general areas: (1) the review criteria used to
develop an assessment of scientific merit; (2) the rating scale; (3) the
comparability of scores across reviewers, review groups, and review
cycles; and (4) how scores are reported.

In the first area, the Subcommittee agreed that the current review
criteria included the principal components to assess scientific merit,
but felt these should be reformulated into three major criteria:
significance, approach, and feasibility. The summary statement critiques
should address each criterion separately, and provide a score for each.
The Subcommittee consultants advised that the reliability of rating
would increase through the disaggregated criterion process, i.e,
discussing, rating, and scoring each criterion separately.

The second area is the rating scale, which currently goes from a high of
1.0 to a low of 5.0, in 0.1 increments, thus providing 41 points on the
scale. The Subcommittee agreed that the number of scale positions should
be commensurate with the potential number of reliable discriminations.
Based on relevant psychological literature, it was recommended that a
seven-step scale be used, with the addition of \"0\" to represent
unacceptability of the characteristic being rated. In addition, the
rating scale should be inverted, so that larger values represent more
favorable characteristics of the criterion being rated.

In the issue of comparability of scores, two approaches were presented
to make scores more comparable across reviewers, groups, and time. Both
of these are employed currently at the NIH. One approach is to instruct
reviewers to anchor the scale positions in an attempt to use scores the
same way and to mean the same thing to different people. NIH uses
adjectives to anchor the scales. In another approach, after the ratings
have been done, scores are processed statistically in an attempt to
achieve uniformity in rating behavior. NIH uses percentiling in an
attempt to achieve uniformity in rating behavior. The Subcommittee
recommended that the scale should be anchored only at the ends. Also,
instead of percentiling by group, which is the current procedure, the
Subcommittee recommended that scores be standardized within each
reviewer.

In the fourth area, reporting priority scores, it was recommended that
scores should be reported with a precision commensurate with the
meaningful information contained in the scores. Two significant digits
were recommended. Even though reviewers rate using one digit, the
resulting values summarizes a group of possibly 10 to 20 scores for each
application, and so two digits are useful to report the priority scores.

Finally, a majority of the committee believed that an overall priority
score should be computed by averaging the three criterion scores. In
response to a question of whether a greater weight is to be placed on
significance as opposed to feasibility, Dr. Stolz said that the
advantages and disadvantages of weighting scores are currently under
discussion.

Dr. Hugh Stamper, Director, Division of Extramural Activities, National
Institute of Mental Health, and Co-chairperson of the Subcommittee, then
commented on the Subcommittee\'s recommendations. He noted that
standardizing scores by reviewer rather than by study section allows a
person to serve and vote more systematically on more than one study
section. When members attend another study section meeting, even for one
grant application, their own voting distributions are used in the
standardized process. He added that each review criterion would be
scored individually, and each criterion along with the overall score
would be reported on the summary statement.

Dr. Nesselroade, the first discussant, pointed out a potential problem
with standardizing priority ratings by reviewers rather than by the
group. If there are two reviewers, one of whom is doing a valid job of
making difficult discriminations among a set of applications and another
reviewer who is not, by standardizing, those two distributions may have
the same variance. Thus, the process may not correct for the differences
between these two types of reviewers. His next comment pertained to how
recommendations would be evaluated. The report mentions the possibility
of having some applications reviewed also by a control study section,
and then determining how well the two groups agree. This is a typical
test to determine the correlation between parallel forms to measure
reliability. Other evaluation methods also could be considered, such as
comes from the generalizability tradition, which would take into account
the idea that each application has a universal score. That might be the
score it would be assigned if it could be given to all possible study
sections. One then could estimate how much variation in these scores is
actually being contributed by having it go to one study section versus
another study section.

Dr. Bakeman, the second discussant, suggested developing a spreadsheet
with histories of reviewers to determine how each is represented in
terms of numbers. There are individual differences, and different people
can validly make different discriminations. When a reviewer joins a
study section, the study section can provide a recommended scale based
on the spreadsheet; but in fact, reviewers can choose their own scale.
For each reviewer, one possibility might be to standardize only over
their last three review cycles. Many individuals change considerably
over time. Basically, Dr. Bakeman thought standardized individual
ratings have some strong advantages over the current system. He felt,
however, that it was arguable whether there should be three or more
review criteria. Furthermore, the division into significance, approach,
and feasibility may lengthen review discussions. He suggested asking
current reviewers\' opinions on this recommendation.

**[Discussion]{.underline}**

The possibility of a fourth criterion \-- innovation or creativity \--
was suggested. The three criteria might tend to favor more conservative
applications from long-term investigators. Dr. Stamper responded that
the committee felt that significance and feasibility included the issue
of innovative, high-risk research. Having creativity as an explicit
criterion would make each reviewer pay attention to this category, which
might tend to balance against the well-established investigator. The DRG
Advisory Committee agreed to make the addition of a creativity or
innovation criterion an explicit recommendation for the Scoring
Subcommittee.

One question pertained to competing renewals, where the investigator\'s
productivity over the past years is a major factor in rating an
application. How are these three criteria used when reviewing a
competing renewal application? Dr. Stamper responded by noting that the
criterion of feasibility is intended to include past progress. However,
it was noted that feasibility versus past productivity are usually not
connected, because feasibility relates to the future and productivity to
the past. This review criterion may need to be rewritten.

Another suggestion was that DRG should provide only the three criterion
scores and not try to create an overall score. There are various means
of weighting of those three criteria among the various Institutes. Also,
with the greater number of criteria, it may take a lot longer to discuss
each application. Dr. Stolz responded that this system is similar to the
way contract proposals are reviewed. Staff who are experienced with
contract reviews said that time should not be an issue, thatreviewers
get accustomed to this kind of format and soon review expeditiously.

Another concern was that people are resistant to change, and the
committee should make recommendations on how these changes will be
introduced. Dr. Stamper agreed, but noted that the committee will not be
overseeing implementation. However the report will include a strong
recommendation for careful testing and evaluation.

Another question concerned the effect of inflated priority scores, or
grade creep, that might result from the proposed seven point scale. Dr.
Stamper responded that the reviewers would know the effect of their own
ratings; it is different when done on an individual basis rather than on
a committee basis. Dr. Luecke added that there was some enthusiasm for
having another recalibration of the current study sections\' voting
patterns and having reviewers spread their ratings so that they provide
greater discrimination. This requires close cooperation with the
Institutes. Dr. Luecke also addressed the issue of standardizing ad hoc
reviewers. How often must they attend study section meetings and how
many applications must they review in order to qualify for
standardization? This concern does not preclude judiciously moving ahead
in that direction.

Dr. Stamper noted that the report, when revised, will go to the
Committee to Improve Peer Review, and then will be submitted to Dr.
Wendy Baldwin for possible implementation.

**XIV. Cooperative Review Effort between the National Institute on
Alcohol Abuse and Alcoholism (NIAAA) and DRG**

> **Dr. Jeanne Ketley\
> Dr. Faye Calhoun\
> Dr. Kenneth Warren**

Dr. Ketley provided some background on this cooperative project. In
October 1992, the research arms of the three Institutes that made up the
ADAMHA agency were brought into the NIH. However, their review
functions, which had been done independently, would not be brought into
the DRG immediately. As part of an effort to merge NIAAA reviewers into
DRG, Dr. Faye Calhoun, now the Deputy Director for Collaborative
Research at NIAAA, suggested that DRG and NIAAA begin working to
integrate part of their biomedical portfolio with the two DRG toxicology
study sections.

Dr. Kenneth Warren, Director, Office of Scientific Affairs, NIAAA,
provided more detail on the ongoing NIAAA-DRG Pilot Peer Review
Integration Program. Dr. Warren explained the creation of NIAAA in 1970,
and its mission to develop and conduct comprehensive health education
and research activities around alcohol abuse and alcoholism. In July
1992, the ADAMHA Reorganization Act reorganized the Alcohol, Drug Abuse
and Mental Health Administration, or ADAMHA, and returned the three
research Institutes to the NIH, leaving a separate organization (SAMHSA)
for the services, training and block grant functions.

After a brief presentation about the organization and activities of
NIAAA, Dr. Warren turned to the issue of peer review integration. The
same law that moved NIAAA back into NIH states that after FY 1996, the
peer review systems utilized by NIAAA and the other two Institutes
within the former ADAMHA would be integrated into the DRG. For the
purpose of the discussion, Dr. Warren concentrated on their two
biomedical committees, ALCB-1 and ALCB-2. The ALCB-1 committee focuses
on research applications related to: alcohol metabolism, including
cytochrome P450 enzyme systems; physiologic and toxicologic mechanisms
of alcohol on various organ systems, including the immune system and
reproductive system; and development as it pertains to alcohol
teratogenesis. ALCB-2 has as its major focus: behavioral teratology,
alcohol-related neurotoxicology and neuroendocrinology, particularly as
it relates to reproductive biology; alcohol-derived neuropathology; and
the study of alcohol\'s actions at the molecular and cellular level with
specific receptor systems, ion channels and signal transduction systems.

The task in developing the integration pilot was to review the structure
of the 18 DRG IRGs, and find a home for the basic research portfolio of
these alcohol-related applications. The Health Promotion and Disease
Prevention IRG seemed the most appropriate, specifically its two
toxicology study sections, because of similarities in their scientific
areas and their focus on adaptive biology and the physiological
consequences of the actions of xenobiotics. For the February 1996 review
round, DRG and NIAAA staff analyzed similar areas of research to
determine how the two portfolios fit together and how the workloads
could be restructured to assure quality review. After evaluating the
process, DRG and NIAAA will make adjustments as needed for the June 1996
review round.

The processes and procedures used by all the review groups are nearly
identical, such that several research areas could be reviewed together.
The SRAs exchanged information about the applications reviewed. They
also had meetings with appropriate program staff and DRG management. In
addition, a subcommittee of their primary constituency organization, the
Research Society on Alcoholism, was formed to consider the implications
of this peer review integration. NIAAA staff addressed members of the
Society in a plenary session at their annual meeting last June. Another
closely affected Institute is NIEHS, and so DRG and NIAAA management
staff traveled to North Carolina to visit and brief both the program
staff and also the Council of NIEHS. Fortunately, both ALCB-2 and
Toxicology-2 met at the same time and within five blocks of each other.
Therefore, they were able to arrange a social gathering to discuss
informally this issue. In February, all four review groups will meet at
the same time and in close proximity to each other. Staff are currently
designing the methods for evaluating the pilot and gaining feedback on
the study.

Dr. Luecke noted that this pilot project came about, in large measure,
because of a groundswell from the SRAs in both organizations. The
interactions grew where they should have been growing, at the grassroots
level. He thanked Dr. Warren and the other extramural staff at NIAAA for
their initiative, and expressed the hope that additional efforts will
happen with the National Institute on Drug Abuse and the National
Institute of Mental Health. Dr. Faye Calhoun added that they are
planning to have senior members of both research communities involved in
the February meeting. They will attend as ad hoc members to help monitor
and evaluate the process.

**XV. General Discussion**

Dr. Luecke brought up the topic of the NIH Reviewers Reserve. The issue
is whether such ad hoc reviewers really have full membership standing,
and how they might participate more fully in a meeting. He suggested
identifying individuals as part of an IRG Reviewers Reserve. These would
be former members of an IRG, who would be willing to come back once or
twice a year to serve on a study section within a specific IRG. There
would be some sort of designation related to the Reviewers Reserve to
identify individuals who are a part of a particular IRG.

Dr. Schachman agreed, suggesting also that these ad hoc individuals
might be available for every study section with a title such as
\"distinguished scientists.\" Some concern was raised with the use of
this term distinguished and the possible interpretation that other
reviewers are not. There might be two such scientists from past study
sections for every study section meeting, who would have some
assignments, and who would vote. They could make enormous contributions.
They bring maturity, historical background, a lot of expertise, and
breadth to the process. However, there was some concern about the title
and its affect on the study section, and a suggestion that this idea be
tried informally at first without a title, and then evaluated.

Dr. Luecke noted that under his IRG reserve concept, more junior members
could be used depending on their expertise. Dr. Braciale stated that the
two suggestions (to build a reviewer reserve within a particular IRG and
to establish a mechanism so that appropriate \"senior\" people can
provide input and support at a particular study section meeting) are
overlapping and not mutually exclusive.

It was suggested that the topic of special ad hoc advisors to an IRG and
to the DRG should be a major agenda item for the next meeting.

In response to a question about how individual applications are assigned
to particular IRGs, Dr. Luecke summarized the current system involving
written referral guidelines that are changed every two years, input from
the Institutes, and assignment by referral officers, who are senior
SRAs, most of whom also manage study sections. In a proposed
modification, the referral officers would judge which IRG would be best,
but then the fine tuning to study sections would occur at the level of
the IRG. Most of the new IRG coordinators will be referral officers.

To what extent can the investigator have input to the referral process?
Dr. Luecke responded that he did not envision a change from current
practices, whereby investigators can ask for up to three Institute
and/or three study section assignments. DRG encourages such initiatives,
which are taken into account when assignments are made. During
discussion, it was noted that as science is increasingly crossing
techniques and crossing disciplines, study sections are using more ad
hoc reviewers. There may be a significantly different study section
composition each meeting than what appears on the standing roster. The
current process appears to work well. Perhaps the flexibility that
exists now for investigators to write letters requesting assignments
should be more widely advertised.

One suggestion\--especially for newly independent investigators who are
worried about finding an appropriate review group\--is that they call an
SRA, before submitting an application, to talk about possible study
sections. Finally, several discussants preferred the current flexibility
and rationale of using a cover letter with an application rather than
checking off boxes on a form.

**XVI. 1996: 50 Years of NIH Peer Review**

> **Dr. Donald Luecke\
> Dr. Suzanne Fisher\
> Dr. John Sherman**

Dr. Luecke remarked that 1996 is the year in which NIH will celebrate 50
years of peer review. The emphasis is not on DRG alone, but on 50 years
of peer review at NIH.

Dr. Suzanne Fisher, Assistant Chief of the Referral Section, and
coordinator of the celebration of 50 years of peer review and 50 years
of the Division of Research Grants, invited everyone to the major event,
an all day symposium in the Natcher conference facility on June 20th.
June 21 will be a predominantly in-house event involving present and
past employees.

The symposium will focus on the past, present and future of peer review.
The organizers hope that this will provide a very positive statement
about peer review. This event will mark the debut of a book called Half
a Century of Peer Review: A History of the Division of Research Grants.
The book is being written by Dr. Richard Mandel, a historian. The
present, will include a report on the current initiatives in DRG. There
has been much discussion about how much emphasis should be on science
versus how much on public policy issues. The organizers want to include
science, because excellent science is the product of peer review. The
goal is to have talks from scientists who have served on study sections
about their experiences in serving and having their applications
reviewed, and also about the resulting scientific achievements. The
various speakers will probably include active research investigators,
university administrators, NIH staff, members of advocacy groups, and
people in the media. The symposium will end with discussions of the
future of peer review by Dr. Varmus and the Director of DRG. The day
will conclude with a reception at the Natcher Conference Center.

The Division\'s efforts are being greatly aided by excellent advice,
logistical support, and financial support from a group called Friends of
DRG. Dr. Fisher then introduced Dr. John Sherman, one of the principal
persons working with the Friends of DRG. Dr. Sherman has held numerous
positions at NIH, including Deputy Director, Acting Director , and
Associate Director for Extramural Programs, and has been Executive Vice
President of the American Association of Medical Colleges. Dr. Sherman
explained that instead of approaching pharmaceutical companies or other
big organizations, they looked to professional societies, voluntary
health organizations, and others interested in the concept of peer
review. They have approached approximately 75 major voluntary health
organizations, professional societies, organizations, institutions and
so forth. So far, the drive has revealed a heartwarming response. The
FASEB organization has already come forth with a sizeable pledge, and
the AAMC has offered their expertise in program planning and management
of meetings as well as in handling all the accounting of funds. Their
goal is to raise \$50,000 to \$75,000.

**XVII. The Next Step for DRG and the DRG Advisory Committee**

> **Dr. Donald Luecke**

Dr. Luecke spoke about the future organization of review into three
groups, each with about six IRGs [(Figure
2)](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/E75BFF5A-A730-4910-96B6-F3E633E47668/images/k165ppjb.gif).
Dr. Luecke then briefly described this organization and staff. With
respect to the DRG Advisory Committee, Dr. Luecke proposed expanding the
membership to 15 advisors to enhance discussions. In addition, he
envisioned bringing on four or five people on an ad hoc basis for each
meeting. Besides the two meetings a year, he proposed forming working
groups to report on such issues as clinical research.

A question was asked about the relationship of the DRG Advisory
Committee to the Peer Review Oversight Group (PROG). Dr. Luecke felt the
Advisory committee would carve out a role looking at the structure, the
function, and the ongoing evaluation of the DRG. Dr. Luecke explained
that PROG\'s role goes beyond DRG.

One suggestion was for PROG to be involved with issues related to
requests for applications (RFAs) and program announcements (PAs). Those
are Institute specific and are not well coordinated. PROG could have a
subcommittee that receives and coordinates information from various
scientific groups A good example is a recent National Cancer Institute
solicitation for research in neurofibromatosis. This is a very broad
disease that includes not only cancer but also learning disabilities,
and therefore crosses Institute boundaries. PROG could thus have a
broader view of not just the peer review process but also the impact of
new directions that the NIH is taking across Institute boundaries. PROG
should not be micromanaging the individual study sections or the IRGs.

To enhance communications between the DRG Advisory Committee and PROG,
in addition to the DRG Director being on PROG, the Chairman of the DRG
Advisory Committee should be on PROG, either as one of the extramural
members or as an ex officio member.

Another recommendation was related to the DRG goal of periodic review of
study sections within the Division. This could be one of the major
responsibilities of the Advisory Committee. Such a task requires careful
study, clear data and strong justification. The Advisory Committee could
oversee that effort, designing it carefully and having the appropriate
advisors. In this process of helping the DRG formulate a quality control
review mechanism, it might be valuable to have the Director and the
Chair of the Committee meet or be in communication on a more regular
basis and use that as a conduit for communications with the membership.
With e-mail and World Wide Web, they could do a fine job of keeping the
membership informed and also communicating with the scientific
community.

Another suggestion was to designate members of a working group of the
DRG Advisory Committee to a particular study section or group of study
sections. They would spend time sitting in study section meetings and
evaluating the process. One proposal was for a five- year review cycle.
Any greater frequency might be too chaotic, and there would not be
enough time to deal with all the study sections. The study sections
would be grouped by IRGs. The main intent would be to have a process
where the study sections and the IRGs are evaluated by the same
standards.

**XVIII. Closing Remarks**

Dr. Braciale thanked his colleagues for their sacrifices in coming here
to help out the DRG. He then thanked Dr. Luecke and the DRG staff for
their willingness and efforts to open up and examine the peer review
process with the Committee.

There being no further business to discuss, the meeting was adjourned at
10:30 a.m. on November 14, 1995.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes and attachments are accurate and complete.

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Samuel H. Joseloff, Ph.D.

Executive Secretary

Division of Research Grants

Advisory Committee

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Donald H. Luecke, M.D.

Acting Director

Division of Research Grants

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Thomas Braciale, M.D., Ph.D.

Chairperson,

Division of Research Grants

Advisory Committee

Note. These minutes will be formally considered by the Advisory
Committee at its next meeting on May 20 & 21, 1996; any corrections or
notations will be incorporated in the minutes at that meeting.

For questions or comments regarding the DRG Advisory Committee Meeting
Minutes, please contact Dr. Samuel Joseloff: <joselofs@drg.nih.gov>.

Last Update: April 18, 1996