## DRG Advisory Committee Minutes May 20 & 21, 1996

[I. Call to
Order](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#I%23I)\
Dr. Thomas Braciale

[II. Report of the Acting
Director](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#II%23II)\
Dr. Donald Luecke

[III. Approval of Minutes for November
meeting](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#III%23III)\
Dr. Thomas Braciale

[IV. Budget
Update](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#IV%23IV)\
Mr. Fred Wong

[V. Information and Analysis
Office](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#V%23V)\
Mr. John Jones\
Mr. James Tucker

[VI. Overview of the Advanced Technology
Branch](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#VI%23VI)\
Ms. Ellen Ring

[VII. Reinvention Status
Report](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#VII%23VII)\
Dr. Anthony Demsey

[VIII. 50 Years of NIH Peer Review: Plans for the
Celebration](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#VIII%23VIII)\
Dr. Suzanne Fisher

[IX. Patient-Oriented
Research](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#IX%23IX)\
Dr. Donald Luecke et. al.

[X. Peer Review Integration
Pilots](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#X%23X)\
Dr. Donna Dean

[XI. DRG Gift
Fund](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XI%23XI)\
Mr. John Jones

[XII. Study Section Trends: Changes in Voting Regulations for Ad Hoc
Reviewers](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XII%23XII)\
Dr. Patricia Straat

[XIII. DRG Home
Page](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XIII%23XIII)\
Mr. James Tucker\
Ms. Judy Baier

[XIV. Review of Study Section Scientific
Boundaries](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XIV%23XIV)\
Dr. Elliot Postow

[XV. Rating of Grant
Applications](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XV%23XV)\
Dr. Constance Atwell

[XVI. Just-in-Time
Concept](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XVI%23XVI)\
Dr. John Mathis

[XVII. Closing
Remarks](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/may96min.html#XVII%23XVII)\
Dr. Thomas Braciale

#### I. Call to Order

#### Dr. Thomas Braciale

Dr. Thomas Braciale, Chairperson of the DRG Advisory Committee, called
to order the 14th meeting of the DRG Advisory Committee. He thanked the
ad hoc advisors for joining the Committee for the meeting, as well as
his colleagues on the Advisory Committee for their continuing support in
their advisory capacity to the Division.

#### II. Report of the Acting Director

#### Dr. Donald Luecke

Dr. Luecke noted that the Division\'s experience with the Advisory
Committee has been an evolving process, and DRG is trying to utilize the
committee in a fuller capacity. He also mentioned that this was the
final meeting for Drs. Thomas Braciale, Roger Bakeman, and John
Starkweather, and thanked them very much for all they had done over the
past years. Next, Dr. Luecke introduced the 13 ad hoc advisors: Drs.
Lucy Anderson, David Center, Don Cleveland, Henry Friedman, Shirley
Hill, Ronald Hines, Ron Lindahl, Mary Moslen, Raymond Novak, Suzanne
Pfeffer, Maharaj Ticku, Judith Turgeon, and Keith Yamamoto. Recognizing
that many of these individuals are current or recent study section
members, Dr. Luecke expressed his gratitude for their time and
enthusiasm in attending the meeting.

He also was deeply grateful for the support shown by the DRG staff over
the last year. This past year has seen significant changes for the
Division, which put considerable stress on the staff. Without exception
the support was outstanding, and much of what they accomplished over the
last year could not have been done without the full cooperation of
everyone.

The government shutdowns, had a tremendous impact on the Division and
other components at NIH. As the major receipt point for grant
applications to NIH and other parts of the Public Health Service, DRG
had a tremendous problem. Thanks to the staff, DRG has caught up with
regard to incoming applications. However, DRG had to postpone
approximately 40 study section meetings. In some cases, study sections
were relocated; in other cases, workshops that were to be held out of
town were canceled.

Progress has been made on the search for a new director of DRG, and the
selection is imminent.\* Dr. Luecke also mentioned some personnel
changes at NIH and internally within DRG, including: Dr. Steven Hyman,
the new Director for the National Institute of Mental Health (NIMH); Dr.
Samuel Wilson, the new Deputy Director of the National Institute of
Environmental Health Sciences (NIEHS); Dr. Peggy McCardle, a former DRG
scientific review administrator (SRA), who is now a special assistant to
Dr. Wendy Baldwin, the NIH Deputy Director of Extramural Research; Dr.
Terry Hoffeld, who has returned to DRG to be the SRA of the Epidemiology
and Disease Control Study Section; Dr. Eugene Zimmerman, who has
returned to DRG to be the SRA of the Allergy and Immunology Study
Section; Mr. Jim Pike, DRG Executive Officer, who has retired (Mr. John
Jones, a former colleague of Dr. Luecke\'s from the National Institute
of Neurological Disorders and Stroke, who has been detailed to DRG as
Acting Executive Officer); Mr. Jim Lowrie, Chief of the Systems Analysis
Section in the former Information Systems Branch, who has retired; Dr.
Jerry Critz, SRA of the Physiological Chemistry, who has retired; and
Dr. Ed Zapolski, SRA of the Metallobiochemistry Study Section, who has
retired.

Dr. Luecke then had some general comments about the evolving role of the
advisory committee. He personally felt that the current size of the
committee \-- seven chartered members \-- is too small; and so, over the
last year and a half, he has brought in additional people for the
discussions. Over time, the Division has grown comfortable sharing ideas
with Committee members and ad hoc advisors. Dr. Luecke emphasized that
DRG management has actively sought their advice and at an early stage in
the decision process. Unlike other advisory committee meetings, the DRG
meeting is entirely open to the public. The minutes are available to the
public on the DRG Home Page.

In response to a question about any residual impacts of the shutdown on
DRG, Dr. Luecke noted some delays in the production of summary
statements, although the Institutes have been understanding and have not
complained. Dr. Demsey added that because of the delay in mailing out
summary statements, the receipt date for amended applications, which
would normally be July 1, is being pushed back to July 15th.

\*In June 1996, Dr. Elvera Ehrenfeld, Dean of Biological Sciences at the
University of California, Irvine, was selected as the new Director of
DRG.

#### III. Approval of Minutes for November meeting

#### Dr. Thomas Braciale

The minutes for the November 13 and 14, 1995, meeting were unanimously
approved.

#### IV. Budget Update

#### Mr. Fred Wong

Mr. Wong, the DRG Budget Officer, noted that the President\'s Fiscal
Year (FY) 1997 budget for NIH is \$12,406,300,000, which is a 3.9%
increase over the estimated FY 1996 budget. The top priority for NIH is
funding investigator-initiated research grants. The number of competing
research project grants would increase by 207, from 6,620 in FY 1996 to
6,827 in FY 1997. The second priority is funding for the replacement of
the current clinical center and its associated laboratories. To
accomplish this goal, \$310,000,000 is set aside in FY 1997.

To accommodate these increases, as well as increases in certain
scientific areas (biology of brain disorders, new approaches to
pathogenesis, new preventive strategies against disease, genetic
medicine, and advanced instrumentation and computers in medicine and
research), some parts of the budget had to be cut. Thus, the decision
was made to limit the direct cost of non-competing grants in future
years to two percent. (In the scientific areas listed above, for FY
1997, the Institute program areas would be responsible for an
\$87,000,000 increase, the NIH Director for a \$99,000,000 increase.)

In other budget matters, the level of full-time equivalents (FTEs) for
the NIH was 15,474 as of FY 1995. NIH will remain stable at that level
for FY 1996 and FY 1997.

There are also rescissions. The first rescission will be a one percent
cap on performance awards in FY 1996. A second will be a
Government-wide, \$500 million rescission for administration, personal
services contracts, and supplies.

One question pertained to the areas of scientific emphasis in the FY
1997 budget. Presumably, these will generate a number of new
applications. What proportion will be reviewed by Institute-based
groups? What proportion by DRG? Dr. Luecke responded that it was too
early to have any specifics, though a number of Institutes have begun to
put in place various plans. National Cancer Institute (NCI) and National
Institute of Allergy and Infectious Diseases (NIAID), for example, are
already moving toward more investigator-initiated RO1 awards. There will
be an increased burden with regard to review centrally; that is, DRG
will undoubtedly do more reviews than in the past, but the actual way in
which that sorts out is not clear at this stage. Dr. Luecke noted in
these particular scientific areas, an illustration that as science
progresses, it tends to encompass the interests and the missions of
multiple Institutes. This is in contradistinction to past trends of
earmarking funds for particular programs within Institutes.

In response to a question about the use of RFAs, which are generally
reviewed in the Institutes, for reviews in these special areas, Dr.
Luecke indicated the likelihood that, the bulk of the reviews would be
for investigator-initiated research. Applications would be in response
to program announcements and reviewed by DRG.

#### V. Information and Analysis Office

#### Mr. John Jones Mr. James Tucker

Mr. John Jones, Acting Executive Officer of DRG, provided some
background on reorganization within DRG. Included among the
recommendations that the Cassman Committee made to Dr. Varmus was that
the NIH-wide information systems responsibility of the DRG should be
transferred to the Office of Extramural Research in the NIH Office of
the Director. That transfer is now taking place. In the meantime, the
Division has reorganized to meet its own needs related to information
systems development and maintenance, as well as information
dissemination to the public. Thus DRG established the Information and
Analysis Office to provide for the information dissemination activities
specific to the Division and has also recommended the establishment of
the Advanced Technology Branch, to address the information systems
development and maintenance remaining with DRG.

Mr. James Tucker, Chief of the Information and Analysis Office, noted
that the Office\'s functions are comprised of many of the functions that
were formerly done in the Statistics Analysis and Evaluation Section and
that portion of the Grants Information Office that stayed within DRG.
The new office is responsible for: providing statistical analysis of
DRG\'s peer review activities; preparing and distributing materials
concerning the DRG peer review process; providing publications support
to the Division\'s professional staff, such as for the publication of
workshop proceedings; generating computer-based graphics and slides for
presentations and publications; maintaining the DRG Worldwide Web home
page; producing and disseminating a periodical entitled Peer Review
Notes; and doing various public relations functions, such as announcing
DRG workshops and preparing articles on DRG for the NIH Record.

The staff is comprised of people with many different areas of expertise.
Three of the staff are from the former Statistical Analysis and
Evaluation Section, one from the former Grants Information Office, one
from the Referral and Review Branch, and one from the former Research
Documentation Section.

#### VI. Overview of the Advanced Technology Branch

#### Ms. Ellen Ring

Ms. Ring gave a brief overview of the DRG Advanced Technology Branch,
whose responsibilities include establishing long-range plans, conducting
studies and analyses, providing end user support, and handling risk
assessment, security, and life cycle planning for all the data
processing systems in DRG. Services are also provided to the Office of
Extramural Research.

The Branch has four sections. The User Support Section contains the help
desk. If anyone in DRG has a computer problem, whether hardware or
software, they first call the help desk. This section also does all the
installation and trouble shooting of all the software, both on the local
area network (LAN) and on the personal computers (PCs). They support all
software and the environment for the DRG Worldwide Web server. They do
the training and the coordination of the training for DRG staff. The
second section, the LAN Management Section, takes care of all the
hardware within DRG. They support all the cabling within the Rockledge
II Building, which had previously been done by the Division of Computer
Resources and Technology. The third section, the Systems Analysis
Section, takes the requirements of users and puts them in some form of
specifications so they can be turned over to the programming group. In
addition, they evaluate the commercially available software. They
develop the systems requirements for security. Finally, they take care
of the data handling communications for the new phone system. The final
section, the Applications Development Section, includes programming
staff. They are also responsible for the DART system, which is used by
most SRAs to track their assignments.

Future activities within the Branch include remote access, which would
allow people to have off-site access to the LAN and dial in as though
they were attached to the LAN. They also have multiple access CD ROM
capability, which means that rather than buying everyone with a PC their
own CD ROM, the CD ROM was put on the LAN for multiple users. They are
also in the process of upgrading all PCs from Windows 3.1 to Windows 95,
and upgrading the DART system. In July, they will have a fax E-mail
capability, so that anyone from the outside can send faxes to people in
DRG. Finally, they are exploring the possibility of electronic receipt
of reviewers\' critiques, to allow reviewers to type on-line their
critiques or to send them as a file.

#### VII. Reinvention Status Report

#### Dr. Anthony Demsey

Dr. Demsey did not go into very much detail on the topic of the RGA
Report and Just-in-Time submission, since these topics would be
discussed later in the meeting. Dr. Demsey noted that the NIH Home Page
includes an entire compendium of ongoing activities, and these are
updated periodically.

In the areas of committee management policy, several changes will have a
great impact on the future of peer review. One of those has to do with
Howard Hughes Medical Institute (HHMI) investigators. Previously, HHMI
investigators were determined to be employees of HHMI, and consequently
were considered in conflict during the review of an application from
another HHMI investigator. Thus an HHMI investigator at Duke University
would have to leave the room during the review of an application from an
HHMI investigator at UC San Diego. Following a determination by legal
counsel, HHMI investigators are no longer considered in conflict with
each other, except based on campus affiliation. In other words, if a
member is an HHMI investigator at Duke University and a Duke University
application comes in, they are in conflict, but not simply by virtue of
their HHMI affiliation.

A second important modification to committee management procedures
relates to the presence on a study section of two members from the same
university. Previously a waiver was required. Now two members on the
same study section can be from the same institution, provided that they
are not in the same department and do not have a supervisor /employee
relationship.

The last change has to do with ad hoc members on study sections.
Historically, ad hoc members at study section meetings, did not have the
full voting rights of regular members. As a consequence, the NIH
Reviewers Reserve was created. Again on the basis of legal counsel\'s
advice, SRAs can now bring additional members on as voting members. The
reviewers will no longer be called ad hocs but rather \"temporary
members,\" and DRG will modify the charters of study sections to enable
the SRAs to appoint temporary members.

The next topic was scoring recalibration. The last time DRG recalibrated
(began the scoring process anew and wrote off all previous scoring
behavior) was in 1988. Since that time, the Division has noticed
\"priority score creep,\" where over a period of time, the scores of a
study section get increasingly better. In a few study sections, for
example, a two priority score point difference (122 to124) takes in five
percentiles (from the 15th to 20th percentile). So, it is difficult for
study sections to have a significant way to distinguish the quality of
all applications and give meaningful scores. As a consequence, many
study sections have asked for an \"open window,\" which is essentially
doing away with all scoring behavior in the past and scoring anew. For
the June cycle of study section meetings, percentiles will be based only
on that round\'s scores. For the October meetings, the percentile base
would be October plus the June scores. Then with the February study
sections meetings, it will be February plus the October plus the June
scores to form the base.

The last item discussed was amended applications. The Division is
developing a pilot study involving applications that need a very limited
amount of information for the study section to feel comfortable giving
them a final review. In the past, all amended applications came in
months later. Now, DRG plans to provide some applicants the opportunity
to send a 3-5 page response (the amended application) directly to the
SRA, who will bring it to the next study section meeting. Also a group
is looking at the issue of how many amendments should be allowed.

##### Discussion

What is the rationale for continuing to include past rounds in the base,
when many study sections have this incessant grade creep? Dr. Demsey
explained that the logic that NIH followed was that, from round to
round, study sections do not necessarily see the same quality of
applications. If NIH percentiled on a round-to-round basis, some
not-so-good applications would perhaps get funded above some the study
section would have thought better in the previous round.

One discussant noted that the problem in the push for calibration exists
only because study sections cluster their own scores. The study sections
have the solution in their own hands to prevent the problem. Also why
does the issue arise now? Why not settle the RGA recommendation first,
because then this would become a non-issue? Dr. Demsey responded that
the RGA committee\'s report is contentious, and DRG could not implement
anything for this June. It is preferable that such major changes in
scoring begin with the June meetings, because these applications go to
councils in October, and are the first ones considered for the following
fiscal year\'s funding. To avoid mixing different systems, NIH prefers
to do recalibrations beginning with the June study section meetings.

In reply to a question of whether, in the upcoming pilot study, amended
applications would be scored during the initial review, Dr. Demsey said
they would be scored, so that the Institute would have the opportunity
to fund the application even at that point before the 3-5 page amendment
is received.

The next question was whether investigators submitting amended
applications would be informed about the recalibration of priority
scores. For example, after having received a 150, they might suddenly
get a 210, and they might feel egregiously harmed. Dr. Demsey stated
that DRG would make use of its Home Page on the Worldwide Web, and, in
fact, the recalibration document is already on the DRG Home Page. But
more importantly, every summary statement will contain a notation
indicating to the applicant that DRG is using a new calibration
procedure, and the priority score and its corresponding percentile may
be quite a bit different from those on a previous application.

One discussant remarked that changing the status of former ad hoc
reviewers might markedly change the influence that temporary members
have on the scoring system. In any study section, as much as one third
of the group could be temporary members, depending upon the necessary
expertise required. They would have no history of the study section
itself, but could influence the process through their votes. Dr. Demsey
admitted that DRG staff are concerned about the percentage of regular
and temporary members, and have not yet worked out specific
implementation procedures. The Scientific Review Administrator (SRA) is
going to be critical in orienting temporary members to study section
activities and behavior, and setting a reasonable median, at least for
the June round.

In response to a question about any intent to substitute temporary
committee members for other members from the Initial Review Group (IRG),
Dr. Demsey remarked that DRG managers are looking at various kinds of
flexible memberships for IRGs and for study sections. They are trying to
provide the SRAs with a full armament of ways to bring people on, but
this is in no way an attempt to diminish the importance of regular study
section members or of keeping a full membership.

Who decides when to request a 3-5 page amended application versus a full
blown resubmission of the application? Dr. Demsey responded that the
study section members would determine this during the review.

#### VIII. 50 Years of NIH Peer Review: Plans for the Celebration

#### Dr. Suzanne Fisher

Dr. Fisher noted that the symposium on June 20 includes current and past
NIH staff as well as scientists. The speakers have served on a variety
of study sections: Virology, Biophysical Chemistry, Molecular Biology,
Experimental Virology, Biochemistry, Bacteriology and Mycology,
Genetics, Medical Biochemistry, Metabolism, Molecular Cytology,
Pathology A, and Pathobiochemistry. The speakers have also served on
Institute review panels, the NIH Reviewers Reserve, and various Advisory
Councils and Boards. Their time of service ranges from 1973 to
appointments extending to FY 2000.

An important part of the symposium will be the afternoon segment, which
will be the major audience participation for questions and comments
about the peer review process. NIH staff who attend will get continuing
education credit. After the full day, participants can relax at a wine
and cheese reception at the NIH Natcher Building.

Dr. Fisher then thanked the Friends of DRG group, especially Dr. Murray
Goldstein and Dr. John Sherman, who have worked very hard to do the fund
raising necessary to have this event, as well as the Association of
American Medical Colleges, which is providing a tremendous amount of
logistical support.

On Friday, June 21st, the Division will have other events back at
Rockledge. These events are planned for present and former staff, and
will be part of a more casual, personal day. The second day will include
a group photograph, a picnic, an awards ceremony, a formal installation
of the pictures of all the directors of DRG, and a series of skits and
performances of work related humor. For the alumni, it will be a chance
to see that life is different than at Westwood. It will be a very
interesting day for people to reminisce about the people who have been
part of DRG.

#### IX. Patient-Oriented Research

#### Dr. Donald Luecke et. al.

Dr. Luecke referred to a report on patient-oriented research prepared by
a working group of the DRG Advisory Committee chaired by Dr. Gordon
Williams. The Williams group concluded from their data, that in those
study sections where there was some concentration of clinical-type
applications, the clinical applications seemed to do somewhat better, as
judged by the percentile rank. It was also noted, that while clinical
applications did not fare as well as basic research applications, the
amendments to clinical applications generally did as well as amended
applications that were not patient oriented. Could the observable
differences in original scores between these two groups of applicants be
more closely related to training and/or preparation practices rather
than to misunderstanding or bias in the review process? While
acknowledging that the study could not answer this question, a majority
of the Williams\' group nonetheless felt that a disappointing initial
experience might discourage some clinically-trained investigators from
submitting an amended application.

After the Williams report, the Division began a tracking system for
clinical applications. Because of the complexity of trying to develop
this tracking system, the Division formed a focus group with the
Oncological Sciences Initial Review Group in DRG. Dr. Luecke noted that
in the focus group, the Division tried to involve not only DRG staff,
but also program staff from the Institutes.

Dr. Bahor also provided some background, noting that in addition to the
Gordon Williams Committee Report, there was recently a report of the
American Society for Clinical Oncology, which identified some of the
same problems in the review, funding, and training of researchers in
clinical oncology. Their definition of clinical oncology, which proved
to be useful, was \"hypothesis-driven research that employs measurements
in whole patients or normal human subjects in conjunction with
laboratory measurements as appropriate on the subjects of clinical
biology, natural history, prevention, screening, diagnosis, therapy or
epidemiology of neoplastic disease.\"

Ms. Diane Bronzert, from NCI, who participated in the oncology focus
group, then provided an overview of the clinical research programs
within NCI, followed by a discussion of the review of clinical oncology
research grant applications in DRG. NCI has been trying to increase the
funding for patient-oriented research. Dr. Klausner, Director of NCI,
transferred considerable funds from other mechanisms into the research
project grant pool this year. As a result, the NCI pay line increased
from 15 percent last year to 23 percent this year. In addition, the
Institute has established an accelerated review process for new RO1
grant applications. When investigators within acertain score range are
sent their summary statements, they have the option of addressing
concerns expressed in the summary statement in a 3-5 page brief. This
statement is then forwarded to the NCI executive committee, along with
comments from program staff, to determine how well the investigators
have addressed the issues. To evaluate this new process, the NCI is
trying to track patient-oriented research that has been supported
following these new procedures.

Ms. Bronzert then summarized some recent reports on the review of
clinical oncology research, from the National Cancer Advisory Board, the
National Coalition on Cancer, and the American Society of Clinical
Oncology. Improved training of clinical investigators was one important
recommendation. Another recommendation is finding mentors for clinical
investigators. One of the biggest problems is the perception of the
clinical oncology research community that their research grant
applications are not seen as competitive by NIH. NCI does not receive
enough applications, because clinical researchers feel that they will
have little chance of success unless in response to an RFA.

With respect to study section assignment and review, some clinical
investigators feel too few clinicians are on study sections reviewing
clinical applications. The quality of clinical oncology grant
applications is also a problem, because quite a few investigators,
especially the new ones, are inexperienced in writing applications.

Problem areas in clinical trials research grants include: the difficulty
in conducting a well-controlled clinical study; the difficulty in
obtaining tissue for laboratory studies; limited access to study
populations; lack of sufficient preliminary data; long review process,
which discourages amended applications; clinical research not being
considered innovative; and the high cost of clinical research.

Dr. Henry Friedman, an assigned discussant, felt it was critical to
separate clinical or patient oriented research into two discrete areas:
(1) phase I or phase II trials, and (2) translational research, where
there is a major laboratory component. The angst that has been generated
by many groups of researchers reflects the fact that people are not
looking at the distinctions within this kind of research. Clinical phase
I or phase II trials can be reviewed very effectively by clinical
investigators experienced in clinical investigations. The RFA mechanism
has often been utilized for review of these applications.

But if those phase I or phase II trials are reviewed by current study
sections, for all the reasons that have been detailed above, they often
fare poorly. There is minimal science frequently in these applications,
which are poorly written. Translational research, where there is a major
scientific component, should be reviewed by experienced scientists, some
but by no means all, of whom need to be clinical investigators. The
objective is to have a cohort of investigator scientists who are
familiar with clinical translation, who can then review the application
the same way they would review a very basic application. In his opinion,
the Experimental Therapeutics-2 Study Section, on which Dr. Friedman has
served, provides a fair and favorable review for translational research.

Dr. David Center, the next discussant, stated that while it is desirable
to stimulate more people to stay in clinical research, people need to do
good research. When NCI funds grants, how does the Institute track them
to determine whether the phase I or phase II trials have been productive
and meaningful? Dr. Center\'s personal experience and observations are
that these types of grants are high risk by their very nature. A very
low percentage of them come out with information that is valuable. That
is the ultimate dilemma in a review process. How does one instruct a
review panel on how to grade what will likely be a very high risk grant,
and to guide the Institute in what would be reasonable and what
unreasonable to fund? Times are tough; money is tight with not much
increase in budget expected. So, reviewers want to recommend projects
that have a higher not a lower probability of success.

Ms. Bronzert noted that NCI is gathering data based on publications. If
there is a follow-up trial, does it lead to a phase II? NCI is trying to
determine the result of research. For example, is it successful if the
drug doesn\'t work but the investigator then develops a similar drug
that works? They are also trying to see if investigators are staying in
research.

Dr. Bakeman, another assigned discussant, wondered whether there is
evidence that clinical trial applications are somehow faring badly
because they are reviewed in the wrong study sections? It is difficult
to track success, not in terms of who gets funded, but in terms of
results, and in figuring out which are the appropriate mechanisms. Dr.
Bakeman felt that patient oriented research is often handled better by
other mechanisms than RO1s.

Dr. Jonasson, the final assigned discussant, wondered if a re-analysis
of clinical research in this context might show that phase I and phase
II success rates are so low that it pulls down and misleads the success
of translational research, which is probably fairly high. Especially in
the era of managed care, research centers simply cannot afford to
support so many of the necessary tests that are associated with phase I
and phase II trials. This is certainly not something that should be
funded out of research monies, but perhaps out of development monies.

Dr. Marcia Litwack, SRA of the Experimental Therapeutics-2 (ET-2) Study
Section, noted that her Study Section, was established to review
translational research. The proposed research that the Study Section
reviews is in the later preclinical stages, ready for translation into
the clinic. Depending on the quality of the application, these have a
fairly good success rate. Much of it is not fundamental or basic
research, but applied research. In preparing reviews that are meant to
be instructive to applicants, who very often lack mentoring at their own
institutions, a major problem, particularly with first-time applicants
but even with established senior investigators, is the poor quality of
the applications.

One major problem is that MDs in managed care settings are often only
given one chance to apply for a grant. If the MD researchers do not get
that grant, they are told to attend to patients and not do research.
Another problem is with clinical investigators who submit a meritorious
project that does not get funded and then become discouraged. People at
their institution tell the investigators that because the research is
patient-oriented, it will not get funded. Do not bother resubmitting.
Dr. Litwack and some program staff have worked very hard to counter this
negative perception and have managed to get some people to resubmit. One
individual whom they convinced to reapply last time got a grant. But it
is hard to overcome that perception.

Dr. Friedman commented that there are two problems; one the group can
solve and one thegroup can not. The problem that can be solved is the
appropriate way to review clinical or patient-oriented research and
improving the mentorship. The problem that cannot be solved is that
clinical research is not being adequately supported financially.

Dr. Braciale ended with two comments. First, the best of both laboratory
as well as patient oriented research will, by definition, have a very
high failure rate, because that is the nature of experimental research.
Second, a high-quality, critical review of an application, which points
out the problems with a trial or with an applied research program, only
stops the research if the Institute program staff adhere strictly to a
priority score in considering payment. Unless the Institutes are willing
to see the value of a clinical project independent of the amount of
scientific information that can be extracted, it is impossible to
effectively deal with the problem at DRG\'s level.

#### X. Peer Review Integration Pilots

#### Dr. Donna Dean

Dr. Dean noted that the peer review integration pilot project, from the
DRG perspective, had two major foci: (1) to proceed with the orderly
integration of review activities from the Institutes, specifically
National Institute on Alcohol Abuse and Alcoholism (NIAAA); and (2) to
demonstrate a process by which the Division, within an initial review
group\'s purview, could examine its review of scientific areas. From
NIAAA\'s perspective, a very strong objective was complete integration
of the appropriate review activities from NIAAA by summer 1996.

The areas of integration activity basically fell into four major areas:
biomedical research on alcohol and other xenobiotics, community
prevention and control, AIDS and related research, and the Small
Business Innovation Research (SBIR) program. DRG already had an initial
review group dedicated solely to AIDS and related research, whose
inherent flexibility allowed the smooth integration of AIDS applications
from NIAAA into DRG with only minimal adjustments. DRG\'s review of SBIR
applications for NIAAA had been accomplished before the time period of
this pilot.

The remaining two areas were far more complex. In the biomedical
research area of alcohol and other xenobiotics, NIAAA had two
subcommittees. Two toxicology study sections represented the most
reasonable focus for interaction within DRG. Within the area of
community prevention and control, NIAAA had an Epidemiology and
Prevention Subcommittee, but within DRG\'s Biobehavioral and Social
Sciences IRG, there was an evolving ad hoc review group focusing on
these same issues.

The pilot study had four phases: (1) planning and identification of some
of the key issues; (2)implementation; (3) adjustments and stabilization;
and (4) formal establishment of new study sections within DRG. In phase
I, there were briefings to both the NIAAA and the NIEHS advisory
councils in September 1995. The Institute staff had already begun
preliminary work at the June 1995 review and council meetings to lay a
foundation for the integration work. In addition, NIAAA staff had worked
with officers and members of the Research Society on Alcoholism.

Phase II, implementation, began when the four study sections met in
October. Pilot study working groups in the biomedical research area and
in community prevention research involved not only NIAAA staff but also
NCI and NIEHS staff. Members and the chairpersons of the involved study
sections met informally.

The major work occurred at a December 6 pilot study group meeting,
wherein working groups from each study section presented preliminary
materials identifying new review configurations. At the end of the day,
four completely new study sections were created \-- Alcohol and
Toxicology 1-4 (ALTX 1-4),\-- and draft referral guidelines were
developed. Members were also distributed among the new study sections.

While the Federal shutdown and the East Coast blizzard led to 5 weeks of
lost time and rescheduling of reviews, the four new Alcohol and
Toxicology Study Sections met in February in Bethesda in the same hotel
at the same time. There was a joint orientation meeting. Senior staff
from both DRG and NIAAA monitored the review meetings. Each review panel
was also extensively debriefed at the end of its meeting. Reviewers were
given questionnaires as part of the evaluation. Staff from NIAAA and DRG
met subsequently to analyze the questionnaires, and share impressions of
the meetings. The questionnaires had been completed by 61 of the 75
reviewers, with the vast majority indicating that the appropriate
expertise was on each of the four ALTX review committees. About 60
percent of respondents felt their ability to follow and contribute to
the discussion was about the same as before. Seventy-two percent
strongly agreed that their comments were treated with respect, and 80
percent that their committee made appropriate recommendations. Overall,
75 percent of the reviewers thought that the committee worked fairly
well.

Some areas of concern were identified, such as heavy metal toxicology,
which will require further attention and adjustments in phase III,
analyzing and refining work, in preparation for the establishment of new
study sections in phase IV.

Dr. Lindahl, the first assigned discussant, felt that the process worked
so well because everybody did their homework ahead of time, thinking not
only about alcohol or toxicology research, but also about how to best
structure the review groups with the expertise needed. Group dynamics
were superb, and having study section chairpersons rotate through each
meeting helped. At some early critical time in the first day, a
non-alcohol person reviewed an alcohol grant as a primary reviewer, and
vice versa, contributing to intergroup dialogue and understanding.

Dr. Ticku, the next discussant, noted that the reviewers were initially
nervous, but as discussions followed, the groups process went smoothly.
There were philosophical disagreements, but this happens routinely on
every study section.

The main concern of Dr. Anderson, the next discussant, was the sudden
disruption in previous group dynamics, in creating four new entities.
However, new group dynamics were not a problem, as the people selected
for these committees had the right attitude toward their peer review
responsibilities. While logistical preparations for the meeting were
compromised for a variety of reasons (related mainly to the snow storm
and the furloughs), no application received less of a fair and expert
review.

Dr. Hines, the next discussant, noted that because of the science area
overlap, the four new study sections have a greater focus of expertise
than the four previous ones. Dr. Hines was concerned about whether it
would be possible in the future to have all four study sections meet at
the same time. Dr. Hines\' final comment pertained to communication,
that DRG needs to communicate effectively the goals and study section
composition to the outside community to alleviate concerns.

Dr. Novak, the next discussant, thought the process went well, but felt
DRG and NIAAA staff need to guard against the science coverage on any
panel becoming too narrowly focused.

Dr. Moslen, the next discussant, stated that her experience overall was
very positive. However, there were lingering concerns among researchers
in the areas of heavy metal, lung, and kidney toxicities about the
appropriateness of assignments.

##### Discussion

In addressing areas of needed expertise, collaboration within IRGs and
among IRGs could be further explored, particularly in the sharing of
reviewers.

Dr. Lindahl indicated that the December 6 meeting was very important.
NIH staff brought in people with seniority and experience on the current
study sections, and told the participants to throw out all
preconceptions. Dr. Anderson added that the reason that the December 6
meeting worked quite well is that for each study section, several people
had given much thought ahead of time to the science, and the workload
balance.

In addressing the question of how four study sections resulted instead
of three or five, Dr. Hines responded that the workload and different
clusters of science led naturally to these groupings. Dr. Dean added
that NIH staff made the implicit assumption that four new study sections
would result unless something really dramatic happened in December. DRG
remains flexible, but for now, four seems reasonable, the best fit at
this particular point, and less controversial to the community at large.

Noting that this experiment has provided DRG with an opportunity to ask
reviewers how well they think the review process has worked, one person
wondered whether DRG had asked the question of other study sections in
recent times. Dr. Luecke responded that DRG has interviewed chairpersons
as they retire from study sections, which has proven to be very helpful.
DRG has not collected data from study sections in a regular or
systematic way, but operated in a collegial mode, in a fairly informal
way with people, so that substantive suggestions are obtained. The role
that reviewers play is critical in terms of allaying the fears that
exist in the outside community as NIH begins revamping with a system
that, for many, is their life support.

#### XI. DRG Gift Fund

#### Mr. John Jones

At its last meeting, the DRG Advisory Committee recommended that the NIH
consider accepting gifts from outside sources to support activities of
the DRG, particularly to permit the SRAs to travel to various scientific
meetings, Mr. John Jones, Acting DRG Executive Officer, explained that
as a result of discussions with the NIH legal counsel, under certain
circumstances DRG can now accept money from outside individuals and
organizations to support the travel of SRAs. As long as there is no
conflict of interest involved and the concept of fairness is applied
appropriately in the circumstances surrounding the acceptance and the
use of the funds, such a gift fund can be accepted. With the gift fund
mechanism, DRG can accept, from outside sources, money for any purpose
that appropriated money can be used. The funds must be unconditional,
that is, the donor must not put any restrictions on the use of the
money. In addition, the gift fund is what is called \"no year\" money.
That is, the money is available to the Division until they use it.
Unlike appropriated monies, it does not lapse at the end of the fiscal
year. DRG employees do not have the authority to solicit monies into the
gift fund account, but they can let people know that the gift fund
exists.

To ensure that there are adequate safeguards against actual or even the
appearance of conflicts of interest, the Division has established
certain policies and procedures. The Director of DRG is the only person
in the organization who can accept monies and approve the use of the
funds. Before funds are accepted, there must be a review process by
senior staff in the Division to ensure that there is no conflict of
interest and that the concept of fairness is applied appropriately.
Finally, senior division staff in the organization will not be allowed
to use gift fund money for their own travel needs.

##### Discussion

The point was made that an unrestricted gift is unlikely to be
attractive to most societies or professional organizations. Societies
and professional organizations would, however, invite and pay for the
attendance of the specific SRAs relevant to their areas of interest.

Also, what is meant by fairness? In response, Mr. Jones gave an example
of a university inviting an SRA to come to its facilities to talk about
the NIH peer review process. If that organization were the only one
represented at the seminar, it could be perceived that that organization
was gaining an unfair advantage. However, if representatives from
several organizations that apply for grant support were going to be in
attendance, it would be perceived to be fair.

Another commentator noted the lack of any detailed description of the
process of deciding what applications would be supported. Mr. Jones
noted that some of the issues to be dealt with are still evolving. The
Division plans to develop such criteria.

One discussant was disturbed by the example given for fairness. It would
presumably benefit NIH, the review process, and the institution to have
an SRA travel to a university to assist investigators. In reality, the
NIH is trying to promote research excellence. Having an SRA visit an
institution, and that institution pay for the visit, would be of great
benefit to the researchers and would promote excellence.

Dr. Leucke responded that if the Division had ample resources, many SRAs
could be sent out to fulfill this desired goal. But with finite
resources, it has to be a judgement call. Thus, if a presentation is
really in lieu of a departmental seminar in a particular institution and
the only people attending that are from that department, that
qualitatively seems different, and less fair, that if other departments
are involved or perhaps other entities outside of the local geographic
area. Not just in DRG, but across the NIH, management is trying to make
sure that when SRAs and other staff go out, there is an opportunity for
others who may not be directly affiliated with that institution or
department to receive the information.

The next question was about whether the Friends of DRG are restricted
from collecting money in support of DRG, once the 50th celebration is
over? Dr. Luecke noted that the mission of the Friends of DRG is solely
to support the 50th celebration. Whether they might be willing to help
DRG later on is an open question.

One discussant wondered whether a single large donor, for example, the
Howard Hughes Institute, might subsequently influence the review
process. Mr. Jones responded that funds would be accepted only after
very careful scrutiny to avoid even the appearance of conflict of
interest.

Dr. Luecke remarked that there are several objectives. One is that NIH
as a whole needs to extend itself into the community more. To some
extent, DRG has been beginning to do some of that. But another important
element is the continuing professional development of the SRAs in the
emerging areas of science. It would not be simply a democratic system,
where the Division would pick SRAs because they had no travel in the
last x number of months. Also, it is not just a means of trying to
substitute for what the budget will not cover. Rather, this would help
the community and, at the same time, help the SRAs do a better job.

Dr. Braciale commented that the major issue is the enhancement of the
SRA in order to enhance the peer review process. The Committee needs to
find out for its next meeting what the available mechanisms are to
support SRA enhancement, both targeted and unrestricted. Secondly,
senior staff should come up with an initial plan with limitations on
this kind of funding, as a starting point to build around and explore.

A final comment was that when DRG develops this plan, to protect the
existing travel funds, the Division should try to think about areas not
covered in current travel plans. The funds could then be complimentary
to the present budget.

#### XII. Study Section Trends: Changes in Voting Regulations for Ad Hoc Reviewers

#### Dr. Patricia Straat

Dr. Straat explained the development of a recent ruling to permit ad hoc
reviewers to vote and assign scores at chartered or standing study
section meetings. Until about 1984, all members of a review committee,
whether chartered or ad hoc, were allowed to make motions, vote, and
assign priority scores. Then in 1984, a reinterpretation of the Federal
Advisory Committee Act by NIH legal counsel ruled that voting by ad hoc
reviewers was contrary to the spirit of the Federal Advisory Committee
Act because study sections had published rosters that did not include ad
hoc reviewers. In addition, the ad hoc reviewers did not undergo the
rigorous screening, mostly involving financial disclosures and possible
conflicts of interest, that was required for regular members. Still
another factor was the observation that in some study sections there was
a preponderance of ad hoc reviewers, and some of them were used over and
over again.

Problems arose from this decision. The declination rate increased as
potential ad hoc reviewers refused invitations to participate without
voting privileges. Even those ad hoc reviewers who attended the meeting
felt that they were being treated unfairly. Since they did the same
work, why shouldn\'t they be allowed to vote?

To circumvent these problems, the NIH Reviewer Reserve (NRR) was created
in about 1987. Reserve members, who could vote, were selected from
retiring study section members and others nominated by NIH staff. One of
the rules for NRR members was that the right to vote required
participation in the entire study section meeting.

At some later point, there was a ruling that ad hoc committees were not
permitted under the Federal Advisory Committee Act. Since all committees
had to be chartered, Special Emphasis Panels were created in 1996. These
were technically chartered committees, but they had no named membership
and there was no requirement for a formal clearance process.
Functionally, the Special Emphasis Panel was a mechanism to permit use
of ad hoc reviewers to serve on panels and vote, just as members of any
other chartered review committee.

However, it was difficult to understand why an individual ad hoc
reviewer could not vote when participating on a standing study section
but could vote when participating in a review group consisting of all ad
hoc reviewers, especially when both groups were chartered. A working
group was convened to consider these inconsistencies and recommend a
more rational policy that would permit full participation of all experts
on a review panel. The committee came up with two recommendations:
first, to allow all reviewers to vote, provided that they fully
participated in the review of the application under consideration; and
second, to discontinue the NRR. The current NRR membership would be
merged with the NIH Consultant File, and no further appointments would
be made. To comply with GSA regulations that only members can vote, and
with advice from the Office of General Counsel, charters of review
committees are being modified to allow the appointment of temporary
members (formerly called ad hoc reviewers).

The need for temporary members is not likely to be eliminated. Temporary
reviewers are needed to substitute in the event that a regular member is
absent. There is also the need to try out candidates for permanent
membership. With the increased scientific diversity on study sections,
SRAs need to bring in people with special expertise in new techniques
and procedures. Finally, with the ever-increasing workload, additional
reviewers are often needed to help handle the higher workloads.

Dr. Straat then provided data on temporary reviewers. From FY 1990 to FY
1995, the total number of applications reviewed by DRG has increased
about 20 percent. The number of applications reviewed in standing study
sections has actually remained relatively constant over the 6 years;
however, the number of applications reviewed by special emphasis panels,
or ad hoc committees, has shown a rather dramatic increase of about 60
percent. In FY 1995, there were 1,057 meetings, of which 285 were
standing study section meetings, and 772 were special emphasis panels.

Dividing the total number of applications reviewed by standing study
sections by the number of meetings, one finds over the past 6 years an
average of 65 applications per standing study section meeting. In
addition, the data show an average of 12 to 13 member- reviewers per
study section, and an average of 5 or 6 ad hoc reviewers per meeting.
This means that, on average, about 31 percent of all reviewers attending
study section meetings were ad hoc reviewers. These average figures are
surprisingly constant over time, and most DRG study sections have ranged
between 10 and 40 percent ad hoc reviewers. Thus, the impact of voting
privileges for temporary members could be significant. And regardless of
whether temporary members are present for the review of all applications
or only one application, they will all be listed on the roster,
identified by an asterisk.

One issue to be discussed is the time commitment of a temporary member.
To maintain continuity and dynamics, temporary members should be
encouraged to attend the entire meeting; but it may not be practical to
make this a requirement. The DRG needs to consider what should be the
minimum participation.

Another issue is the frequency of attendance at the same study section
meeting. The new regulations indicate that if the same individual serves
more than three times within a two-year period on a given committee,
that individual should be considered for membership.

Should the Division limit the percent participation of temporary members
and, if so, what is a reasonable percent? Clearly, if 50 percent or more
of the members are temporary, the impact will be much greater than if it
is only 10 percent. Apparently a new document has just become available
proposing a 50 percent limit for temporary members.

Finally, there are tracking considerations. Temporary members, according
to the document, can vote provided they fully participate. SRAs will be
responsible for monitoring the level of participation by temporary
members and determining that they satisfy the criteria for full
participation whenever reviewers vote or assign a score. There will also
be tracking requirements for temporary members with regard to geographic
areas, minority status, women, and the appropriate balance of review
perspectives.

##### Discussion

In response to a question about what percentage of ad hocs become
regular members, Dr. Luecke noted that the standard practice is for the
SRAs to use people in an ad hoc capacity prior to membership.

One discussant noted that one criterion for full participation of
temporary members is having read the applications. However, when review
groups have about 100 applications per meeting, it is not possible to
read all the applications. Dr. Luecke responded that certainly for most
reviewers that it is impossible to read all the applications. His
understanding of this concept is that full participation would include
applications they read and applications where they participated in the
discussion.

Dr. Braciale noted that the situation seems in flux, and he suggested
that the Avisory Committee deal with this issue again at their next
meeting in November.

In response to a question of why some study sections need so many ad hoc
reviewers, Dr. Straat noted that perhaps this was a reflection of the
diversity of the group. Dr. Straat added that the Division should look
at whether the same study sections repeatedly have high percentages of
ad hoc reviewers or whether it happens only occasionally within any
given study section.

#### XIII. DRG Home Page

#### Mr. James Tucker Ms. Judy Baier

Mr. Tucker explained that under the leadership of Ms. Baier, DRG
recently did a major renovation of its Home Page. Mr. Tucker and Ms.
Baier then demonstrated the information currently available on the DRG
Home Page: late breaking news items; general information about DRG,
including the phone listing of DRG staff; news and events (the DRG
Advisory Committee agenda and membership, the minutes from the November
1995 Advisory Committee meeting); the revised receipt date for amended
applications; the latest issue of the Peer Review Notes; information on
the DRG 50th anniversary celebration; publications (NIH Extramural
Trends and Women in NIH Extramural Grant Programs); referral and review
(dates to remember, review process changes, study section rosters,
workshop reports); and links to other Internet resources.

As of April 1, 1996, the Home Page had been used 27,642 times, and usage
has been increasing each month. The daily average was 921 information
requests. The most popular items were the phone directory, referral and
review, general information on DRG, and news and events. While the vast
majority of users were from the United States, there were users from 14
other countries around the world.

To help in the development of its Home Page, the Division established an
internal advisory committee. This committee\'s charge is to make sure
that information is rapidly and accurately disseminated to the public,
and also to provide DRG staff with an efficient means of information
exchange with NIH and the extramural community.

One future issue concerning the web is the archiving of information. A
tremendous amount of information is available. How long should the
information remain on the Home Page before being archived?

Another issue is the elimination of duplication within NIH. DRG is also
looking at ways to automate the updating of information. They would like
to increase the use of interactive documents for the exchange of
information. Finally, the Division needs to look at security issues. If
information is sensitive, it must be protected, so that only the
appropriate individuals can see it.

#### XIV. Review of Study Section Scientific Boundaries

#### Dr. Elliot Postow

Dr. Postow discussed the efforts of the Division to ensure that study
sections have the appropriate breadth and depth so that applications are
reviewed fairly and comprehensively. DRG addressed this problem during a
Survey of Science meeting, which took place in September of 1995 and was
co-chaired by Dr. Donna Dean and Dr. Keith Yamamoto. That meeting led to
the recommendation that the Division review its study sections on a
periodic basis. The review guidelines and competency rosters of the
study sections would be evaluated to ensure that they are appropriate
for the review of applications being submitted to the NIH.

DRG is proposing a one and a half day meeting to look at four issues.
Are the scientific boundaries of the IRG, i.e., the referral guidelines,
appropriate? Is the body of science that comprises the IRG divided into
logical units (study sections), and are these logical units too broadly
or too narrowly defined? Are the study section members representative
of, and expert in, all of the areas of science that the study section
includes? Finally, are the names of the study sections logical and
appropriate?

The Division is recommending a pilot program with the Endocrinology IRG.
This IRG is being proposed for several reasons. They participated in the
Survey of Science meeting, and therefore have some experience with the
process. It is a relatively small IRG, with four SRAs and five study
sections, and covers a discrete area of science. There is a high degree
of overlap among the study sections within the IRG. Finally, the SRAs in
this IRG are all experienced.

The Division is undecided about whether to provide the committee with
abstracts of grant applications that recently have been reviewed by the
Endocrinology Study Sections. DRG envisions the meeting to include
members of the scientific community as well as program officials at the
NIH Institutes. They do not want observers, because it would change the
dynamics of the meeting. The Division wants specific recommendations.

Dr. Cleveland, the first discussant, agreed that a review of how DRG
structures the overall reviews is long overdue, as the boundaries have
changed, and there are areas of out-moded approaches. As to whether
study section titles should be descriptive or non-descriptive, he
thought the community cared less about the title and more about the
reviewers\' expertise. Descriptive names quickly become non-descriptive.
In general, Dr. Cleveland felt this re-evaluation would benefit most
investigators.

Dr. Turgeon, the next assigned discussant, noted that if DRG bases its
decisions only on a historical aspect, there may be a problem,
especially in areas where the boundaries have changed greatly. Dr.
Turgeon agreed that Endocrinology is a manageable unit that has a lot of
problems that could be addressed by such a pilot program.

Dr. Yamamoto, the next discussant, also agreed with starting with
Endocrinology, but he had a concern with the attendees. If current study
section members and the SRAs of those study sections are primary
evaluators, there will be a very strong force toward conservatism.
Therefore, Dr. Yamamoto recommended selecting respected scientists whose
work crosses into that area, perhaps past chairpersons or outstanding
members of those study sections, as well as others who have not been on
the study section, but are recognized experts in the field.

He also stated that it would be valuable to provide participants copies
of summary statements. The most useful summary statements would be for
applications that did extremely well in the study section. That would
provide a view of what that study section considers to be the top work
in the field. A few summary statements of applications that just missed
the pay line or had a score of about 200 would give a view of what the
study section considers to be middle of the road work in that area.
Those kinds of materials would allow an evaluating group that has a
broad perspective to make a clearer statement about whether that panel
has the appropriate expertise.

##### Discussion

Dr. Theil suggested the selection of applications with a large number of
collaborative efforts in areas that are traditionally outside the area
of endocrinology. It would be useful to get the abstracts of other NIH
grants of the principal investigator and collaborators: (1) to see how
these are inter-digitated with the proposed research under review; and
(2) to evaluate how the collaborations impact on the progress of
science. Then DRG staff when deciding how to structure future panels
will have a breadth of information.

Another suggestion would be to have two separate meetings: the first,
with members of the Endocrinology Study Sections to review the current
status of those committees; the second, an independent evaluation along
the lines of Dr. Yamamoto\'s recommendations. Another person agreed that
it would be very useful to have a mix of persons who have a long history
with the group as well as newcomers who have no particular bias about
what should be in their particular group.

But another discussant felt it was important to have a meeting with some
of the current members, perhaps the chairpersons and one other member
from each group, as well as individuals who were chairpersons or members
three or four years ago to look at what has happened in the period of
time since they have left. The ideas of providing the summary statements
is a good one, but the abstracts are also useful.

What do DRG staff think will come out of the meeting? Will there be
major revisions or is the science in such flux that there will not
really be restructuring of the study sections? Dr. Postow responded that
the IRGs will probably remain, but hopefully, DRG will develop a more
logical organization of the responsibilities of study sections. In a
system that has been running for 50 years with little mid-course
correction, there have to be some weaknesses, and Dr. Postow hoped to
uncover those weaknesses.

Dr. Dean added that areas that probably need further examination include
reproductive biology endocrinology, cancer biology endocrinology, and
receptor biochemistry, i.e., structural biology. Another more
transcending issue is how to deal with basic science, translational
research, and the clinical implications of science.

The comment was made that the Division is really dealing with \"customer
satisfaction.\" The customers are the grantees. So, some review of the
recommendations of the in-house group by an external group would add
another dimension to this study. Dr. Postow noted that there are two
groups of \"customers\": the Institutes, as represented by their program
directors, and the applicant community. The reviewers would represent
the applicant community.

Dr. Luecke noted that the Division needs to bring in people who are
willing to explore new boundaries, new ways of examining scientific
relationships, but not necessarily to exclude people on the basis of
prior experience on a cognizant study section. He believes that the
process of accommodating to change will improve if DRG is successful in
getting people who are acknowledged leaders in the field to participate
in some reviews that fall outside of their previous study section
experience. This broader experience would be useful in future
evaluations of the IRG and recommendations to the DRG for change.

Dr. Braciale added that he thought the meeting might result in some
suggestions about the way in which to design the study sections within
the IRG, such that there perhaps is some kind of deliberate overlap
built in.

One discussant remarked that a major challenge is that a good review and
a good review group need to have a great breadth of expertise, because
in a given study section, some applications will use a great number of
different approaches and techniques. But at the same time, the study
section needs to have great focus of expertise, because an applicant is
going to be doing specific procedures, and a reviewer needs to be able
to evaluate those procedures.

One approach to this issue would be to mandate that an IRG include a
certain number of people from other IRGs that have overlapping areas of
science.

#### XV. [Rating of Grant Applications](http://018.nihmcmspauth.cit.nih.gov/NR/rdonlyres/335A9CF6-C813-480F-93EA-02C286B51D63/8581/rgatalk.htm)

#### Dr. Constance Atwell

Dr. Atwell started her presentation with background on the [report of
the Subcommittee on the Rating of Grant
Applications](http://www.nih.gov/grants/dder/rga.htm). This was a
subcommittee of one of the trans-NIH extramural reinvention committees,
the Committee on Improving Peer Review. One of the major activities of
the committee was to look at a specific aspect of peer review, which is
how NIH actually scores applications and then reports the results of
that scoring process. The subcommittee first met in December 1994. They
were charged with looking at the rating process, especially using
psychometrics and decision making, i.e., the science associated with
assigning ratings, numerical indices, to complex dimensions. They
recognized that the existing process was partly responsive to that
literature. So, the task was to reexamine the process, with assistance
from outside consultants, and see if things could be improved.

The subcommittee first looked at the criteria by which applications were
reviewed. They concluded that there was nothing wrong with the current
criteria, which are listed in the application kit. However, the
subcommittee thought that the focus could be sharpened up in terms of
the review. So, they decided to put more emphasis upon one existing
criterion, namely, the scientific significance of the application.
Scientific significance is whether the proposed project would make a
fundamental, original contribution or change the way scientists think
about a field. Included in this concept was creativity and innovation.

The second criterion is the experimental approach, the methodology, the
details of the research plan, all of the particulars of the research as
it is to be conducted. In most summary statements, at least 75 percent,
if not more, of the critique deals with this aspect of an application.

The third criterion, feasibility, relates to how likely the proposed
project is to be carried out successfully. The indicators included the
availability of resources and equipment, access to special populations
or reagents, and also the track record of the investigators. With the
track records, NIH is, in a way, gambling that what the investigators
have done in the past will predict what they will do in the future.

Other recommendations of the subcommittee included the rating scale. In
NIH peer review ratings, a lower number means better quality. So, the
subcommittee recommended that NIH make it a little more intuitively
obvious that a bigger number means more of the item in question, namely,
scientific merit.

It is known from the psychological literature that in complex judgments,
between 5 and 10 discriminations at a time can be reliably made. In
theory, however, NIH currently uses a 41-point scale. In fact, people do
not use all those numbers because they simply cannot rate according to
all those numbers. If one looks at the distributions, most people are
using about 10 numbers. The subcommittee recommended that a limited
scale be used with each criterion.

The standardization process of scores is a controversial issue. The
issue is how to take into account the varying behavior of different
individuals and different groups. Currently, NIH uses the percentile
system. Before that, NIH had a normalized scoring system. These are
attempts to take into account that group behavior or dynamics can
influence significantly the evaluation and scores. With the changes in
the more flexible arrangements between study sections, as well as
anticipated changes, the subcommittee is recommending that the
standardization be done by individual rather than by group. Individuals
will develop a history of scores and their own distributions will create
a scoring pattern. Then if the individuals do not spread their scores
naturally, the scores will be spread for them. This, of course, brings
up the issue of whether individual voters are influenced so much by the
behavior of the group that they switch their behavior to go along with
that group.

One of the most controversial recommendations relates to the overall
score. Does there need to be one? If there does need to be one, should
it be calculated on the basis of individual scores for each criterion,
or should reviewers make a global rating of scientific merit, as they do
now? The recommendation, based on the research findings in the
literature, was that the reviewers should not calculate overall scores.

Their report was delivered to Dr. Wendy Baldwin in December of 1995.
Since then, there has been extensive discussion within NIH, including,
among others, the RPC (the review policy staff), POPOF (the program
staff), and EPMC (the extramural policy and coordination managers).

Four, internal study groups have been created to look at each
recommendation and to think about what the implications would be if the
recommendations were implemented. They are also to develop an
implementation plan for each recommendation if it were to be adopted.
This is to be completed by August 1, 1996.

Additionally, comments will be solicited from the scientific community
via the NIH Home Page. It is also likely that there will be a more
focused group established that may meet, possibly via the Internet, to
discuss things in more detail. The Peer Review Oversight Group could
consider this issue.

The timing is such that decisions need to be made probably in January
1977 to affect the FY 1998 funding of applications. But the report was
written explicitly such that it is not a package; any of the
recommendations can be adopted independently of the other ones. In fact,
Dr. Atwell felt that the assumption is that not everything that is
recommended would ever be implemented. These are ideas that are thrown
out to be considered and some of them, perhaps, adopted. Dr. Yamamoto,
the assigned discussant, distributed a handout explaining his views. His
goal was to develop a rating methodology that assesses for each
application all the contributing elements that define research
excellence, and yet retains sufficient flexibility to permit the study
section to weigh the individual characteristics of each element. NIH
needs a system that is standardized, yet has flexibility built into it.

Dr. Yamamoto\'s plan covered review criteria and rating procedures. Dr.
Yamamoto recommended four review criteria. In the first criterion, which
is significance, the issues are as follows: Does this study address an
important biological problem? If the aims of the application are
achieved, how will scientific knowledge be advanced? Feasibility, the
second review criterion, includes whether the methodological strategies
are appropriate and the studies well designed and controlled. Did the
applicant identify risky or problematic elements in the application and
make provision for the possible failure of these elements? The next
criterion is creativity and innovation. This is an important criterion
because the applicant community is concerned about the current process
favoring established investigators with large laboratories. Fourth,
would be the investigator and environment. Is the investigator qualified
and well suited to carry out the work? Does the scientific environment
in which the work will be done contribute to the probability of success?
Do the experiments take advantage of the unique features of the
scientific environment?

In the discussion of rating procedures, Dr. Yamamoto recommended that
reviewers not only explicitly comment on each of the four criteria with
a paragraph or more, but also provide a letter grade. A letter grade
requires that the reviewers make a commitment about what they think
about each review criterion. It would help reviewers, for when they were
ready to make an overall scored evaluation, the reviewers would
explicitly consider the ratings in each area. In addition, for the
applicant, this procedure could provide critical information. For
example, with revised applications that are still not very interesting
to the study section, the tendency is for reviewers to talk about
technical flaws in the application. Those get corrected with each round,
and the applicant becomes increasingly frustrated as the scores often do
not improve. If, instead, the investigator got a \"D\" on impact and an
A on feasibility, he or she would know that the study section questioned
whether it would make any difference tomorrow if one knew the answers to
all these questions today.

On the overall score, Dr. Yamamoto recommended that a single,
quantitative score be assigned. The reviewers would be encouraged to
apply their own judgment about what kind of weight to give each
criterion. The reason for that is that each application may differ in
the importance of the various criteria.

##### Discussion

In response to a question about how an overall priority score would be
derived, Dr. Atwell noted that the subcommittee had recommended that if
there were to be an overall score, it should be derived from the
criteria scores, rather than be assigned separately by the reviewers.
This was consistent with what the majority of the subcommittee believed
to be in the literature.

It was noted that most fellowship committees already use multiple
categories, which are not weighted evenly but impact on one another.
Reviewers quite easily deal with giving each an internal rating, but
coming up with an overall score that is much more meaningful than
looking at each one in isolation. The same principle can be applied to
this situation.

One person was concerned with the proposal for individual
standardization of reviewers. One beneficial aspect of a study section
is the interaction among the members. They listen carefully to each
other\'s proposed scores and points of view. If a member of a study
section constantly rates the same, the others stop listening to that
person, who loses validity in the group. New or temporary members learn
very quickly that in order for the group to respect their opinions, they
have to spread their scores.

The same person stressed the importance of assigning a single score at
the end. Reviewers evaluate a large group of applications that are in
the boring category. With the split, averaged scores, it may be
impossible to distinguish uninteresting research from, for example, high
risk/high impact applications. While it may be useful to ask reviewers
to be more specific in the categories, in the end, one should respect
the competence and expertise of the reviewers to distinguish overall
merit. This will be more useful to Institute program staff than an
arithmetic average of three or four scores.

Two major problems with standardizing within reviewers are what to do
with streamlining and what to do with temporary reviewers who do not
have much of a record. Nonetheless, the standardization of individuals
may be appealing, and can be done in parallel with the current system
for a period of time, for all or some study sections.

In response to a question about the utility of the review product for
program staff, Dr. Atwell noted that the major goal of this study was to
help the Institutes make better decisions. They were concerned about
Institutes slavishly following a three-decimal-point score that probably
does not reflect that level of precision. Dr. Atwell noted that there
are other activities going on in terms of the decision-making process,
how to use whatever the output of peer review is to make funding
decisions.

One discussant stated that the two proposals seem to have a common core
consensus, with the Yamamoto proposal clearly more conservative. Both
aim to provide the Institutes and the principal investigators with a
clearer description of where the review panel ranks the application, not
just on scientific merit but also on other relevant criteria. In both
proposals, there is a stress on significance. NIH needs to encourage or
insist that review panels write comments and quantify them with
NIH-approved adjectives or letter grades, so that the principal
investigator really has a sense of how the application was ranked.

For investigators at all ranks, one important predictor of future
success is recent success. However, on many summary statements, this
topic is confined to the end with very modest importance applied to it.
Both proposals stress the significance of recent success. One discussant
agreed with the practice of several study sections, to ask reviewers how
they would rank each application in the pile of applications to which
they are assigned, and what they thought about the overall quality of
the applications. This information gives the members around the table a
much firmer view very quickly of where this application is likely to
rank, and leads to a spreading of scores.

The discussant did not like the idea of individual standardization of
scores, because reviewers who serve on multiple panels adopt the scoring
system of the panel on which they are serving. It is inappropriate not
to do this. Standardization is not useful in such circumstances.
Finally, the discussant also disagreed with asking reviewers not to give
an overall ranking.

One person, however, expressed concern about the over-emphasis on track
record. Although clearly it is important that the investigator be well
trained and experienced in the field, documentation of past productivity
almost insists that \"those who have get,\" and \"those who do not will
never have a chance of getting.\" NIH must not seem to perpetuate an old
boy network where grants are awarded to those who have been successful
in the past.

The importance of getting input from the biomedical research community
was noted. Perhaps this could be accomplished electronically through the
NIH Home Page. One attendee noted that Dr. Baldwin intends to do this.

A final suggestion was to designate a working group from the DRG
Advisory Committee to a particular study section or group of study
sections. They would spend time sitting in study section meetings and
evaluating the process. A five-year review cycle was recommended. Any
greater frequency might be too chaotic, and there would not be enough
time to deal with all the study sections.

#### XVI. Just-in-Time Concept

#### Dr. John Mathis 

This concept originated with the realization that some data are not
needed early in the review process but are needed later for making an
award. Since NIH is only funding 25 percent of the applications
submitted, segregating the kind of data collected would save at both
ends of the process. Currently, NIH is burdening applicants and their
institutions, and burdening NIH staff with unneeded information and
paper.

With so many different grant mechanisms, does NIH really need the same
data at the same time for all of mechanisms? Four Institutes started
about a year and a half ago with various experiments, largely around
reviews of applications in response to requests for applications (RFAs).
While there were variations among Institutes, in general: (1) for all
personnel, they asked for the names, roles and percent effort on the
project, but not salaries; (2) for consultants, they asked for the role
on the project and the time involvement; (3) for budget, they did not
ask for a detailed budget, only for the total direct cost each year, and
an expanded narrative to explain the budget; (4) for other support,
there was no separate section, but other support information would be
listed in the biographical sketch, to indicate how it related to their
ability to carry on this particular project; and (5) the checklist was
eliminated. They then distributed questionnaires to the reviewers, to
Institute staff involved in this project, and to the applicants. The
responses were generally positive; the new process saved time and did
not affect the quality of review.

Consequently, in March 1996, NIH published in the [NIH
Guide](gopher://gopher.nih.gov:70/0R5949-12384-/gopherlib/data/nih-guide/OLDGUIDES/96.03.29.data)
the following recommendations. First, all Institutes and Centers should
use just-in-time procedures routinely for RFAs. Second, for the FIRST
(R29) award and the career (K) awards, just-in-time procedures should be
applied across the NIH. These two mechanisms were selected because of
their tight budget limits (most of the award is used for salaries). The
only change from the pilot program was in some of the mentored K awards.
Reviewers needed information about the mentor\'s other support to
understand if this was an appropriate place for the awardee to receive
training. In fact, this is not an other support issue so much as a
resources and environment issue. So, the information on the mentor\'s
support should be put in the resources and environment section, not in
the other support section.

Dr. Mathis emphasized that NIH is going about this experiment very
cautiously. There is no sense that any time in the near future,
just-in-time will be applied to other kinds of mechanisms.

Dr. Pfeffer, the assigned discussant, found the goals of the
just-in-time project to be outstanding and supported them fully.
However, knowing the specific aims of a given individual\'s other
support is important. For example, if an investigator has three other
grants using a procedure on frogs and proposes to do something similar
on sea urchins, that impacts on her decision as to whether this is
moving the field forward or is redundant. It is fine to include this in
the biographical sketch, but it is important for a reviewer to see the
specific aims. In times of tight funding, the review panel would like to
have input in the overlap issue.

Dr. Luecke suggested that the DRG Advisory Committee consider at a
future meeting the sorts of things necessary and not necessary for the
initial review of RO1s as related to the just-in-time concept.

#### XVII. Closing Remarks

Dr. Braciale thanked his colleagues for assisting the DRG, as well as
Dr. Luecke and the DRG staff for their willingness and efforts to
examine thoroughly the peer review process with the Committee.

There being no further business to discuss, the meeting was adjourned at
11:30 a.m. on May 21, 1996.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes and attachments are accurate and complete.

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Samuel H. Joseloff, Ph.D.

Executive Secretary

Division of Research Grants

Advisory Committee

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Donald H. Luecke, M.D.

Acting Director

Division of Research Grants

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Thomas Braciale, M.D., Ph.D.

Chairperson,

Division of Research Grants

Advisory Committee

Note. These minutes will be formally considered by the Advisory
Committee at its next meeting on November 18 & 19, 1996; any corrections
or notations will be incorporated in the minutes at that meeting.

For questions or comments regarding the DRG Advisory Committee Meeting
Minutes, please contact Dr. Samuel Joseloff: <joselofs@drg.nih.gov>.

Last Update: September 11, 1996