## DRG Advisory Committee Minutes November 18 & 19, 1996

[I. Call to Order](http://www.csr.nih.gov/drgac/nov96min.htm#I.%23I.)\
Dr. Keith Yamamoto

[II. Approval of
Minutes](http://www.csr.nih.gov/drgac/nov96min.htm#II.%23II.)\
Dr. Keith Yamamoto

[III. Director\'s Report: Introduction of the New DRG
Director](http://www.csr.nih.gov/drgac/nov96min.htm#III.%23III.)\
Dr. Ruth Kirschstein

> [Remarks](http://www.csr.nih.gov/drgac/nov96min.htm#Remarks%23Remarks)\
> Dr. Ellie Ehrenfeld\
> Dr. Donald Luecke

[IV. NIH Budget](http://www.csr.nih.gov/drgac/nov96min.htm#IV.%23IV.)\
Mr. Fred Wong

[V. Peer Review Oversight Group (PROG) Recap of First
Meeting](http://www.csr.nih.gov/drgac/nov96min.htm#V.%23V.)\
Dr. Keith Yamamoto

[VI. Review of the Criterion Pilot Study for
PROG](http://www.csr.nih.gov/drgac/nov96min.htm#VI.%23VI.)\
Dr. Elliot Postow

[VII. Study Section Approaches to Application Review and Scoring-Open
Window and Recap of
Results](http://www.csr.nih.gov/drgac/nov96min.htm#VII.%23VII.)\
Dr. Donna Dean\
Dr. Camilla Day

[VIII. Piloting IRG
Oversight](http://www.csr.nih.gov/drgac/nov96min.htm#VIII.%23VIII.)\
Dr. Donald Luecke\
Dr. Gerald Greenhouse

[IX. A Quick Turn-around Pilot for Amendments (3-5 page
responses)](http://www.csr.nih.gov/drgac/nov96min.htm#IX.%23IX.)\
Dr. Anthony Demsey

[X. New NIH Policy on
Amendments](http://www.csr.nih.gov/drgac/nov96min.htm#X.%23X.)\
Dr. Anthony Demsey

[XI. Patient-Oriented Review (Oncological Sciences
IRG)](http://www.csr.nih.gov/drgac/nov96min.htm#XI.%23XI.)\
Dr. Marcia Litwack

[XII. Neurosciences-Institute
Perspective](http://www.csr.nih.gov/drgac/nov96min.htm#XII.%23XII.)\
Dr. Zach Hall\
Dr. Richard Hodes

[XIII. Integration
Activities](http://www.csr.nih.gov/drgac/nov96min.htm#XIII.%23XIII.)\
Dr. Elliot Postow

[XIV. Review of Individual NRSA (Fellowship)
Applications](http://www.csr.nih.gov/drgac/nov96min.htm#XIV.%23XIV.)\
Dr. Nancy Pearson

[XV. Closing Remarks and Final
Discussions](http://www.csr.nih.gov/drgac/nov96min.htm#XV.%23XV.)\
Dr. Keith Yamamoto and\
The Advisory Committee

[XVI. Closing
Remarks](http://www.csr.nih.gov/drgac/nov96min.htm#XVI.%23XVI.)\
Dr. Keith Yamamoto

#### I. Call to Order

#### Dr. Keith Yamamoto

Dr. Yamamoto, the Chairperson of the Committee, called to order the 15th
meeting of the DRG Advisory Committee, welcoming all the participants
and thanking everyone for taking the time out from their busy schedules
to attend the meeting and stay involved in the DRG and in the peer
review process at NIH. Peer review is possibly the most important NIH
accomplishment, and it depends solely upon the volunteer efforts of the
best scientists in the community. The participants then introduced
themselves, briefly explaining their experiences with NIH peer review.

#### II. Approval of Minutes to the May 1996 Meeting

Dr. Cleveland recommended that in future minutes, the speakers should be
identified instead of having the discussions anonymous. Dr. Yamamoto
also felt that it would be useful for the Division\'s permanent records
to know the identities of individuals making recommendations. This
suggestion was endorsed by the committee.

Dr. Cleveland\'s second comment was that in the [paperwork reduction
section](http://www.csr.nih.gov/drgac/May96min.htm#XVI) the minutes did
not appropriately reflect the discussion. Dr. Cleveland agreed to
provide Dr. Joseloff with some language for this section. Finally, Dr.
Yamamoto noted that in the [discussion of rating
criteria](http://www.csr.nih.gov/drgac/May96min.htm#XV), the first
criterion should have been \"impact,\" rather than \"significance.\"

#### III. Director\'s Report: Introduction of the new DRG Director 

#### Dr. Ruth Kirschstein

Dr. Ruth Kirschstein, Deputy Director of NIH, began by thanking and
leading a round of applause for Dr. Luecke for his incredibly good job
as Acting Director of DRG. Dr. Luecke had become Deputy Director of the
National Institute on Deafness and Other Communication Disorders.

Then Dr. Kirschstein welcomed and officially introduced DRG\'s new
Director, Dr. Ellie Ehrenfeld, whom she knew both as a colleague and a
friend. Dr. Kirschstein noted that Dr. Ehrenfeld had served NIH in all
of the capacities needed for having a full understanding of peer review,
and had had a broad panoply of scientific and administrative
experiences.

#### Remarks

#### Dr. Ellie Ehrenfeld Dr. Donald Luecke

Dr. Ehrenfeld noted that her appointment officially started here on
December 31. Since September, she had been coming to DRG about a week a
month. Despite the time limitations, she had been able to get quite a
bit done, in large part due to some effective help from her special
assistant, Linda Engel, as well assistance as from everybody at DRG at
all levels.

Dr. Ehrenfeld had already begun to work in some areas, and ongoing
projects were in progress to try to make the review process more
efficient. Dr. Ehrenfeld vowed to try to move along these projects as
quickly as possible. Probably the newest initiative is an NIH
Campus-wide effort to take a look at and possibly reinvent the process
of receipt and referral. As the receipt and referral center at the NIH,
DRG will be a leader in that initiative.

One area where Dr. Ehrenfeld felt she would need the most help is the
analysis of the composition of study sections. The following questions
are relevant to this topic. Is the best science going to too small a
subset of study sections? Is NIH sacrificing the breadth needed to
evaluate the importance of individual research projects to get the depth
needed to evaluate technical expertise? Is NIH tracking the changing and
often multidisciplinary approaches that are the current way of doing
science? Does NIH appreciate the new and emerging areas of science that
need to be encouraged to provide a base for tomorrow\'s achievements?

To answer those kinds of questions, DRG needed to engage the scientific
community in ways markedly different from what had been done in the
past. Dr. Ehrenfeld had already started meeting with professional
societies, and was planning to bring on, as IPAs, scientists who would
come for a short time and help evaluate peer review.

Communication with the NIH Institutes and Centers needed to be expanded,
and she planned to speak at several Advisory Councils. Mechanisms needed
to be developed to strengthen the relationship between SRAs, and
Institute program staff. Dr. Ehrenfeld was not challenging the
separation of program and review, a concept that she supported, but the
two groups had a great deal to discuss.

The first efforts will be in the field of neuroscience because of the
mandated integration of the former ADAMHA research Institutes with NIH.
But the issue will eventually have to be dealt with in all of the areas
of science throughout the DRG.

On behalf of the Division, Dr. Luecke welcomed Dr. Ehrenfeld, and
thanked Dr. Kirschstein for her comments. Noting that the Director
position will require Dr. Ehrenfeld\'s well-known energy and enthusiasm,
Dr. Luecke wished her every success in her venture.

Dr. Luecke emphasized the critical role that outside advice had played
over the 50 years of DRG and peer review. The current system could not
exist without the dedicated efforts of individuals like those
participating in this meeting. Dr. Luecke expressed his gratitude to
those serving on the Advisory Committee, particularly Drs. Elizabeth
Theil, Olga Jonasson, and Leo Davis, the current members, and Dr. Keith
Yamamoto, the Chairperson, for demonstrating that it is possible to be
critical but at the same time supportive of what goes on in the
Division.

Dr. Leucke then mentioned some recent DRG personnel changes. Dr. Suzanne
Fisher was serving as the Acting Chief of the Referral Section and Linda
Engel as Special Assistant to the Director. Dr. Christine Melchior is
the new SRA for the Alcohol and Toxicology Study Section, subcommittees
one and three, Dr. Michael Micklin, the SRA of the Human Development and
Aging Study Section, subcommittee two, and Dr. Syed Quadri, the SRA for
the Special Emphasis Panel assigned to the Oncological Sciences Group.
Two DRG staff had recently retired: Dr. Keith Murray, SRA for the
Biopsychology Study Section; and Patricia Bailey, Chief of the
Administrative Services Office. Finally, Dr. Leucke thanked the DRG
staff for their support, as a group and as individuals. They had shown
themselves willing and able to accept a great many changes over the last
few years.

Dr. Yamamoto added his personal thanks to Don Luecke. As Acting Director
of DRG, he did an outstanding job of opening up the process to the
outside community, recognizing that while the workings of the DRG are
excellent, the Division could still benefit from close scrutiny.
Finally, Dr. Yamamoto added his personal welcome to Dr. Ehrenfeld, and
stated that he looked forward to working with her in the future.

#### IV. FY 1997 NIH Budget

#### Mr. Fred Wong

For FY 1997, Mr. Wong, Chief of the DRG Financial Management Office,
explained the House Allowance, and compared those figures with the FY
1996 enacted budget. While the House figures are not the most recent,
the total NIH budget, \-- \$12,747,203,000, \-- has not changed and
represents a 6.9 percent increase over the 1996 budget.

Within the FY 1997 appropriations level, the AIDS monies will go
directly to the Institutes. Also included in the 1997 appropriations is
a downpayment of \$90 million for the Margo Hatfield Clinical Research
Center. The full cost of the new clinical center will be \$310 million.

Another significant point is that research management and support (RM&S)
has been held to the FY 1996 level, which is approximately \$480
million. (The Institutes are assessed under the RM&S budget category to
support DRG, which is a central services activity and does not receive
direct appropriations.)

Also, in FY 1997 NIH anticipates two recisions, one having to do with
administrative costs, and the second having to do with legislative
affairs. The first one is more relevant to DRG.

#### V. Peer Review Oversight Group (PROG) Recap of First Meeting

#### Dr. Keith Yamamoto

Dr. Yamamoto explained that the Peer Review Oversight Group (PROG) had
been proposed in the Cassman Report. Recognizing that peer review at the
NIH extends beyond DRG, the Cassman Committee felt that coordination and
oversight would be useful when addressing issues of peer review policy
that are common to the entire NIH. The Committee is chaired by Dr. Wendy
Baldwin, the Deputy Director for Extramural Research, and is split
evenly between NIH people and members of the extramural community. The
first meeting took place in July. During that meeting, the Committee
determined what its missions should be, and how it would interface with
other parts of the NIH. The agenda focused on two main items: the
[Rating of Grant Applications (RGA)
Report](http://www.nih.gov/grants/peer/rga.htm); and the integration of
the Alcohol, Drug Abuse, and Mental Health Institutes\' peer review
process with the peer review process of NIH.

Most of the first day included a general discussion of the kinds of
topics that PROG may want to consider. Some of these topics included:
the quality of review and how it could be measured; similarities and
differences between the DRG and the Institute review processes; how
science leads to specific review groups and the expertise within these
groups; the identification of high-risk research; and how to combat the
innate conservatism in science and the review groups.

There was a clear recognition of the areas that would be of common
concern to PROG and the DRG Advisory Committee. Such common concerns
were: selection, training, and supervision of SRAs; roles of review and
program staff; reviewer selection, approval, and training; travel to
scientific meetings for SRAs; balance between new and senior or
distinguished reviewers; communication among study sections; and the
role in training of study section chairpersons.

With respect to the RGA Report, PROG endorsed using the four criteria
put forth \-- impact, feasibility, investigator, and environment.
Whether there would be adjective descriptors or letter grades to assess
each of those criteria was a matter of debate, and there was
considerable difference of opinion within PROG on this issue. There was
enthusiasm for having reviewers make their own decisions about the
importance of individual scores. There was not much discussion about the
number of points in the rating scale or about the possible reversal of
the scale so that the better grants would get a higher score.

As a point of clarification, Dr. Theil asked whether the recommendation
was that each reviewer give an overall individual score? Dr. Yamamoto
replied that the recommendation was that the process remain as it is
now, with each reviewer assigning a global, scientific merit score. In
the original RGA proposal, the recommendation was for no overall score
or a score to be assigned using an algorithm. The reviewer and
investigator community unanimously rejected this idea.

#### VI. Review of the Criterion Pilot Study for PROG

#### Dr. Elliot Postow

Dr. Postow noted that the RGA report was generated by an NIH group who
spent over a year in a detailed study of the process by which grant
applications are reviewed and rated. They made a series of far-reaching
recommendations, which were then debated by the scientific community.
For some recommendations, such as rating an application by criteria, NIH
staff felt they needed experimental data. So they initiated the pilot
study to look at the concept of review by criterion. When evaluating
this concept, NIH was interested in utility and in burden. Under
utility, does review by criterion create a better review? Does it focus
the reviewer\'s attention better on the important aspects of the
application? Does it allow reviewers to communicate their views more
efficiently? In terms of burden, is the process too cumbersome? Does it
take more time to review by criterion?

In the pilot study, they evaluated a set of four criteria that Dr.
Yamamoto recommended, as well as a set of three criteria recommended in
the RGA report. The four criteria set included: (1) impact, the
importance or significance of the work to basic, behavioral or
biomedical science; (2) innovativeness, the creativity of the work,
whether it is profoundly new and interesting; (3) feasibility, whether
the proposed methodology is logical, whether the methodologies follow
the hypothesis, whether the investigator has included alternate
approaches, whether preliminary data are included, and whether, based on
past accomplishments, the investigator is likely to be successful in
performing the research; and (4) investigator/environment, the research
environment, including both the human and physical resources needed to
accomplish the project. The three criteria set is similar, except that
innovativeness is not separate but instead included under the impact
criterion.

The pilot included only research project grant (R01) and FIRST (R29)
applications. Reviewers assigned a global score to each application as
usual. However, in the pilot the reviewers comments were separated into
three of four sections, i.e., impact, innovativeness, feasibility and
investigator/resources. In addition, reviewers used one of three systems
to rate the applications according to each of the criteria. These rating
systems were: (1) the standard numerical range, using 1.0 to 5.0 (2)
letter grade, using A,B,C, and D or (3) not providing scores for the
individual criteria. Assignment of rating system was randomized with the
constraint that all three reviewers/reader on a given application used a
different rating system.

The pilot was evaluated based on a debriefing of the reviewers at the
end of the meeting, conducted by Drs. Wendy Baldwin, and Peggy McArdle
and the results of a questionnaire given to reviewers. The preliminary
results of the analyses were to be presented to PROG on November 20th.

Eight study sections participated in this pilot, half using four
criteria, the other half using three criteria They ranged from
biochemical to behavioral study sections, including clinical study
sections. The pilot included about 182 reviewers and 450 applications.

##### Assigned Discussants 

Dr. Burgess preferred four to three criteria, because he felt this
option would better reflect the review evaluations and discussions. Dr.
Burgess then discussed another issue, training new study section members
and chairpersons. One useful training technique might be to provide
newcomers with an outstanding application accompanied by an outstanding
review of that application, so that they could see what is expected of
them.

The second assigned discussant, Dr. Cleveland, felt that significant
progress would be made in the review process with either three or four
criteria. He was glad, however, that the consensus was that NIH would
not standardize scores by reviewers. Finally, the resources and
investigator criterion should include a more direct charge to the
reviewers to evaluate whether a particular applicant has the experience
(a senior investigator) or promise (a more junior investigator) to bring
the proposed work to fruition.

Dr. Wickens, the third assigned discussant, noted on the general
enthusiasm in the Molecular Biology Study Section for using criteria,
with a preference for four criteria. However, the reviewers recognized
the difficulty in some instances in balancing creativity against
feasibility.

##### Discussion 

Dr. Pfeffer commented on the guidance given their study section. Under
the impact criterion, for example, their SRA, Dr. Greenhouse, asked the
reviewers to consider whether they felt the proposed project would move
the field vertically or horizontally, which is a useful test. Dr.
Pfeffer liked rating each criterion separately, and felt that this
procedure gave the applicant a more precise picture of how the study
section ranked the application.

Dr. Spralding believed that reviewers have always used these criteria to
review applications, and therefore this change would be a relatively
small improvement to the current system. In his experience, the reason
most investigators do not have their projects funded is that with the
increasing number of fine applications and restricted budgets, the
investigator\'s projects often just missed the pay line.

Dr. Ehrenfeld felt that the key criterion was the quality of the
reviewers. Most reviewers are good scientists with the expertise to
evaluate the applications; however, some are inexperienced, and some
have personal flaws. More explicit criteria for evaluating applications
would help the average and beginning reviewers to organize their
thoughts and balance the various criteria. It would also be helpful for
applicants to have a clearer statement of exactly what the reviewers
admired. Dr. Ehrenfeld also thought that the Advisory Committee should
wait for the data from the PROG pilot program before reaching any
conclusions about numerical versus letter scoring.

Dr. Jonasson stressed the impact on the applicant. Being as explicit as
possible within the broad criteria helps the applicant to be more aware
of exactly what is expected, which should improve not only the mechanics
of the grant application, but also the thought processes that precede
the application.

Dr. Pollock stated that if the ultimate goal was to improve science by
giving investigators better feedback, structured descriptive data would
be helpful, because such information would force a new reviewer to step
back from the application and place the work in a broader context. A new
reviewer tends to focus on the technical aspects of how the science will
be done and can easily miss the forest for the trees.

Dr. Yamamoto noted that criterion rating would encourage reviewers to
explain the thinking behind the words that they were putting down.
Reviewers tended to sugar coat harsh evaluations, which made it
difficult for the applicant to perceive what the reviewer had in mind.
One main problem with current reviews that were not done by the best
reviewers is that some summary statements become a discussion of
technicalities, such as whether the concentration was right and whether
this plasma would be effective. So the applicant patched up those
misdemeanors, in a revised application, fully assuming that now that
everything has been fixed word for word from the summary statement,
money should be forthcoming. Then they got another three pages of
problems. The real reason behind all of this was that the reviewers were
not impressed by the impact of the entire project.

Dr. Wickens thought the system might be useful for simplifying a problem
where the evaluations were not in concordance with the scores. With the
criteria system, it would become more apparent whether there was
correspondence between the overall score and the separate criteria.

With respect to the data of the pilot study being presented first at the
PROG meeting on November 20th, Dr. Theil wondered whether a process
could be developed so that the DRG Advisory Committee might interact
with PROG directly by looking at the data when available and then
sending comments to PROG.

Dr. Cleveland was enthusiastic about having different reviewers weight
the different criteria differently, because reviewers cannot read the
technical detail of every application. They can, however, read part of
each application, and can assess its impact or creativity. Criterion
rating would allow the entire study section to participate in the review
process in a much more open way.

Dr. Cleveland also suggested that during the review of applications,
reviewers should first discuss significance, investigator, resources,
and creativity and then discuss feasibility. Otherwise, the technical
aspects would overpower the discussions, as they do currently.

Dr. Yamamoto remarked that the criteria were not independent, but were
related to each other. For this reason, it would not be useful to try to
generate an algorithm for each criterion score. For an individual
application, the two assigned reviewers might diverge rather
dramatically, not only in how they arrived at their final score but even
in what they thought about the individual criteria. This is beneficial
in a judgement process, and certainly useful for the applicants to know
if they decide to revise their applications. Hopefully, this would
result in a better review and better applications in the future.

#### VII. Study Section Approaches to Application Review and Scoring - Open Window and Recap of Results

#### Dr. Donna Dean Dr. Camilla Day

Dr. Donna Dean, Acting Chief of the DRG Referral and Review Branch,
provided a historical framework for the discussions of scoring behavior.
She reviewed the records starting in 1952 to see if the current concern
with spreading scores was a relatively recent issue or a perennial
problem for the Division. Even in 1952, study sections were exhorted to
spread their scores and use the same standards regardless of subject
matter or Institute. They were specifically directed not to assign
scores according to the perceived funding levels of the Institutes.

As a memorandum to the Director of the NIH from the Director of DRG
showed, DRG was still grappling with this issue sixteen years later. The
DRG Director noted that in some instances, SRAs (then called Executive
Secretaries) were convinced that their study sections had altered their
behavior in assigning priority scores. While the SRAs had been able to
reverse this trend with the proper admonitions; the DRG Director felt
that his staff might still not be able to completely control the
situation despite the maximum efforts of staff.

In that same year, DRG developed a process that is still used today. The
Division examined the median priority score for every study section as
soon as possible after the meeting. If the median score had improved
appreciably, the situation was discussed individually with the SRA. If
the study section had altered its scores inappropriately, the SRA was
instructed to discuss the issue with the study section. Then a general
admonition was to be repeated before every review cycle.

The Division\'s most recent directive was at the January 1996 meetings,
where DRG again exhorted the study sections to spread their scores. In
some study sections, the priority scores were so compressed that funding
by the Institutes was within five to eight priority score points. The
Division was also concerned that reviewers might feel that they needed
to make their preliminary written critiques match the compressed scores,
and therefore, the full benefit a reviewer\'s judgment on an application
would be compromised.

Dr. Dean pointed out that DRG had recalibrated its scores at least three
times in recent times: for the May, June 1988 study section cycles, when
the NIH across-the-board percentiling began; for the October/November
1991 review round, when the not recommended for further consideration
procedure was implemented; and for the June 1996 meetings.

Dr. Camilla Day, SRA of the Biological Sciences-2 Study Section,
reported on a forum given by the SRAs about this problem of score
spreading. Different study sections had different styles, and sometimes
what was appropriate for one study section was inappropriate for another
because the science and mix were different. But some common ideas
emerged from this forum. One is that collaboration between the study
section, chairperson, and SRA was absolutely essential. In all study
sections whose scores were spread, the SRAs provided the study section
with clear benchmarks of how they scored. In some study sections,
members monitor each other in terms of, for example, being an easy
grader. Since it appears that scrunching (compressing scores) is a
natural tendency, both study section members and DRG SRAs should be
constantly alert to this problem, even in study sections where scores
have been well spread in the past.

##### Assigned Discussants

Dr. Pfeffer, the first assigned discussant, stated that if reviewers
believed that everyone else was spreading scores, it was easier for them
to spread the scores. Reviewers had to trust each other. It was always a
psychological battle between I want to see this application funded\' and
let\'s all participate and provide useful information to the Institute
so that the top science can be funded.\' Reviewers would only do the
latter if they feel that everybody was playing by the same rules.

In their study section, reviewers were asked to explain: their
enthusiasm for the application in general; where the application ranked
among the applications they reviewed that round; and whether this was a
strong group of applications compared with other groups seen in the
past. Ranking helped the group establish a common language and a
reference to help give those reviewers a standard to use.

Dr. Theil, the second assigned discussant, brought up the fact that in
the rapidly changing world of science, there has been a huge quantum
leap in how science is done and the speed with which people can get
information. A particular study section might get many outstanding
proposals, and if they were forced to spread the scores, NIH may miss
the opportunity to support some truly excellent science.

The next assigned discussant, Dr. Van Houten, explained her study
section\'s experiences with spreading scores and avoiding scrunching.
First, they did not review applications by alphabetical order, and mixed
up the disciplines, which kept everyone\'s attention. The study section
also stopped two times to recalibrate, and to check that they were
actually staying decompressed. Their third procedure was dividing the
applications one more time after streamlining. This division was called
the best and the rest. This was a way of discussing, in random order,
applications of similar merit, dividing the best applications from the
others assigned. This could be done at the beginning of the study
section, right after streamlining and could also be done without
conflict of interest complications.

Another important consideration for Dr. Van Houten was the agenda. Even
though they mixed up the disciplines and did not use alphabetical order,
the study section could still accommodate the Institute program staff,
who needed to have a schedule of when their program\'s applications
would be reviewed, by blocking out chunks of time on each day. They
could be flexible because almost all of their applications were assigned
to one Institute. This may be more difficult for other study sections.

Dr. Wickens, the final discussant, noted that his study section had a
guide sheet linking priority scores with probable percentile rankings,
which proved useful when assigning ratings. Also at the start of the
discussion of an application, the reviewers recommended a rating,
sometimes with a short description of any dilemmas and possible
discussion points. Dr. Wickens also felt it would be useful at the end
of the meeting to revisit applications that are bunched together.

##### Discussion

Dr. Ehrenfeld felt it was important to collect data from the DRG as to
whether a small subset of study sections were currently receiving the
majority of what are considered \'hot\', super good applications. If all
the exciting applications ended up in the same few study sections, NIH
would lose the opportunity to fund some of the best science, and end up
funding the best applications from other study sections, which may be
more mediocre science.

Dr. Pollock wondered whether every member of the study section could be
given a score distribution sheet at the beginning of the meeting. After
every 10 applications were reviewed, tick marks could be put down so
that people could see the normative curve that their group was
generating. That might help influence voting behavior.

Dr. Dean noted that in some study sections, SRAs have kept a running
tally either on paper or by a computer program, and have taken it upon
themselves, or had the chairperson, be the conscience of the study
section and warn the reviewers against scrunching their scores. In some
cases, the chair and the SRA did it themselves; in other cases, the
reviewers were given their own sheets to keep track of.

Dr. Yamamoto asked if those tactics were helpful. Dr. Dean responded
that in an informal survey involving 23 SRAs who experimented with such
a system, some found it extremely helpful. However, 3 SRAs said that no
matter what they did, it did not work.

Dr. Spralding found that some reviewers were concerned that low scores
would be discouraging to applicants. Since percentiles were what really
matter in the system, it was much better to use the whole range. Also,
reviewers did not have to vote in agreement with the primary and
secondary reviewer. Reviewers could be independent, and spread their
scores even when the study section did not.

One, less common source of improved scores was with individuals who
represented a certain subfield where, they felt, not enough grants were
being funded. For this situation, the SRA or another member of the study
section should speak with the person.

Dr. Cleveland noted that recalibrating scores would unfund two rounds of
applications. Although the quality of a round of applications might
fluctuate from round to round, Dr. Cleveland suggested that the NIH
might be better off percentiling based only on one round of scores. Then
a study section could spread their scores unilaterally every time they
met.

In terms of the scrunching of scores, individual scores for primary and
secondary reviewers would be useful, because most study section members
take the advice of these assigned reviewers. So the real question was
whether primary and secondary reviewers thought that all of their
applications were outstanding, and whether they could defend this
position with the study section.

Dr. Pfeffer added that such discussions about the quality of one\'s of
applications should start with the first application. The sooner the
study section achieved that common language about the pile of
applications, the sooner everyone around the table would be an equal
colleague. Most reviewers really listen to assigned reviewers who, they
trust, are spreading their scores.

Dr. Pfeffer felt uncomfortable reevaluating an application to spread the
scores, because this would be unfair to anybody who was not given the
chance to be reevaluated. There is some randomness to the way NIH
evaluates applications. If an investigator is the first person
discussed, it may be easier to get a higher score than the last person
discussed. This is part of the luck of the draw.

Dr. Wickens mentioned that his study section dealt with that random
selection problem by reevaluating everyone. Conflict of interest was not
a problem; any reviewer in possible conflict either left the room or did
not participate in the discussions. Another suggestion was that each
assigned reviewer send the recommended scores to the SRA prior to the
meeting. The SRA and possibly the chairperson then could look at these
scores, identify individuals with too many very high scores, and thus be
aware of that potential problem.

Dr. Dean believed that in some instances SRAs had asked reviewers to
send in recommended ratings in advance of the meeting. In the future,
DRG might probe this issue further. The SRAs administer the peer review
process, because they are the people who are legally responsible for the
sanctity and integrity of this process. What must every study section do
and what are the acceptable boundaries within which study sections can
operate? Those are very difficult questions because it means stepping
back and rethinking some fundamental principles of peer review.

Dr. Dean wondered whether scrunching of applications was limited to
certain disciplines or subfields. Dr. Van Houten noted that in her study
section, with roughly four disciplines, their scrunching problem seemed
to cross all the disciplines. It seemed to cross all of the disciplines.
Despite the best efforts of trying to undo the clumping, once reviewers
have scrunched, they look at past history and scrunch again.

Dr. Ehrenfeld was grateful to Dr. Dean for bringing up the question of
permissible variation in behavior among different study sections. DRG
needs to deal with this issue in the future. Another issue was that
recommendations must be transmitted to the SRAs in full rather than as a
brief memo stating that the Division is going to do something from now
on. The success of implementing any new recommendations is going to
depend upon the SRAs understanding the context and benefiting from the
value of discussions. DRG needs to develop a better structure for
dialogues to occur.

Dr. Demsey commented that some study sections round after round have
successfully maintained their spread of scores. They must be brought
into the discussion, because they must be doing something right.

#### VIII. Piloting IRG Oversight

#### Dr. Donald Luecke Dr. Gerald Greenhouse 

Dr. Luecke explained that several years ago, pressure came from the
newly elected President Clinton and the Executive Branch of the
Government to reduce the number of advisory committees. So overnight,
approximately 85 chartered study sections became 18 chartered initial
review groups (IRGs). The study sections became, for all practical
purposes, subcommittees of IRG. Degrees of flexibility exist with this
arrangement. For example, the chairperson of any study section grouped
under an IRG may serve as the chairperson of that IRG.

During several meetings with the DRG Advisory Committee, there had been
interest in inviting distinguished investigators to come back, to
improve the quality of the reviews and hopefully also to have some
additional input into other matters that concern that IRG. The Division,
therefore, decided to do a pilot study involving the Cell Development
Function IRG, whose coordinator is Dr. Greenhouse. Dr. Cleveland would
serve as the chairperson of that IRG. The number of members was not
fixed. One objective is to develop recommendations about ways to change
the boundaries among study sections or to look at the issue of disparate
quality in scientific areas.

Dr. Greenhouse saw this pilot project as a chance to formalize and to
expand at the IRG level what groups of SRAs have been doing all along,
to use the expertise and advice of the scientific community. The new IRG
charter allowed reviewers to meet as an IRG, rather than just as a study
section. SRAs could supplement the group with non-members rather
liberally and still have an official meeting. The Cell Development and
Function IRG has eight study sections, six that handle the standard
R01s, R29s and fellowships and two that are more specialized. In the
beginning, Dr. Greenhouse felt they needed the most help with the R01,
R29 groups.

Three basic areas would be very helpful. The first, and probably the
simplest, is in identification and recruitment of reviewers. It would be
to their advantage to think about recruiting on an IRG-wide basis rather
than in the more narrow way of each study section fighting for its own
little corner of the reviewer market. The second area would have to do
with the guidelines of the IRG, the breadth and depth of the science
within a particular IRG, and then how they have divided up that workload
among the various study sections. The third area would be sitting in on
meetings to give the IRG feedback on whether the meetings were
accomplishing what they were supposed to, and whether it can be done in
a better way. They would try to meet once a year at least, at a big
national meeting, and there would also be opportunities for smaller
groups to meet at some of the more specialized conferences.

Dr. Cleveland, the first assigned discussant, felt that Dr.
Greenhouse\'s goals were both modest and practical, and should help in
the recruitment of reviewers. Having more flexibility in what they ask
of reviewers should prove useful in identifying appropriate people and
getting them to serve on study sections. The other assigned discussants,
Dr. Hynes, Dr. Pfeffer, and Dr. Davis, agreed with Dr. Cleveland\'s
assessment.

##### Discussion

Dr. Schachman had been enthusiastic for some time about bringing back
senior people who could make a real contribution to the evaluation of
science, and also help alleviate some of the problems that have been
brought to his attention as ombudsman to the outside scientific
community. He hoped to see the practice expanded soon to other IRGs.

#### IX. A Quick Turn-around Pilot for Amendments (3 - 5 page responses)

#### Dr. Anthony Demsey 

Dr. Demsey provided some background on the increase in amended
applications over the past 8 years. In FY 1987, amended applications
accounted for about 29 percent of all applications reviewed in DRG study
sections. By FY 1995, close to 38 percent of the DRG applications were
amended. In FY 1995 alone, the Division reviewed 15,743 amendments of
original applications (A-1s) and 750 subsequent amendments (A-2 to A-8).
The success rates for R01 applications for FY 1995 were 14 per cent for
new applications, 40.9 percent for amended applications. The success
rate improved slightly from A-1s (amendments of original applications)
to A-2\'s (second amendments).

Faced with workload and budget pressures, the NIH performed various
experiments to decrease the time from submission of an application to
its potential award. One concept being explored is to allow some
applicants to respond to their summary statements in a very short time
frame. In this pilot study involving several study sections, after the
review process, if the study section found any application for which, in
their opinion, a limited amount of information would create a
significant difference, they rated the application, but also identified
it as a special case. Although the study section could have identified
any application, the ones identified thus far have had priority scores
and percentile rankings close to the Institute payline.

After the review meeting, these applicants were offered the opportunity
to respond to the critique with a 3 to 5-page document, which would go
to the next study section meeting. The 3 to 5-page response and the
previous application were considered to be an amended (A-1) application.

In the pilot study, surprisingly few applications were so identified.
The Cellular Biology and Physiology -1 Study Section had only two such
applications, the Molecular Cytology Study Section only one application,
and the Experimental Therapeutics -2 Study Section three applications.
The results were mixed, but generally positive. The two Cellular Biology
and Physiology applications improved dramatically \-- from the 29th to
the 9th percentile and from the 25th to the 5th percentile \-- but the
situation was not clear with the Molecular Cytology Study Section
application, and only two of the three Experimental Therapeutics
applications fared well. The applications will be reviewed by the
National Advisory Councils or Boards in January.

#### X. New NIH Policy on Amendments

#### Dr. Anthony Demsey

Dr. Demsey discussed another recent issue related to amendments
\--limiting the number of amended applications that an applicant may
submit. NIH is one of the few organizations that supports biomedical and
behavioral research and allows submission of amended applications. After
considerable discussion, the NIH decided to limit the number of amended
applications to two. This limitation was not a workload issue, because
the applicants were free to reformulate the projects and submit them as
new applications. Instead the concern was rooted in the concept of a
tutorial. After several amendments, a study section often became so
invested in directing the research that they acted like a tutor and lost
much of their objectivity.

The second rationale for the limitation was that statistical analyses
had showed that the longer it takes for an original application to be
funded, the less likely it is that the project will be continued after
the original budget period.

With the new policy, even though the study section might get the same
application more than twice, they would be asked, to the degree they
can, to consider it as a new application. The new policy will go into
effect for the October 1 and November 1 receipt dates.

##### Discussion 

Dr. Cleveland stated that based on his study section experiences, there
would always be only a small number of applications eligible for a quick
response. With these few cases, Dr. Cleveland felt that the NIH might
even be misleading applicants into thinking that the study section was
almost guaranteeing that a brief response would substantially improve
the priority score.

Dr. Demsey noted that there is a degree of sentiment among researchers
that this policy is unfair unless everyone has the option to respond in
3 to 5 pages. Dr. Cleveland responded that while most people might be in
favor of the opportunity being extended to all applicants, there is a
workload problem, and such an expansion might not be feasible at
present.

Dr. Demsey then remarked that the National Cancer Institute has
developed a similar process Within a certain percentile window, any
applicant can submit a 3 to 5 page response to the summary statement.
Their 3 to 5-page responses do not go back to the peer reviewers for
consideration, but instead go to an Institute executive committee where
the funding decision is made.

Dr. Litwack noted that the National Cancer Institute program staff are
pleased with the results and reception by the biomedical research
community. Approximately 50 to 60 percent of these 3 to 5 page responses
have been paid.

Dr. Van Houten wondered about the instructions given to the study
section, because reviewers are not supposed to be rating according to
funding considerations. At what level does the study section find out
about funding, or are they not involved in the decision at all?

Dr. Demsey responded that reviewers do not know the percentile ratings.
They are asked to identify those applications that have a specific
weakness, regardless of the score or ranking.

Dr. Ehrenfeld felt that this was an area that should be discussed more
closely with the Institutes. The National Cancer Institute has an ease
of handling the issue, because they do not rereview the applications;
they can make an executive decision about funding. There is considerable
variability in how these rules will be interpreted and whether they will
be fair to all investigators.

In response to Dr. Yamamoto\'s question about the status of the pilot
program, Dr. Demsey noted that the 3 to 5 page reviews will go to the
Institutes\' January Councils. Subsequent to getting some feedback from
the applicants, the pilot will probably be discussed at an NIH
Extramural Program Management Committee (EPMC) meeting. Dr. Demsey\'s
prediction was that this program would be greeted with enthusiasm and
expanded. The expansion would be done only after discussions between the
Institutes and DRG.

Dr. Ehrenfeld wondered how the Division planed to evaluate this pilot.
The numbers are small. Most of the people who got their grants funded,
would undoubtedly think it is great. Also they would not be talking to
the people who were not subjected to it.

Based on the various concerns expressed by the Advisory Committee, Dr.
Yamamoto suggested that the topic be reconsidered in a future meeting.

Dr. Manis noted that applicants, especially the ones on the borderline
of getting funded, currently often come back with their own rebuttal to
the Council or to the Institute. This pilot program seems to be the same
mechanism, and therefore redundant.

Dr. Demsey responded that there was a key difference in timeliness;
rebuttals rarely were able to be reviewed in the same review round.
Except for the National Cancer Institute, most Institutes responded to
such rebuttals by deferring the application for rereview.

Dr. Thiel felt that if the Advisory Committee were going to revisit this
topic, they should get information about the impact on the Institute
programs.

#### XI. Patient-Oriented Review (Oncological Sciences IRG)

#### Dr. Marcia Litwack

Dr. Litwack, SRA of the Experimental Therapeutics -2 Study Section,
traced the origins of her study section to complaints in the early 1980s
from clinical oncologists and NCI staff that clinical oncology
applications were not getting a fair review in existing study sections.
The study section was chartered in 1986.

The applications reviewed by this study section are concerned with
cancer therapy, from pre-clinical studies to clinical applications, many
of which include phase one, phase two, and, on rare occasion, phase
three clinical trials. The applications cover chemotherapy,
immunotherapy, gene therapy, bone marrow and peripheral stem cell
transplantation, and surgical oncology. The study section membership,
originally 17, was increased to 20 two years ago due to the broader area
of science being covered and the increased number of applications. Even
then, additional reviewers are usually invited either to cover areas of
science not adequately covered by the study section membership or more
frequently to handle overloads in specific areas of science.

At the most recent study section meeting, there were 16 M.D.s \-- 3
surgical oncologists, 3 pediatric oncologists, and 10 medical
oncologists. Many of the Ph.D.s on the study section are involved in
translational research as well as basic research and are in clinical
departments. Because most of the applications involve the spectrum of
pre-clinical to clinical research, a membership with broad expertise was
used.

Dr. Litwack used the following definition of patient-oriented clinical
research: research conducted with human subjects or on material of human
origin, such as tissues, specimens or cognitive phenomena, for which an
investigator or colleague directly interacts with human subjects.\"
Using this definition, the percentage of these applications in the
Experimental Therapeutics-2 Study Section ranged from 20 percent to 61
percent. The average has been about 40 percent. Data from the last three
meetings (February, June and October, 1996) indicate that
patient-oriented applications are holding their own, and even seem to
have a slight edge over non-patient research in her study section.

Dr. Pollock, the first assigned discussant, felt that one strength of
the Experimental Therapeutics Study Section is its diversity of
membership (M.D.s, M.D./Ph.D.s, and Ph.D.s). With that array of
expertise, the group could make good judgments about whether the
research was moving forward in the direction of practical applicability.

Dr. Pollock also thought it extremely important that all the members of
the study section have NIH funding. Otherwise, the larger oncology
community would look askance at those reviews. As translational and
patient-oriented cancer research changes, there is a tremendous
opportunity to use ad hoc reviewers to participate in these activities
without having to reconstitute an entirely new study section.

Finally, Dr. Pollock believed that the underserved community\' is
actually being well served, and that few clinical trials that deserve to
be funded were not funded.

Dr. Jonasson, the second assigned discussant, noted that with the
increasing emphasis on scientific evidence, there will be more need for
clinical trials. However, in the foreseeable future, there will be great
difficulty in funding these clinical trials because of the expense. They
require a large infrastructure. Dr. Jonasson wondered whether some
mechanism could be developed to facilitate a partnership with industry.
Dr. Jonasson next brought up the critical need to include
patient-centered outcome research, which would include the actual impact
on patients. It is extremely important to determine whether quality of
life or just life has been extended. Finally, with respect to the
difficulty in recruiting qualified reviewers to the study sections, Dr.
Jonasson mentioned that many qualified reviewers who are willing, and
capable are never considered. She personally brought a box of about 100
CVs to the DRG about a year and a half ago of qualified, funded
individuals, all of whom indicated their willingness to serve. Not one
has been appointed.

##### Discussion

Dr. Luecke remarked that these concerns need to be addressed soon in an
ongoing, comprehensive way. Some time ago, the DRG Advisory Committee
sponsored a study on the review of patient-oriented research. It is
appropriate to rereview the report from that study, especially in light
of considerable new data.

Dr. Ehrenfeld noted that in addition to the available data, there are
widely differing perceptions in the patient-oriented research community.
It would be very useful for DRG to bring in someone with good
credibility in the community, with good expertise and contacts with the
appropriate societies, to work in DRG for a specified period of time on
this topic and coordinate activities in this area.

Dr. Litwack provided some data related to her study section. In the 0 to
25 percentile range, for the 5/26 Council, of the 9 patient-oriented
research applications, 4 involved phase one or phase two clinical
trials. For the 10/96 Council, of the 6 patient-oriented research
applications, 4 involved phase one and phase two trials. For the last
study section meeting, of the 11 patient-oriented research applications,
6 were phase one and phase two trial.. So when this clinical research is
well done, it does well in review.

Dr. Robert Browning, Chief of the National Cancer Institute (NCI) Grants
Review Branch, explained that from the NCI point of view, one of the
major discouragements, was that while they had been trying to develop
mechanisms for encouraging clinical researchers, but after the
clinicians are funded once, they submit an application to a DRG study
section, become discouraged when they are not funded, and go into
private practice.

Dr. Litwack, however, saw the problem to be an unfounded perception in
the clinical community that they do not do well in the study sections,
and so they just don\'t send in applications. Also if there is a problem
the first time, clinicians tend to give up rather than examine carefully
the summary statement and possibly revise the application.

##### Open Discussion (Criterion Scoring and Other Topics)

Dr. Van Houten wondered whether collapsing the ratings from 41 to 7
would automatically increase scrunching. Dr. Yamamoto responded that
this reduction in numbers would definitely greatly increase the number
of high scores, and being able to sort those out by some other way would
be important. That was something NIH needs to consider.

Dr. Thiel agreed that a 41 point range was too large, but she also felt
that 7 might be too narrow a range. The exact number needed further
study and discussion.

Dr. Pfeffer stated that one major strength of the criterion based
scoring is that it will provide the applicant, review group, and
Institutes with a much clearer picture of the basis for the overall
score. However, the scientific community at large has had major concerns
about the fairness of the application procedures. Dr. Pfeffer thought
that the community would be pleased to see the criterion based scoring,
but that there would be a lot of unhappiness if they sensed that NIH was
simply reversing the order of the numbers without any real
justification. She thought it would also generate a great deal of
confusion if NIH reversed the order or changes the numbering system at a
time when some study sections are already making good strides towards
spreading their scores.

Dr. Spradling stated that given the difficulties in the investigative
community, NIH should move forward with changes that would be viewed as
significant steps toward some improvement in the situation. These two
changes in the numbering system seemed very minor to Dr. Spradling, and
he thought they would be viewed as almost cosmetic changes and not well
received.

Dr. Burgess agreed, noting that the reduction to 7 numbers might produce
so many ties, that program staff would be making most of the funding
decisions, taking these away from study sections. Priority scores and
percentile rankings would become less significant.

Dr. Ehrenfeld noted that of the 10 recommendations that were in the
report, the one on criterion scoring was universally felt to be
important. Several other recommendations could be deferred or tabled.
Two other recommendations were largely rejected. There was a consensus
to maintain global scoring and reject multiple scores. The other
recommendation rejected was normalizing reviewers.

#### XII. Neurosciences-Institute Perspective

#### Dr. Zach Hall Dr. Richard Hodes 

Dr. Hall, Director of the National Institute of Neurological Disorders
and Stroke (NINDS), provided some background to the reintegration of the
Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA), NINDS,
and DRG. The timing of the integration had been sparked in part, by the
joint arrival of Dr. Steve Hyman, Director of the National Institute on
Mental Health and Dr. Ellie Ehrenfeld, Director of DRG. Under Dr.
Hyman\'s initiative, a group of Institute directors and other staff
discussed how to reorganize neuroscience review across all the various
Institutes interested in this topic.

Dr. Hall felt that the principles by which neuroscience applications
might be reviewed could be the start of a larger cooperative working
with DRG, in which they were concerned not only with review but also
with application assignment. In the past, unfortunately, there had often
been an arena for competition between the Institutes; and there had been
a lot of unease, about communications with DRG to get applications
assigned to the appropriate Institutes.

The general principles that they agreed on were clear. First, they felt
that study sections should not be either Institute directed or technique
directed, but should be organized around broad areas. The broad areas
should be redundant and overlapping so that applicants would have more
than one choice of study section. Second, study section review
responsibilities should be flexible. There should be some mechanism to
review a study section after several years to see if they are responding
to new developments. They felt that broad scientific groupings would be
one appropriate way. That is, one study section could deal with cellular
molecular neuroscience, another with integrative neuroscience, another
with behavior neuroscience. But then there should be some groups that
would cut across all areas, such as memory, learning, and plasticity.

They were concerned that their clinical research applications be
reviewed by study sections with sufficient expertise in clinical
research. On the other hand, they felt it would be a mistake to
segregate clinical research completely. The interchange around the study
section table among people with different backgrounds was often
tremendously valuable. They were eager to involve the extramural
community in this process. Change is always anxiety producing, and so at
the annual neurosciences meeting, they organized a meeting at which a
number of Institute directors were present. Attendance was poor because
the meeting was slightly awkwardly placed, but the poor attendance also
showed there was no groundswell of angry grantees worried about the
future. In that sense it was constructive.

It is critical to recruit first rate people into the important jobs in
review and program, both in DRG and in the Institutes. He had had
discussions with Dr. Elliott Postow and other DRG staff about
recruitment efforts, and Dr. Hall was appreciative of their collegial
and consultive attitude.

Dr. Richard Hodes, Director of the National Institute on Aging,
reinforced what Dr. Hall said, emphasizing that the collegiality across
Institutes made processes such as this easier and, on a personal level,
made serving as director of an Institute an enormously gratifying
experience. Dr. Hodes noted that as areas become more specialized, the
applicants and program staff, on the one hand, to ensure an appreciative
hearing for their work, look for peers who are more and more
specialized. On the other hand, the Institutes fear that kind of
specialization and isolation will be counter-productive to the quality
of science overall. They hope to provide sufficient depth and expertise
in areas while the breadth remains, so that they do not lose sight of
broader perspectives and the cross-fertilization of scientific input
that can occur in peer review.

#### XIII. Integration Activities

#### Dr. Elliot Postow

Dr. Postow discussed the implementation of the [neurosciences
integration process](http://www.csr.nih.gov/REVIEW/neurorev.htm). The
goal is to develop an approach to peer review in which the most
meritorious science is funded. The stakeholders in this process are the
applicants and their scientific organizations, the Institute program
staff, the review staff of DRG and the Institutes, the study section
members and the advisory councils of the Institutes.

A group of extramural scientists from five Institutes (NIA, NICHD, NIDA,
NIMH, and NINDS) and DRG is developing the process by which an array of
new neuroscience study sections will be organized. The goal is to create
a process in which stakeholders have an input, they plan to open a
special electronic mailbox through which members of the scientific
community can provide their views. They hoped to begin with applications
to be reviewed at June and July 1998 study sections. A topic left for
future discussion is which grant mechanisms will be reviewed by DRG and
which by the Institutes, and whether this would vary from Institute to
Institute. They also are going to have to discuss the interface between
neuroscience and behavioral sciences.

In DRG, the vast majority, but not all, of neuroscience research was
reviewed in the neuroscience study sections. Therefore, in some cases,
the impact on other DRG study sections must be considered.

##### Discussion 

Dr. Hall commented on the remarkable spirit of collegiality among those
Institutes involved with brain research, which he thought, augered well
for future developments in this area at the NIH. They are concerned with
cooperation through the entire process. For example, Institute paylines
vary, and Institutes with very good pay lines tend to attract more grant
applications. They are trying to work together to have mechanisms that
will allow funding of deserving science wherever it is.

Dr. Postow stated that philosophically DRG is starting off with
basically a clean slate, with no special treatment for any study
section. DRG is looking at this subject from the point of view of the
science, and is trying to create an array of study sections that can
provide grant applications a fair and competent review.

Dr. Ehrenfeld commented that in light of the Advisory Committee\'s
discussion about the need to examine the study section boundaries and
organization because of the changing nature of science, she welcomed the
challenge to reorganize the neurosciences review. This might provide the
ability to develop mechanisms and evaluate systems, which could be
applied throughout DRG in other areas.

#### XIV. Review of Individual NRSA (Fellowship) Applications

#### Dr. Nancy Pearson

Dr. Pearson began by briefly discussing the three major criteria for the
review of postdoctoral fellowship applications: (1) whether the
candidate has the qualifications for a career as an independent
investigator, as shown in the recommendations, previous research, and
quality and number of publications: (2) the scientific merit as well as
the training potential of the proposed research project; and (3) the
training resources and environment including the sponsor\'s training
record, current grants to support the project, and expertise in the
area. Each criterion carries significant, although not necessarily
equal, weight in determining the priority score. No algorithm is used to
figure out the final priority score based on these subcategories.

In terms of the recent history of the review of postdoctoral
fellowships, prior to the spring of 1994, 11 study sections were
dedicated solely to the review of fellowships. Then in 1994 and 1995,
three SRAs who were in charge of the postdoctoral fellowship study
sections either retired or left their positions for other jobs. Since
the Division was downsizing at the time, the decision was made not to
replace these positions but to distribute fellowships in study sections
that reviewed research grants (R01) that had appropriate expertise.

Beside reflecting downsizing, there were potential advantages to merging
the review of fellowships and R01 applications. It was felt that
fellowships would be reviewed in a more focused scientific manner, since
some fellowship study sections had covered extremely broad areas of
science. Also, the former ADAMHA research Institutes had been reviewing
fellowships in R01 study sections, and since part of the review function
of these Institutes was going to be absorbed by DRG, this would result
in a consistent procedure for review of postdoctoral fellowships. This
change in fellowship procedures was discussed at the May 1994 DRG
Advisory Committee meeting, and the responses were positive. Shortly
after that, all study sections that reviewed exclusively fellowships
were disbanded.

Occurring in parallel with the change in fellowship review was the
formation of IRGs. One result of the new IRGs, in terms of fellowship
review, was that each IRG decided on the best way to review their
fellowship applications according to whatever factors they thought were
important. As a result, some IRGs reviewed fellowships in R01 study
sections; other IRGs reviewed their fellowships in special emphasis
panels that reviewed exclusively postdoctoral fellowships, and some IRGs
used a combination of both procedures.

In all cases where fellowships were reviewed in R01 study sections,
fellowships were grouped together, so that the SRA could orientate the
study section members to the specific fellowship review criteria. Dr.
Pearson then presented data on the number of fellowship applications
reviewed by the various study sections for the October 1996 Institute
Advisory Council meetings.

Dr. Pearson concluded with three questions: (1) Should postdoctoral
fellowships be reviewed in the same study sections as research grants or
in study sections that exclusively reviewed post-doctoral fellowship
applications? (2) If postdoctoral fellowship applications are reviewed
in R01 study sections, is it necessary to have a minimal number of
fellowships to get a quality review? If so, how would an SRA determine
that number? (3) In determining the final priority score, how much
weight should be given to the various sub-categories?

Dr. Hamkalo, the first assigned discussant, had polled study section
members and colleagues at the University of California at Irvine. The
one benefit noted by a majority of the people polled was that the
reviewers in the R01 study section often had more overall scientific
expertise. Thus they were more able to place the postdoctoral project in
a larger picture.

One major drawback was the perception that fellowship applications often
got a rather perfunctory review. They were sometimes reviewed at the end
of the study section meeting when reviewers were often fatigued or
distracted by travel arrangements. There was also a strong sense that
often flaws were noted solely in terms of the scientific merit and not
placed in balance with the candidate\'s training potential, training
resources, and environment.

Some respondents felt that there should be a minimum number of
fellowship applications per study section, but the number was not
stated. One solution was carried out recently in the Biochemistry Study
Section. During the study section, the panel agreed to meet for three
days. On the first day, a subset of the regular members plus some adhoc
reviewers reviewed post-doctoral fellowship applications. After they
were finished, the ad-hocs left and the remainder of the regular group
appeared, and reviewed the R01 and other applications.

There was also some concern from several study section members that they
get better guidelines for weighting the categories to get an overall
score. Still another concern was the time needed for resubmission of a
fellowship application, where often what is needed is just some fixing
up of the written proposal. Fellowships might be an appropriate place to
consider 3 to 5 page responses to the summary statements.

Dr. Burgess, the second assigned discussant, also felt that there was
not enough guidance for the study section members, and when they
reviewed the fellowship applications together at the end of the meeting,
the reviews in too many cases, were rushed and perfunctory. Dr. Burgess
also preferred the experiment in the Biochemistry Study Section of
coming a day earlier to review fellowships. He agreed that regular study
sections have the expertise in dealing with the mentors, which is a
benefit, but in general tended to focus too heavily on the scientific
merit of the proposal instead of the training environment. The amended
application of 3 to 5 page responses to the summary statement should be
useful with fellowships.

Dr. Schachman, the third assigned discussant, felt there needed to be a
substantial number of fellowship applications, a \"critical mass,\" and
that having only a few postdoctoral fellowships tacked onto a study
section is a move in the wrong direction. In regular study sections,
reviewers tend to go into the details of the scientific proposal, and
feasibility becomes the dominant part of the overall appraisal, rather
than looking at the candidates\' background, the general areas in which
that individual wants to work, and the overall environment of the
laboratory to which that person wants to go. Fellowship applications in
study sections with similar expertise could be combined and reviewed by
one study section.

##### Discussion

Dr. Demsey noted that DRG has already considered Dr. Schachman\'s
suggestion, for example with the Visual Sciences-A, Visual Sciences-B,
and Visual Sciences-C Study Sections. Individually, the study sections
have small fellowship numbers, but together in one study section the
number could be adequate. Dr. Demsey also was concerned by the practice
of reviewing all fellowship applications on the last day of the meeting.
Perhaps even the second day is not good enough because the reviewers
have gotten into R01 mindset by then. Dr. Demsey suggested scheduling
fellowships either at the beginning of the meeting or the night before.

Dr. Cleveland stated that some Institute program staff want to be able
to recommend funding for promising young candidates who are likely to be
the next generation of leaders in American science, and they don\'t
believe that is how the applications are now being ranked. Dr. Cleveland
also had seen long technical diatribes on the proposed research from
candidates who were quite promising. If they went back to postdoctoral
fellowship study sections, there would be difficulties in getting
scientists to serve on those study sections. Within an IRG, however,
reviewers from existing study sections, or subcommittees, could be asked
to serve once or twice a year. The final point was that the criteria for
these candidates are vague. For example, they often are not certain
about the degree to which the candidate participated in the publications
listed. Also they often are not certain about the letters of reference
or to what degree the candidate actually wrote the application.

Dr. Van Houten noted that in her study section, reviewers work in good
faith to evaluate fellowships, but they are confused as to how to rate
the criteria, and spend a great deal of time asking for clarification.

Dr. Ehrenfeld remarked that one problem is that fellowships have
increasingly become supplements to scientists\' research grants, and too
often the principal investigators are writing those applications. They
are trying to get additional funding for their laboratory, and they will
pick up somebody they think can do the technical work. For the review of
fellowship applications, it is important to find senior people with good
records of training, who have the overview, breadth, and perspective of
the field. The workload is less; reviewing postdoctoral fellowship
applications is not as labor-intensive as reviewing R01s. With the right
kind of publicity and education, DRG ought to be able to get a group of
outstanding scientists to review postdoctoral fellowships, and whether
the review is tacked onto a regular study section meeting or a separate
group is probably largely a cost issue.

Dr. Manis felt that the reviewers, once they sensed exactly how they
were supposed to treat fellowship applications, did a good job with
them, but there was a problem with ad hoc reviewers who often did not
read the appropriate instructions and so wrote an R01 type review for
fellowship.

Dr. Pearson thought that fellowship reviews could and did attract senior
investigators who felt that they were influencing who would become
successful researchers. Also senior investigators who serve on
fellowship study sections did not have to worry about their own R01
applications being switched to another study section. It is also useful
to bring in newer investigators because they are closer to the
postdoctoral experience.

Dr. Walter Schaffer, Director of the Research Training and Special
Programs Office, NIH Office of Extramural Research, brought up the issue
of the interdigitation of scores. When applications are spread over a
large number of review groups, DRG is dependent upon the scoring
distributions being constant from one group to the other. Especially
with a large program with, for example, 50 applications, which are
reviewed by four or five different review groups, figuring out,
especially close to the funding line, which ones are better and how to
rank them, becomes a problem. This also causes a problem for program
officials attending the review meetings, because of scheduling
conflicts. Another problem was with the way percentiles were assigned to
fellowships. The fellowships were compared to an R01 base for each
group, and sometimes the distribution of scores for the fellowship
applications was nowhere near what is was for research applications.

Dr. Dean wondered if DRG should sequester the fellowships into 18
Special Emphasis Panels (1 for each IRG). That would build in
flexibility and create opportunities for bringing in reviewers in
whatever way the IRG thinks best.

Dr. Syed Amir, SRA of the Endocrinology Study Section, stated that his
IRG had reviewed fellowship applications using the flexibility inherent
in the IRG structure; and their experience had been extremely positive.
They avoided the problem that usually occurs when fellowship
applications are reviewed in R01 study sections by sending information
before the study section meeting, and also separating the fellowship
review from that for R01s by reviewing fellowships in Special Emphasis
Panels (SEPs). Dr. Amir did not find any particular problem in getting
the reviewers for the fellowships, because the workload is less than for
R01\'s.

Dr. Cleveland, felt that for the review of fellowships, they needed a
broad spectrum, from young investigators to senior investigators, and
that the collective wisdom of such a group would be able to decide
quickly among the applicants who are more likely to be the next research
leaders.

Dr. Yamamoto noted that private foundations that award fellowships use
senior as well as junior investigators. Part of the current
dissatisfaction, Dr. Yamamoto felt, was culture and education, giving
the reviewers better instructions. He hoped that the culture would
eventually change, so that reviewers would regard this as equivalent to
their other responsibilities and recognize training as one of the most
important things that they can accomplish.

#### XV. Closing Remarks and Final Discussions

#### Dr. Keith Yamamoto and the Advisory Committee

In his closing remarks, Dr. Yamamoto complimented the group on the
breadth and depth of their discussions, and he hoped that the Advisory
Committee would represent for the DRG Director a source of support for
peer review and the Division. He thanked Dr. Luecke for his role in
making this happen, and led the Committee in a round of applause for Dr.
Luecke for a job well done.

Dr. Yamamoto summarized some of the major discussion topics, including:
establishing open lines of communication with the Peer Review Oversight
Group (PROG), defining points of common interest and potential
interactions; increasing the efficiency of review, especially for
revised applications; encouraging experimentation with scoring
practices, but maintaining the integrity of NIH peer review; continuing
efforts like the [neuroscience integration
project](http://www.csr.nih.gov/REVIEW/neurorev.htm); periodically
reviewing the structure and review guidelines of the IRGs and study
sections; examining further the complex issue of patient oriented
review; and improving the review of fellowship applications.

In the ensuing discussion, Dr. Van Houten proposed establishing a
working group to examine the guidelines and boundaries of the study
sections. Her study section is confused as to its limits and guidelines,
and needs some direction.

Dr. Manis added that even the chairperson of a study section is not
given many specific guidelines. His study section has experimented with
different methodologies, and while they have been pleased with the
results, this should be looked at by DRG management.

Dr. Ehrenfeld stated that she planned to meet with some study section
members soon to discuss some of these acute issues. There needs to be
more coordination with the various study section experiments, for there
are policy constraints, as well as some confusion about whether
something is policy, practice, or an interpretation of practice. This is
largely a communication and clarification issue, and she would
appreciate the participation and help of the Advisory Committee.

Dr. Thiel then proposed establishing a working group to discuss the
issue of the rapid turn-around of applications and to compare DRG\'s
ideas with the experiences of NCI. Dr. Hamkalo added that she would like
to see the discussions include postdoctoral fellowship applications.

Dr. Pollock praised the recent interactions of NIH staff with the
neuroscience community at an annual professional society meeting. These
types of liaisons should be continued and developed. For example,
patient oriented research could be part of presentations to some of the
large professional organizations such as the American College of
Surgeons and the American College of Physicians.

Dr. Wickens wondered whether there should be a new mechanism, apart from
the FIRST (R29) award, to fund beginning investigators with a fairly
large budget. Dr. Luecke responded that this topic was currently under
discussion at the NIH with a committee co-chaired by Dr. Ehrenfeld and
Dr. Marvin Cassman, Director of NIGMS. Dr. Ehrenfeld offered to present
the status of the committee\'s efforts and recommendations at the next
meeting of the DRG Advisory Committee.

#### XVI. Closing Remarks

Dr. Yamamoto thanked his colleagues for assisting the DRG, as well as
Dr. Luecke and the DRG staff for their willingness and efforts to
examine thoroughly the peer review process with the Committee.

There being no further discussion and no further business, the meeting
was adjourned at 11:45 a.m. on November 21, 1996.

We do hereby certify that, to the best of our knowledge, the foregoing
minutes and attachments are accurate and complete.

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Samuel H. Joseloff, Ph.D.

Executive Secretary

Division of Research Grants

Advisory Committee

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Ellie Ehrenfeld, Ph.D.

Director Designate

Division of Research Grants

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Donald H. Luecke

Acting Director

Division of Research Grants

/s/

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Keith R. Yamamoto, Ph.D.

Chairperson

Division of Research Grants

Advisory Committee

Note. These minutes will be formally considered by the Advisory
Committee at its next meeting on April 28 & 29, 1997. Any corrections or
notations will be incorporated in the minutes at that meeting.

![](media/image1.png){width="0.4166666666666667in"
height="0.4166666666666667in"}For questions or comments regarding the
DRG Advisory Committee Meeting Minutes, please contact Dr. Samuel
Joseloff <joselofs@drg.nih.gov>

Last Update: April 8, 1997