**DISCUSSION OF THE [REPORT ON THE RATING OF GRANT
APPLICATIONS](http://www.nih.gov/grants/peer/rga.htm)**

**Excerpted from the Minutes of the Meeting of:**

**DIVISION OF RESEARCH GRANTS**

**ADVISORY COMMITTEE**

May 21, 1996

Natcher Building, National Institutes of Health

Bethesda, Maryland

**ADVISORY COMMITTEE MEMBERS AND AD HOC PARTICIPANTS**

**THOMAS BRACIALE, MD, PhD, CHAIR.** Professor and Director, Beirne
Carter Center for Immunology Research, University of Virginia,
Charlottesville, Virginia

**LUCY ANDERSON, PhD,** Chief, Perinatal Carcinogenesis Section,
Laboratory of Comparative Carcinogenesis, NCI, Frederick, Maryland

**ROGER BAKEMAN, PhD,** Professor, Department of Psychology, Georgia
State University, Atlanta, Georgia

**DAVID CENTER, MD**, Professor/Chief, Pulmonary & Critical Care
Medicine, Boston University School of Medicine, Boston, Massachusetts

**DON CLEVELAND, PhD,** Professor, Department of Medicine, University of
California School of Medicine, La Jolla, California

**LEODIS DAVIS, PhD,** Professor, Department of Chemistry, University of
Iowa, Iowa City, Iowa

**HENRY FRIEDMAN, MD,** Professor/Chief, Division of Pediatric
Neuro/Oncology, Department of Pediatrics, Duke University Medical
Center, Durham, North Carolina

**SHIRLEY HILL, PhD,** Professor of Psychiatry, Psychology and Human
Genetics, Western Psychiatric Institute and Clinic, Pittsburgh,
Pennsylvania

**RONALD HINES, PhD,** Professor, Department of Pharmacology, Wayne
State University School of Medicine, Detroit, Michigan

**OLGA JONASSON, MD,** Director, Education and Surgical Services,
American College of Surgery, Chicago, Illinois

**RON LINDAHL, PhD,** Professor/Chairman, Department of Biochemistry and
Molecular Biology, University of South Dakota, Vermillion, South Dakota

**MARY TREINEN MOSLEN, PhD,** William C. Levin Professor of
Environmental Toxicology. Director, Toxicology Training Program,
Department of Pathology, University of Texas, Medical Branch at
Galveston, Galveston, Texas

**RAYMOND NOVAK, PhD,** Director, Institute of Chemical Toxicology,
Wayne State University, Detroit, Michigan

**SUZANNE PFEFFER, PhD,** Associate Professor, Department of
Biochemistry, Stanford University School of Medicine, Stanford,
California

**JOHN STARKWEATHER, PhD,** Professor Emeritus, University of
California, San Francisco, California

**ELIZABETH THEIL, PhD,** Professor of Biochemistry and Physics,
Department of Biochemistry, North Carolina State University, Raleigh,
North Carolina

**MAHARAJ TICKU, PhD,** Professor, Department of Pharmacology &
Psychiatry, University of Texas, Health Science Center, San Antonio,
Texas

**JUDITH TURGEON, PhD,** Professor, Department of Human Physiology,
University of California School of Medicine, Davis, California

**KEITH YAMAMOTO, PhD,** Professor/Chairman, Department of Cellular and
Molecular Pharmacology, University of California, San Francisco, San
Francisco, California

**SAMUEL JOSELOFF, PhD, EXECUTIVE SECRETARY.** Division of Research
Grants, NIH, Bethesda, Maryland

### Proceedings:

**DR. ATWELL**: I noted that at your last meeting you discussed the
report on the Rating of Grant Applications. But your leadership felt it
was important to give you an update on where we are with this, and that
is what I am here to do today.

For those of you who may not be tuned in to some of this, I will start
with just a little bit of background on the report.

The committee that dealt with this was actually a subcommittee of one of
the trans-NIH extramural reinvention committees, the Committee on
Improving Peer Review. Don Luecke and I have chaired that committee for
what seems like 15 years, but I don\'t think it has really been that
long. And Tony Demsey has stepped in since Don has had other duties
assigned.

The Improving Peer Review committee has had a number of tasks, one of
which, of course, is streamlining, which you have already dealt with.
Another is the amended applications issue that is now being dealt with
in a joint program and peer review committee. We also have a new report
coming out on flexible appointments to initial review groups that
complements what you have just been hearing about in terms of
rearranging the structure of review groups, in terms of the negotiated
terms of service of members.

But one of the activities that has really consumed us is the task to
look at a very specific aspect of peer review, which is how we actually
score applications, and then report the results of that scoring process.

Now, since the report has come out, there have certainly been, from my
perspective, some misinterpretations of what we were about. One of those
was that we were trying to fix peer review by tinkering with scales and
criteria. Of course, that was never the intention of the project. It was
never the assignment of the project. So, I want to make that clear in
the very beginning.

The group first met, actually, in December of 1994. So, it is almost two
years ago. We were charged with looking at the rating process,
especially with the knowledge of psychometrics and decision making. That
is, there is a science associated with assigning ratings, numerical
indices, to complex dimensions. We all recognized that our existing
process was really partly responsive to that literature. So, the task
was really to re-look at what we were doing and see if things could be
improved.

The committee met regularly and all too often during 1995. This was
really a very intensive committee. We also had ourselves supplemented by
consultation with outside consultants. In fact, Roger Bakeman, one of
your members, participated in the fast and furious e-mail on this topic.
The concepts that were developed were discussed as we went along with
program staff and review staff. So, there was a kind of dialogue as we
were developing our thoughts on this.

The report that we actually came up with was, in fact, delivered to
Wendy Baldwin in December of 1995. Then there was extensive discussion
with some of the in-house \-- by that I mean within NIH \-- committees
that would be affected by this, namely, virtually everyone at NIH.

It certainly was a peer review issue. And any changes in how scoring is
done would very much affect program. In fact, from the very beginning we
were very much aware that program staff, those people who were actually
going to make funding decisions on the basis of what is reported out,
had to be involved in the process. So, the committee in fact did,
indeed, have program people on it. It wasn\'t just from the peer review
perspective itself. We rather viewed program staff as one of the major
consumers, in a sense, of what we were about.

Our report was also then discussed by the RPC (the review policy
committee), POPOF (the program staff), and EPMC, the extramural policy
and coordination people at NIH, like me.

This slide just shows some of the external consultants, from the
essentially psychological and psychometric community, who were involved
with us.

The first thing we decided was to look at the criteria by which
applications were reviewed. Obviously, this is fundamental. In fact, we
concluded that there was nothing wrong with the criteria that we
currently use. They are all listed in the application kit and the
reviewers generally use them. However, we thought that, in fact, the
focus could be sharpened up in terms of the review.

What has happened, as funds have gotten tight, is that in some sense,
reviewers sometimes find it very difficult to make distinctions among
lots of grant applications that are, in fact, very good. So, in fact,
what happens is they start to look for what is wrong with an application
rather than what is right with an application.

So, we decided that we would like to put perhaps more emphasis upon one
of the criteria that is already there; namely, the scientific
significance of the application.

There has been some confusion about what is meant by the scientific
significance. Some people assume that it means the programmatic
relevance, importance to the mission of the funding institute. That is
not, in fact, what is meant by significance here. What is meant is if
the application, if the project is done in the way it says it will be
done, and it is successful, how big an increment to our knowledge is
likely to result. Is this going to be just a little notch in an already
well-established area? Is this going to make a fundamental, original
contribution? Is this going to change the way we think about a field?
That is the aspect of this that we were particularly concerned about.

Included in this was the notion of creativity and innovation. I know
that in your previous discussion of this, the group recommended that
there, in fact, be another criterion added to highlight even further the
importance of creativity and innovation in research; that we not just
have good grant applications that are more of the same, but not really
ground-breaking, getting really good scores. This was always part of
what we considered to be part of the significance. That, of course, is a
debatable issue.

Now the second criterion is the one that I think, in fact, mostly now,
in my opinion, drives most of the review, and this is the experimental
approach, the methodology, the details of the research plan, all of the
particulars of the research as it is to be conducted. If you look at
summary statements, you will certainly find that probably at least 75
percent, if not more, of the critique deals with this aspect of an
application. So, there is really nothing new in here. This is what is
the methodology.

Now, what we are calling feasibility \-- that may not be such a good
term and we may need to rethink that \-- what we are calling feasibility
is, if the research plan is actually okay and it is a really important
thing to do, as judged through the significance and the approach
criteria, how likely is it that this is going to be carried out
successfully. The indicators that we subsumed under this were the usual
\-- availability of resources and equipment, access to special
populations, reagents \-- but also in this case we considered the track
record of the applicant to be a crucial aspect of feasibility.

When we look at track record, what are we doing? We are kind of gambling
that what they have done in the past is going to be a predictor of what
they are going to do in the future. When we say somebody has been
productive in the past, we mean that it is a good bet that they are
likely to be productive in the future. For us, that came together under
how likely is it that this is going to be carried out successfully.

In the case of beginning investigators, of course you have your
references and all that, in the FIRST Award, and nothing would change
there. Previous progress on a competing renewal would be considered as
part of that track record. In the case of beginners or a new area for an
old investigator, the pilot data. Again, that is a clue that things, in
fact, are likely to go well.

So, this is again just a repackaging of things that we already use to
assess the likelihood of success of a project.

The rating scale is one of the other recommendations of the committee.
There are some things that we consider trivial like the first one: NIH
is the only system we could think of where a lower number means better.
So, we in fact would recommend that we go along with the rest of the
world and make it a little more intuitively obvious that a bigger number
means more of the item in question, namely, scientific merit.

The next part is, in fact, one of the keys of what we were doing in the
first place, which was asking what kind of judgements people can make
with what kind of reliability.

It is known from the psychological literature, both from very basic
sensory attributes to complex judgements, that the range is somewhere
between five and ten discriminations at a time that can be reliably
made, and we are recommending that that be done, in this case, on each
criterion.

We are not aiming for any particular number, so it is just that we had
to limit the range people use. In theory we currently have a 41-point
scale. In fact, people don\'t use those numbers because, in fact, they
can\'t rate according to all those numbers. So, we end up, in fact, if
you look at the distributions, using about 10 numbers. So, this is the
recommendation based on the literature as well as peoples\' actual
behavior.

The third item here, anchoring, really has to do with whether you have
descriptors at lots of points along there, or you just say the most and
the least of whatever aspect you are talking about.

The standardization process of scores is a controversial issue. The
group, of course, is faced with the issue of what do you do to take into
account the varying behavior of different individuals and different
groups. Currently, of course, we have the percentile system. Before that
we had a normalized scoring system. All these are just attempts to take
into account that group behavior or dynamic can influence significantly
the actual evaluation and scores that are assigned.

With the changes in the more flexible arrangements between study
sections that you are hearing about, with people coming and going and
serving on different groups and a number of the other changes that we
anticipate, as well as just looking at issues concerning
standardization, the group is recommending that the standardization be
done by individual rather than by group. An individual will develop a
history of scores and their own distribution will create a scoring
pattern. Then if they don\'t spread their scores naturally, they will be
spread for them.

Now this, of course, does bring up a number of other issues. The issue
of are individual voters influenced so much by the behavior of the group
that they are in that a person who may be a very tough grader in one
study section goes to another study section and sees that everybody else
is being a lot more lenient. So, they then switch their behavior to go
along with that group. If you were doing it by individual, how would
that affect things.

So, there are issues concerning this; issues about who should vote on an
application. Currently \-- and we are not, ourselves, proposing anything
different \-- everybody gets to vote. Everybody\'s vote counts equally.
But you could establish things in a different way.

We are also recommending that the scores be reported out on the same
scale that is used by the reviewers. That is, whatever they have, the
0-10, 0-7 scale, that they be converted back to that for each criterion.

One of the most controversial recommendations was the overall score.
Does there need to be one, first of all? If there does need to be one,
should it be calculated on the basis of individual scores for each
criterion, or should reviewers make a global rating of scientific merit,
as they do now. The recommendation was that they not do that, on the
basis of the literature that says that, while people indeed believe that
they are very good at making overall ratings, in fact they do better if
they rate by disaggregated criteria.

The current status of activities is, as I say, that the report has been
delivered to Wendy Baldwin. Now internally there is considerable further
discussion being generated. There are four study groups that have been
created, with people mostly other than those who originally were
involved in generating the report. The four study groups will deal with
the criteria, with the rating scale, with the standardization and with
the overall score issue. There is a liaison to each of those groups from
the original panel, but they are liaisons to it. They are not chairing
those committees.

A [resume of the full report](http://www.nih.gov/grants/dder/rga.htm) as
well as the [full text](http://www.nih.gov/grants/dder/rga.htm#download)
are now being put up on the NIH Home Page. I believe you have already
had distributed to you the overview that will be going up on the Home
Page.

These internal study groups are being asked not to redo the report or to
make different or separate recommendations, but rather, to look at each
of the recommendations relevant to their group\'s charge and to think
about what the implications would be if each recommendation were
implemented. Is there additional information that would be needed by the
decision maker \-- such as Wendy or Harold \-- that would be necessary
to decide whether a particular recommendation should be adopted? Are
there any studies that should be generated that we could do relatively
quickly? They are also to develop an implementation plan for each
recommendation if it were to be adopted. There is no assumption that any
of them would be adopted. But if they were, how would it best be done to
prevent too much disruption to the system. Now this is to be completed
by August 1st of 1996. Obviously, this is a quick process.

Additionally, comments will be solicited from the scientific community
via the Home Page after it goes up. People, when they read the
[overview](http://www.nih.gov/grants/dder/rga.htm), would be hyperlinked
to the [full report](http://www.nih.gov/grants/dder/rga.htm#download).
From there, they could make comments through the Internet that will be
looked at by all of us and by Wendy. It is also likely that there will
be a more focused group established that may engage in perhaps a virtual
meeting via the Internet to discuss things in more detail.

The Peer Review Oversight Group would be the obvious group to consider
these things, but it is not yet active. So, we are not quite sure about
the timing of that. But my guess is that eventually it would be
considered by them.

The timing is such that a decision would need to be made about these
things probably in January of 1977 if we were to affect fiscal year 1998
funding of applications. So, the time frame on these things gets quite
strung out.

Now, it is important to bear in mind that the report was written
explicitly such that it is not a package, that any of the
recommendations could be adopted independently of any of the other ones.
Revised criteria could certainly be implemented without hardly any
change at all. You could just review by criterion, write the summary
statements in that way, but not vote by criterion, for example. The
rating scale itself could be changed without changing anything else,
just using our current system but switching the rating scale.
Standardization processes could be changed without changing anything
else.

So, we hope that the community, when they think about it, will bear that
in mind, that we are not talking about an all-or-none deal here. I think
the assumption is that not everything that is recommended, in fact,
would ever be implemented. These are ideas that are thrown out to be
considered and some of them perhaps adopted.

This is just the timing issue that I mentioned a moment ago. And at this
point, I will stop. I think you have discussants. I will certainly take
questions in terms of the process for how we go on from here.

**DR. BRACIALE**: Thank you, Dr. Atwell. What I would like to do is hear
from Dr. Yamamoto. First, I have one question to you and that is, I see
the advantage of this rating system change for program staff and
institutes. But how do these rating changes benefit the extramural
community or, indeed, even the Division of Research Grants?

**DR. ATWELL**: The issue in terms of feedback to applicants, I think,
is a real one. In terms of the criteria, for example, making it clear
that a particular application, while it may be absolutely perfectly
designed and feasible, may not be all that significant. Getting that
message across is now a very difficult one to do. We certainly encourage
reviewers to make statements about that now, but how often have you seen
statements like that in a summary statement? If you rate by those
criteria or if you were asked to make explicit statements about that in
a separate section, I think you would be getting clearer messages to the
applicants.

In terms of some of the others, I am not sure it would matter, because
you are going to get the same kind of information. But you are right. We
think it would help the program and councils to get a better handle on
what is going on.

Also, different institutes may have different values in terms of how
they want to weight things. We can talk about weighting the criteria. We
have recommended essentially that they be equal, but that certainly is
up for grabs, and anyone could do it any way they wish, or whether it is
to be an NIH-wide dictum or whether it would be left to institute
discretion.

For the reviewers, I think it would have impact, especially issues about
the standardization. There, depending on if you did what we suggested,
for example, and standardized by individual, then an individual\'s
voting behavior becomes crucial. It would not take reviewers long to
realize that if they voted all good scores, that they would be
essentially disenfranchising themselves; that if you vote all, say,
sixes and sevens, sevens being the best, that that would probably be
normalized back to a four average and you would have essentially taken
yourself out of the influence. So, reviewers are smart people and they
would understand that.

**DR. YAMAMOTO**: A set of questions. First, I wasn\'t clear on your
recommendation not to arrive at an overall score. You would score the
individual criteria and then what would you do?

**DR. ATWELL**: This is what is open for discussion. The group is
recommending that if there is an overall score \-- we didn\'t say that
there had to be \-- if there were, that we recommended that it be
derived from the criteria scores, not be rated separately. It is not
that there is interaction among the components or anything in that way,
to make it kind of overall a better deal. So, I would personally prefer
a derived score.

Part of the problem here is that we are all used to making global
assessments. We think we are pretty good at it, and by and large we are.
Nobody is claiming that bad science is being funded or that absolutely
stellar science is going unfunded in any kind of a major way. We are
just not into making it be as rational as we can.

People believe that they are very good at this. One of the things that
happens is that everybody believes they are an expert on their own
behavior and everybody knows in their guts that they can do this. They
are good at this. You know, I can make these judgements! But when you
actually subject some of this to study, it turns out that people in fact
do make more consistent, reliable judgements, even repeating their own
judgements, if they do it by criterion.

**DR. YAMAMOTO**: I understand that. I just didn\'t understand what you
would do with the criterion score in terms of making a final judgement
about the grant application.

**DR. ATWELL**: My guess personally is that there will be an overall
score. That is just a personal guess. If there is not an overall score,
what would be reported is that there were three scores, a matrix. Then
the institutes would have to either themselves combine them or use them
individually as a matrix for deciding who gets funded.

Personally, I think that would be unwieldy and that is not going to
happen. But that is a personal opinion. My guess is that there is going
to be an overall score because of the vast numbers of applications that
have to be interdigitated. That is not inherent in what is being done.
You could give the matrix of three scores. Some program directors, in
fact, on our committee were very much in favor of doing just that.

**DR. THEIL**: Is this recommendation #4 that we are talking about, the
one that says reviewers should not make global ratings of scientific
merit?

**DR. ATWELL**: There is a difference between not making a global rating
and having an overall score. What I was referring to was whether there
would be an overall score at all. The committee did recommend that if
there were to be an overall score, that it be derived from the criterion
scores, rather than assigned separately by the reviewers and just give
their overall rating. That is to be consistent with what the majority of
the group believed to be what was being said in the literature. Now,
having said that, I will tell you that there was not unanimity on that
within the committee, nor amongst the experts. Maybe Dr. Bakeman would
like to speak to that.

**DR. BRACIALE**: Actually, I would very much like to hear from Dr.
Bakeman, but I would like to hear Dr. Yamamoto first and then we can
have a general discussion on the two positions.

**DR. YAMAMOTO**: I passed around a handout (reproduced below). Really
it is a modest variant on what we just heard from Connie (Atwell). But I
think in my view it has some elements that I think are worth
considering.

> ***RATING OF GRANT APPLICATIONS***
>
> *A proposal for discussion by the DRG Advisory Committee*
>
> ***THE ISSUE**. Especially in times of constrained resources,
> evaluation of grant applications can become imbalanced, as applicants,
> study sections and councils are acutely aware of the importance of
> \"cost efficiency\". Consequently, the probability of success of a
> research plan (\"feasibility\") may take on an overriding importance
> in judging proposals and making funding decisions*
>
> ***GOAL**. Develop a rating methodology that assesses for each grant
> application all of the contributing elements that define research
> excellence, yet retains sufficient flexibility to permit the study
> section to weigh the individual characteristics of each application.*
>
> ***BACKGROUND**. The Rating of Grant Applications (RGA) subcommittee
> of the Committee on Improving Peer Review (IPR) has prepared a draft
> report, not yet finalized, with several principal recommendations
> relevant to this issue (RGA recommendations concerning the rating
> scale will not be addressed here). In brief, the recommendations are:
> \[a\] to assess applications on three criteria\-- significance,
> approach, feasibility; \[b\] to numerically score each criterion;
> \[c\] to use a common algorithm\-- the subcommittee favored the
> arithmetic average\--to compute overall merit.*
>
> ***PROPOSAL**. The RGA subcommittee proposal has substantial merit.
> What follows is a variant of that scheme with alterations in the
> criteria and rating procedure that may contribute to achieving the
> goal.*
>
> *I. Review Criteria*
>
> *A. Impact*
>
> *Does this study address an important biological problem? If the aims
> of the application are achieved, how will scientific knowledge be
> advanced? What will be the effect of these studies on the concepts or
> methods that drive this field?*
>
> *B. Feasibility*
>
> *Are the methodological strategies appropriate and the experiments
> well designed and controlled? Does the applicant identify risky or
> problematic elements in the application, and make provisions for
> possible failure of these elements?*
>
> *C. Creativity/Innovation*
>
> *Does this proposal create new juxtapositions of ideas and concepts,
> challenge existing paradigms, develop new methodologies or
> technologies?*
>
> *D. Investigator/Environment*
>
> *Is the investigator qualified and well suited to carry out this work?
> Does the scientific environment in which the work will be done
> contribute to the probability of success? Do the proposed experiments
> take advantage of unique features of the scientific environment?*
>
> *II. Rating Procedures*
>
> *A. Individual criteria*
>
> *In addition to narrative comment on each review criterion, reviewers
> would assign a letter grade \--e.g. A through D, supplemented with
> range descriptors (e.g. A=top 5%; B=5-25%; C=25-50%; D=bottom 50%)\--
> to each. Thus, reviewers would declare an explicit, but only
> semi-quantitative, judgment on each criterion, which will be useful
> both to the applicant and for arriving at the overall quantitative
> judgment.*
>
> *B. Overall score*
>
> *Reviewers would assign a single quantitative score to each
> application, using the recalibrated scale of 1.0-2.5 plus 50%
> unscored. Reviewers would be encouraged to apply their judgment to the
> relative importance attached to each criterion in arriving at the
> overall score, recognizing that each application might have unique
> characteristics that may make a particular criterion more or less
> important.*

\-

**DR. YAMAMOTO**: The issue, as I see it, and as I outlined here, takes
a slightly different slant from what you just heard from Connie. That is
that in my view, in times of constrained resources, what is given the
overwhelming strength in review is the feasibility side. Study sections
and councils have become aware of themselves as sort of cost efficiency
experts, and they really look to see what the probability of a series of
experiments working is.

The goal as I see it \-- and again, I think this constitutes a real
challenge \-- is to develop a rating methodology that assesses for each
grant applicant all the contributing elements that define research
excellence, yet \-- this is important \-- yet retains sufficient
flexibility to permit the study section to weigh the individual
characteristics of each element. So, you want something that is
standardized, yet has flexibility built into it.

We just heard a very good background from Connie. I don\'t need to
re-state it. But the proposal that I am making actually does not deal
with the rating scale issue. I might mention that at the end, while I am
thinking about it, but really, instead, I will deal with the criteria
issue.

So, the recommendations that are relevant that we have heard, I think,
are three; to assess the applications on three criteria \--
significance, approach and feasibility \-- to numerically score each of
those criteria, and to use a common algorithm. In the draft report that
I read, the subcommittee favored an arithmetic average which gave the
overall merit of the applications. I suggest the following variant that
has two elements, one covering review criteria and the second covering
rating procedures.

As for the review criteria, I recommend four. Again, this is sort of a
re-carving of what we just heard.

I should just say something first about the significance point that you
made. You said it correctly, that the applicant community as well as the
study sections have come to view significance largely as a statement of
how the work relates to the application. It is going to be difficult to
retrain the community to get them to use the term in a different way.

Also, impact, which was included in yours. Does this study address an
important biological problem? If the aims of the application are
achieved, how will scientific knowledge be advanced? What will be the
effect of these studies on the concepts or methods that drive this
field?

Feasibility, are the methodological strategies appropriate and the
studies well designed and controlled? Did the applicant identify risky
or problematic elements in the application and make provision for the
possible failure of these elements?

Creativity and innovation, I think, is an important one because I think
the applicant community to some extent is concerned about this in this
race toward feasibility. I think it does favor established investigators
with large laboratories. I think it is important to make this explicit.
This (DRG Advisory) Committee recommended this additional criterion
before (at its November 1995 meeting). Does the proposal create new
juxtapositions of ideas and concepts, challenges, develop new
methodologies or technologies?

Fourth, rather than track record, I called it investigator and
environment. Is the investigator qualified and well suited to carry out
the work. That is going to roll in both track record and, if there is no
track record, either because the investigator is new or is an
established investigator entering a new field, is it the kind of person
who has shown that they are able to make new advances in areas in which
they didn\'t previously have their expertise.

Does the scientific environment in which the work will be done
contribute to the probability of success? So, somebody doesn\'t normally
do genetics, but he is in an institution where there is a lab next door
that he is going to be able to use.

Do the experiments take advantage of the unique features of the
scientific environment? If somebody is writing an application from an
undergraduate institution, did they design an application that is well
suited to undergraduate people in their laboratory getting out the work.

In the rating procedures area, I have two points to make. One relates to
the individual criteria. What I recommend is that reviewers not only
explicitly comment on each of those four areas, with a paragraph or more
on each of those four areas. In addition, I recommend a letter grade; a
letter grade requires that the reviewer actually make a commitment about
what he or she thinks about that particular explicit review criterion.

They could be, as I indicated, supplemented with range descriptors.
Another one that I like would be to have the top five percent be the As,
so that it is really skewed. It is not evenly cut at all. Then five to
twenty-five percent be a B, twenty-five to fifty be a C and then the
bottom half be a D.

So, the reviewer would have to take a stand, but it would only be
semi-quantitative. But I think it would help when the reviewer reaches
the point of making an overall scored evaluation, he or she would
explicitly consider what they had done in each of these areas. I think
that is really important and useful. In addition, for the applicant I
think it could provide really critical information.

All of us who have sat on study sections are used to seeing grant
applications slide through on the A-1 and A-2 and A-3 where really the
feeling of the study section is that it is not very interesting. The
tendency is for reviewers not to really hit a grant applicant with that,
some statement like your work is boring. So, it doesn\'t get done. So,
instead, we talk about technical flaws in the application. Those get
corrected with each round and the applicant becomes increasingly
frustrated as the scores commonly tend to go down. If, instead, the
person got a D on impact and an A on feasibility, he or she would know
that the study section really questioned whether it would make any
difference tomorrow if you knew the answers to all these questions
today.

On the overall score, I would recommend that a single, quantitative
score be assigned. I was using this recalibrated scale notion. I don\'t
know what has happened with that, but it was distributed. Even the 1.0
to 2.5 plus 50 percent unscored would reduce the 41 criteria points to
17 or 16 numericals. That plus everything else would help move things in
the right direction and I think would represent less of a drastic change
to the study section members who are used to using the criteria.

In any case, whatever scoring system is used, the reviewers will be
probably encouraged or mandated to apply their own judgement about what
kind of weight they would give each of the criteria. The reason for that
is that each application is going to differ in what the importance is of
the individual criteria. I really value a lot the study section members
who apply that judgement. I think they do it well. Especially if they
have divided their reviews in this way, they will look at each of these
criteria separately and weigh them and make a judgement about whether in
this particular case they think the idea is so creative and original and
important in that way, that they want to go out on the limb and give it
a high numerical score, et cetera.

So, I think it is important to come up with an overall score. I don\'t
want it to be the arithmetic mean. I think that would tie the hands of
the study section members unnecessarily. I would rather have them apply
their subjective wisdom to making their evaluation.

**DR. ATWELL**: Many of those same points were ones that were expressed
within our committee, and as I said it was not unanimous within the
committee. We also did discuss using a letter scale for the criteria
rather than numerical.

**DR. BRACIALE**: Let\'s get some comments on Dr. Atwell\'s proposal and
possibly Dr. Yamamoto\'s counter-proposal.

**DR. TURGEON**: Relative to the last point about whether we have three
or four criteria and whether they are weighted, most fellowship
committees already use multiple categories and they are not weighted
evenly. They are not independent variables. They impact on one another.

Those committee members quite easily deal with evaluating each and
giving each an internal rating, but coming up with an overall score that
I think is much more meaningful than looking at each one in isolation. I
think that same principle can be applied to this situation.

Also, just a procedural question, who comprises the four study groups?
Are these study section members?

**DR. ATWELL**: No, these are NIH staff people. The external comments
are going to come via the Internet and probably a focus group \-- I am
not sure calling it a focus group is the right idea \-- but it will be
some kind of a dialogue that Wendy will set up.

**PARTICIPANT**: Actually, we are setting that up now. We have got about
43 names from all of the ICDs or as many as wanted to contribute names.
We will have a short orientation teleconference for them and let them
look at the documents.

We hope to have sort of a bulletin board set up where they can pose
their comments, look at each other\'s comments, re-pose comments. Then
we will have an hour\'s teleconference where they can discuss those
comments.

**DR. ATWELL**: Most of the people that were recommended, I think, by
the institutes are either current or recently past study section
members.

**DR. PFEFFER**: I want to comment on the scoring and rating of the
applications. I think that clearly one of the goals of this whole
project is to give better information to applicants. I think that both
of the proposals are excellent in that regard. I happen to prefer the
grading on the individual criteria. But I am concerned about the rating.

I know that another goal of this project was to deal with the issue of
spreading of scores, so that the final score actually reflects where
that application rates relative to others and to give a more valid
indication. I don\'t think it matters if you go from a 1.0 to a 2.5 or a
0 to 10. I think you will still get bunching up of scores.

It is an interesting idea to do individual standardization per reviewer.
But I have a real problem with that, and I will tell you what it is. One
of the wonderful aspects of the study section is the collegiality among
the members. We listen to each other\'s scores. We listen to each
other\'s points of views. If you have a member on a study section
constantly rating 8, 8, 8, 4, 4, 4, whatever the number is, you stop
listening to that person. They lose validity in the group. So, they
suffer. It is not only that their scores are less contributing to the
final outcome for that application, I lose that person in terms of being
able to understand what they are saying about this application.

What has worked very well on the review panel that I serve on is, we ask
three questions of everyone as they present each application. How do you
generally feel about this application. But just as importantly, where
does it sit in your pile that you did and how big is your pile? Do you
have a good pile this round or a weak pile this round? Right away when
somebody says, this is a 1.5 but it is seven out of fifteen, we say, oh,
hold on here, that is not possible, because that means that half of your
applications is over 1.5. This has worked extremely well for us. We
learn about each other. But our scores on that review panel have spread
and it is a cultural distinction. But immediately, a new member, a
temporary member coming on learns very quickly that in order for us to
really see how to respect their opinions, they have to move things
around.

So, my own view would be strongly against a computer standardization,
even though if we could do it right on site so that the computer would
put the real score up on the board the rest of the panel would benefit
from that, because I need to know really where does that application
rate relative to the whole group that we are discussing.

I also want to point out the importance of a single score at the end and
not splitting the scores and averaging them, and I will give you an
example why that is so important. We handle a large group of
applications that are in the boring category. They get a 1.0
feasibility, 1.0 approach, 5.0 in terms of impact. We have a large
number of those. I can tell you, I have immediately made them impossible
to distinguish, because if you did a numerical average of those three,
it would be very different from the other kinds that we get, which are
what we call high risk, high gain. It would be a 0.5 on the impact and
maybe a 3.0 on the feasibility. But in terms of the overall importance
of what we want, as a group of reviewers, to see supported, the average
would come out very different. That is because we are weighting these
different variables differently.

I totally support the idea that Keith and your panel have suggested, to
ask the reviewers to be more specific in specific categories and to give
some evaluation. But in the end, respect the beliefs of the reviewers,
the beliefs and the opinions of the panel members to give you an overall
merit. In the end, I think you will have better discrimination of those
boring groups, which you really need to discriminate, not as an average
of three or four criteria, and you will also allow the superstar, high
risk, high gains, to move up in the group. I think in the end that is
going to be more beneficial to the program directors than an average of
three scores that you may think has given you more information.

**DR. NOVAK**: I would just like to make a few comments. First of all, I
think this is an extremely important area and I am very happy to see
that it is being addressed. I do like Dr. Yamamoto\'s guidelines and I
agree really with what he has said in terms of technical feasibility on
a boring grant still resulting in a boring score.

The only two points I would like to add is, number one, I think the
review guidelines should be tailored to each type of grant application.
Right now it appears as though we focus primarily on RO1s. But some of
the issues that have come up in the past day addressing translational
and clinical research need to be thought about in terms of different
grants, and especially grants that have different objectives, such as a
KO8, which is a physician/scientist career development award.

The second thing I would just like to mention is that in one of the
guidelines, going rapidly through the list that Dr. Yamamoto circulated,
is that we should have some area of emphasis in there on human health
relevance. Again, this would be important from the perspective of
translational or clinical research.

I would also like to add that, in addition to providing information to
reviewers in these different areas as the applications are reviewed, I
also think that these guidelines, first of all, serve to assist new
reviewers coming onto study sections. It is very difficult to begin this
process without some detailed instructions, and even with those, it is
difficult.

All of a sudden one day you show up as an ad hoc reviewer, and you see
25 people staring at you at a table and you think, oh, my gosh, did I
really do this correctly. So, having these guidelines really helps from
that perspective.

Secondarily, it also helped to establish a baseline or, if you will, a
commonality for emphasis in discussion. It is more difficult, I think,
when these things are established, to get side-tracked on relatively
minor issues that may, in the end, because of the length of discussion
devoted to them, detract from the overall emphasis and sometimes,
unfortunately, from the score.

So, as I conclude my remarks I would like to say I really do like a lot
of the remarks that have been put forward and I think there are some
crucial areas that should be examined further, and perhaps in the
committees, if former members or existing members of study sections are
allowed to have input, this can be further refined to help both DRG,
IRGs and study section members as well as the applicants.

**DR. BAKEMAN**: I find myself in the wonderful position of agreeing
with most of the comments that have already been made. What I think they
lead to logically is to a rejection of the first four recommendations
made by the RGA committee.

All of this can argue for a single summary score. They argue that way,
that reviewers may give a single score. They argue that way because we
recognize that criteria vary from content area to content area, from
grant to grant to grant, from time to time, and we want that judgement
to fully remain to the people who know most, the reviewers.

Given that, you don\'t want to have people rating separately and then
averaging, because then whatever algorithm is used will always be
controversial.

What you do want instead is each study section \-- which I am sure
happens now \-- discussing what are the criteria, what are the existing
list of criteria now to guide the study section. It is what, seven,
eight or nine in number? Any intelligent group that sits down to argue
about criteria will come up with a list that is defensible. Thus, I
don\'t think that that list, whether it is the current committee\'s or
yours, Keith, should be cast in stone across the DRG. But each study
section discuss their criteria, but then come to the single rating.

I would also add that not everybody reads the psychometric or decision
making literature as necessitating separate judgements. The question is,
by and large to some extent, does it generalize to groups of
scientifically trained people making complicated decisions. I don\'t
read it as necessitating that. Others do. I think it is always the
question with behavioral research, I might add, as to what is the
appropriate group that a given body of literature generalizes to.

I think we want to keep the flexibility, we want to keep the
intelligence about criteria within the study section, within the
reviewers. The way to do that is to appreciate the conversations that
the committee had, to reject these three criteria as three separate
judgements. This is the first part.

The rating scale is not controversial. That is fine; whatever. We will
adjust, whether it is seven scales, five or seven, plus or minus two.

The third proposal is very fascinating. Let me say that I find it very
intellectually appealing, but at the same time see problems. The two
major problems with standardizing within reviewers is, what do you do
with streamlining, and what do you do with temporary reviewers who
don\'t have a record?

I make two final points. I find the standardization of individuals
appealing, but it is not a matter of deciding, yes, we are going to do
it beginning on a certain date. It is only arithmetic. So, all that
needs to be done is it needs to be done in parallel with the current
system for a period of time, for all or some study sections. There are
two systems in parallel, you see, because what standardization within
reviewers requires is record keeping, and that is not done currently,
which is why we can\'t look at what has been done the last five years.
Unless records are kept \-- if I am wrong on that and there are, indeed,
the individual data for each individual \-- but I have been told that is
not kept.

**DR. PFEFFER**: It is not kept, but I have seen print outs.

**DR. BAKEMAN**: So, it means we have to begin to keep that data now for
a period of specified time and we look at the reports we generate.

But let me make my final, and I think really important, point. Who is
the consumer of the ratings and the remarks that are made? In the
narrative where we have to accept the responsibility for summary
judgement, that is where we really have to say, and no quantitative
system will absolve us of the responsibility of saying, slightly dull.
If those are our judgements, we must say that.

So, who are the consumers? Where do these numbers and narratives go
forward to? They go forward to institutes who then make decisions. They
tended in the past \-- as I understand it, some more than others \-- to
make it fairly mechanically: I must not deviate from this numerical list
I get, rank ordered in the end to three digits of improbable precision.
There is a responsibility that has to be taken for making judgements. I
think the focus and the tremendous energy spent on this report may have
been deflected energy spent where it might also have been profitably
spent in saying, how do we ask institutes who make funding decisions to
bring in and justify more of the qualitative concerns that the system
was designed to do. Yet it seems like they have been \-- and tell me if
I am wrong on this \-- have perhaps been evaded somewhat by, as I say,
misplaced precision. I think that is the kind of conversation that I
would like to see proceed.

**DR. ATWELL**: Actually, the major goal of any of this is to help us
have better decisions made. Yes, the concern about slavishly following a
three-decimal-point score that probably does not reflect that level of
precision was a concern.

There are other activities going on also in terms of the decision-making
process, for how you use whatever the output of peer review is, to make
your funding decisions. So, I think there is, indeed, a great deal of
activity around NIH concerned with just that issue. A couple of
institutes that were well known for doing down-the-line funding have now
taken a different look. They are now putting out specified priorities.
They are setting aside a portion of their competing funds for
applications that are in those areas of priority. So, they are taking
that into account.

This project actually, I think, contributes to that as well, Dr.
Yamamoto\'s or the committee\'s versions of giving more information
about relevant characteristics of the application that can help program
make those decisions, independently of the three digit score or whatever
score you want. So, when a program person looks at an application,
let\'s say you go to the overall score and it is a 125, a 150, a 7.2,
whatever it comes out to be, there is more information available to the
program person to make those decisions than just that number. It could
be certainly on the basis of letter scores that are distributed across
the criteria. It could be in terms of numbers and it could be any number
of ways. But the whole point is to do exactly what you were suggesting
be done.

**DR. BRACIALE**: Let me just second Roger\'s very important point, and
that is that the consumer of the rating score is the institute.
Therefore, at this point, what one has to do is deal with the reality
\-- or the institute has to deal with the reality \-- that the study
section and the DRG cannot give them the kind of fine discrimination
that they have in the past.

Perhaps a top five percent, the next 25 percent, is the best that we
will be able to do. Therefore, you need to tell the reviewers the kind
of criteria that you need to have in some kind of evaluation to at least
allow you to put applications in that kind of grouping.

**DR. ATWELL**: You also, of course, never take the place of the
specific words that are used in the review. I mean, the specific
characteristics of the individual proposal still have to be highlighted.
That is another way, of course, that programs have always been able to
make distinctions and will continue to be.

**DR. THEIL**: I just had a short question. It seems that the decision
has been made to make available very widely the report of the committee
by putting it on the Home Page of the NIH. I wondered, given the fact
that there seems to be quite a bit of discussion here and support for
the alternative proposal \-- not alternative, but say somewhat modulated
\-- proposal of Dr. Yamamoto, is there some way to get that out also as
a possible source of discussion to the community, or just how are we
going to proceed mechanically in this regard?

**DR. ATWELL**: We can certainly discuss this with Wendy. Obviously
there is no technological barrier to that.

**DR. CLEVELAND**: The two proposals now in front of us seem to me to
have a common core consensus, with the Yamamoto proposal clearly the
more conservative one. The one that you have just discussed has much
more novel aspects. The key features of both are, as I see it, that what
we want to provide to both the institutes and the PIs a clearer
description of where the review panel really ranks the application, not
just on technical merit where I think there is a consensus around the
table that the current summary statements often focus far too heavily,
but also to deal with other equally relevant criteria. That is what I
would like to focus on now.

In both of the current recommendations, there is a stress on
significance \-- however we title that: e.g., impact \-- and I think
that, again, what I heard around the table and what I would strongly
concur with is that it is a key feature that is absent from many summary
statements. We need to encourage or insist that review panels write such
comments and quantify them with NIH-approved adjectives or letter
grades, so that the PI really has a sense of how the application was
ranked.

Then secondly, we have discussed the issue of feasibility, and in your
proposal that encompassed the PI\'s credentials and track record, and in
Keith\'s, he split it slightly differently. That is what I would like to
speak to, actually even raising the rank of that from the fourth of the
criteria in Keith\'s and from the third of the criteria that you
proposed.

It would seem to me that for investigators, at all ranks \-- be they the
most junior, the middle of their careers or the senior ones \-- one of
the strongest, maybe the strongest criterion for future success is
recent success. That is almost, at least on the panels I have served on
and the pink sheets I have seen, confined to the end with very modest
criteria applied if at all as to how that influences the likelihood of
future success.

I would applaud both of these for raising significance. I would argue to
raise it even higher.

I also wanted to concur with and support enthusiastically, a proposal
made by Dr. Pfeffer, that one of the problems in the review panels in
which I have served is how do we get individual standardization, a topic
which I have to say I found in the proposal from the committee to be
shockingly new, and a terrible idea. I will address that directly in a
moment.

On a more positive note, what Suzanne mentioned was, if you make
reviewers at the table tell you how they rank the application in their
pile of applications and what they thought this round of the overall
quality of the applications they read, that gives the members around the
table a much firmer view very quickly of where this application is
likely to rank. Yes, 50 percent of your applications may fall in the
upper 10 percentile if you happen to have an unusually good set of grant
applications. But that cannot happen round after round.

You very quickly get a spreading to the scores by insistence on that.
And I think that if we could direct the SRAs and the chairpersons of the
study sections to insist on that, that the program officers are going to
see scores that are much better spread and they are going to see that
immediately. I think that if we could implement that sort of internal
discussion for the June 1 study sections where you have re-calibration
and this window of opportunity to re-spread the scores, that you can
actually get much of a spread \-- maybe not all, but you can make a big
step in the right direction using the time right at hand.

So, lastly, the idea of standardization of individual scores, or scores
of individual reviewers, this is, as I said before \-- I hate to be too
conservative in my use of adjectives \-- but it really is a terrible
idea.

For those of us who seem unable to say no and serve on multiple panels
from time to time, you have to adopt the scoring system of the panel on
which you are serving. It is impossible not to, and inappropriate not
to. Therefore, if you look at my scores on different panels, you are
going to see they are very different. If you look at my scores on the
panel on which I am about to be chairperson, they are very different
from the time we re-normalized. So, you just can\'t tell.

So, I think what you really want to know and what everybody around the
table wants to know is, are you rating the grant applications like the
rest of us around the table. The only way to know that is to make the
reviewers speak up. Where does this rank in your pile of applications.
And if yours is an unusual pile because they are so good or so
unfortunate this time, that you speak up and say so.

The last issue, I am afraid I have to say I also agree, I also believe
it to be a terrible idea, and that is to ask reviewers **not** to give
an overall ranking. That seemed to me to come from an absence of
understanding of how the discussion really works at the table. We
discussed this yesterday. No, not every reviewer has read every
application. Indeed, they have, to use my words, read **through** the
application. They do participate in the discussion at the table but it
is really at the table.

It is a question of, is it really true that this is a boring
application, or is the significance of this application really highly
compromised? Is the record of achievement of this PI so modest that the
likelihood of future success is itself so modest? That is what you want
to hear at the table and that is what you want summarized in that final
score. And I don\'t see how you are going to get that by a simple
numerical average.

**DR. BRACIALE**: Any last comments?

**DR. JONASSON**: I think this is an important discussion and I would
like to comment just briefly on Dr. Yamamoto\'s proposal and then on
what Dr. Cleveland has just discussed.

I think it is very important that there still be \-- whether it is
called impact or significance \-- an emphasis on health. After all, this
is the National Institutes of Health. The work that we support, I think,
should be clearly identified as being associated with the health of the
nation and I think that is a very important characteristic that uniquely
sets aside the research that is conducted under the auspices of the
National Institutes of Health.

Then I would like to express real concern about what I feel has been an
over-emphasis on track record. This disagrees partly with Dr. Cleveland
on that aspect. Although clearly it is important that the investigator
be well trained and hopefully experienced in the field, documentation of
past productivity almost insists upon \"those who have get,\" and those
who don\'t will never have a chance of getting.

I think that is pernicious in the system as it stands today. I would
hope that if, in fact, we do want to emphasize the qualifications of the
investigators, which I think is important, that we do so in a very
objective way that does not perpetuate the old boys network and the
system that the grants consistently be awarded to those who have had
grants in the past and had high productivity in granting. Somehow we
should develop criteria that are more inclusive than exclusive in that
regard.

**DR. CLEVELAND**: I would argue that past productivity has nothing to
do with past funding record. It is simply whether the candidate is a
promising candidate, whether it be a senior scientist or a beginning
grant or KO8. It is the qualifications of that applicant for the
particular award that they are writing for.

**DR. CENTER**: One last thing is procedural. Have you gone through the
process of determining how you are going to instruct the SRAs and the
study section chairpersons on how they will modulate the discussion so
that the reviewers all actually bring out on the table the either three
or four criteria?

That will, by definition, change a little bit the way each of the
natures of various study sections go on. So, if you are going to do the
experiment, I think you need to inform or have some instructional period
as to how you want to have each reviewer get up, discuss the various
criteria, and then how you vote on that. Do you vote on them separately?
Do you say, okay, at the end, put your score down A, B, C, D or
whatever? But that aspect of it is going to be important because there
is going to be a lot of confusion at the beginning. Not only that, it is
going to make a difference as to how the results come out.

**DR. BRACIALE**: And maybe at the end, a lot of confusion.

**DR. ATWELL**: There has been no decision to do any of this yet. So, we
are obviously not at the implementation phase. Obviously, those things
would have to be considered, whatever form is adopted. The four study
groups were asked to think about things like that, if something were to
be adopted or a version like Dr. Yamamoto\'s or whatever. But we are not
at the implementation stage. This is really conceptual, before we go
forward and do it.

**DR. BRACIALE**: I would like to recommend that the minutes of this
portion of our meeting, as well as Dr. Yamamoto\'s proposal, be
forwarded to your committee and to Dr. Baldwin\'s, so that these
alternative points of view get some airing.

**DR. YAMAMOTO**: I wanted to just comment on the health points that
have been made. I think those are very important points. I didn\'t
include an explicit statement about health relevance in my queries under
\"impact\" because of what I think has happened to significance sections
of grant applications, where everyone is saying how their studies on
whole chromosomal proteins will cure cancer. And of course, they might.
They might not. They might cure something else.

Since we still know so little about biology and disease that we really
don\'t know where the basic research impacts will come or not come,
whereas with research that is explicitly translational or explicitly
clinical, it is easier to make those statements. So, I think both of
those issues are encompassed on the query, what will be the effect of
these studies on the concepts or methods that drive this field.

So, for the very basic ones where we don\'t yet know what the
applications will be, this would encourage the applicant to respond
hopefully appropriately. And for those doing translational or clinical
research, those applicants would respond in a way that would, of course,
point out their applications to health benefit.

I think an explicit statement that will say how will your work apply to
some health issue will just generate a lot of the kinds of statements
right now that just don\'t really mean very much. It is not to say that
it is not important that we are mindful of this, that we are indeed
funded by the National Institutes of Health, but it is also true that
the next E. coli study may tell us a lot about cancer. We have seen this
happen over and over again. So, that was the issue there.

The last point I want to make is addressed back to Connie, and that is
something that Judy (Turgeon) brought up about who was actually sitting
on these study groups. I thought that you responded that it is NIH staff
people which I think is fine, but I do really think you would gain by
going outside and getting some outside community, extramural grantees,
study section members, people from this group, onto those study groups
as well.

In talking around these ideas, while I think that the RGA panel had a
point that innovation and creativity, for example, are elements that are
built into the current criteria, there is a strong feeling in the
outside community that those points have lost weight in the way that
grants are evaluated. As I talked about these four criteria in places I
was going around \-- this would be anecdotal \-- there has been really
universal support for that and an emphasis on the importance of these
various points. So, you might get a different feel if you go outside the
Beltway, as it were, and get some people from the extramural sector.

**DR. ATWELL**: Of course we are trying to get it externally. The
difficulty with doing that in these study groups is that these are, as
you see, to be finished by August 1st. This is a very intensive
involvement. What might be possible would be to have people participate
electronically. I don\'t think people are going to want to spend,
frankly, the time that our staff is going to spend on this. But they
could do it electronically, perhaps as we did with the other as well. We
will think about it.

**PARTICIPANT**: We need to talk to Dr. Baldwin about that. She
definitely wants the opinions of the scientific community. She is hoping
that the response to the Home Page, that disseminating the fact that it
is out there and that everyone is welcome to respond, and
teleconferences for discussions, will get that input.

**DR. ATWELL**: I think Dr. Yamamoto\'s point was that by having the
staff and external people talking together at the same time, that we
might have a better understanding on both sides. So, I think it is
something we should certainly consider.

I would also just like to point out that the original what we call
significance criterion did have appended to it a statement, \"\...and
thereby improve the health of the nation.\" That apparently led to
confusion about what was meant by what actually Keith has now called
impact; that is really the issue and responsibility in the institutes
for their mission, that absolutely, if things are going to improve the
health of the nation. That is different from what the study sections are
asked to do, in terms of evaluating scientific and technical merit and
clinical importance, if it is a clinical project.

**DR. BRACIALE**: Thank you, Dr. Atwell.

Please send questions and comments to <dder@nih.gov>.

Last revised: June 12, 1996