# Presentation: 062957

## Rating of Grant Applications

**Brent Stanfield, Ph.D.**
**Acting Director**
**Center for Scientific Review**

## Rating of Grant Applications at NIHBrief History* 

**June 1950 **
- DRG begins using 1-to-5 priority rating scale, along with rating criteria
- Applications rated by various study sections are interdigitated on a master priority list for the Health Council allowing payment of grants in order of appraised merit
- 1-to-5 priority rating scales were quickly adopted by Cancer and Mental Health

**September 1980 **
- NHLBI suggests percentiling
- 1987 percentiling becomes accepted as the NIH priority score normalization convention

*** Source**: Division of Research Grants: A Half Century of Peer Review, Mandel, 1996

## 1996 Report on Rating of Grant Applications Background

-     **Developed by a subcommittee of the Committee on Improving Peer Review (IPR), chaired by Dr. Constance Atwell**

- IPR Committee was charged with reviewing all aspects of the NIH peer review system and recommending changes that would result in improvements and/or streamlining of that system.

- During the course of IPR’s discussions, a number of issues arose that appeared to be related to how reviewers assign ratings of scientific merit (priority scores) to applications.

- In the Fall of 1994, the Committee on Rating of Grant Applications (**RGA**) was formed to examine issues pertaining to this general topic.
- The Committee included primarily NIH staff (one member from NSF) and enlisted the aid of several experts in the fields of psychometrics and decision-making

## 1996 Report on Rating of Grant Applications Background...cont.

**Major issues addressed by the RGA Committee were:**
- Review criteria and how they are used by reviewers
- The scale on which reviewers make their quantitative ratings
- Standardization of individual of reviewer ratings based on voting record to enhance comparability of scores
- The necessity of pilot testing any new procedures before they are adopted

## RGA Committee Recommendations Review Criteria

**Reformulate review criteria**** **

**Conduct reviews criterion by criterion **** **

**Provide separate** **numerical rating for each criterion**

**Eliminate global rating of scientific merit **** **

**Four Primary Recommendations**

**NIH Should:**

## Significance
Impact on knowledge and ultimately health 
Approach
Conceptual framework, design, methods analyses as appropriate to aims 
Feasibility
Investigators’ experience, track record, training
Preliminary data /past progress, resources

- Impact on knowledge and ultimately health
**Approach**
- Conceptual framework, design, methods analyses as appropriate to aims
**Feasibility**
- Investigators’ experience, track record, training
- Preliminary data /past progress, resources

**RGA Committee Recommendations ****Review Criteria****...****cont****.**

## RGA Committee Recommendations Rating Scale

**RGA Committee Recommendations ****Rating Scale**

**Scale values**
- Larger scale values represent greater degrees of the characteristic being rated and smaller values represent smaller degrees
- In other words, the higher the value the _***better***_ the score
**Number of scale values = number of reliable judgments (0-7 or 0-10)**
- An eight-step scale (0-7) was recommended on the basis of the psychometric literature; however, a maximum of 11 steps (0-10) was suggested as acceptable.
**Rating scale anchored only at ends**
- The performance of end-anchors should be evaluated and other approaches to anchoring should be investigated as needed.

## RGA Committee Recommendations Calculation, Standardization, & Reporting of Scores

**Scores standardized on each criterion within reviewer, averaged across reviewers**
- There may be both advantages and disadvantages of such an approach

**Scores reported on 8-pt scale used in original ratings**
- 2 significant digits

**If a single score is needed for overall merit**
- Compute from 3 criterion scores using common algorithm for all applications
- Committee favored arithmetic average

## Ways RGA Committee Recommendations Relate to Each Other

**Revised criteria could be implemented without any other changes**** **
**Rating scale alone could be changed**** **
**Analyses and recommendations regarding summarizing and reporting ratings could be applied to any scale measuring any aspect of scientific merit**

## PROG Discussions

**In 1996, The Peer Review Oversight Group (PROG), a group charged with considering trans-NIH issues and a predecessor to PRAC considered the recommendations of the** **RGA Committee**

- PROG initially tabled most recommendations in the RGA report focusing primarily on the questions:
- Should specific review criteria be used in review?
-  If so, what should the criteria be?
- Should individual criteria should be scored?
- Should reviewers continue to assign a global score or should an overall score be derived from sub-scores assigned to individual criteria?

## PROG Discussions...Cont. (1)

- PROG eventually put forward four review criteria for consideration:
**IMPACT**: The extent to which, if successfully carried out, the project will make an original and important contribution
**FEASIBILITY**: The extent to which the conceptual framework, design, methods, and analyses are adequately developed, well integrated, and appropriate for the aims of the project
**INVESTIGATOR/ENVIRONMENT**: The extent to which the investigators, available resources, institutional commitment, and any other unique features will contribute to the success of the proposed research
**INNOVATION/CREATIVITY:** The extent to which the project employs novel concepts, approaches, or methodology
- Harold Varmis modified these criteria slightly to develop the five criteria used today (**Significance, Approach, Innovation, Investigator, Environment**)
- Addition of “INNOVATION” as a criteria substantially complicates assigning scores to individual criteria and then using individual scores to derive a global score

## PROG Discussions...Cont. (2)

- While PROG did take up the RGA Committee’s recommendation of using criteria to structure written critiques, it was not enthusiastic about other recommendations put forward by the committee
- No changes were recommended in the basic numerical (1.0-to-5.0) scoring system
- The system of having reviewers assign a global score to each application was endorsed and continued
- There was generally low enthusiasm for the idea of assigning scores to individual criteria or deriving an overall score from sub-scores and these practices were not adopted
- The recommendation of standardizing by reviewer did not meet with enthusiasm and was never revisited

## PRAC Discussion

**Should the way reviewers consider review criteria be reconsidered?**
- Global ratings have the potential of incorporating criteria not explicitly defined and accepted and some would argue that global ratings are less valid than scores computed from criterion rankings
- Would independent quantitative assessments of each criterion be more fair or informative?
- How would the funding ICs deal with the absence of a global rating?
**Should the current rating scale be reconsidered?**
- Polarity of the current 1 to 5 scale (with 1 best and 5 worst) is the inverse of the more widely accepted convention
- 41 point scale (i.e. 1 to 5 with tenth of a point intervals) may be excessive; validity and reliability of a 41 point scale are not supported in the literature
**Should reviewers’ ratings be “handicapped”?**
- Could possibly attenuate impact of “harsh” or “easy” scorers
- But there is a great variety in the “history” of reviewers and in the applications they review