# Presentation: 649153

## Providing Information on the Dispersion of Reviewer Ratings  A Consideration of Some Alternatives:Standard DeviationsRangesGraphical RepresentationsPresentation to the Peer Review Advisory Committee1/23/06

## Summary of Working Group’s Activity

- The NIH Office of Extramural Research formed a working group to:
-   Consider whether there might be more information contained in reviewers’ scores, not captured by priority and percentile scores alone, that might be useful to either applicants or NIH staff
-   Consider how such information might be presented
- The working group examined:
-   Distributions of individual reviewer scores given to a sample of applications reviewed by CSR study sections
-   Alternative methods for characterizing those distributions (various measures of dispersion)
-   Alternative methods for presenting this information (numerically, graphically)

## Conclusions

-   When applied across a broad range of applications and study sections, no numerical measure of dispersion adequately represents the score distributions.
-   A straightforward way of communicating information about the distribution of scores would be to display them graphically.
-   The working group uncovered additional issues which will need to be addressed.

## Examples

- Numerical measures of dispersion:
- Can fail to capture differences in the underlying distributions
- May not work well with small study sections
- May not be comparable across applications reviewed by different study sections (e.g., study sections differing in size)
- Can suggest differences that aren’t otherwise apparent
- May not be grasped intuitively

## Idealized Score Distribution

## Actual Score Distribution

## A.  Fail to capture differences in the underlying distributions Example 1

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 70%

- 130

- 140

- 150

- 160

- 170

- 180

- 190

- 200

- 210

**Reviewer Score**

**% of Scores**

- s.d. = 14

- s.d. = 12

## B. May not work well with small study sectionsExample with 5 Reviewers

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 70%

- 80%

- 90%

- 100%

- 140

- 150

- 160

- 170

- 180

- 190

- 200

**Reviewer Score**

**% of Scores**

- IQR = 0

-  IQR = 0

## C.  May not be comparable across study sections Example with small and large study sections

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 70%

- 150

- 160

- 170

- 180

- 190

- 200

- 210

- 220

- 230

- 240

**Reviewer Score**

**% of Scores**

- s.d.: 26

- n = 5

- s.d.: 26

- n = 26

## D.  Can suggest differences that aren’t otherwise apparentExample of similar (?) distributions with different standard deviations

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 130

- 140

- 150

- 160

- 170

- 180

- 190

- 200

- 210

**Reviewer Score**

**% of Scores**

- s.d. = 14

- s.d. = 9

## E. May be hard to grasp intuitivelyStandard deviations differ by a factor of 2

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 70%

- 80%

- 100

- 120

- 140

- 160

- 180

- 200

- 220

- 240

**Reviewer Score**

**% of Scores**

- s.d. = 11

- s.d. = 22

## Graphical/Tabular Representations

| Score | Number of Scores | % of Scores |
| --- | --- | --- |
| 1.0 | 0 | 0 |
| 1.1 | 0 | 0 |
| 1.2 | 0 | 0 |
| 1.3 | 0 | 0 |
| 1.4 | 4 | 13 |
| 1.5 | 13 | 41 |
| 1.6 | 12 | 38 |
| 1.7 | 3 | 9 |
| 1.8 | 0 | 0 |
| 1.9 | 0 | 0 |
| 2.0 | 0 | 0 |
| 2.1 | 0 | 0 |
| 2.2 | 0 | 0 |
| 2.3 | 0 | 0 |
| 2.4 | 0 | 0 |
| 2.5 | 0 | 0 |

## Additional Issues Raised

-   Are these measures of dispersion (either statistical or graphical) informative in the absence of more information on the reasons for differences in viewpoints among reviewers?
-   With graphical representations, would reviewers (particularly in small study sections) have concerns about maintaining anonymity/confidentiality?
-   Will knowledge that the distribution of individual scores is to be shared, either with staff or the applicant, have any detrimental effects on reviewers’ scoring behavior?

## Next Steps

- Assess benefits and costs of dispersion reporting
- Clarify need, usage, and potential value to PIs, staff
- Fully explore unintended consequences
- Develop cost estimates
- Determine if there is sufficient benefit:cost to proceed
- Consider a pilot approach with a volunteer Panel or IC
- Requires change to information systems
- Consider a “virtual” pilot
- Present simulated/archival data to staff, PIs via survey or focus group
- Easy next step, but does not address feasibility