# Presentation: 663172

## PEER REVIEW ADVISORY COMMITTEE

**PEER REVIEW ADVISORY COMMITTEE**
**General Principles of Application Clustering**
**Don Schneider, Ph.D.**

**National Institutes of Health****U.S. Department of Health and Human Services**

**December 3, 2007**

## Clustering Topics

**Spread of Integrated Review Group (IRG) review assignments**
**Overall review outcomes in low & high clustered IRGs**
**New investigator review outcomes in low & high clustered IRGs**
**Discussion items**

## Clustering and Peer Review

**Review by people of the same rank (equals)**
**Broad sense – any research scientists who appreciate/understand the scientific problem**
**Narrow sense – scientists with direct experience in the area of proposed research**
**Clustering of similar applications (bunching of the same sort together) lends itself to peer review in the narrow sense**

## Clustering Factors

**Panel on Scientific Boundaries for Review (PSBR) set a very high bar 30% (Jan 2000 report)**
**PSBR de-clustered some communities**
**Some areas of science, esp. new areas, bridge existing study sections, e.g., genetics of human behavior**
**Some areas of science are inherently diverse, e.g., complications of diabetes**
**Research in some areas of science is diminishing, e.g., thyroid metabolism**
**Extreme clustering essentially establishes an entitlement and is counter to broad study sections**

## Numbers of Applications

**CSR reviews more than 15,000 applications a cycle**
**Study sections average about 80 applications a cycle**
**The PSBR 30% clustering would be a set of about 24 applications**
**“****Low” clustering here is an average 5 or fewer applications per IRG per cycle**
**Three subsets to be examined, each about 200 applications per cycle, 1.3% of total**

## Possible Actions To Promote Clustering

**Cluster better within Integrated Review Groups (IRGs)**
**Cluster within fewer IRGs**
**Form a Special Emphasis Panel (SEP)**
**Constitute a Working Group to consider formation of a new chartered study section**

## Possible Actions by Program

**Designate low represented areas high program priority (“just pay them”)**
**Write Program Announcements (PAs)/Requests For Applications (RFAs)**

## PRAC – Clustering Working Group

**Aim – to examine clustering and to develop recommendations for PRAC & CSR Director **
**First meeting 11/30/2007**
**Noni Byrnes, CSR**
**Patricia Greenwel, CSR (Data mining and graphics)**
**Ann Hagan, NIGMS**
**Leslie Leinwand, PRAC, Colorado**
**Don Schneider, CSR**
**Ross Shonat, CSR**
**Phil Smith, NIDDK**

## (44)

**(44)**

**(37)**

**(22)**

**(169)**

**(36)**

**(22)**

**(9)**

**(116)**

**(35)**

**(4)**

**(258)**

**(129)**

**(13)**

**(52)**

**(18)**

**(16)**

**(185)**

**(104)**

**(15)**

## R01s in CSR IRGs

**0**

**20**

**40**

**60**

**80**

**0**

**20**

**40**

**60**

**80**

**Standard Distribution**

**Percentile**

**Cumulative Percent**

**R01s in CSR IRGs**

## Subset A R01s in CSR IRGs*

**20**

**40**

**60**

**80**

**20**

**40**

**60**

**80**

**Subset A R01s in CSR IRGs***

**Cumulative Percent**

**Percentile**

**Trend in High **
**Representation IRGs**

***2005-2007, excluding**
**population-based IRGs**

## Subset A R01s in CSR IRGs*

**20**

**40**

**60**

**80**

**20**

**40**

**60**

**80**

**Subset A R01s in CSR IRGs***

**Cumulative Percent**

**Percentile**

***2005-2007, excluding**
**population-based IRGs**

**Trend in Low **
**Representation IRGs**

## Subset A R01s in CSR IRGs*

**0**

**20**

**40**

**60**

**80**

**0**

**20**

**40**

**60**

**80**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

**Percentile**

**Cumulative Percent**

**Subset A R01s in CSR IRGs***

***2005-2007, excluding**
**population-based IRGs**

## CSR “Scatter Plot” Committee

**Aim – to provide QVR search guidelines, plot instructions, and statistical tool for measuring significance**
**Sam Edwards**
**Brian Hoshaw**
**Malgorzata Klosek**
**Kristin McNamara**
**Marc Rigas**
**Don Schneider**
**Chris Sempos**

## Subset B review assignments, 2005-2007 Councils

**200**

**400**

**600**

**800**

**1000**

**1200**

**# of Applications**

**BDA**

**BST**

**HOP**

**IMM**

**MDCN**

**MOSS**

**GGG**

**BCMB**

**CB**

**IDM**

**SBIB**

**BDCN**

**IFCN**

**ONC**

**RPHB**

**Subset B review assignments, 2005-2007 Councils**

**(9)**

**(6)**

**(12)**

**(1)**

**(6)**

**(40)**

**(25)**

**(21)**

**(1062)**

**(3)**

**(73)**

**(6)**

**(2)**

**(9)**

**(14)**

## Subset B R01s in CSR IRGs*

- 10

- 20

- 30

- 40

- 50

- 60

- 70

- 20

- 40

- 60

- 80

**Cumulative Percent**

**Percentile**

**Subset B R01s in CSR IRGs***

***2005-2007, excluding**
**population-based IRGs**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

## DIG

**100**

**200**

**300**

**400**

**DIG**

**IFCN**

**BDA**

**BDCN**

**CVS**

**BST**

**EMNR**

**GGG**

**HOP**

**IMM**

**MDCN**

**MOSS**

**ONC**

**RES**

**RPHB**

**Subset C review assignments, 2005-2007 Councils**

**# of Applications**

**(10)**

**(3)**

**(12)**

**(9)**

**(185)**

**(11)**

**(23)**

**(122)**

**(12)**

**(380)**

**(10)**

**(3)**

**(17)**

**(100)**

**(4)**

## Subset C R01s in CSR IRGs*

**20**

**40**

**60**

**80**

**20**

**40**

**60**

**80**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

**Percentile**

**Cumulative Percent**

***2005-2007, excluding**
**population-based IRGs**

**Subset C R01s in CSR IRGs***

## R01s in CSR IRGs

**5**

**10**

**15**

**20**

**25**

**30**

**35**

**40**

**45**

**50**

**20**

**40**

**60**

**80**

**Cumulative Percent**

**Percentile**

**R01s in CSR IRGs**

**Standard Distribution**
**(all PIs)**

**Standard Distribution**
**(only new PIs)**

## Subset A R01s in CSR IRGs*

**5**

**10**

**15**

**20**

**25**

**30**

**35**

**40**

**45**

**50**

**20**

**40**

**60**

**80**

**Cumulative Percent**

***2005-2007, excluding**
**population-based IRGs**

**Percentile**

**Subset A R01s in CSR IRGs***
**New Investigators only**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

## Subset B R01s in CSR IRGs*

**Cumulative Percent**

**Subset B R01s in CSR IRGs***
**New Investigators only**

***2005-2007, excluding**
**population-based IRGs**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

**Percentile**

## Subset C R01s in CSR IRGs*

**population-based IRGs**

**Subset C R01s in CSR IRGs***
**New Investigators only**

**Trend in High **
**Representation IRGs**

**Trend in Low **
**Representation IRGs**

**Standard Distribution**

**Cumulative Percent**

**Percentile**

## Conclusions

**Clustering is a complex matter, but advantageous to some**
**Low clustering presents challenges (many variables) and perhaps occasional advantages**
**Tools for assessing clustering and for determining statistical significance of clustering are needed and are under development (by CSR “Scatter Plot” Committee)**

## Discussion items

**What should our clustering goal be?**
**Is low clustering generally a disadvantage?**
**Do we need a more comprehensive study of clustering, including vulnerable applicants, with broad SRO and PO input?**
**Should CSR provide interim realignment solutions, such as reversible SEPs?**

## Discussion