# Presentation: 110898

## PEER REVIEW ADVISORY COMMITTEE

**PEER REVIEW ADVISORY COMMITTEE**
**Clustering of Grant Applications for Review**
**Don Schneider, Ph.D.**

**National Institutes of Health****U.S. Department of Health and Human Services**

**April 30, 2008**

## Clustering of applications raises questions

**Clustering of applications raises questions**

**How to express peer review outcomes in consistent manner**
**How to compare different review outcomes, with statistical significance**
**How to identify “orphan” applications**
**How to promote optimal fairness**

## Peer Review implies clustering

**Peer Review implies clustering**

**Review by people of the same rank (equals)**
**Broad sense – any research scientists who appreciate/understand the scientific problem**
**Narrow sense – scientists with direct experience in the area of proposed research**
**Clustering of very similar applications (bunching of the same sort together) lends itself to peer review in the narrow sense**

## PSBR & recent trends altered clustering

**PSBR & recent trends altered clustering**

**Panel on Scientific Boundaries for Review (PSBR) set a very high bar, 30 % clustering (Jan 2000 report)**
**PSBR de-clustered some communities**
**Some areas of science, esp. new areas, bridge existing study sections, e.g., genetics of human behavior**
**Some areas of science are inherently diverse, e.g., complications of diabetes**
**Research in some areas of science is diminishing, e.g., thyroid metabolism**
**Extreme clustering essentially establishes an entitlement and is counter to broad study sections**

## PRAC – Clustering Working Group

**PRAC – Clustering Working Group**

**[met 11/30/2007; PRAC 12/3/2007]**
**Noni Byrnes, CSR**
**Patricia Greenwel, CSR**
**Ann Hagan, NIGMS**
**Leslie Leinwand, PRAC, Colorado**
**Don Schneider, CSR**
**Ross Shonat, CSR**
**Phil Smith, NIDDK**

## PRAC recommended comprehensive study

**PRAC recommended comprehensive study**

**Develop statistical tool for assessing differences in review outcomes**
**Identify orphan applications; “just ask – staff can tell you”**

## CSR formed “Scatter Plot” Committee

**CSR formed “Scatter Plot” Committee**

**Sam Edwards (QVR Searches)**
**Brian Hoshaw (Scatter Plots)**
**Andrea Kopstein (Planning, Analysis, & Evaluation)**
**Kristin McNamara (Organizer)**
**Chris Sempos (Statistical Tool – Odds Ratios)**
**Plus: Malgorzata Klosek, Tina McIntyre, Marc Rigas, Don Schneider, & Anita Miller Sostek**
**Methods shared with Chiefs, and established a team of Statistical Analysis System (SAS) point persons, one per review division**

## Scatter Plot of select High & Low Rep R01s

**Scatter Plot of select High & Low Rep R01s**

## Odds Ratio for High & Low Rep study sections

**Odds Ratio for High & Low Rep study sections**

## Multivariate Logistic Regression example: high vs. low groups

**Multivariate Logistic Regression example: high vs. low groups**

## Multivariate Logistic Regression: Odds Score < 15%

**Multivariate Logistic Regression: Odds Score **_**<**_** 15%**

## Odds Ratio for R01s receiving < 15% scores:high vs. low

**Odds Ratio for R01s receiving **_**<**_** 15% scores:****high vs. low**

## Interpret significant differences with caution

**Interpret significant differences with ****caution**

**Multiple explanations may account for significant differences, including degree of clustering and quality of science – ****do not jump to conclusions**
**Significant results should be the starting point for discussion of how to investigate further – ****be thorough**
**Nonetheless, the Odds Ratio does provide a very useful tool of value in assessing scatter plots, e.g., Skeletal Muscle and Exercise Physiology (SMEP)**

## Comparing Review Outcomes 2001-03 vs. 2005-07

**Comparing Review Outcomes 2001-03 vs. 2005-07**

## Limitation of Odds Ratio

**Limitation of Odds Ratio**

**QUESTION: For odds ratio of 2, with 95% confidence limit, how many data points (applications) are needed, minimally?**
**ANSWER: About 100**

## R01 Orphan applications across CSR

**R01 Orphan applications across CSR**

**Definitions: fewer than 5 applications on the topic per IRG per cycle vs. two or fewer per study section per cycle**
**Topic included in guidelines, but expertise normally not present (broad vs. narrow)**
**Orphan topics or applications?**

## Asking/Getting answers has complications

**Asking/Getting answers has complications**

**Discussion about what is an orphan application**
**Significant fraction of SROs do not respond; another significant fraction report no orphans**
**Orphans are scattered – a basis for comparison may be arbitrary**

## SUMMARY - Comments from CSR SROs

**SUMMARY - Comments from CSR SROs**

**Several SROs of regular study sections have no orphan R01s**
**Most SROs suggest that two or fewer applications on a topic per cycle is about right for defining an orphan**
**Selected orphan topics are: sociocultural factors affecting bone density, circadian rhythms, Archaea, sleep disorders, immunology of larynx, vision & driving, blind navigation, transcranial magnetic stimulation, toxic agents and pregnancy, toxic agents and heart, massage therapy, host defense in transplantation, modeling of wound healing, tobacco toxicology, sepsis, alternative medicine, pain, monkeys, pure theory such as modeling of brain oscillations, animal communications, motion sickness, gastrointestinal epidemiology, surfactant biophysics, enzyme nitrosylation, neuro-AIDS and end-organ diseases like enteritis, AIDS and rare opportunistic infections, models to test toxicity, prevention of diabetic foot ulcer, brain vascular development, Hashimoto’s thyroiditis, EPR imaging**

## Interim conclusion is that analysis of orphan applications is challenging

**Interim conclusion is that analysis of orphan applications is challenging**

**The Odds Ratio provides a useful statistic for evaluating significant differences in “Scatter Plots”**
**95% Confidence in Odds Ratios require about 100 applications/data points**
**Orphan applications have low numbers, 1-2 per study section per cycle**
**To apply statistics to orphan applications, one must accumulate data over about 50 cycles (>15 years) or group orphan applications**
**CSR is in process of identifying orphan applications (next step poll program staff) and considering ways to group them**

## Discussion