# Presentation: 047914

## CSR Review of Member Conflict Applications

- Working Group Members:
- Samuel Edwards, SRA, CMIA Study Section, CHR
- Anshumali Chaudhari, SRA, VCMB Study Section
- Valerie Durrant, SRA, HOP-B Special Emphasis Panel
- Marc Rigas, SRA, BDMA Study Section

## Acknowledgements

- Teresa Lindquist, Program Analyst Office of Planning, Evaluation and Analysis, CSR
- QVR Development Team (NIH/CIT), particularly Heather Allison and Don Tiedemann
- CSR IRG Chiefs
- Michael Martin, Director, Division of Physiology and Pathology, CSR

## Overview

- What is the Law and what is Policy
- What is Practice
- Demographics of Member SEPs
- Comparison of member applications reviewed in Member SEPs and regular study sections with all CSR-reviewed applications
- Conclusions and Recommendations

## Peer Review Regulations

- Revised February 4, 2004
- 42 CFR 52h.5d: “When a peer review group meets regularly, it is assumed that a relationship among individual reviewers in the group exists and that the group as a whole may not be objective about evaluating the work of one of its members. In such a case, a member’s application or proposal shall be reviewed by another qualified review group to ensure that a competent and objective review is obtained”
- Does not define what constitutes a different review group (previous version stated: no more than 50% members)

**Notes:**
Previous version of regulations specifically stated that the other review group to review members’ applications may contain no more than 50% members. This was intentionally left out of the current regulations to give ICs more flexibility to define these alternate review groups.
During public comment, two individuals stated they thought this particular rule for reviewing member applications was too restrictive and would disadvantage study section members.  NIH countered that all available data indicated that members have not been disadvantaged by this practice.

## CSR Policy

- Member applications can be reviewed in an appropriate standing study section or special emphasis panel (SEP)
- SEP Composition
- Limited number (<50%) may be current or recent members of the conflicted member’s study section
- Chair of SEP cannot be a member of conflicted member’s study section
- SEP percentiling rules (30 – 50%)

**Notes:**
SEP PERCENTILING RULE: <30% members, percentiled against CSR base, 30-50% study section members, percentiled against parent study section.

## What is practice

- Distribution of review of member applications (combined 2005 council rounds)

## The proportion of member applications reviewed in regular study sections (SRG), member SEPS, and others have remained relatively constant

## Survey of IRG chiefs of review of member applications

- Member applications are reviewed in a variety of ways within and across IRGs, mostly sister study sections and small phone SEPs.
- 95% of IRGs use multiple formats to review member applications.
- Appropriate expertise is the driving factor for how applications are reviewed

## Percent of IRGs that reported using various formats 

- Percent of IRGs that reported using various formats
- for the review of member applications *(Note: Chiefs *
*could select more than one option)*

## Most IRGs review some or all member applications in groups (either sister study sections or SEPS) that do not contain any other current or recent members of the conflict study section

**Notes:**
Percentiling practices follow CSR policy; <30% members percentiled against CSR All, 30-50% are percentiled against parent study section.

## MEM SEPS are held throughout the round with the peak being 1 month after the peak for regular study section meetings

## Most (approx. 90%) Member SEPs review 1 – 5 applications

**Notes:**
Note that Toni Scarpa has recently instructed SRAs that no SEP should have fewer than 3 applications.  

## Seniority is similar for Member SEPs and standing study section meetings

## Overview

- What is the Law and what is Policy
- What is Practice
- Demographics of Member SEPs
- Comparison of member applications reviewed in Member SEPs and regular study sections with all CSR-reviewed applications
- Conclusions and Recommendations

## How do study section member applications fare?

- Examined only R01 applications reviewed by CSR
- CSR Member SEP based on ZRG1 prefix and M suffix in the meeting code (plus Decade coding (02 – 07) beginning 2005)
- Compared scores (percentiles) for member applications reviewed in Member SEPs and standing study sections (SRGs), and to all R01 applications reviewed in CSR (CSR All).
- Percentage of applications in the
- Top 10%
- Top 20%
- Bottom 70% (including streamlined applications)
- Differences in Type 1 and 2 applications?
- Differences for applications using human subjects?

## Comparison of scoring of Types 1 and 2 applications reviewed in 2005 in Member SEPs and SRG meetings and to CSR All

- Type 1 applications

- Type 2 applications

- Cumulative Percent

- Percentile

- Percentile

## Types 1 and 2 applications reviewed in Member SEPs and Standing Study Section meetings (SRG) for Years 2000 – 2005 compared to CSR All

- 0

- Type 1 applications

- Percentage of applications reviewed

## Fewer member applications reviewed in Member SEPs are streamlined compared to those reviewed in standing study section (SRG) meetings

## Types 1 and 2 applications the year prior to beginning SS service compared to those reviewed in Member SEPs or study section (SRG) meetings and to CSR All 

- Types 1 and 2 applications the year prior to beginning SS service compared to those reviewed in Member SEPs or study section (SRG) meetings and to CSR All

- Percentage of applications reviewed

## Comparison of applications reviewed in small SEPs (Series 90) to member SEPs and to CSR all

## Conclusions

- Member conflict applications comprise a significant segment of applications.
- Lots of members + high productivity of members = many member conflict applications
- Members are providing a service to NIH
- Goal is a fair review for member applications given that _by regulation_ they cannot be reviewed in their parent study section, which may be the most appropriate study section

## Conclusions:  Key findings

- No SYSTEMATIC bias in the review of members’ applications and, in particular, with the use of member SEPs.
- Composition (seniority) is similar
- The percentage of Type 2 applications that receive scores in the top 10 percentile are very similar when reviewed in member SEPS or in regular study sections
- Members receive comparable scores before and during SS service.
- Member applications do better—likely because members are excellent scientists who write good applications.

## The current system provides appropriate means for reviewing member applications

- IRGs differ, study sections differ, and the review of member applications will differ accordingly.
- Flexibility is important to ensure most appropriate review venue for member applications.
- IRG chiefs are integral!

## Concerns with member SEPs

- Confidentiality of reviewers in small SEPs – adequate protection?
- Designer reviews?
- SRA workload

## Recent changes

- Encouraged to schedule meetings earlier
- Encouraged to avoid 1 or 2 application meetings