# Presentation: 352923

## Pilot-Online Review of R21 Applications

- Daniel F. McDonald, Ph.D.
- Chief, Musculoskeletal Oral & Skin Sciences and Renal & Urological Sciences IRGs
- Center for Scientific Review, NIH

## I. Objectives

- To assess alternate review paradigm given increasing overall CSR review burden and ultimately limited human/financial resources
- To relieve review burden on standing skeletal biology study sections

## Slide 3

## II. Meeting Mechanics

- Choose application pool
- Administrative review (SRA)
- Recruit reviewers
- Make assignments
- Critique prep (reviewers)
- IAR submission Phase

- “Triaging” process
- Initiation of chat room
- Monitoring of discussions
- Establishment of final levels of enthusiasm
- Procurement of individual reviewer votes

## Chatroom Setup

## III. Post-meeting Activities

- Critique revision (reviewers via IAR)
- Score/Code entry/release
- Summary Statement generation/release
- Process assessment
- 		> CSR personnel (SRA, IT support, senior staff)
- 		> Reviewers
- 		> Program staff

## Summary of Review Load

- 52 *R21*, 4 *R03* and 2 *R15* applications
- 194 assignments for 46 reviewers
- 21 of the 58 applications went unscored

## Number of Reviewers Participating

- 137 assignments to 37 scored applications
- Average of 3.3 reviewers assigned to each scored application

## Summary of Comment Source

- Average of 3.7 members commented on a given scored application in electronic panels
- 					vs.
- For a regular study section, average number of members participating in a discussion per application = 4.6
- 	*[N = 75 study sections, with a mean of 19.7 members per study section; Unpublished observations made in DRG study sections in 1989]*

## Summary of Comment Source (cont’d)

- Average of:
- 5.9 comments per application from assigned reviewers
- 0.8 comments from unassigned reviewers [almost always the Chair]
- 6.8 total comments per application

## Summary of Scoring Profile

- 7 of 37 scored applications were scored by all reviewers
- The other 30 applications were scored by all but 1 or 2 reviewers
- Conflicts counted as scores to differentiate from those that didn't score at all

## Slide 12

## Data Analysis Suggests:

- Number of comments tended to increase with range of initial scores.
- Once the initial score spread from the assigned reviewers gets above some figure between 0.5 and 1.0, there is an increase in the number of comments that are made.
- The additional comments are all from the assigned reviewers, not the unassigned reviewers.**  **

## Slide 14

## Data Analysis Suggests (cont’d):

- Final score not closely related to number of comments.
- Range of final scores from assigned reviewers was much narrower than their initial scores.
- Final score range from assigned reviewers narrower than from all reviewers.
- Initial range of scores from assigned reviewers broader than final range from all reviewers.

## Slide 16

## Slide 17

## Slide 18

## Slide 19

## IV. Preliminary Post-meeting Reflections (SRA)

- ADVANTAGES
- Maximum flexibility in construction
- Recruiting tool
- Convenience for potential reviewers
- Substantial reviewer cost savings

- SHORTCOMINGS
- Unfamiliar approach to reviewers
- Added security vulnerabilities
- No real-time face-to-face interaction
- Restricted involvement of subgroup of panel members in any particular evaluation
- More time-consuming for all parties (SRA/Chair/panelists)
- Meeting size constraints