# Presentation: 184931

## STRUCTURED CRITIQUES

***September 26, 2005***

- Associate Director for
- Scientific Review and Policy
- DEA, NIAID

-  Hortencia Hornbeak, Ph.D.

**P****EER REVIEW ****A****DVISORY ****C****OMMITTEE ****M****EETING**

## Outline for Today’s Discussion

- Multi-pronged approach to shortening review cycle
- Principal drivers of structured critiques
- Structured critique formats
- Structured critiques-award mechanisms
- Reviewer guidance
- Changing reviewer behavior
- Resources needed

## Features of NIH Peer Review thatMeet the Test of Time

- Management of conflict of interest (core value)
- Secure appropriate expertise (core value)
- Develop collective expert advice through discussion
- Provide guidance/feedback to applicant

## Multi-pronged Approach

- Electronic submission of applications
- Knowledge management solutions in referral/recruitment
- Electronic recruitment of reviewers for SEPs
- Internet assisted review (IAR)
- Structured critiques
- Abbreviated summary statements
- Council approval independent of scheduled meetings

## Structured Critiques:  Principal Drivers

- Accelerated review for AIDS initiatives (mid 1980s)
- Hyper-accelerated review of Innovation grants for HIV/AIDS hyper-accelerated review
- Large increase in funds ($1.5 B) for biodefense research (2003)
- Five review cycles (73 days per cycle) per FY
- 60 days from receipt to review

## Structured Critiques:  Principal Drivers (cont.)

- Hyper-accelerated review of Bioshield initiatives- Project Bioshield Legislation 2004
- No FACA rules
- Outside IMPAC II
-  Constraints on FY funding
- Lag time to hire and train staff

## Five Review Cycles:  FY03-04

- 73 days per cycle for most reviews
- SRA team work
- Flexible SRA assignments
- Advance electronic recruitment of reviewers for SEPs
- 	based on letters of intent
- Structured critiques
- IAR assembly of abbreviated summary statements
- Administrative review report to GMB and Program

## Five Review Cycles: FY03-04 (cont.)

- Lessons learned
- Secure appropriate expertise with large numbers of potential reviewers in conflict
- Tools developed to manage large reviews
- Structured critiques
- Practical guidance for reviewers
- Administrative workbooks
- Reviewer Support Site
- Publication search macro for conflicts

## Five Review Cycles: FY03-04 (cont.)

- Effective flexible working teams
- Conflict of interest management
- Development of staff training resources
- Effective communications with stakeholders
- Pre-meeting teleconferences to educate reviewers
- Increase use of teleconferences for reviews

## STRUCTURED CRITIQUE FORMAT

## Structured Critique - Benefits

**Goal**
- Improve utility for stakeholders by:
- More focused discussions
- More concise written evaluations
- Facilitated preparation of abbreviated summary statements

## Structured Critique - Challenges

- Requires change in behavior of reviewers
- May adversely affect “tutorial” aspect
- May not capture adequately complexity and/or subtleties
- May lead to applicant appeal

##  Structured Critiques: Simple and Complex Mechanisms

- R03
- R21
- R01/U01
- P54      Regional Centers  - two-tiered review
- P01/U19
- UC6      Biocontainment laboratories

## Structured Critique Template

- Tailored for each initiative
- Initiative-specific review criteria
- Use short phrases to describe each review criteria
- Based on the five NIH review criteria as a scaffold for initiative specific criteria

## Critique Structured Format

**Significance**:
- Strength # 1   [strongest]      (1-2 lines)
- Justification/explanation  (2-4 lines)
- Strength # 2                          (1-2 lines)
- Justification/explanation  (2-4 lines)
- Strength # X                          (1-2 lines)
- Justification/explanation  (2-4 lines)

## Weakness # 1  [most serious weakness]  (1-2 lines)
Justification/explanation                       (2-4 lines)
Weakness # 2                                            (1-2 lines)  
Justification/explanation                       (2-4 lines)
Weakness # X                                           (1-2 lines)  
Justification/explanation                       (2-4 lines)

- Justification/explanation                       (2-4 lines)
- Weakness # 2                                            (1-2 lines)
- Justification/explanation                       (2-4 lines)
- Weakness # X                                           (1-2 lines)
- Justification/explanation                       (2-4 lines)

**Critique Structured Format (cont.)**

## Continue with format in previous slides for:
Approach
Innovation
Investigator
Environment Criteria

**Approach**
**Innovation**
**Investigator**
**Environment Criteria**

**Critique Structured Format (cont.)**

## EXAMPLE OF A STRUCTURED CRITIQUE

## Example of Structured Critique

**RESUME AND SUMMARY OF DISCUSSION:**  This  (adjectival rating) application entitled “_                       _" was submitted by  (organization),  Principal Investigator. The applicants propose to  (one sentence  summary of specific aim(s). The principal strengths identified by reviewers include _               _, _                 _, _               _, and _                _. The major weaknesses are _                 _, _                 _, _               _, and _                _. After discussion, the panel concluded that _                                  _,  and scored the application in the _        _ range.

## Example of Structured Critique (cont.)

**OVERALL EVALUATION **
- This is an excellent application in response to the RFA...... It is based on solid hypotheses and preliminary evidence that XX inhibitors can protect from toxin-mediated pathogenicity of _*B. anthracis*_. The team is highly experienced and has worked together to establish the background for the project. Goals are clearly laid out, experiments are generally well described, and there is a high probability of significant advances in therapy of anthrax-related toxicity in patients.

## Example of Structured Critique (cont.)

**CRITIQUE**** **
- Significance
**Strength 1** - novel approach to block enzymatic conversion of a bacterial XX protoxin in several bacteria, including _*B. anthracis*_
**Strength 2** - excellent likelihood of successful development of lead compounds
**Strength 3** - excellent opportunity to test lead compounds directly in toxin-treated animals
**Weakness 1** - unknown toxicity related to furin inhibitors

## Approach 
Strength 1 - solid rationale and preliminary evidence for Pseudomonas-related toxin XX, including protection of toxin-induced death in mice
Strength 2 - lead compound XXX has potent binding to target
Strength 3 - excellent team to conduct studies 

**Strength 1** - solid rationale and preliminary evidence for Pseudomonas-related toxin XX, including protection of toxin-induced death in mice
**Strength 2** - lead compound XXX has potent binding to target
**Strength 3** - excellent team to conduct studies

**Example of Structured Critique (cont.)**

## Example of Structured Critique (cont.)

- Approach (cont.)
**Weakness 1** - Is furin block in vivo sufficient to cure infection or only protect from XX-mediated toxicity?
**Weakness 2** - in preliminary results, the PI concludes that “XXX actually enters cells”, but the data only demonstrate that “biotinylated XX” enters cells - they are quite different substances. To further understand the apparent lack of cytotoxicity of XXX, permeabilized cells should be used.

## Innovation 
Strength 1 - detailed mechanistic study of XXX effect on XX toxicity to RAW cells 
Strength 2 - study of kinetics and time course of protection from XX and cell-surface bound iodinated XX 
Strength 3 - design of XX analogs including synthetic peptides, polyamines and related peptidomimetics 
Strength 4 - in vivo studies examining efficacy of XXX on TNF production and on XX induced toxicity in animal models 
Weakness 1 –the approach not innovative and could benefit from collaborative expertise for these studies.

**Strength 1** - detailed mechanistic study of XXX effect on XX toxicity to RAW cells
**Strength 2** - study of kinetics and time course of protection from XX and cell-surface bound iodinated XX
**Strength 3** - design of XX analogs including synthetic peptides, polyamines and related peptidomimetics
**Strength 4** - in vivo studies examining efficacy of XXX on TNF production and on XX induced toxicity in animal models
**Weakness 1** –the approach not innovative and could benefit from collaborative expertise for these studies.

**Example of Structured Critique (cont.)**

## Investigator 
	The PI is experienced in peptide hormone studies and peptide processing. Collaborator Dr. XX is experienced in peptide synthesis. The team has successfully worked together in the discovery of polyarginine furin inhibitors. The application would be strengthen by additional collaborative expertise for XX studies
Environment 
    Lab facilities of the PI are adequate for the studies. Presumably those of Dr. XX’s institution are also adequate. 

**	**The PI is experienced in peptide hormone studies and peptide processing. Collaborator Dr. XX is experienced in peptide synthesis. The team has successfully worked together in the discovery of polyarginine furin inhibitors. The application would be strengthen by additional collaborative expertise for XX studies
- Environment
-     Lab facilities of the PI are adequate for the studies. Presumably those of Dr. XX’s institution are also adequate.

**Example of Structured Critique (cont.)**

## Budget
    Appropriate for the level of effort proposed.
Animal Welfare 
    Comments:  A letter of approval of an animal protocol was included. It does not indicate if both mice and rats are approved for the study. The PI does not address the “five questions” about animal usage. Nature of anesthesia is not indicated. 

-     Appropriate for the level of effort proposed.
- Animal Welfare** **
-     Comments:  A letter of approval of an animal protocol was included. It does not indicate if both mice and rats are approved for the study. The PI does not address the “five questions” about animal usage. Nature of anesthesia is not indicated.

**Example of Structured Critique (cont.)**

## STRUCTURED CRITIQUES: MECHANISM-SPECIFIC EXAMPLES

## Structured Critiques: Simple to Complex Initiatives

- R21- Rapid Response Grant Program on Bioterrorism-Related Research
- U01/U19 RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense
- Biodefense Countermeasure Development

## Structured Critiques:  Benefits – All Mechanisms

- More focused assessments of the applications
- Increased time for discussion of competitive applications
- Enabled committee to manage larger number of applications per SEP.
- Faster SS completion for awarded grants

## Structured Critiques - R21

- RFA Rapid Response Grant Program on Bioterrorism-Related Research  (2002)
- _Mechanism_				_R21_
- Number of applications  		303
- Review committees		3 SEPs
- Number of members/SEP	25
- Streamlined applications		33 percent

## Structured Critiques - R21 (cont.)

- Electronic critique templates
- 3 SRAs assigned
- Resume and summary of discussion templates
- Faster SS completion – 4 weeks

##  Structured Critiques - U01s and U19s

- RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense  (2003)
- _Mechanism_				_U01, U19_
- Number of applications  		186
- Review committees			3 SEPs
- Number of members/SEP		40
- Streamlined applications		33 percent
- Electronic critique templates

## Structured Critiques - U01s and U19s (cont.)

- Resume and summary of discussion templates
- Templates for resume and summary of discussion
- 3  SRAs assigned
- Faster SS completion - 3 weeks

## Structure Critiques - Project Bioshield

- Biodefense Countermeasure Development
-     Hyper-accelerated Review (2004)
- _Mechanism_			           _N/A (small grants)_
- Number of applications  		66
- Letter of intent/program
- Pre-approval
- Review committees		1 SEP
- Number of members/SEP	23
- Streamlined applications		60 percent

## Structure Critique - Project Bioshield (cont.)

- Teleconference review
- Receipt to award 		         60 day
- Electronic critique templates
- Enabled committee to manage the large workload in a hyper-accelerated time frame
- Faster SS completion  - 2 weeks
- Not standard summary statements

## Guidance Provided to Reviewers

- Orientation teleconference with members from multiple committee for standardization
- Electronic critique template tailored to the initiative.
- Instructions for Reviewer Manual tailored for each initiative or SEP
- Overlapping reviewers among the review panels

## Changing Behavior of Reviewer

- Clearly stated expectations
- Focused rational rationale and objectives
- Required buy-in from all stake holders
- NIH
- NIAID
- reviewers
- Make it easy across the board

## Resources Needed to Decrease Time

- Sufficient numbers of trained staff
- Effective SRA/GC Work teams
- Flexible staff assignments
- Versatile and state of the art IT support

## Summary

- Preserve features that make NIH Peer Review work with any changes adapted
- Manage conflict of interest (core value)
- Secure appropriate expertise (core value)
- Develop collective expert advice through discussion
- Provide guidance/feedback to applicant
- Structured critiques are applicable to many award mechanisms: simple to complex

## Summary (Cont.)

- Reviewers must be properly oriented in pre-review teleconferences
- Reviewers do adapt to change
- Structured critiques decrease time for summary statement preparation
- Resulting product (SS) more useful to stakeholders

## Summary (cont.)

- Structured critiques along with other innovations
-      can shorten the review cycle
- Electronic submission of applications
- Knowledge management solutions in referral
- Electronic recruitment of reviewers for SEPs
- Internet assisted review (IAR)
- Abbreviated summary statements
- Council approval independent of scheduled meetings

## Questions?