# Presentation: 071079

## Peer Review at the National Institutes of Health 

- Peer Review at the National Institutes of Health
*What happens to your grant application after submission?*
*Joy Gibson**	**	**	**	**	**gibsonj@csr.nih/gov*
*	**	**	**	**	**	**301-435-4522*

- National Center for Minority Health and Health Disparities
- December 15, 2008

- National Institutes of HealthU.S. Department of Health and Human Services

## The Review Process at NIH
Tips on Grant Preparation
Electronic Submission 
Enhancing Peer Review

**The Review Process at NIH**
**Tips on Grant Preparation**
**Electronic Submission **
**Enhancing Peer Review**

- Goals of Today’s Talk:

## National Institutes of Health

**National Institutes of Health**

- National Institute
- on Alcohol Abuse
- and Alcoholism

- National Institute
- of Arthritis and
- Musculoskeletal
- and Skin Diseases

- National Cancer
- Institute

- National Institute
- on Drug Abuse

- National Institute
- of Environmental
- Health Sciences

- National Institute
- on Aging

- National Institute
- of Child Health
- and Human
- Development

- National Institute on
- Deafness and Other
- Communication
- Disorders

- National Eye
- Institute

- National Human
- Genome Research
- Institute

- National Institute
- of Mental Health

- National Institute
- of Neurological
- Disorders and
- Stroke

- National Institute
- of General
- Medical Sciences

- National Institute
- of Nursing Research

- National Library
- of Medicine

**Center for **
**Scientific Review**

- National Center
- for Complementary
- and Alternative
- Medicine

- National Institute
- of Allergy and
- Infectious Diseases

- Fogarty
- International
- Center

- National Center
- for Research
- Resources

- Clinical Center

**National Center on **
**Minority Health and**
**Health Disparities**

- National Institute of
- Biomedical Imaging
- and Bioengineering

**Office of the Director**

- Center for
- Information
- Technology

- National Heart,
- Lung, and Blood
- Institute

- National Institute
- of Dental and
- Craniofacial
- Research

- National Institute
- of Diabetes and
- Digestive and
- Kidney Diseases

## Review Process for a Research Grant

**National Institutes of Health**

- Center for Scientific Review

**Study Section**

**Institute**

**Advisory Councils and Boards**

**Institute Director**

**Assigns to IC &     IRG/Study Section**

**Reviews for    Scientific Merit**

**Evaluates for    Relevance**

**Recommends    Action**

**Takes Final Action**

**Research**
**Grant Application**

**School or Other**
**Research Center**

**Initiates**
**Research Idea**

**Conducts**
**Research**

**Allocates Funds**

***Submits Application***

## Dual Review System for Grant Applications

-   **Scientific Review Group (SRG) at CSR**
**Provides Initial Scientific Merit Review of Grant Applications**
**Rates Applications and Makes Recommendations for Appropriate Level of Support and Duration of Award**

**Dual Review System for Grant Applications**

***Second Level of Review***
**  ****Council**
**Assesses Quality of  SRG review of Grant Applications**
**Makes Recommendation to Institute Staff on Funding**
**Evaluates Program Priorities and Relevance**
**Advises on Policy**

## Serves as central receipt point for most PHS grant applications
Assigns applications to CSR Integrated Review Groups/Study Sections or Institute Scientific Review Groups
Assigns applications to NIH Institute(s) as potential funding component(s)
Conducts initial scientific merit review of most research applications submitted to the NIH in about 220 Study Sections and regularly recurring special emphasis panels

**Assigns applications to CSR Integrated Review Groups/Study Sections or Institute Scientific Review Groups**
**Assigns applications to NIH Institute(s) as potential funding component(s)**
**Conducts initial scientific merit review of most research applications submitted to the NIH in about 220 Study Sections and regularly recurring special emphasis panels**

**         ****Center for Scientific Review**

## CSR Mission Statement

**To see that NIH grant applications receive **
**fair, independent, expert, and timely reviews - **
**free from inappropriate influences - so the **
**Institutes and Centers within NIH can fund the **
**most promising research. **

## CSR Review Divisions with 25 IRGs

## Assignment to CSR Study Sections

**Standing Study Sections**** when the subject matter of the application matches the referral guidelines for the study section**
**Ad Hoc Special Emphasis Panels (SEPs)**** when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAS)**

**Within an IRG, applications are assigned for review to:**

## Assignment to Institutes

**This assignment is based on a match between the research proposed and the overall mission of the Institute or Center**
**Where applications are appropriate for more than one Institute or Center, multiple assignments are made**

**Applications are referred to an Institute or Center as the potential funding component:**

## The Study Section

## Peer Review in CSR

**CSR Study Sections are managed by a Scientific Review Officer (SR0) who is a professional, usually at the MD, Ph.D. MD/PhD level, whose scientific background is close to the expertise of the study section**
**Each CSR standing study section has 12-40 members who are primarily from academia**
**As many as 60-100 applications are reviewed at each study section meeting**

## Scientific Review Officer

**Performs administrative and technical review of applications to ensure completeness and accuracy**
**Selects reviewers based on broad input**
**Manages study sections**
**Prepares summary statements**
**Provides requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards**

## Study Section Member

**Associate Professor or higher**
**1 Renewed or 2 R01 grants**
**Representation:**
**Gender**
**Minority**
**Geography**
**Institution****	****	****	****	****	**

## Selection of Peer Reviewers

**Selection of Peer Reviewers**

- Non-Doctoral

- Scientific Community

- Non-Research

- Research
- Capability

- Active and Productive
- Researchers

## Criteria for Selection of Peer Reviewers

**Criteria for Selection of Peer Reviewers**

- Scientific Community

- Active and Productive Researchers

- Non-Doctoral

- Non-Research

- Active & Productive      Researchers

- Research Capability

- Doctoral or Equivalent Degree

- Interest in Serving

- Expertise in Discipline of Review Group and Specialization Needed

- Meet Internal Administrative Considerations

- Non-Federal

- Employment

- Gender &

- Ethnic

- Status

- Geography

- Institutional

- Affiliation

## The Study Section Meeting

## Each CSR standing study section has ~23-40 members who are primarily from academia

CSR standing study sections convene face-to-face meetings

As many as 60-100 applications are reviewed by each study section

**CSR standing study sections convene face-to-face meetings**
**As many as 60-100 applications are reviewed by each study section**

**CSR Study Sections**

## REVIEW CRITERIA:

**Significance:**** ****Does the study address an important problem?  How will scientific knowledge or clinical practice be advanced?**
**Approach:**** ****Are design and methods well-developed and appropriate?  Are problem areas addressed?**
**Innovation:**** ****Are there novel concepts or approaches?  Are the aims original and innovative?**
**Investigators:**** ****Is the team of investigators appropriately trained?**
**Environment:**** ****Does the scientific environment contribute to the probability of success?  Are there unique features of the scientific environment or subject populations?**
**Overall Evaluation: ****An application need not be strong in all categories to be judged likely to have a major scientific impact.  In addition, an investigator may propose to carry out important work that by its nature is not innovative, but is essential to move a field forward, or improve clinical decisions or outcomes.**

## Important Considerations
Is the proposed study exempt from human subject review?
Are there any apparent risks* to the human subjects?
Are the protections adequate?
What are the potential benefits to the subjects and to mankind?
Are the inclusions of minorities and both genders adequately addressed?

**Is the proposed study exempt from human subject review?**
**Are there any apparent risks* to the human subjects?**
**Are the protections adequate?**
**What are the potential benefits to the subjects and to mankind?**
**Are the inclusions of minorities and both genders adequately addressed?**

- *”Risks” include the possibility of physical, psychological,
-  or social injury resulting from research.

**Research Involving Human Subjects**

## Important Considerations
Will the anticipated results be for the good of society?
Will the work be planned and performed by qualified scientists?
Will the animals be treated so as to avoid any unnecessary discomfort, pain, anxiety, or poor health?
Species chosen
Animals in short supply

**Will the anticipated results be for the good of society?**
**Will the work be planned and performed by qualified scientists?**
**Will the animals be treated so as to avoid any unnecessary discomfort, pain, anxiety, or poor health?**
**Species chosen**
**Animals in short supply**

**Animal Welfare**

## For each study section, applications in the upper half generally are scored from 1.0-3.0, with 1.0 the best score.  Scores as low as 5.0 are possible. 
Individual scores are averaged and multiplied by 100 to give the final priority score

**Individual scores are averaged and multiplied by 100 to give the final priority score**

- plus past two meetings

**Priority Scores/Percentile Rank**

##   Summary of Review Discussion
  Essentially Unedited Critiques
  Budget Recommendations
  Administrative Notes
  Priority Score and Percentile Ranking

**  ****Essentially Unedited Critiques**
**  ****Budget Recommendations**
**  ****Administrative Notes**
**  ****Priority Score and Percentile Ranking**

**Summary Statement**

- Results are documented by SRO in a summary
- statement and forwarded to the PI and the assigned NIH Institute or Center, where
- a funding decision is made.
- The Summary Statement Contains

## New Review Mechanisms

**Electronic Reviews**
**Telephone Enhanced Discussions**
**Video Enhanced Discussions**
**Asynchronous Electronic Discussions**
**Two Stage Review**

## The Application

## Read instructions
Never assume that reviewers “will know what you mean”
Refer to literature thoroughly 
State rationale of proposed investigation
Include well-designed tables and figures
Present an organized, lucid write-up
Obtain pre-review  from faculty at your institution
Don’t neglect to address human subjects and animal issues, as these could affect your score.

**Read instructions**
**Never assume that reviewers “will know what you mean”**
**Refer to literature thoroughly **
**State rationale of proposed investigation**
**Include well-designed tables and figures**
**Present an organized, lucid write-up**
**Obtain pre-review  from faculty at your institution**
**Don’t neglect to address human subjects and animal issues, as these could affect your score.**

- NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm

**When Preparing an Application**

## Center for Scientific Review Referral of Applications

**Investigators may include a cover letter regarding:**
**Specific study section(s) for review**
**Specific institutes for funding**
**Scientific areas critical to understanding the application**
**Individuals with whom there is a conflict of interest **
**It is NOT appropriate to include a list of potential reviewers by name**

**Notes:**
REFERRAL OF APPLICATIONS 
With the grant application, it is entirely appropriate for an investigator to include a cover letter, as a PDF attachment for electronic submissions, that addresses: [1] ICs most likely to be interested in in the scientific area being studied; if the investigator has discussed the application with a specific Program Director at an Institute or Center, this information should be included; [2] the IRG(s) or study section(s) to which the application should be directed – if the investigator has spoken with a Scientific Review Administrator, this information should be included; [3] a list of the scientific areas critical to understanding the application – it is also helpful to IDENTIFY MULTIDISCIPLINARY APPLICATIONS & to clearly identify the what he / she feels is the main disciplinary or methodological thrust of the application; and [4] it is appropriate / acceptable to mention individuals (by name) with whom there is a conflict of interest and who should not be considered as reviewers.

It is NOT appropriate, however, to provide a list of potential reviewers by name!

## Lack of new or original ideas
Absence of an acceptable scientific rationale
Lack of experience in the essential methodology
Questionable reasoning in experimental approach
Uncritical approach
Diffuse, superficial, or unfocused research plan
Lack of sufficient experimental detail
Lack of knowledge of published relevant work
Unrealistically large amount of work
Uncertainty concerning future directions

**Lack of new or original ideas**
**Absence of an acceptable scientific rationale**
**Lack of experience in the essential methodology**
**Questionable reasoning in experimental approach**
**Uncritical approach**
**Diffuse, superficial, or unfocused research plan**
**Lack of sufficient experimental detail**
**Lack of knowledge of published relevant work**
**Unrealistically large amount of work**
**Uncertainty concerning future directions**

**Common Problems in Applications**

## Electronic Submission

## NIH – Two System Process

**Grants.gov**
**Grants.gov hosts standardized federal forms SF424 (R&R) and agency-specific forms (PHS 398)**
**Electronic application through Grants.gov using these forms is mandatory for most FOAs.  Grants.gov checks the application for federal-wide requirements. **

**eRA Commons**
**NIH retrieves the data from Grants.gov and checks the application against NIH-specific requirements**
**eRA Commons allows applicants to electronically track the status of submissions and to receive/transmit application and award information**

**Notes:**
The electronic application process for NIH involves using standardized federal forms and some agency-specific forms provided online by Grants.gov.  Once the forms are submitted to Grants.gov, the data is retrieved by NIH systems and checked against the rules and requirements for the Funding Opportunity Announcement.  The vast majority of NIH Funding Opportunities require electronic submission through Grants.gov.  Within NIH’s system, called the eRA Commons, Signing Officials and Principal Investigators can check the status of their applications and view the assembled application image.  Each one of these systems has registration requirements that are independent of one another.

## Software Requirements

**Grants.gov Forms Viewer Software (Free)**
**New Adobe Acrobat Viewer mandatory**
**	****http://www.grants.gov/resources/download_software.jsp**
**PDF generation software – list provided on Grants.gov website ****http://www.grants.gov/agencies/software.jsp#3**

**Notes:**
In order to view the electronic application forms, you must download the required Forms Viewer that is available at no charge from the Grants.gov website.  Mac users have a special viewer that they will need to use until Grants.gov releases a new viewer from Adobe Acrobat which will also be free of charge.  This is anticipated for release at the end of summer 2007 or early fall.  Applicants also need to have the ability to generate PDF (portable document format) documents to attach all of the required information to the application forms.  Grants.gov provides a list of PDF software that is compatible on their website.

## Where to Find More Information

- Electronic Submission Website
- http://era.nih.gov/ElectronicReceipt

**Where to Find More Information**

## Enhancing Peer Review

**Enhancing Peer Review**

##  Improving the Quality and Transparency of the Peer Review Process

**February 2009 Submission**
**Shorten summary statements, follow template for each criterion**
**Change the rating system**
- Use 1-9 integers
- Score each criterion
- Provide score for all applications (even those not discussed)
**February 2010**
**Shorten applications, aligning with review criteria**
- Impact, investigator, innovation (if applicable), research strategy, facilities

## Slide 35