# Presentation: 002553

##  A Short Course in the Peer Review of Grant ApplicationsReview Branch, NIDDK (originally developed by NIDA-Office of Extramural Affairs and adapted from a presentation by Dr. William Grace, NIDA/NIH )

## A Short Course in the Peer Review of Grant Applications

- Welcome to the NIH scientific review community.  This course is designed to give you a quick orientation to your role in the grant review process.
- Participating in this course will minimize the time you spend as you prepare your written critiques of applications.

## Table of Contents

- _[Before the Meeting-Your Review Packet](slide4.xml)_
- _[Before the Meeting-Purpose of the Review](slide9.xml)_
**Before the Meeting-Reviewing, Critiques and Scoring**
- _[Before the Meeting-Internet Assisted Review](slide23.xml)_
- _[At the Meeting-Protocol](slide24.xml)_
- _[At the Meeting-Procedures](slide25.xml)_
- _[At the Meeting-Turning in Materials](slide29.xml)_
- _[After the Meeting-Confidentiality](slide31.xml)_
- _[After the Meeting-Reimbursement Forms](slide32.xml)_

## Before the Meeting—Review Packet

- You will receive a package with the applications, review instructions, and logistics information for travel and lodging.
- Open the package and read the instructions **immediately**.
- This course cannot cover everything. It is important for you to read the instructions in the packet carefully, so that you can fully understand what is required, and can give the application a fair and full consideration.  Reading the information upon receipt avoids last minute problems with travel and with your reviews.
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Review Packet

- Check applications for conflicts of interest.  Make sure you have no financial interest in the application or applicant organization.
- Keep material confidential.  This means you should not share ANY information about the applications with ANYONE (apart from review staff), including colleagues or graduate students.  NIH wants your independent judgment of the application.  Plus, the confidentiality of the applicant must be protected.
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Review Packet

- Make your travel arrangements early. Contact the travel contactor following the information contained in your package. You must use this contractor to arrange for your travel. Failure to do so will result in you receiving only up to the government rate for your travel, which may be less than your cost.
- Hotel arrangements are handled through a separate contractor. You will receive an email from the contractor with additional details.
- Please advise the Scientific Review Administrator if you are traveling by car or if you have any unusual travel requirements as soon as possible.

- Hotels have reservation cut-off dates, the travel contractor has requirements, and, in general, last-minute arrangements cause headaches for you and us.
- Get the arrangements made early and avoid problems.
- Call your SRA if you have any special needs or
- questions.
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Review Packet

- Reimbursement
-  The NIH is billed directly for hotel room charges and travel.
- Your reimbursement is handled through an electronic funds transfer.
-  In general, you will not be required to submit receipts unless you encounter an unusual situation during the meeting.
- Your reimbursement includes a fixed fee to cover transportation and incidentals (such as parking and taxis), a daily fee for meals and incidental expenses, and the standard NIH honorarium for each meeting day.
- To receive your reimbursement, you must obtain a personal Dun and Bradstreet number (DUNS number) and then register with the government using the  Central Contractor Registration system (CCR).
- 							Return to TOC

## Before the Meeting—Review Packet

- DUNS Number and CCR Registration
- Please read and follow the instructions on the CD that is included in your package or on the web: http://cms.csr.nih.gov/NR/rdonlyres/E1342054-F3C7-4503-9587-66CF1F472A0B/7680/DUNSCCRInstructionswithAlert1.pdf.
- To receive your reimbursement you must obtain a personal DUNS number and then register with CCR.
- When obtaining a DUNS number be sure to “Opt Out” of the mailing list. You may obtain a DUNS number using either the telephone or on the internet.
- You will use the internet when registering with CCR.
- This is a one time process - if you already have a DUNS number and are registered with CCR, you are all set.
- 							Return to TOC

## Before the Meeting—Purpose of Review

- You should know that the NIH separates the functions of “review” and “program”.
- Review staff manage the scientific and technical aspects of the review and are required to maintain impartiality.  Therefore, in consideration of confidentiality and conflict of interest concerns, you should not discuss the application or its review– even after it is complete—with anyone except review staff.
- Program staff work closely with applicants and are considered to have an interest in the outcome of the review.
- 	_[Return to TOC  ](slide3.xml)_

## Before the Meeting—Purpose of Review

- Make sure you clearly understand the purpose of the review.
- The purpose of the review is to advise the NIH staff and council(s) on the scientific merit of the applications.  You provide this advice through: (1) your written critiques; (2) your comments at the review meeting, and (3) the score you assign the application.
- 	_[Return to TOC](slide3.xml)_

## Question

- Does the review group make funding decisions?

- Answer

- No.
- The funding results or pay plan is based on the priority score and/or percentile and availability of funds. Program staff may recommend [to the institute director through the advisory council] an exception to the priority score plan with sufficient justification, for example, scientific or Congressional priorities.
- 	_[Return to TOC](slide3.xml)__[ ](slide3.xml)__[  ](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Assess the merit of the applications.
- You will want to read the abstracts of all the applications on the CD to be familiar with them, but you will want to be very knowledgeable about the applications specifically assigned to you. You will write a critique for these and present the critiques at the meeting.
- The length of the critique will vary, but for most applications, a few pages is sufficient.
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Follow the instructions you receive to write the critiques, and make comments only on the review criteria.  For most applications, these criteria include Significance, Approach, Innovation, Investigator, Environment, and other issues related to sample composition and protection of subjects.  You may also be asked to comment on the budget and data sharing.
*Some types of applications have unique review criteria; again, make sure you check the instructions.*
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Write your critiques. Evaluate strengths and weaknesses, do not just re-tell what is proposed.
- Keep comments scholarly and concise.
- Be fair and objective; discuss the application and not the applicant (except under the “Investigator” criterion).
- 	_[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Organize your writing by addressing each review criterion separately.
- Remember to stay focused on main issues; there is no need to “prove” that you covered every detail.
- 	_[Return to TOC](slide3.xml)_

## Question

- The application makes it clear that the proposed methods of data collection are included at the suggestion of NIH staff.  How does that affect your assessment of the approach?

- Answer

- It does not affect your evaluation.  You are asked to give your opinion on the merit of the ideas in the application, regardless of the source (e.g., staff suggestion, previous review committee) of those ideas.
- 	_[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- For each application assigned to you, decide on a preliminary score.
- Consider  the application’s strengths and weaknesses in regards to only the review criteria, and use the table on the next slide to anchor your scores. In the actual review meeting, you may modify your score as you hear other perspectives.
- Try very hard to use the full range of scores. Score compression or “bunching’ makes it difficult for program staff to evaluate what the review group identified as scientific merit. Don’t be afraid to give the application that you liked a 1.2, or one that you did not like a 4.5.
- _[Return to TOC](slide3.xml)_

## Scoring Anchors 

- _[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- You need to be ready to suggest a score at the meeting for your assigned applications.  Each review group has slight variations on how it matches a number with the merit of an application.  In all cases, you’ll use a 1-5 scale.  “1” is outstanding, “5” is acceptable.  At some review meetings the poorer applications, (“3-5” scores) are designated “unscored.” In this case, you’ll suggest a score from 1.0 to 3.0 for the remaining applications under discussion.
- When scoring, use increments of  0.1 (e.g., 1.3, 1.4, 2.7 - but not 2.72).
- _[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Certain applications are always discussed and scored (K Series, Program Projects).
- Your SRA will notify you if the “unscoring process” will be used during the review meeting.
- Unscored applications receive a preliminary score before the meeting but are not formally discussed or scored during the meeting.
- Your written critiques will be included in the summary statement that is provided for the applicant.
- 	_[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- Even in a pool of terrific applications, there’s always a bottom half of those that aren’t just quite as terrific as the other 50%.  NIH asks you to designate those as unscored. This procedure is called “streamlining” and needs to be applied evenly across the NIH for it to be fair.
- Preliminary scores for unscored applications are generally from 3-5.
- These applications will not be discussed during the review meeting. When appropriate, recommend “Unscored” if the application is in the poorer (lower) half of the distribution of the group of applications being considered.  (The next slide will say more about this term “group of applications being considered”.)
- 	_[Return to TOC](slide3.xml)_

## Before the Meeting—Reviewing, Critiques and Scoring

- The term “group of applications being considered” means different things at different times.
- For standing review committees, the reference group of applications for unscoring is usually all those in the current and past two meetings.  For an RFA, it is usually all those in the current meeting.
- Sometimes the group of applications being reviewed at a meeting is very small or of exceptional quality.  In such cases, unscoring 50% may not be in the best interests of science.  If you think this is the case, check with the SRA.
- Return to TOC

## Before the Meeting—Internet Assisted Review

- Internet Assisted Review (IAR) may be used to submit scores and critiques electronically for the meeting.
- IAR has three phases: *Upload, Read, and Edit*. Each phase is given a deadline by the SRA.
- During the *Upload Phase* the reviewers will enter their written critiques and preliminary scores into the IAR system.
- During the *Read Phase* you will be able to download and read the other reviewers critiques prior to the review meeting.
- During the *Edit Phase* you will be able to modify your critiques (after the review meeting if necessary).
- This system will: focus and accelerate the discussions at the review meeting, provide a fair and balanced review for the applicant, and get everyone out on time!
- Return to TOC

## At the Meeting—Protocol

- Arrive on time, stay to the end.
- We know that you are busy and that reviews take time.  Still, to develop consistent scoring norms, to allow a common understanding of any policy changes, and to promote maximum fairness to the applicants, we ask you to be present when the meeting starts.  By staying until the end, a consistent scoring pool is maintained and all scores within a meeting can be compared. However, inform the SRA if time is a factor.
- Return to TOC

## At the Meeting—Procedures

- Applications will be discussed one at a time, except the “unscored” ones.
- For those discussed, the Chair will ask for the  recommended score from all the assigned reviewers prior to the discussion of the application.
- Present only the main ideas of your critiques. Your colleagues will admire your brevity and focus.
- Return to TOC

## At the Meeting—Procedures

- Procedures vary among groups, but present only the main ideas from your critiques.  You should be able to do this in 5-7 minutes.
- If you present after another reviewer and have no new information to add, it’s fine to just say “I agree and have nothing to add.”
- 	Return to TOC

## At the Meeting—Procedures

- After the critiques are presented, the Chair will lead general discussion.
- As you participate in the discussion, remember that other reviewers will assume you have expertise and are credible, so there is no need to “prove” yourself. In other words, don’t “nit pick.” Stick to main issues.   Stick to the review criteria.
- After discussion, the Chair will lead final scoring of each application.
- Return to TOC

## At the Meeting—Procedures

- While you don’t have to “prove” yourself, neither do you have to be modest.  If you have been assigned an application or want to comment, do so without remarks such as “This isn’t my area...” or “I’m not an expert...”. The committee will understand that you speak from your own perspective and may not be knowledgeable about all aspects of the application. (For example, a statistics expert isn’t expected to know dosing schedules).
- 	Return to TOC

## At the Meeting—Turning in Materials

- Give an electronic copy of your critiques on a diskette, or CD to the support staff immediately at the end of the meeting or after each application is finished.  You will turn in all your scores on a score sheet at the end of the meeting. You will also turn in certification that you had no conflicts of interest in the review meeting.
- 	Return to TOC

## At the Meeting—Turning in Materials

- It is important that you turn in your written critique right after the meeting or review. If possible you should not wait to send them via Email. If you have a laptop edit your critiques before you leave the meeting. You can transfer them to an NIH computer at the meeting using a thumb flash drive.
- Edited comments can also be uploaded through Internet Assisted Review during the *Edit Phase.*
*	*Return to TOC

## After the Meeting—Confidentiality

- Maintain confidentiality of the meeting.
- You may be asked by other NIH staff, applicants, or colleagues how the meeting proceeded.  Apart from very general comments such as “We worked hard,” or “I enjoyed it,” please say nothing to anyone outside the review staff. As far as the meeting goes, your memory banks should be erased.
- 	Return to TOC

## After the Meeting—Reimbursement

- Under the new system, you do not have to turn in receipts in most cases (see Reimbursement).
- Turn in receipts for exceptions to the standard reimbursement quickly.  The Federal government has audit guidelines that NIH is expected to meet in providing reimbursement, and it will help us very much if you do not delay in submitting your receipts.
- 	Return to TOC

## After the Meeting—Thanks!

- Reviewers play an extremely important role in furthering the scientific goals of the NIH and the public health goals of the Department of Health and Human Services.  NIH hopes you will be proud of your contribution.  Please know that you have our sincere gratitude for your help!** **
**	**Return to TOC