**INSTITUTE OF EDUCATION SCIENCES**

**Procedures for Peer Review of Grant Applications**

***Adopted by the National Board for Education Sciences on***

***January 24, 2006***

**INSTITUTE OF eDUCATION sCIENCES**

**Grant Application Review**

Under the Education Sciences Reform Act, activities of the Institute of
Education Sciences (Institute) that are carried out through grants,
contracts, or cooperative agreements, at a minimum, shall be awarded on
a competitive basis and, when practicable, through a process of peer
review. Further, the Director is required to establish a peer review
system, involving highly qualified individuals with an in-depth
knowledge of the subject to be investigated, for reviewing and
evaluating all applications for grants and cooperative agreements that
exceed \$100,000, and for evaluating and assessing the products of
research by all recipients of grants and cooperative agreements under
this Act.

See Appendix A for relevant sections of the legislation.

***Overview***

The Standards and Review Office is responsible for the Institute\'s
scientific peer review process. Standards and Review staff work with a
contractor that handles the logistics and manages the electronic system
for grant application submission and review. The review process includes
(a) pre-submission procedures that enable the Institute to plan for
specific review sessions, (b) application processing procedures, (c)
panel selection and appointment, (d) pre-review panel procedures, (e)
panel meeting, and (f) post-review process.

***Pre-Submission Procedures***

***Letters of Intent***

In the Request for Applications, the Institute encourages potential
applicants to send a letter indicating their intent to submit an
application. As described in the Request for Applications, the Letter of
Intent is optional, is not binding, and does not enter into the review
of subsequent applications. The information that it contains allows
Institute staff to estimate the potential workload to plan the review.

The Letter of Intent must be submitted electronically by the date listed
at the beginning of the Request for Applications, using the instructions
provided at the submission web site listed in the Request for
Applications.

The Letter of Intent includes a descriptive title, the goal that the
application will address, and a brief description of the research
project; the name, institutional affiliation, address, telephone number
and e-mail address of the principal investigator(s); and the name and
institutional affiliation of any key collaborators. The Letter of Intent
indicates the duration of the proposed project and provides an estimated
budget request by year, and a total budget request.

***Letters of Intent Receipt and Report***

The contractor provides a report on the Letters of Intent. This report
details the number of Letters of Intent received for each competition
and includes a .pdf file containing the letters received for each
competition.

***Review of Letters of Intent***

Institute staff members review the Letters of Intent to estimate the
numbers of applications within each competition and the distribution of
applications across goals. Institute staff members also check the
letters to identify particular applications that may require additional
expertise on the review panel.

The Letters of Intent are examined to identify potential applicants who
may be considering the submission of applications that are likely to be
non-responsive to the Request for Applications (e.g., individuals
seeking support to implement a program in their school or individuals
seeking scholarship funds to support their education). Program officers
contact these individuals to clarify the Request for Applications and,
when appropriate, to suggest alternative sources of funding.

***Processing Applications***

***Submission of Applications***

Application forms and instructions for the electronic submission of
applications are available at the web site identified in the Request for
Applications approximately one month prior to the application due date.
This date is announced in the Request for Applications. Information
about the electronic submission procedures that must be followed and the
software that will be required are available on the web site.

***Receipt of Applications***

Applicants must submit their application electronically by 8:00 p.m.
Eastern time on the application receipt date announced in the Request
for Applications, using the ED standard forms and the instructions
provided on the application submission web site.

***Screening of Applications***

Applications are screened for compliance with the application rules
(e.g., page length and formatting requirements, completion of all parts
of the application), responsiveness to the Request for Applications, and
identification of resubmissions.

*Compliance*. Each application is screened for compliance with the
application rules. Staff record when the application was submitted and
verify that the required applicants components have been submitted and
that each component meets format requirements and length restrictions.

*Responsiveness*. Institute staff members screen each application for
responsiveness to the Request for Applications. Applications are
identified (a) that may have been submitted to the wrong competitions or
to the wrong goal; (b) in which text in the abstract or narrative
suggest that the applicant meant to submit to a particular goal but the
application is currently listed under a different goal; or (c) that may
be non-responsive.

Staff members identify applications that may have been submitted to the
wrong competition or to the wrong goal and send a report to the Senior
Review Officer. If the Senior Review Officer concurs, then the applicant
is contacted to verify the competition or goal to which the application
has been submitted. On occasion, the Institute suggests that the
application is more appropriate for a different competition or goal
(e.g., in instances in which an application would be non-responsive for
a particular competition but responsive for an alternate competition).
In such cases, the Institute contacts the applicant to discuss the
appropriate placement of the application.

The review for responsiveness is not an evaluation of the quality of the
proposal or of a component of the proposal. Non-responsive applications
are those that do not meet the basic requirements with respect to
restrictions on the target sample, type of intervention or assessment,
and provision of detailed descriptions of research and data analytic
procedures. Staff members identify applications that may be
non-responsive and send reports to the Senior Review Officer. The Senior
Review Officer reviews each identified application and makes a final
determination of the responsiveness of the application. The Institute
takes a conservative approach to responsiveness screening, and
relatively few applications are judged to be non-responsive.

*Resubmission*. Institute staff members screen each application for text
in the abstract, narrative, or Appendix A that indicates that the
applicant meant the application to be a resubmission but did not check
the appropriate resubmission box on the application form. Applications
classified as possible resubmissions that do not appear on the
resubmission report for the competition are marked and a report is sent
to the Senior Review Officer for review. The Senior Review Officer
reviews each identified application, and for those appearing to be
resubmissions, the applicant is contacted. If the applicant confirms
that the application is a resubmission, reviews from the previous year
are sent along with the application to reviewers.

***Assignment of Applications to Panel***

Applications are assigned to panel according to the match between the
overall expertise of reviewers on each panel and the content and
methodological approach proposed in each application.

***Assignment of Applications to Reviewers within Panels***

*Initial Conflict of Interest Determination*. Prior to assignment of
applications, Institute staff review each application to identify
possible conflicts of interest by comparing the individuals associated
with each application (principal investigator, key personnel,
consultants, advisory board members) against the individuals on each
panel and the applicant institutions with the organizational
affiliations of each panel member. In addition, any obvious professional
collaboration between individuals associated with each application and
panel members is noted. Further discussion of conflict of interest,
including definitions of types of conflicts of interest, appears in the
section below on Appointing Scientific Review Panels.

*Assignment of Applications*. Applications are assigned to reviewers
according to the match between the expertise of each reviewer and the
content and methodological approach proposed in each application, with
consideration given to the balance of applications across reviewers.

*Number and Type of Reviewers*. Each application is assigned to at least
2 reviewers. Applications to conduct randomized trials submitted to Goal
3 (efficacy evaluations) or Goal 4 (effectiveness evaluations) have 3
reviewers, one of which is an experienced methodologist in the
implementation and evaluation of randomized trials. Applications to
conduct studies implementing single-subject methodologies are assigned
at least one reviewer who is experienced in the implementation and
analysis of single-subject studies.

***Selection and Appointment of Scientific Review Panels***

***Overview of Panel Composition and the Roles of Panel Members***

Panels are composed of a Panel Chair, a Scientific Review Administrator,
and panel members. The Deputy Director for Science appoints the Panel
Chair from among the members of the panel. The Scientific Review
Administrator is a non-voting member of the panel and is typically a
Standards and Review staff member or contractor staff member. The
Scientific Review Administrator understands the peer review process and
is familiar with all of the procedural rules governing the process. The
Panel Chair and Scientific Review Administrator work together to
facilitate an orderly peer review meeting and ensure that all
applications receive a fair review by the panel.

***Selection of Panel Members***

The Institute identifies highly qualified potential reviewers who have
the scientific expertise to evaluate the proposals on the criteria
listed in the Request for Applications. Potential reviewers are
identified primarily on the basis of quality of the research they have
conducted and published in scientific peer-reviewed journals and the
degree to which they are an in-depth expert in the research methods and
subject matter that are relevant to the applications dealt with by any
given panel. The diversity that individuals will bring to a panel on
disciplinary, institutional, and other dimensions is also considered.
The Institute uses 3 types of reviewers on its panels: principal panel
members, rotating panel members, and ad hoc panel members.

*Principal panel member*. Principal panel members serve 1 to 3 year
terms on a review panel or until the specific research grant competition
assigned to the panel is discontinued. Only standing panels (see below)
have principal panel members. The Director reviews and approves all
appointments of principal panel members.

For a particular panel review meeting, the Institute may assign a
principal panel member to serve on an alternate peer review panel as a
principal panel member rather than on the principal panel member\'s
regular panel. Such actions would occur, for example, if the
scientist\'s expertise were needed on the alternate panel for that
meeting.

*Rotating panel member*. Rotating panel members are appointed to a
particular panel (standing panel or single session panel) for one review
session and serve as full members of the panel, receiving a full
assignment of applications to review, attending the panel review
meeting, and scoring all applications considered by the panel except
those for which the individual has a conflict of interest.

*Ad hoc panel member*. Ad hoc panel members are appointed to a
particular panel (standing panel or single session panel) for one review
session and are typically assigned 2 to 4 applications to review. Ad hoc
panel members participate in the full panel review meeting on those
applications for which they served as a primary reviewer. They do not
participate in and are not present for the discussion and scoring of any
application for which they were not a primary reviewer. Their
participation is typically via teleconference. Ad hoc reviewers are
recruited when the number of applications received is greater than what
a panel can efficiently handle or when particular expertise is needed
for a specific application.

***Types of Panels***

The Institute has two types of Scientific Review Panels: standing panels
and single session panels. In Appendix B are lists of the Institute\'s
initial standing panels and single session panels for the fall 2005
review session.

*Standing panels*. Standing panels are scientific review panels to which
panel members may be appointed for multiple, consecutive review
sessions.

Standing panels may be composed of principal panel members, rotating
panel members, and ad hoc panel members. To promote continuity in
standards and procedures across review sessions, at least 50 percent and
up to 100 percent of the members of a full panel are principal panel
members. A typical full panel is limited to 15 to 20 members, in order
to facilitate discussion of the applications. The remaining members of
the full panel are rotating panel members or ad hoc panel members who
serve on a panel for a specific grant review session. Generally, no more
than 3 or 4 ad hoc panel members are used to supplement a panel.

For each meeting of a peer review panel, the Deputy Director for Science
shall appoint an individual to serve as chair of the panel.

*Single session panels*. Single session panels are appointed for one
review meeting. These panels may be composed of rotating panel members
and ad hoc panel members. A typical full panel is limited to 15 to 20
members, in order to facilitate discussion of the applications.
Generally, no more than 3 or 4 ad hoc panel members are used to
supplement a panel.

***Recruitment of Panel Members***

The identification of potential panel members is a continuous process
involving an annual call for nominations from the IES Board,
consideration of IES report external reviewers, solicitation of
suggestions from IES and Center experts, literature review, and
networking with known experts in relevant fields. After receiving the
Director's approval (for potential principal panel members), the Deputy
Director for Science generally sends the initial letter (electronic
mail) of invitation to all potential reviewers (including rotating and
ad hoc). This contact is followed by phone calls and electronic mail to
the potential reviewer from the Institute staff member responsible for
the panel.

*Appointment letters.* Following the acceptance of panel members to
standing panels, the Deputy Director for Science sends appointment
letters to panel members indicating the term of their appointment.

***Conflicts of Interest***

The Institute is concerned with three types of conflicts of interest,
which are described below: (a) personal financial interest, (b) personal
or professional relationship, and (c) professional relationship with the
applicant\'s institution.

*Personal financial interest*. The Institute considers reviewers to be
in conflict if they can benefit financially from the outcome of the
review. An individual may not serve on a panel if the individual is part
of an application that is being considered by the panel, regardless of
the scope of the individual\'s role on the application (e.g., principal
investigator, consultant, advisory board member). An individual may not
serve on a panel if his or her spouse or partner, child, household
member, or other relative with whom he or she has a close relationship
has a financial interest in the outcome of the review. An individual may
not serve on a panel if he or she has a financial interest in a
for-profit organization that has an application being considered by the
panel.

For principal panel members, this means that the individual would not
serve as a reviewer for that peer review session, but could serve on the
panel for future sessions and could serve on an alternate panel for that
particular peer review session.

*Personal or professional relationship*. Reviewers may not serve as a
primary reviewer for and must recuse themselves from participating in
the panel discussion and scoring of any application from individuals
with whom they have a close personal or professional relationship (e.g.,
collaborators on current projects, recent students, recent professors,
personal friends). Judgments are determined on the basis of recency,
frequency, and strength of the working relationship between the member
and the applicant as reflected, for example, in co-authored
publications. If the close personal relationship is one that constitutes
a potential personal financial interest, then the individual may not
serve on the panel as described in the previous section.

Reviewers may not participate in the review (either as primary reviewers
or as part of the panel discussion and scoring) of applications from
individuals with whom they have professional differences that could
reasonably be viewed as affecting the objectivity of their review.

For conflicts of interest falling under this category, staff at the
panel meeting records the reviewer's absence from the panel discussion.
The reviewer does not receive access to the application or to the
reviews of the application.

*Professional relationship with an applicant\'s institution*. Reviewers
may not serve as a primary reviewer for and must recuse themselves from
participating in the panel discussion and scoring of any application
submitted by other individuals from the institution or organization that
employs them.

For conflicts of interest falling under this category, staff at the
panel meeting records the reviewer's absence from the panel discussion.
The reviewer does not receive access to the application or to the
reviews of the application.

***Pre-Panel Meeting Process***

***Contact with Panel Members***

After individuals have agreed to participate on a panel, their names are
given to the logistics contractor so that all necessary forms are
completed (e.g., Conflict of Interest, Consulting Agreement) prior to
the release of applications to the panel and travel arrangements can be
made in a timely fashion.

*Introduction to Contractor.* Following an individual\'s acceptance to
participate in a panel review, the Institute staff member responsible
for the panel sends an acknowledgement letter (electronic mail) to the
panel member that includes an introduction to the logistics contractor.

*Initial Contact from Contractor*. The logistics contractor sends an
introductory letter (electronic mail) to each panel member. The
contractor provides information on the electronic peer review system and
sends all forms that need to be completed by the reviewers (e.g.,
Conflict of Interest).

***Instructions to Panel Members***

There are two primary means for providing guidance to panel members
prior to the actual panel review session -- the pre-meeting orientation
teleconference and the peer reviewer handbook.

*Pre-Meeting Orientation Teleconference*. Approximately 6 to 7 weeks
before the panel review session, a pre-meeting orientation
teleconference is conducted with Institute staff, the logistics
contractor, and the panel members. Generally 2 to 3 teleconference times
are offered for each panel so that panel members may participate at a
convenient time. The purpose of the pre-meeting orientation is to
provide reviewers with an overview of the peer review process, identify
key points in the relevant Requests for Applications, familiarize the
reviewers with the scoring system, emphasize the need for looking over
the applications as soon as they receive them to identify any conflicts
of interest with assigned applications, and stress the importance of
completing reviews by the due date so that we can triage the
applications in time for reviewers to familiarize themselves with all of
the applications to be discussed at the review session.

*Peer Review Handbook*. Each panel member receives a Peer Review
Handbook. The Handbook provides detailed information about the
responsibilities of the panel members and the panel chair.

***Release of Applications to Reviewers***

About 6 to 7 weeks prior to the panel meeting, each reviewer receives
hard copies of the applications that have been assigned to the reviewer.
At this time, each reviewer is also given access to those applications
that have been assigned to him or her through the electronic peer review
system. In addition, if any of the assigned applications are
resubmissions, the reviewer will receive the previous year\'s reviews
for that application.

***Review Criteria***

All of the Institute\'s regular research competitions have a common goal
structure that categorizes the types of research projects the Institute
funds and the requirements for projects submitted under each goal. These
requirements are in the Requests for Applications. The research
standards of the Institute are, in part, embodied in the articulation of
specific methodological requirements in the Requests for Applications.
The standards are also enacted through the judgments of the scientific
reviewers. As such, the standards of the Institute are reflected in the
quality and type of scientists selected for the scientific review
panels.

The review criteria are described in each Request for Applications. With
some exceptions (e.g., training grants, research and development
centers), the criteria are the same. Reviewers are expected to assess
the following aspects of an application in order to judge the likelihood
that the proposed research will have a substantial impact on the pursuit
of that goal:

*Significance*. Does the applicant make a compelling case for the
potential contribution of the project to the solution of an education
problem? For development projects, does the applicant present a strong
rationale -- including a theoretical foundation and prior empirical
evidence -- justifying the development of the selected intervention or
assessment? For cases in which the applicant proposes to evaluate an
intervention, does the applicant present a strong rationale justifying
the need to evaluate the selected intervention (e.g., does prior
evidence suggest that the intervention is likely to substantially
improve student learning and achievement)?

*Research Plan*. The primary methodological issue is whether the
research plan is appropriate for answering the research questions or
testing the proposed hypotheses. Reviewers are asked to consider whether
the applicant presents (a) clear hypotheses or research questions; (b)
clear descriptions of and strong rationales for the sample, the measures
(including information on the reliability and validity of measures),
data collection procedures, and research design; and (c) a detailed and
well-justified data analysis plan.

*Personnel*. Does the description of the personnel make it apparent that
the principal investigator, project director, and other key personnel
possess the training and experience and will commit sufficient time to
competently implement the proposed research?

*Resources*. Does the applicant have the facilities, equipment,
supplies, and other resources required to support the proposed
activities? Do the commitments of each partner show support for the
implementation and success of the project?

***Review of Applications by the Primary Reviewers***

Typically, each panel member is assigned to be a primary reviewer for
about 8 applications. The primary reviewers prepare written critiques of
the applications to which they have been assigned, addressing the
strengths and weaknesses of each application. Reviews are submitted
electronically through the electronic peer review system.

***Scoring of Applications***

Primary reviewers assign a score for each review criterion
(significance, research plan, personnel, and resources) and an overall
quality score for each application. Reviewers are asked to score each
application against an ideal application, as opposed to evaluating
applications against the other applications submitted to a competition.

*Criteria Scores*. Each criterion is rated on a 7-point scale from 1
(poor) to 7 (excellent), where intermediate values are treated as equal
steps along the scale.

*Overall Scores*. Each reviewer rates the overall quality of the
application on a scale from 5.0 (poor) to 1.0 (outstanding). Reviewers
are given the adjectival equivalents listed below to help anchor their
scores:

  --------------------------------- -------------------------------------
  **Overall Score Range**           **Adjectival Equivalent**

  1.0 to 1.5                        Outstanding

  1.6 to 2.0                        Excellent

  2.1 to 2.5                        Very Good

  2.6 to 3.0                        Good

  3.1 to 4.0                        Fair

  4.1 to 5.0                        Poor
  --------------------------------- -------------------------------------

The overall score is not an average of the criterion scores or a
mathematical derivative of the criterion scores. Each scientific
reviewer is asked to thoughtfully consider each aspect of the
application and make a determination of the overall quality of the
proposal.

***Triage of Applications***

To facilitate the discussion by the full panel of the most competitive
applications, the Institute generally employs a system of triaging
applications so that only the most competitive applications are
discussed and scored by the full panel at the panel review meeting.
Determination of the most competitive applications is based on the
preliminary rank order of the average overall scores of the
applications.

*Average Overall Score Report.* Based on the overall scores assigned by
primary reviewers, an average overall score for each application is
calculated and is used to generate a preliminary rank order of
applications. The Institute asks all reviewers to submit their scores
and critiques of the applications 10 business days prior to the panel
meeting so that the determination of the applications to be considered
at the panel meeting is completed with sufficient time for reviewers to
become familiar with any of the applications for which they were not a
primary reviewer.

[]{.mark}

*Triage Cut Point*. In general, the triage cut point is set so that the
full panel discusses about 25 applications. Triage cut points may vary
across competitions when a panel reviews applications for more than one
competition.

[]{.mark}

*Panel Nominations of Applications to be Considered by the Full Panel*.
At the beginning of each panel meeting, panel members are given the
opportunity to nominate for consideration by the full panel any
application that they reviewed and judged to be of equal merit to the
applications being considered by the full panel on the basis of their
average overall score. This process is described in the section on the
procedures of the panel meeting.

[]{.mark}

*Discrepant Scores*. Primary reviewers may reach different judgments
(i.e., scores) about the merit of a particular application. Institute
staff members calculate the difference in overall scores between
reviewers for each application and then the average of the differences
between scores across applications. For applications receiving only 2
primary reviews, applications that have widely discrepant scores are
identified and a third review by a panel member is obtained. All three
reviewers\' scores are then used to calculate the average overall scores
of such applications.

***Applications to be Discussed by the Full Panel***

After the triage has been determined, panel members are notified of the
applications to be discussed and scored by the full panel. Notification
is done as soon as possible to allow panel members sufficient time to
become familiar with the applications that will be discussed by the full
panel.

***Panel Meeting***

***Review of Procedures***

The evening before a panel review session, the Deputy Director for
Science meets with the panel chairpersons to go over the panel meeting
procedures and discuss any issues that have arisen during the
application review period.

***Plenary Session***

Typically the panel sessions begin with a short plenary session during
which members of all of the panels meet together. The Deputy Director
for Science provides a brief overview of the competitions for the panel
members. At the end of the Plenary Session, panels are dismissed to
convene their separate review meetings.

***Panel Orientation***

Each panel review session begins with a brief introduction and
orientation. Panel members and all observers introduce themselves. The
order of procedures for the panel and materials in the panel notebook
are reviewed.

***Nomination of Triaged Applications for Discussion by the Full
Panel***

The panel chair begins by asking panel members if anyone wishes to
nominate for consideration by the full panel any applications that they
reviewed that did not survive triage but in their opinion were of equal
scientific and technical merit to applications that are being considered
by the full panel.

If a panel member chooses to nominate an application, he or she is given
2 minutes to present the case for including the application in the panel
review. A vote of the full panel occurs and a simple majority vote
determines whether the application is added to the panel\'s agenda.

***Order of Review***

Institute staff determines the order of review before the panel meeting.
Typically, applications are first sorted by competition goal (e.g.,
Development, Efficacy, Effectiveness), and then, within each goal,
applications are sorted by competition (e.g., Reading and Writing
Education Research, Reading and Writing Special Education Research).
Within each goal and competition category, applications are randomly
ordered to the extent possible (e.g., consideration is generally given
to arranging the order so that the same reviewer is not a primary
reviewer on consecutive applications).

*Primary reviewers*. In the Order of Review, the primary reviewers are
assigned to be Reviewer 1, Reviewer 2, or Reviewer 3. In the panel
meeting, Reviewer 1 presents a short overview of the application in
addition to his or her critique of the application. Reviewers 2 and 3
are asked not to repeat what Reviewer 1 has said, but to provide
additional comments and to note where they might disagree with Reviewer
1.

*Note Taker*. For each application, a panel member other than one of the
primary reviewers for an application is assigned the task of being the
note taker who provides a summary of the panel discussion.

***Review of Applications***

The review of applications begins when the chair identifies the
application to be reviewed and asks anyone who has a conflict of
interest to leave the room.

***Scores of Primary Reviewers***

The scores of the Primary Reviewers are presented to the panel.

***Presentation of Application and Critiques by Primary Reviewers***

Reviewer 1 presents a brief description of the application and his or
her critique of the application. Reviewer 2 then provides any additional
comments about the application. If there is a third reviewer, Reviewer 3
also gives his or her critique.

During these presentations the primary reviewers are invited to express
any strong personal points of view or personal perspectives, positive or
negative, with respect to the application.

***Panel Discussion***

After the presentations of the primary reviewers, the entire panel
discusses the merits of the application. Any panelist intending to
comment is invited to express any strong points of view or personal
perspectives, positive or negative, with respect to the application
being discussed. The chair and the Scientific Review Administrator are
responsible for ensuring that the discussion focuses on the scientific
and technical merit of the application. The chair is responsible for
ensuring that the application is reviewed according to the criteria
described in the Request for Applications.

***Comments on Budget***

After the discussion of the scientific and technical merit of the
application, the primary reviewers may offer comments on the
appropriateness of the budget.

***Summary of Panel Discussion by Note Taker***

The note taker reads his or her summary of the panel discussion. The
chair is responsible for ensuring that the summary reflects the issues
that were raised by the panel. After hearing the summary, the chair asks
panel members if they concur with the summary as presented or if any
additions or corrections are necessary.

*Minority report*. On occasion, 2 or more panel members disagree with
the majority view of the application as presented by the note taker. In
such instances, these reviewers may submit a comment reflecting their
view of the application.

***Revisit Scores of Primary Reviewers***

After the panel discussion, the primary reviewers are given the
opportunity to change any of their original scores.

*Revision of reviews*. If a primary reviewer substantially changes his
or her scores based on the panel discussion, the reviewer is asked to
make any necessary revisions to his or her written review so that the
scores are aligned with the review.

***Confidential Scoring by All Panel Members***

After the primary reviewers have indicated any changes in their scores,
all panel members, acting independently and individually, score the
application. A scoring sheet is provided in the review panel notebook
with applications listed in order of review. At the end of each day, the
scoring sheets are collected.

***Funding Enthusiasm Scores***

After all of the applications have been discussed and scored, reviewers
are asked to give each application a funding recommendation: (a) highly
recommend funding, (b) recommend funding, or (c) do not recommend
funding.

**Post-Review Session Procedures**

***Summary Reports***

The Standards and Review Office provides two basic reports to the
program officers after each panel meeting.

*Overall score report*. The Final Scientific Merit Report includes
average criteria and average overall scores for each application. The
averages are based on the scores provided by the full panel, excluding
any members who were in conflict.

*Funding enthusiasm report*. The Funding Enthusiasm Ratings Report is
the average of the funding enthusiasm ratings provided by the full
panel, excluding any members who were in conflict.

***Feedback to Applicants***

All applicants receive copies of the primary reviewers\' critiques. For
applications receiving full panel discussion, a summary of those
comments about the application is also sent. These materials are sent to
the principal investigator and to the applicant institution\'s
authorized representative.

***Procedures for Responding to Applicant Inquiries***

Applicants who have questions regarding the review of their application
are directed first to the cognizant program officer. If an applicant
questions whether his or her proposal received appropriate treatment and
is not satisfied with the program officer\'s feedback, the inquiry is
directed to the Deputy Director for Science. The Deputy Director for
Science reviews the process under which the proposal was handled to
determine if there was anything irregular or inappropriate in the
handling of the application, which is the only basis on which the
results of a grant competition review can be put aside. A re-review of
an application is the only remedy available for an application that the
Deputy Director for Science judges to have been mishandled. A re-review
consists of a review of the same application, not a revised version,
during the next scheduled panel review meeting for recurring
competitions. The re-review will occur without panel access to the
reviews from the contested review session. In instances where
competitions occur infrequently, the application will be re-reviewed
during the fiscal year when the next competition to which the
application was submitted occurs.

**Appendix A**

**Legislation Relevant to Scientific Peer Review**

**Legislation Relevant to Scientific Peer Review**

**20 U.S.C. 9501 et. seq.**

**Education Sciences Reform Act of 2002**

**[SECTION 102. DEFINITIONS]{.underline}**

\(18\) [Scientifically based research standards]{.underline}.

> \(A\) The term \"scientifically based research standards\" means
> research standards that---
>
> \(i\) apply rigorous, systematic, and objective methodology to obtain
> reliable and valid knowledge relevant to education activities and
> programs; and
>
> \(ii\) present findings and make claims that are appropriate to and
> supported by the methods that have been employed.
>
> \(B\) The term includes, appropriate to the research being
> conducted---
>
> \(i\) employing systematic, empirical methods that draw on observation
> or experiment;
>
> \(ii\) involving data analyses that are adequate to support the
> general findings;
>
> \(iii\) relying on measurements or observational methods that provide
> reliable data;
>
> \(iv\) making claims of causal relationships only in random assignment
> experiments or other designs (to the extent such designs substantially
> eliminate plausible competing explanations for the obtained results);
>
> \(v\) ensuring that studies and methods are presented in sufficient
> detail and clarity to allow for replication or, at a minimum, to offer
> the opportunity to build systematically on the findings of the
> research;
>
> \(vi\) obtaining acceptance by a peer-reviewed journal or approval by
> a panel of independent experts through a comparably rigorous,
> objective, and scientific review; and
>
> \(vii\) using research designs and methods appropriate to the research
> question posed.

\(19\) [Scientifically valid education evaluation]{.underline}. The term
\"scientifically valid education evaluation\" means an evaluation
that---

> \(A\) adheres to the highest possible standards of quality with
> respect to research design and statistical analysis;
>
> \(B\) provides an adequate description of the programs evaluated and,
> to the extent possible, examines the relationship between program
> implementation and program impacts;
>
> \(C\) provides an analysis of the results achieved by the program with
> respect to its projected effects;
>
> \(D\) employs experimental designs using random assignment, when
> feasible, and other research methodologies that allow for the
> strongest possible causal inferences when random assignment is not
> feasible; and
>
> \(E\) may study program implementation through a combination of
> scientifically valid and reliable methods.

\(20\) [Scientifically valid research]{.underline}. The term
\"scientifically valid research\" includes applied research, basic
research, and field-initiated research in which the rationale, design,
and interpretation are soundly developed in accordance with
scientifically based research standards.

**PART A -- THE INSTITUTE OF EDUCATION SCIENCES**

**[SECTION 114. OFFICE OF THE DIRECTOR]{.underline}**

\(f\) [DUTIES]{.underline}. The duties of the Director shall include the
following:

> \(2\) To ensure the methodology applied in conducting research,
> development, evaluation, and statistical analysis is consistent with
> the standards for such activities under this title.
>
> \(7\) To ensure that activities conducted or supported by the
> Institute are objective, secular, neutral, and nonideological and are
> free of partisan political influence and racial, cultural, gender, or
> regional bias.
>
> \(g\) [EXPERT GUIDANCE AND ASSISTANCE]{.underline}. The Director may
> establish technical and scientific peer-review groups and scientific
> program advisory committees for research and evaluations that the
> Director determines are necessary to carry out the requirements of
> this title. The Director shall appoint such personnel, except that
> officers and employees of the United States shall comprise no more
> than ¼ of the members of any such group or committee and shall not
> receive additional compensation for their service as members of such a
> group or committee. The Director shall ensure that reviewers are
> highly qualified and capable to appraise education research and
> development projects. The Federal Advisory Committee Act (5 U.S.C.
> App.) shall not apply to a peer-review group or an advisory committee
> established under this subsection.

**[SECTION 116. NATIONAL BOARD FOR EDUCATION SCIENCES.]{.underline}**

\(b\) [DUTIES]{.underline}. The duties of the Board shall be the
following:

> \(3\) To review and approve procedures for the technical and
> scientific peer review of the activities of the Institute.

**[SECTION 120. COMPETITIVE AWARDS.]{.underline}**

Activities carried out under this Act through grants, contracts, or
cooperative agreements, at a minimum, shall be awarded on a competitive
basis and, when practicable, through a process of peer review.

**PART B -- NATIONAL CENTER FOR EDUCATION RESEARCH**

**[SECTION 134. STANDARDS FOR CONDUCT AND EVALUATION OF
RESEARCH.]{.underline}**

\(a\) [GENERAL]{.underline}. In carrying out this part, the Research
Commissioner shall---

> \(1\) ensure that all research conducted under the direction of the
> Research Center follows scientifically based research standards;
>
> \(2\) develop such other standards as may be necessary to govern the
> conduct and evaluation of all research, development, and wide
> dissemination activities carried out by the Research Center to assure
> that such activities meet the highest standards of professional
> excellence;
>
> \(3\) review the procedures utilized by the National Institutes of
> Health, the National Science Foundation, and other Federal departments
> or agencies engaged in research and development, and actively solicit
> recommendations from research organizations and members of the general
> public in the development of the standards described in paragraph (2);
> and
>
> \(4\) ensure that all research complies with Federal guidelines
> relating to research misconduct.
>
> b\) [PEER REVIEW]{.underline}.
>
> \(1\) In General. The Director shall establish a peer review system,
> involving highly qualified individuals with an in-depth knowledge of
> the subject to be investigated, for reviewing and evaluating all
> applications for grants and cooperative agreements that exceed
> \$100,000, and for evaluating and assessing the products of research
> by all recipients of grants and cooperative agreements under this Act.
>
> \(2\) Evaluation. The Research Commissioner shall
>
> \(A\) develop the procedures to be used in evaluating applications for
> research grants, cooperative agreements, and contracts, and specify
> the criteria and factors (including, as applicable, the use of
> longitudinal data linking test scores, enrollment, and graduation
> rates over time) which shall be considered in making such evaluations.

**PART E -- NATIONAL CENTER FOR SPECIAL EDUCATION RESEARCH**

**[SECTION 175. ESTABLISHMENT]{.underline}**

\(c\) [APPLICABILITY OF EDUCATION SCIENCES REFORM ACT OF
2002]{.underline}. Parts A and F, and the standards for peer review of
applications and for the conduct and evaluation of research under
sections 133(a) and 134, respectively, shall apply to the Secretary, the
Director, and the Commissioner in carrying out this part.