**Archived Information**

## 

## Chapter II

**Peer Review Process**

The *Standards for Conduct and Evaluation of Activities Carried Out by
the Office of Educational Research and Improvement (OERI) for Evaluation
of Recipients of Grants, Cooperative Agreements, and Contracts* mandate
that the Regional Educational Laboratories (and other programs funded by
OERI) be evaluated through a peer review process. The *Standards*
provide specific guidance for the selection of peer review panels,
including the knowledge and expertise needed to serve as a peer
reviewer.

This chapter describes the peer review process that was implemented as
part of the Interim Evaluation of the Regional Educational Laboratories.
The peer review process was developed through three initial stages: (1)
recruitment, (2) panel assignment, and (3) training. Each of these
activities will be discussed in further detail in the sections that
follow. The actual review process conducted by the panelists, and the
resulting evaluation reports, is discussed in Chapter III
(Implementation of the Evaluation).

### Recruitment of Peer Reviewers

According to the *Standards*, each peer reviewer must have the necessary
knowledge and expertise in the area of the project being reviewed to
evaluate the performance of a recipient. This experience may include:

-   *Expert knowledge of subject matter in the area of the activities to
    be reviewed*

-   *Expert knowledge of theory or methods or both in the area of the
    activities to be reviewed*

-   *Practical experience in the area of the activities or type of
    institution or both to be reviewed*

-   *Knowledge of a broad range of educational policies and practices*

-   *Experience in managing complex organizations*

-   *Expertise and experience in evaluation theory and practice*

In addition to these characteristics, peer reviewers must also be free
of any conflict of interest.

In an effort to attract a broad pool of qualified peer review
candidates, DIR used a multi-pronged approach to recruitment. First,
nominations were solicited from a number of sources, including the Lab
Directors, the U.S. Department of Education (PES, OERI, NERPP Board),
and DIR's Technical Work Group. Additionally, "snowball" approach was
implemented by asking candidates to recommend others who may be
interested in serving as a peer reviewer. DIR also made direct contact
with specific individuals representing a broad variety of educational,
business and professional organizations (e.g., The Business Roundtable,
the Prichard Committee, the National PTA) requesting the nomination of
qualified candidates.

To ensure broad representation among the candidate pool, DIR posted a
call for peer reviewers with various professional educational and
evaluation organizations (see Table II.1). Members of these associations
were invited to submit a resume or vitae to nominate themselves as a
peer review candidate.

> **Table II.1**
>
> **Selected Educational and Evaluation Associations**
>
> **Who Posted a Call for Peer Reviewers**

-   National Association of Elementary School Principals (NAESP)

-   National Center for Research on Evaluation, Standards, and Student
    Testing (CRESST)

-   National Association of Secondary School Principals (NASSP)

-   American Evaluation Association (AEA)

-   American Education Research Association (AERA)

-   American Association of School Administrators (AASA)

-   National Education Association (NEA)

-   Association for Supervision and Curriculum Development (ASCD)

    Close to 300 peer reviewer nominations were received. A resume or
    vitae was obtained from each potential candidate and reviewed for
    relevant experience and expertise. A standard interview protocol
    (Appendix II) was also completed with each candidate (via telephone)
    to gather additional information on the individual's areas of
    expertise, screen for any real or perceived conflicts of interest,
    and discuss the responsibilities of serving as a peer reviewer in
    this Interim Evaluation. Reference checks were also conducted,
    particularly on individuals who had self-nominated.

    Over the course of the recruitment and screening process, 160
    individuals declined an invitation to participate in the Interim
    Evaluation (Figure II.1). In most cases, schedule conflicts during
    the anticipated time frame for the on-site visits or the inability
    to commit 10 days of time (five in preparation and five on-site)
    were cited as reasons. In a limited number of cases, individuals
    declined to participate due to the level of financial remuneration.
    [^1] A small number of individuals were eliminated from further
    consideration due to real or perceived conflicts of interest.

    After adding a few new nominees recommended by the DIR Technical
    Work Group, a list of qualified peer review candidates (n=135) was
    sent to the Department of Education for review and comment.
    Specifically, ED was asked to identify:

```{=html}
<!-- -->
```
-   Individuals known to ED who were particularly well qualified to
    serve as a peer review panelist in general

-   Individuals known to ED who were particularly qualified to serve as
    a peer reviewer in a specific capacity, or for a specific Laboratory

```{=html}
<!-- -->
```
-   Individuals known to ED who would not be appropriate to participate
    as a peer reviewer due to previous difficulties, conflict with a
    specific Laboratory, etc.

**Figure II.1**

**Overview of Peer Review Recruitment and Panel Assignment**

![](media/image1.wmf)

\* Due to family emergencies, illness, death, or time constraints

![](media/image1.wmf)\*\* Includes 31 panelists initially assigned to a
panel plus 3 new reviewers added as replacements

DIR revised the list of candidates based on this feedback and submitted
it to the Assistant Secretary of OERI for approval. Supporting materials
provided with the slate of reviewers included a matrix of
qualifications, a table with brief biographical sketches, and each
individual's full curriculum vitae or resume. Upon receipt of approval
from ED, initial notification letters were sent to individuals who were
still under consideration. Those candidates who were no longer being
considered were also notified of their status at that time.

The approved list of peer review candidates (N=123) was subsequently
forwarded to the Labs for comment. Specifically, each Lab was asked to
identify:

-   Individuals who had a potential conflict of interest evaluating
    their site

-   Individuals who had a potential conflict of interest affecting their
    ability to evaluate any Laboratory

-   Individuals whom they considered to be highly qualified to serve as
    a peer review panelist, as well as those who would be a good panel
    chair

    Justification was required for all requested modifications to the
    peer review candidate list. As shown in Figure II.1, the final pool
    of candidates included 117 individuals. It was from this group of
    highly qualified individuals that the actual reviewers were selected
    and assigned to specific Lab review panels.

#### Panel Assignment

Because the S*tandards* call for independent evaluations to be prepared
by each peer reviewer, DIR felt that it was important that a broad range
of perspectives and expertise be represented on each Lab review panel.
While the panels were tailored to match the unique characteristics of
each Laboratory, each panel was configured to contain, at a minimum, one
or more individuals with the following characteristics:

-   Research Expertise

-   Evaluation Methodology Expertise

-   Practical Experience (e.g., educational practitioner)

-   Specialty Area Expertise (in the specialty area of the
    Laboratory)[^2]

-   Knowledge of the Labs or Lab-like activities

-   Experience managing complex organizations

-   Educational Policy & Practice Expertise

```{=html}
<!-- -->
```
-   Knowledge in content areas of Lab focus in addition to that of the
    specialty area

    In most cases, individual panel members possessed multiple areas of
    competence with a particular strength in one area. The ultimate goal
    was to configure a panel with a complementary balance of skills that
    collectively embodied the necessary knowledge and expertise to
    evaluate the assigned Lab.

When making the initial panel assignments, DIR attempted to include at
least one individual on each panel with knowledge of the region in which
the Lab was located. Diversity among panel members (with respect to
gender, race, and ethnicity) was also considered. Preference was given
to peer review candidates that were highly recommended by ED or by the
Labs, those who had been rated by DIR to have particularly strong skills
or expertise, and to those who were able to serve on two panels.[^3]

DIR originally proposed to configure each Lab review panel with five
members and retain a pool of alternates from which to fill any position
that was vacated prior to the site visit. Upon the recommendation of the
TWG, DIR increased the membership of each panel to six reviewers to
allow for attrition.[^4] By starting with six panelists, DIR was fairly
well assured that a trained panel, who had had the opportunity to review
advance materials pertaining to the Lab prior to arriving on-site, would
indeed be able to complete the evaluation for each site, even in the
event of a panelist needing to withdraw for an unforeseen reason.

The first step in assigning reviewers to Lab-specific panels was to
determine when each site visit would take place. DIR contacted the Labs
and requested that they submit their availability to host a week-long
site visit during the same ten-week time frame that the peer review
candidates were asked about their availability. Site visit dates were
set for the Interim Evaluation of each Laboratory based on feedback
received from the Labs. Once these dates were established, DIR
configured draft panels for each Laboratory evaluation based on the
desired mix of skills and expertise and the availability of peer
reviewers. Care was taken to not have panelists performing back to back
evaluations (i.e., one week after the other) in order to allow
sufficient time for the reviewer to complete the first evaluation before
preparing for the second.

Draft panel assignments were submitted to ED for review and comment.
Each Lab was also given an opportunity to review the list of panelists
assigned to evaluate their Lab. Since ED and the Labs had already been
asked to identify peer review candidates who had a conflict with a
particular Lab, DIR did not anticipate major revisions based on this
review process. In fact, only minor adjustments were made and the
revised panel assignments were submitted to ED for approval.

After final approval of the panel assignments was received from ED, DIR
proposed a chair for each panel. In addition to preparing an individual
evaluation report, panel chairs were asked to facilitate the on-site
data collection process and panel discussion sessions, conduct an exit
interview at the end of the site visit, and prepare a synthesis report
highlighting the major findings of the panelists. In consultation with
the TWG, DIR determined that in considering nominees for panel chair,
the candidate(s) should possess skills and have demonstrated experience
in the following areas:

-   Peer Review and/or Program Evaluation

-   Leadership

-   Group facilitation

```{=html}
<!-- -->
```
-   Report writing

```{=html}
<!-- -->
```
-   Broad knowledge of educational policies and practice

-   Familiarity with the work of the Laboratories

Overall, 40 peer reviewers were assigned to fill the 60 panel slots
(i.e., six panelists at each of 10 Labs). During the course of the
training and evaluation process, nine peer reviewers withdrew from the
evaluation due to family emergencies, illness, death, or scheduling
conflicts. Three of these individuals were replaced with new reviewers
while some of the remaining open slots were filled by reviewers who were
willing to take on a second Lab assignment. Some of the vacancies were
left open, with five of the Labs being evaluated by a five-member panel
instead of six. In the end, 34 individuals conducted evaluation site
visits. Selected demographic characteristics of this group are provided
in Table II.2. A final roster of peer reviewers with brief biographical
sketches is provided in Appendix III.

+-----------------------------------------------------------------------+
| **Table II.2**                                                        |
|                                                                       |
| **Final Panel Member Characteristics**                                |
|                                                                       |
| -   Gender composition                                                |
|                                                                       |
| > 50% male                                                            |
| >                                                                     |
| > 50% female                                                          |
|                                                                       |
| -   Racial/Ethnic composition                                         |
|                                                                       |
| > 73% white                                                           |
| >                                                                     |
| > 18% black                                                           |
| >                                                                     |
| > 6% Asian or Pacific Islander                                        |
| >                                                                     |
| > 3% Native American                                                  |
|                                                                       |
| -   Professional affiliation                                          |
|                                                                       |
| > 38% higher education                                                |
| >                                                                     |
| > 32% private research consultants                                    |
| >                                                                     |
| > 24% K-12 practitioners                                              |
| >                                                                     |
| > 6% community representatives                                        |
+-----------------------------------------------------------------------+

#### Training

The training provided to review panelists was designed to ensure a
common base of understanding of the evaluation framework, the peer
review process and reporting responsibilities. The peer reviewer
training was conducted through a series of three teleconferences, with a
total of five hours of training provided. Each peer reviewer was given a
training manual containing detailed outlines for each of the training
sessions, along with the evaluation framework, a list of peer reviewer
responsibilities, the Phase III *Standards*, brief summaries of each of
the Labs, review worksheets, the evaluation report format and other
materials to support the training and evaluation process.

Each training session was focused on a different aspect of peer reviewer
preparation. The objectives for each of the sessions are provided in
Table II.3. Training Sessions #1 and #3 were designed to be
Lab-specific, while Training Session #2 was a general session on the
evaluation framework. DIR offered Session #2 six times over a 10-day
period, during which panelists had the option of participating in
whatever session best fit their schedule. Training Session #3 was held
approximately two weeks prior to the scheduled site visit. During this
session, the Lab was invited to make a brief (20 minute) presentation to
provide whatever contextual information they felt was important for the
panelists to have as they began to review the advance materials for the
evaluation.

+-----------------------------------------------------------------------+
| ## Table II.3                                                         |
|                                                                       |
| ###### Objectives of the Peer Reviewer Training Sessions              |
|                                                                       |
| ####### Session #1                                                    |
|                                                                       |
| -   Introduce DIR staff and panel members                             |
|                                                                       |
| -   Provide background on the REL program and the Interim Evaluation  |
|                                                                       |
| -   Review the roles and responsibilities of panelists and panel      |
|     chairs                                                            |
|                                                                       |
| -   Provide an overview of the remaining training sessions            |
|                                                                       |
| -   Schedule Training Session #3 (Lab-specific training)              |
|                                                                       |
| ####### Session #2                                                    |
|                                                                       |
| -   Provide reviewers with a working understanding of the evaluation  |
|     framework, including evaluation questions, evaluation indicators, |
|     and possible data sources                                         |
|                                                                       |
| -   Introduce panel members to the review worksheets and the          |
|     reporting format                                                  |
|                                                                       |
| ####### Session #3                                                    |
|                                                                       |
| -   Provide detailed contextual and factual information on assigned   |
|     Laboratories                                                      |
|                                                                       |
| -   Resolve questions related to evaluation indicators and data       |
|     sources                                                           |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Review worksheets and other forms including panelists\' final     |
|     report format                                                     |
|                                                                       |
| -   Review the advance materials received                             |
+-----------------------------------------------------------------------+

In addition to the basic training provided to all peer reviewers, panel
chairs were provided with an additional one-hour session to review the
roles and responsibilities of panel chairs, including facilitation of
Training Session #3, and to clarify any procedural issues. DIR staff
also met with the panel chair on-site before the evaluation visit began
to answer any last-minute questions and to review expectations.

[^1]: ^1^ Peer reviewers were offered honoraria of \$2500. Panel chairs
    were offered slightly more, \$3600, to compensate

    for the additional time required to serve in that capacity.

[^2]: ^2^ Specialty areas of the Laboratories include: Language and
    Cultural Diversity; Rural Education; Urban Education;

    Curriculum, Learning & Instruction; Technology; School Change
    Processes; Early Childhood Education; and

    Assessment and Accountability.

[^3]: ^3^ DIR determined that the inclusion of panelists who could
    perform more than one evaluation was important to

    provide consistency across evaluations.

[^4]: ^4^ It was agreed that a review could proceed with either a five-
    or six-member panel. Replacements would be

    necessary only when the panel membership dropped below five.