Archived Information

Interim Evaluation of the Regional Educational Laboratories

Individual Evaluation Plan for the\
Northwest Regional Education Laboratory (NWREL)

March 3, 1999

The purpose of this document is to describe the plans to implement the
Interim Evaluation of the Regional Educational Laboratories at the
Northwest Regional Education Laboratory (NWREL). This plan contains the
most up-to-date information on the schedule of activities to implement
the evaluation at this Laboratory, the current panel members, and the
proposed data sources. The data sources are based on information
obtained from NWREL's nominated signature works, their inventory of
products and services, and the other data items needed to examine
overall Laboratory operations.

# Schedule of Activities for NWREL Evaluation

  ---------------------------------------------------- ------------------
  **Activity**                                         **Proposed Date**

  Approval of Individual Evaluation Plan by PES        03/08/99

  Receipt of advance materials from NWREL              03/15/99

  Distribution of advance materials to panel members   03/15/99

  Training Session #3, including NWREL presentation    03/17/99

  On-site panel visit                                  04/05/99 --
                                                       04/09/99

  Individual evaluation reports submitted to DIR       04/09/99

  Synthesis report submitted to DIR by panel chair     04/16/99

  Review of synthesis report by panel members          04/19/99 --
                                                       04/23/99

  Final synthesis report submitted to DIR by panel     04/30/99
  chair                                                

  Synthesis and individual reports sent to PES and     05/05/99
  NWREL                                                

  NWREL response to evaluation report submitted to DIR 05/21/99

  DIR submission of NWREL Interim Evaluation Reports   05/26/99
  (including individual reports, synthesis report, and 
  Lab response) to ED                                  
  ---------------------------------------------------- ------------------

**B. Current Participants in NWREL Evaluation**

The following persons are assigned to serve as members of the NWREL
review panel:

> Diane Lassman (Chair)
>
> Robert Egbert
>
> Pi Irwin
>
> Michael Jennings
>
> Morris Lai
>
> Haroldie Spriggs

# C. Data Sources to be used in the NWREL Evaluation

The Evaluation Framework approved by ED contains associated evaluation
questions and indicators to guide the conduct of the Interim Evaluation
and reporting of results. The specific data sources to be used for the
evaluation of the NWREL have been developed after receiving input from
the Laboratory, PES, and the OERI Program Officer regarding the
nominated Signature Works and the NWREL Inventory of Products and
Services.

The remainder of this section identifies the specific data items or data
collection activities to be included in the NWREL evaluation plan. Most
of the publications and other hard-copy materials specified as data
sources are to be provided to the peer reviewers as part of their
advance materials. Peer reviewers with Internet access will also be able
to examine the NWREL's web-site before arriving on the site visit.

The following data sources are grouped by whether they are: (1)
associated with a Signature Work; (2) additional items selected from the
Inventory that are not part of the selected Signature Works; or (3)
materials to be reviewed in conjunction with overall Laboratory
operations. Summary descriptions of the two selected Signature Works, as
described by NWREL, are attached to this document.

Signature Work 1: Trait-based Assessment and Instruction---Reading and
Oral Communications

This Signature Work as currently defined differs slightly from the
initial submission made by the NWREL. For this Signature Work, the peer
reviewers will focus on those aspects of the trait-based assessment
model that are specifically related to reading and oral communications.
The writing and mathematics assessment activities that are also a part
of the trait-based assessment work at NWREL will not be reviewed by this
panel.

Specific publications and materials related to this Signature Work have
been selected from the NWREL Inventory of Products and Services. The
list of proposed items for advance review by the panelists includes:

-   *Assessment Bibliography and Training Materials (AE2)*

-   *Reading Resources Annotated Bibliography (AE6)*

-   *Reading Assessment Process (AE7)*

-   *Speaking and Listening Bibliography (AE8)*

-   *Speaking and Listening Paper (AE9)*

-   *Reading Assessment Workbook (AE15)*

-   *Integrated Language Arts Assessment Workbook (AE17)*

-   *Promotional/other literature on Assessment Resource Library (AE19)*

-   *6-traits Website (AE25)*

-   *Journey of a Reader (video)*

Other materials that NWREL should provide regarding Signature Work 1 for
advance review by the panel include:

-   References to Signature Work 1 in NWREL Technical Proposal, response
    to questions, and initial project plans

-   References to Signature Work 1 in contract modifications

-   References to Signature Work 1 in quarterly reports

-   References to Signature Work 1 in annual plans and updates

-   Deliverable List for Signature Work 1

-   Design documents for Signature Work 1, including design team meeting
    agenda and notes

-   Documentation of needs assessment related to Signature Work 1,
    including minutes and agenda of reading advisory conference,
    evidence of revisions based on customer needs, and other sources
    identifying need for reading and oral communication trait-based
    assessment models

-   Documentation of review of products by peer review and expert panels
    as part of QA processes

-   Drafts of rubrics, publications, and training materials to
    demonstrate examples of QA processes

-   List of contracts, institutes, and on-going partner sites for
    Signature Work 1 (including contact information)

-   Lists of states and districts using reading trait models in
    large-scale assessment (including contact information)

-   Evaluations of pilot sessions for reading and oral communications
    models

-   Evaluations of institutes and workshops regarding reading and oral
    communications models

-   Rater statistics and current research on technical characteristics
    of reading and oral communication trait models

-   Summary of teacher feedback groups

-   Documentation of revisions of materials resulting from pilot testing
    and customer feedback

-   Client thank you letters regarding reading and oral communications
    models

-   Utilization statistics, including materials disseminated, materials
    requested, web-site counts, product sales

-   Product reviews by outside experts

-   List of publications in journals, etc. of materials related to
    Signature Work 1

-   List of presentations made related to Signature Work 1

On-site activities for evaluation of Signature Work 1 may include
interviews with:

-   REL project staff involved with Signature Work 1

-   Partners in R&D sites

-   Users in implementation sites

-   OERI Program Officer

On-site observations of selected activities may also be included,
specifically:

-   Assessment Resource Library

-   Implementation site visit

Signature Work 2 ---Onward to Excellence (OTE) II

Because the OTE model has been used by this Laboratory for many years,
we asked Laboratory staff to ensure that they were providing products
and reports to the peer reviewers that reflected work conducted in the
current contract period. Specific publications and materials related to
this Signature Work have been selected from the NWREL Inventory of
Products and Services. The list of proposed items for advance review by
the panelists includes:

-   *OTE II training materials, including trainer notes, participant
    materials, and workbooks (SIP7)*

-   *Alaska OTE Study of Rural Systemic Reform: Year Two Progress
    Report: Second Draft Case Studies of Alaska Sites (SIP28)*

-   *Alaska OTE Research Team materials for three workshops: Getting
    Started, Monitoring Progress, and Looking at Student Progress
    (SIP29)*

-   *OTE II External Study Team workshop materials (SIP30)*

-   *"Onward to Excellence II: A Systemic Reform Mode." (SIP43)---book
    chapter*

-   *"Onward to Excellence II: Carrying the Flame" (SIP44)*

-   *"Onward to Excellence II\-\--School Improvement Process: Services,
    Activities, Timeline, Costs" (SIP45)*

-   *"Onward to Excellence II---The Road to Reform: What to Expect Along
    the Way" (SIP46)*

-   *"Onward to Excellence II---The Evolution of Onward to Excellence I
    to II: Making Changes" (SIP47)*

-   *"Make a Difference: Onward to Excellence" (SIP48)*

-   *There's a Real Buzz on Around Here About Education: The Snoqualmie
    Valley School District Story (SIP49)*

-   *The Catalyst for Change: The Bruce, Mississippi Story (SIP50)*

-   *Closing the Gap: The Southwest Region School District Story
    (SIP51)*

-   *Scaling Up Onward to Excellence II: A Systemic Reform Model
    (SIP55)*

Other materials that NWREL should provide regarding Signature Work 2 for
advance review by the panel includes:

-   References to Signature Work 2 in NWREL Technical Proposal, response
    to questions, and initial project plans

-   References to Signature Work 2 in contract modifications

-   References to Signature Work 2 in quarterly reports

-   References to Signature Work 2 in annual plans and updates

-   Deliverable List for Signature Work 2

-   Evidence of agreements and contracts with other entities for support
    of pilot testing and implementation of OTE II (e.g., Grant Wood Area
    Education Agency, Weyerhaeuser Company Foundation, Regional
    Assistance Center, AKRAC, MISTA Partnership)

-   Design documents for Signature Work 2 to be selected from *Tools and
    Strategies for School Change: A Framework for Development; Onward to
    Excellence: Design for OTE Adaptation---Process and Service; and
    Design Specifications: Tools and Strategies for School Change.*

-   Documentation of needs assessment related to Signature Work 2,
    including minutes and agenda of advisory councils (e.g., School
    Improvement Advisory Committee Meetings), evidence of revisions
    based on customer needs, and other sources identifying need for OTE
    II, selected from *Enhanced Case Description of Scaling Up
    Continuous Improvement: Onward to Excellence in Mississippi; and
    Barriers to Scaling Up School Change.*

-   Documentation of review of products by peer review, advisory
    councils, and expert panels as part of QA processes

-   Drafts of publications, and/or training materials to demonstrate
    examples of QA processes

-   List of contracts and on-going partner sites (including contact
    information)

-   Evaluations of OTE II implementation

-   Summaries of user feedback

-   Documentation of revisions of materials resulting from pilot testing
    and customer feedback, including sections of *Mississippi Onward to
    Excellence Impact Study,* and notes from AOTE Training
    Development/Planning Workshop

-   *Onward to Excellence II Pilot Description: Summary of Major
    Features and Preliminary Outcomes of the Project*

-   Utilization statistics, including materials disseminated, materials
    requested, web-site counts, product sales

-   Product reviews by outside experts

-   List of publications in journals, etc. of materials related to
    Signature Work 2

-   List of presentations made at regional and national meetings related
    to Signature Work 2

On-site data collection activities in relation to Signature Work 2 may
include interviews with:

-   REL project staff involved with Signature Work 2

-   Partners in R&D sites

-   Users in implementation sites

-   OERI Program Officer

On-site observations of selected activities may include, if feasible
within the time available:

-   Implementation site visit

# Additional Items Selected from the Inventory of Products and Services

In addition to the Signature Works data sources, additional materials
will be provided in advance to the peer reviewers. The following
materials have been selected from the NWREL Inventory to provide
additional breadth to the evaluation:

-   Latest 1998 issue of *Northwest Education* magazine and *Northwest
    Report* newsletter (DC1 and DC2)

-   *Policy Analysis Charter schools policy options (PPD14)*

-   *Catalog of School Reform Models (First Edition) (PPD32)*

-   *Addendum to the Catalog of School Reform Models (PPD37)*

-   *Supporting documents (e.g., agenda, attendance list, proceedings)
    from 1998 National School Improvement Conference (SIP69)*

-   *Strengthening Community Education: Building Partnerships Workbook
    (RE1)*

-   Supporting documents (e.g., agenda, attendance list, proceedings)
    from selected 1998 community meetings and onsite workshops for
    partner sites (RE8)

-   Selected, representative item from *It's Good Teaching* series

-   *Toolkit 98* (one copy to be made available at the Laboratory;
    advance copies are not desired for the peer reviewers due to the
    size of the document).

# Materials for Overall Laboratory Operations

Other materials will be collected and provided to the peer reviewers so
that they can examine the overall operations of NWREL. The following
materials will be provided to the peer reviewers in advance of the site
visit:

-   Original RFP

-   NWREL Technical Proposal and response to technical questions

-   Contract modifications

-   Quarterly reports

-   Annual plans and updates

-   Summary of REL funding, by source

-   Organizational chart

-   Board roster and affiliations

-   Selected Board meeting agenda and minutes, especially annual Board
    planning retreat and follow-up meeting

-   1998 Strategic Plan and Status Report

-   1998 Internal Evaluation Report

-   1998 Customer Satisfaction Survey Results

-   QA procedures documentation

-   Needs assessment procedures documentation

-   Customer feedback procedures documentation

-   List of staff presentations at relevant regional and national
    meetings

-   List of staff publications in external venues

-   Inventory of products and services from the current contract period

On-site interviews regarding overall NWREL operations will be conducted
and should include:

-   Staff interviews

-   Board interviews or focus groups

-   OERI Program Officer interview