Archived Information

**Interim Evaluation of the Northwest Regional Educational Laboratory**

I.  **Brief Overview of Laboratory**

The North West Regional Educational Laboratory (NWREL) was one of the
original (1966) RELs. In the thirty-three years of its existence, the
NWREL has had three Executive Directors -- Dr. Fish, Dr. Raths and Dr.
Simon-McWilliams. The NWREL also has had three Board Chairs, whose terms
have corresponded roughly with those of Executive Directors. A recent
change in the structure and operating procedures of the Board of
Directors has disconnected the Executive Director and Board Chair terms.

The NWREL includes five states -- Alaska, Idaho, Montana, Oregon, and
Washington. It covers twenty-seven percent of the land mass of the
United State but has only four percent of the people -- approximately
11,000,000. Thus, although the region has a few major cities, much of it
has only limited numbers of people and large areas are largely
unoccupied.

The substantive work of the Laboratory is divided into (a) Training and
Technical Assistance and (b) Research and Development. The Training and
Technical Assistance work is further subdivided among seven centers --
Equity, Mathematics and Science Education, National Mentoring, National
School Safety, Technology, Community and Education Volunteer Services,
and Comprehensive. Each center has its own operating budget and from
five to ten professional staff members.

The Laboratory's research and development work is subdivided among five
programs: Assessment and Evaluation, Rural Education Program, Education
and Work, School Improvement, and Child and Family. Both the seven
Centers and the five Programs report to the

Laboratory Associate Director, Dr. Carol Thomas.

The administrative functions of the NWREL, including Finance, Human
Resources, Development and Communications, and Planning and Program
Development report directly to the Executive Director/CEO, Dr. Ethel
Simon-McWilliams.

In order to coordinate and track the work of the seven centers and five
programs, Dr. Thomas holds monthly half-day meetings of the twelve
leaders of the centers and programs. The meeting agendas are organized
primarily around cross-cutting issues, thus providing opportunity for
resource sharing and coordination. Opportunity also is given for each
center/program leader to keep others current on activities and
achievements.

In addition to the monthly meetings, each center/program leader submits
a monthly report of activities and future plans. To each five to eight
page monthly report are attached important publications and other
documents. Dr. Thomas provides informal written comments and feedback on
each report.

A review of the March 1999 child and family program report showed that
it includes personnel/staffing, budgets, proposals and contracts,
deliverables and upcoming deliverables, program implementation
activities, dissemination, future plans, and programs and activities.
Each of these sub-reports contains useful and appropriately detailed
information, including, for example, account numbers and both due dates
for deliverables and the dates they were actually delivered.

Through the combination of monthly meetings and reports, as well as
informal meetings and contacts, Dr. Thomas is able to provide leadership
and coordination to the twelve very active centers and programs.

According to figures provided to the Panel, funds for the NWREL
operations during

fiscal years 1996, 1997 and 1998 came from the OERI Regional
Laboratories fund (\$14.8 M \--34 % of the total), other Department of
Education sources (\$15.5 M -- 36% of the total; Corporation for
National Service (\$2.3M -- 5 % of the total), and state and local
sources (\$10.7 M -- 25% of the total.) (When the figures were broken
down by program, the total was somewhat different.)

**II. Implementation and Management**

A.  **To what extent is the REL doing what they were approved to do
    during their first three contract years?**

The NWREL is doing very well what it was approved to do during its first
three contract years. The work was carefully planned and it is being
conducted in a thoughtful and constructive manner. Additional work also
is being completed by the Laboratory as an institution under other
grants and contracts, some of them being supplementary to the OERI
Laboratory funding for the same general tasks and others being for new
and, usually, related work.

> **1. Strengths**

As indicated above, the monthly reports now being provided to the
Associate Executive Director and then discussed with the respective
center and program heads, permits tracking the performance of both the
Centers and the Programs. The tracking process enables both the NWREL
administration and the staff to be confident that the Laboratory is
accomplishing what it was approved to complete during the first three
contract years. These reports also permit anticipating future
expectations and budgetary needs. In addition the reports provide a
strong basis for discussions and for thinking that move across centers
and programs.

**2. Areas of needed improvement**

As the work of the progresses, changes and improvements will be made in
various

processes for being sure that work is planned and conducted in a timely
and efficient manner, but I don't see any specific areas of needed
improvement at the present time.

> **3. Recommendations for improvement**

None noted.

B.  **To what extent is the REL using a self-monitoring process to plan
    and adapt activities in response to feedback and customer needs?**

NWREL conducts a biennial regional educational needs assessment, the
most recent one being completed in 1997. Another needs assessment is
scheduled for later this year. Each needs assessment has two major
strands (a) a telephone survey that is contracted to an independent
agency, and (b) one or more focus groups in each of the five states in
the Region. The Board and staff use the results of these needs
assessments in planning future Laboratory programs and directions.

In addition to this major needs assessment, the Laboratory regularly
conducts follow up studies of workshops and all major training
activities. It also keeps track of the sales of books and other
publications and conducts surveys of current and developing needs for a
variety of services and publications.

> **1. Strengths**

The biennial survey is an especially important effort and leads to a
quite useful publication \-- [Northwest Trends Shaping Education: The
1997 Regional Education Needs Assessment]{.underline}. This publication
contains basic information about schools in the region. It also reports
trends that were revealed in the telephone survey and focus groups as
well as information from various documents published by the National
Center for Educational Statistics (NCES), the Bureau of the Census, the
Anne E. Casey Foundation and others. Besides information about the
region as a unit, the volume has a major section for each of the
Region's five states. In addition to a wealth of information, the volume
has a section that presents and discusses a set of implications of this
information for the work of the Laboratory (pp. 47-48 of the report).

> **2. Areas of needed improvement**

Additional written and audiovisual information about the excitement of
school people with the Laboratory's programs and publications might be
helpful. This comment is based on experiences in the visit that this
panel made to the Laboratory in which the importance of NWREL programs
to school people was made clear in the various focus group discussions.
It is one thing when people are invited to present prepared
testimonials; it is quite another thing when they are invited to "speak"
to a panel and then are presented with a set of questions for their
responses, as our panel did. In each instance the persons in the focus
group represented the Laboratory extraordinarily effectively.

> **3. Recommendations for improvement**

Laboratory staff persons and those "users" who participated in the focus
groups made extremely powerful statements to our panel, but our panel
was composed of six persons who likely will have relatively little
opportunity to speak with potential users about what they have seen and
heard. For future reference, possible some videotapes or transcriptions
of panel discussions such as the ones we experienced would be valuable
for discussions with potential clients of the Laboratory.

**III. Quality**

> **To what extent is the REL developing high quality products and
> services?**
>
> Overall, the NWREL appears to be producing high quality products and
> services. This is

judged by the demand for both their products and services, by
examination of products and by discussions with teachers,
administrators, and others familiar with their services.

> **1. Strengths**

Examples of strengths in the NWREL program are the Trait-Based
Assessment for Writing, Trait-based Assessment for Reading, and the
anticipated Trait-Based Assessment for Mathematics. Trait-Based
Assessment for Writing was developed in the early 1980s and was
immediately popular with elementary and secondary teachers. With the
recently developing interest in "all children learning to read" the
preparation of a Trait-Based Assessment Reading program was a natural
follow up to Trait-Based Assessment in Writing. NWREL recognized the
potential and quickly moved to conduct thorough literature searches
followed by discussions with teachers and others. This preparatory work
led to the identification of six traits of a good reader, traits that
appear to parallel Bloom's taxonomy, and that teachers can be taught to
use in identifying ways in which children have progressed or not
progressed in learning to read.

The NWREL work with Trait-Based Assessment in Writing anticipated a
need. The later Trait-Based Assessment work has responded quickly and
effectively to more recent needs.

> **2. Areas of needed improvement**

The development and distribution processes being used by NWREL are quite
effective. School and community requirements change over time. The needs
sensing work that the NWREL is conducting will assist it in anticipating
future demands. I do not see any great needs for improvement.

> **3. Recommendations for improvement**
>
> None.

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

# **1. Strengths**

As noted above the NWREL has in place a strong needs sensing process.
When a need is identified, potential users are involved in the
development process. A good example of this process is the one used in
the development work with Trait-Based Assessment in Written and Oral
Communication. Dr. Lesley Thompson described this process to our panel.

After conducting a thorough review of the literature and finding
characteristics of effective readers that were mentioned with high
frequency in the scholarly literature, Dr. Thompson interviewed a number
of reading content specialists and then observed and talked with many
classroom teachers. This process permitted her to combine the research
information with the practical experience of those working regularly
with students learning to read. As she and her colleagues worked with
the approximately thirty skills of good readers that were mentioned,
they found that the skills tended to "collect" in groups. Ultimately,
the NWREL group identified six traits that good readers demonstrate.

The skills that good readers demonstrate then became the Six Traits that
are the foundation for the NWREL publications and training in Reading
Assessment. Their experience with NWREL Trait-Based Reading Assessment
was discussed with our panel by teachers and reading specialists from
Everett, Washington to Nevada to Iowa. In each instance, the visitors
mentioned the importance of the "common language" that the Traits
provide for teachers, teacher trainers, and classroom students. The
visitors also indicated that the students made better progress when
working with the Traits. (No one presented data to support these
observations.)

> **2. Areas of needed improvement**

More data are needed on teacher preparation and competence in using
Trait-Based Assessment in the teaching of reading. The testimonials are
useful, but more data also are needed on child progress in reading using
this approach.

# **3. Recommendations for improvement** 

NWREL should devote some of its own resources or obtain additional
funding for the issues noted above. The Trait-Based Assessment approach
to teaching reading is so promising that serious validation work is
critical.

**B. To what extent is the REL focused on customer needs?**

See Responses to Question IV.A.

**V. Outcomes and Impact**

A.  **To what extent is the REL's work contributing to improved student
    success, particularly in intensive implementation sites.**

Student success is difficult to measure and the contribution of a given
aspect of the student's life, including specific portions of the school
experience, to greater academic success is especially difficult to
assess. Nevertheless, the NWREL does have some evidence that both the
Six Trait Writing Program and the School Improvement Program do have a
positive impact on students, including academic progress. A great deal
more work needs to be done in this entire area -- by everyone, not just
NWREL \-- including the development and testing of instruments to use in
place of or in addition to multiple choice achievement tests.

**1. Strengths**

As suggested above, NWREL has some limited information about student
success with

the Six-Trait Assessment Writing program. One teacher\\leader reported
that use of the Six-Trait Assessment of Reading program has changed her
classroom and she told of third grade students who now voluntarily read
from an hour to an hour and a half each night.

## **2. Areas of needed improvement and Recommendations for Improvement** 

Each education program should collect information about program use as
well as well as child success in the program. NWREL is no exception.
Stating a rule is easy; living with it is much more difficult.
Nevertheless, Laboratories are in a better position than are schools and
individual teachers to design and conduct data collection and analysis
efforts and NWREL should devote more resources to doing so.

B.  **To what extent does the Laboratory assist states and localities to
    implement comprehensive school improvement strategies?**

Onward toward Excellence is NWREL's contribution to comprehensive school
improvement. Many schools in the Northwest Region are using this model,
but even more schools outside of the region also have begun to adopt it.
One facilitating reason has been that Weyerhauser Education Foundation
asked NWREL to make its "model" available in Mississippi. When
Weyerhauser began lumbering operations in Mississippi, they recognized
the need for better educated workers than was typical in Mississippi.
When they learned of NWREL's success with its Onward to Excellence
program, they made their request and then supported it with funding.

> **1. Strengths**

In Onward to Excellence, the student is always at the center. This keeps
the focus on the student. Schools that use the approach appear to be
successful if they implement it seriously. Of the 33 Mississippi schools
(with an average of about 50 to 70 percent of the children eligible for
free or reduced price lunches) using the approach, about seven were said
to implement the model rather completely. In those seven schools,
improvement is beginning to appear. One of the problems is a high
turnover rate in leadership even in these schools. Mississippi state
officials noted that the poorest schools were the ones making the
greatest gains.

> **2. Areas of Needed Improvement**

Onward to Excellence appears to be moving forward well. One suggestion
was that some other Regional Laboratories might have cooperated more
fully NWREL in their regions in training and building the capacity to
training on their own.

> **3. Recommendations for improvement**
>
> None.

C.  **To what extent has the REL made progress in establishing a
    regional and national reputation in its specialty area?**

In the school renewal area, the NWREL has established a good reputation
both in the region and nationally. This is demonstrated by the number of
schools in both the five state area and in other states outside of the
region that have adopted the On to Excellence approach.

> **1. Strengths**
>
> Model is believed to work and has been widely adopted.
>
> **2. Areas of needed improvement**
>
> More evaluation work.
>
> **3. Recommendations for improvement**

No specific recommendations.

**VI. Overall Evaluation of Total Laboratory Programs, Products and
Services**

Overall, the NWREL has excellent programs, products and services. In
fact, separating programs, products and services from each other is
essentially impossible. A program usually is made up of a combination of
products and services, and though not as clearly inseparable from each
other as programs are from the other two, products and services also
often are closely related. An example of a very strong combination of
program, product and service would be the six trait writing and reading
programs. Though developmental work led to the development of first the
six trait writing and then the six trait reading programs with their
related materials -- but without the service component, the materials
would not have been programs.

**VIII. Broad Summary of Strengths, Areas for Improvement, and
Strategies for Improvement**

The NWREL is a strong Regional Educational Laboratory. The Laboratories
came into being in a turbulent time, and, along with the other
structures conceived under the Elementary and Secondary Education Act,
they represented a major experiment. Many of the original Laboratories
disappeared within a few years. NWREL has operated effectively and
continuously since 1966. By continuing formal and informal needs
assessing and responding with strong programs to the needs of schools in
the region, the Laboratory has experienced continuing success.

My primary concerns are with the recognition that teachers and
administrators now being prepared in our colleges and universities do
not have either enough information about Laboratories and Laboratory
programs or close enough ties to Laboratories to make good use of this
resource. I would urge that both the individual Laboratories and the
OERI make greater provision for inclusion of colleges and universities
in Laboratory planning, program development, and dissemination.