Archived Information

**Interim Evaluation of the Northwest Regional Educational Laboratory**

# 

## I. Brief Overview of Laboratory

During the week of April 5-9, 1999, I served on a panel comprised of six
individuals charged with the responsibility to conduct a comprehensive
on-site evaluation of the Northwest Regional Educational Laboratory
(NWREL) at its headquarters in Portland, Oregon. NWREL, one of ten
regional educational laboratories in the country, provides educational
services to a designated region that includes Alaska, Idaho, Montana,
Oregon, and Washington.

Prior to the site visit, I reviewed all print material submitted by the
Laboratory via the evaluation contractor, Decision Information
Resources, Inc. During the visit to the Lab, panel members listened to
presentations by Lab staff members and engaged in discussions with board
members, Laboratory clients, and focus groups (trainers and users). We
also interviewed the Laboratory program officer who represents the
Laboratory's funding agent, the United States Department of Education,
Office of Educational Research and Improvement (OERI). Panel members
were also given a guided tour of the Laboratory facilities.

## Implementation and Management

A.  To what extent is the REL doing what they were approved to do during
    their first three contract years?

**1. Strengths**

NWREL, hereafter referred to as the Lab, has been involved in providing
educational services in its designated region and beyond since 1966.
During this time, the Lab has developed a credible and exemplary
reputation as a provider of education services. To date, the Lab appears
to be meeting all expectations with regard to its contract obligations.
Specifically: (1) the contractual activities of the Lab are based on an
extensive needs assessment of the region and the work scope under each
task is based on such needs; (2) staff members appear to be qualified to
carry out proposed tasks; (3) the organizational and management
structure appear to be appropriate for carrying out Lab activities; (4)
the Lab appears to be implementing the scope of work according to its
approved contract and in instances where changes were necessary,
modifications with appropriate justifications were requested in a timely
manner; and (5) the fiscal resources of NWREL, which are derived from a
variety of sources (other federal, state agencies, local schools, higher
education) in addition to OERI, are used to leverage a comprehensive,
systemic education improvement program that has regional as well as
national impact.

Following an assessment of the Lab governing board, it was restructured
and now has 26 members. Under its original plan, the board had 45
members including the Chief State School Officers (CSSO) of each state
in the Lab region. Discussions with four of the board members, including
the chair, indicate the new configuration is much more effective for
carrying out board responsibilities. The board established eight
committees to oversee and address issues regarding the overall operation
of the Lab, including OERI funded Lab activities. The committees appear
to be appropriate for the purposes of board oversight. A review of the
minutes of board meetings indicates that appropriate action is taken on
committee matters as necessary.

The Lab network appears to be extensive and appropriate for the purposes
of its activities. The network includes state education agencies, local
education agencies, universities, other educational laboratories as well
as educational organizations. Discussions with board members and clients
indicate that the Lab is providing effective services as requested both
within and beyond its region.

**2. Areas of needed improvement**

A major factor in Lab program operations is its governing board. This
board appears to be functioning in an effective and efficient manner.
However, it is interesting to note that the board does not appear to
have parent representation. Even student representatives are not
precluded from serving on the board. While the Lab is not required to
include either of these role groups on its board, it would appear that
such inclusion would be in the best interest of the Lab since its
primary constituents are students who ultimately receive and hopefully
benefit from Lab services. In response to this concern, Lab staff stated
that parents serve on advisory committees. However, this does not give
this role group a voice in the governance of the Lab.

With regard to board representation by CSSOs, only four of the five
appear to be members of the board. Apparently the CSSO from the state of
Oregon is not on board at this time. The panel members learned that in
all instances, the CSSOs have designated personal representatives to
represent them on the board and, thus, do not appear to be active
participants in board activities. In lieu of CSSO participation at the
board meetings, they are convened via telephone on a quarterly basis to
discuss issues regarding Lab activities in the region. While the
designated representatives are no doubt capable of representing the
position, it seems that the [visibility of the CSSO]{.underline} is
vital to the Lab's efforts to effectively implement school change on a
statewide basis.

During a discussion on the use and impact of the Lab's specialty area
(school change model), a question was asked regarding the Lab's
collaboration with other Labs in regions where the school change model
is being implemented. Curiously, it does not appear that much
collaboration takes place between Labs when one Lab provides services in
another Lab's region. It would seem that when implementing and providing
services\--particularly in the specialty areas\--which all Labs have,
there would be more collaboration among the Labs.

The Lab (in response to a technical question) stated its intent to use
some of its resources to provide professional development for staff in
its efforts to effectively develop and implement activities under its
scope of work. Such activities were to be provided on-site as well as
through seminars, conferences, and college course work outside the
Laboratory. The extent to which such staff development activities have
occurred or are occurring was not readily apparent in reviewing the
available material. In response to this issue, staff reported that
\$70,000 were spent for staff development activities. An Employee
Assistance Program is now available to staff as a result of a
cooperative arrangement between the Lab and a service agent in the area.
If not already provided, the Lab may wish to consider providing
information on staff development in the quarterly reports that are sent
to OERI.

In seeking to understand the Lab's overall governance and management
plans, I reviewed a number of items addressing this area. The Lab's
report on institutional performance is an example of items reviewed.
While there is a heavy emphasis on the impact of Lab activities, ("Who
does NWREL represent?" or "How well know is NWREL...?" or "How does
NWREL mobilize...?") there is very little information regarding how well
staff members function in their prescribed roles. There is even a
question as to how well the board represents its constituencies. It
would seem that a key question to be asked by the Lab is\-\--How well is
Lab staff meeting the goals and objectives of the program? While it is
likely that staff performance is an important and integral factor in Lab
operations, it appears to be a glaring oversight when reviewing material
that describes the institution as an entity. Recognizing the need for
confidentially, staff may wish to consider how to include information
regarding overall staff productivity in its institutional evaluation
reports. This probably gets at the heart of question 10 in the Lab's
"Response to Technical Questions." (What areas of staff development are
anticipated for the proposed contract period and how are these areas
related to proposed staff responsibilities and management objectives?)

In reviewing the charts that show the schedule of activities in the
technical proposal, as well as other organization and management
material, there does not appear to be an indication of when regular
examinations will occur to determine whether program activities are
being accomplished as forecast. Given the massive amount or work that is
generated at the Lab, a chart showing the intervals for examination of
accomplishments (discrepancy analysis) would provide a clearer picture
of the institutional management of the Lab. Although Lab staff made it
clear that each director oversees the planning, development, and
implementation of activities in his or her program or center, it is not
clear how such information is provided to the executive committee,
governing board, and OERI.

**3. Recommendations for improvement**

The Lab should consider the impact of the presence of representatives
from the primary recipients or beneficiaries of schooling activities on
its governing board. It seems feasible to consider not only parent role
group participation on the board, but also student representation---even
if on a limited but focused basis.

Given the large annual budget provided by OERI as well as funds from
other sources, it seems appropriate that CSSOs become more visible on
the Lab board. To the extent possible, direct involvement by CSSOs in
Lab operations should be encouraged.

In reporting on institutional evaluation, consider including a section
that addresses the productivity of Lab staff in accomplishing Lab goals
and objectives, i.e.: How well is Lab staff performing in accomplishing
the goals and objectives of the organization?

In preparing schedules of activities, Lab should provide information
that shows key intervals at which major Lab tasks and or accomplishments
will be examined to determine whether they are taking place as planned.

Collaboration among Labs, particularly in the Task 7 specialty areas,
should be addressed. It is very likely that individual Labs can benefit
from support provided by the Lab in whose region services are being
provided.

Comment: It was noted that reports of site visits from OERI staff were
not available. Such reports are vital in the overall assessment of Lab
performance. This recommendation is not directed to the Lab but rather
to OERI, the funding agent. Lab site visits should be followed up with a
report that indicates whether or not the Lab is accomplishing its goals
and objectives under the funded scope of work.

B. To what extent is the REL using a self-monitoring process to plan and
adapt activities in response to feedback and customer needs?

**1. Strengths**

With regard to using a self-monitoring process, the Lab appears to be
meeting expectations under this evaluation item. There is a policy in
place that directs the Lab to develop and implement procedures to assure
the quality of Lab work as well as the expectation that the board will
review results of such work. The procedure regarding this area requires
that the Lab establish processes to assure the quality of its products
and services. In keeping with these policies and procedures, the Lab has
instituted a variety of self-monitoring strategies.

One of the primary vehicles for self-monitoring of Lab services appears
to be the "Professional Activity Reporting System" which is designed to
track the "the number and nature" of services provided to myriad
educational constituents. Additionally, the Lab relies upon such data as
demographic studies, needs clarification, information requests,
research-based input, public input via telephone surveys and educational
gatherings in the region. The Lab appears to carry out an extensive
technology based communication system that is both internal and
external. It enables staff to interact with constituents in its vast
region via a local area network and a web site. Given the large number
of "hits" on the web site, Lab staff are able to assess customer needs
by the requests for services and products.

In efforts to determine how well it is addressing its self-monitoring
processes, the Lab commissioned Dr. Alfred Rasp to conduct an
independent study of its needs assessment and institutional evaluation
processes. The study, conducted in 1997, contained a number of
well-founded recommendations that, if addressed and implemented, would
strengthen Lab operations in the area of self-monitoring.

Additional measures to assure quality self-monitoring processes were
described in quarterly reports. They included the convening of a
three-person panel to review the Lab's overall institutional evaluation
and needs identification processes. This panel stressed the need for the
Lab to collect objective data with "candor induction" the goal. The Lab
also conducted a comprehensive needs assessment in 1997. The report,
"Northwest Trends Shaping Education," describes 12 trends that are
shaping education in the region served by the Lab as well as
implications for Laboratory work. In 1997, a telephone survey was
conducted by the Gilmore Research Group to determine program
satisfaction among Lab clients. These evaluation studies and a number of
other activities clearly indicate the Lab's concern for assuring that
effective self-monitoring processes are in place.

**2. Areas of needed improvement**

Although it is evident the Lab is using extensive self-monitoring
processes and is seeking to determine how well it is doing in this area,
it is not clear that the findings noted in commissioned studies are
being used to improve self-monitoring processes. For instance, in the
quarterly report dated April-June 1997, a statement is made that
"preliminary findings (in the study by Dr. Rasp) were presented to the
board, and recommendations are being considered for strengthening these
processes." The Executive Cabinet was to review the final written report
on July 3 of next quarter and recommend a course of action. In reviewing
subsequent quarterly reports, I found no information about action taken,
if any, with regard to the findings and recommendations of the Rasp
report. This appears to be the case with other studies conducted to
examine self-monitoring processes. While it is very likely that action
was taken on the recommendations, there does not appear to be any
records or reports addressing the specific action taken for each
recommendation or finding.

In the summary of findings of the February 1998 "NWREL Board Post
Self-Assessment," which addresses the board's role in approving and
monitoring the organization's programs and services, (which of course
would include Lab self-monitoring processes), a board member responded:
"We certainly hear strengths---sometimes hearing about program
weaknesses would be revealing." Another respondent stated: "When program
directors (make) presentations at the Board meetings, they are obviously
going to dwell on their successes..." Such statements suggest that the
board is not fully aware of the strengths and weaknesses of the Lab's
major programs. Management staff at the Lab are aware of this concern as
indicated by this statement: "This area of (board) responsibility has
shown only slight improvement during the past year. Respondents continue
to be concerned with knowing the strengths and weaknesses of each major
program." In discussions with Lab staff, it is clear there is concern
about this issue. While the Lab has taken steps to address this issue,
there does not appear to be any information (in quarterly reports or
separate reports) describing the results of efforts to resolve this
issue.

**3. Recommendations for improvement**

It is recommended that Lab staff consider strategies for formally
reporting any actions that respond to the findings in independent
studies and evaluation reports commissioned by the Lab. If recommended
actions are not taken, this should be reported as well, with an
explanation for such decisions.

As the Lab determines and implements concrete and effective ways to keep
board members apprised of Lab operations, including strengths and
weaknesses, lessons learned should be reported and shared since it is
likely other similar education providers are struggling with the same
issue.

## III. Quality

**To what extent is the REL developing high quality products and
services?**

**1. Strengths**

The planning, development, and implementation of high quality products
and services is, without a doubt, one of the hallmarks in this Lab's
long history of providing educational services in its region and
throughout the country. Two signature works provided by the Lab for
review are quite overwhelming, so much so that it was virtually
impossible to thoroughly review the hundreds of pages contained in each
of the binders submitted for review. Strengths noted in the review of
the signature works as well as other products/services and during
discussions with Lab staff, and clients include the following: (1) The
signature works and other products are being used extensively throughout
the Lab region and beyond. (2) Comprehensive quality control procedures
are utilized to ensure that products are of the highest quality. Before
products are disseminated, a variety of steps are employed in testing
for quality including internal and external reviews, pilot tests,
follow-up surveys, and consultations with target audiences and
unsolicited comments from users. The Lab's standards for developing
quality products and services include methodology that is basic to
exemplary research and development. Development plans also consider
techniques for scaling up activities to reach a larger client pool.
Examples of this effort were noted when staff members responsible for
the two signature works I reviewed presented information that shows the
extent to which the products are used in the region and beyond. (3) Lab
staff appear to be heavily involved in research and development and have
published numerous publications or have been cited in other publications
such as the Association for Supervision and Curriculum Development.
Staff members have also presented their work in conferences, workshop,
etc. and have received national recognition for their work.

There is no question, in my view, that Lab staff members are thoroughly
steeped in the knowledge base undergirding their products and services.
What was even more evident was the pride and joy they displayed in
discussing the work they have accomplished. This "affect" seems to
permeate the staff from the executive director to program/center
directors to staff.

The Lab's research and development products are not limited to showcase
print material. Products and services are also designed to be delivered
via distance learning technology including the Internet. It appears that
such technology is extensively used, probably to a large degree by
necessity because of the demographics and vastness of the region served
by the Lab.

The five areas in which the Lab is targeting its research and
development activities are assessment and evaluation, early childhood
education, rural education, school change, and community based learning.
The Lab appears to be making progress in these areas of research and is
relying on feedback derived from its partner sites in assuring that
efforts result in exemplary products and services.

It was noted that the Oregon partner site for research and development
in rural education elected to end its partnership relationship and not
continue with year four activities. An explanation of or reason for
termination of the partnership was not given. It is likely that this is
not an unusual occurrence. Yet, there was no information provided that
would indicate the Lab has contingency plans for such occurrences.

A survey of client satisfaction with Lab products showed that the impact
of the products ranged from a 33 percent "change in the implementation
of practices" and 33 percent in the "quality of professional practices
were improved" to a 67 percent impact in "decision making and/or
planning was facilitated." This was a small sample, but it is evidence
that the Lab recognizes the importance of gathering quality assurance
data (whether in small or large samples) in its research and development
activities.

Generally, the Lab\'s attention to developing high quality products and
services exceed expectations. It has generated a reputation for being
prolific in developing high profile products and services that are in
demand at both the regional and national levels.

**2. Areas of Needed Improvement/Recommendation**

In instances where partner sites terminate their relationship with the
Lab, the events causing or leading to termination should be documented
and analyzed. Lessons learned may provide information that could prevent
similar occurrences with other sites in the future. If such steps are
already used, an explanation should be provided when reporting such
occurrences.

## IV. Utility

A. To what extent are the products and services provided by the
Laboratory useful to, and used by customers?

The Lab's marketing capabilities appear to be very good. A number of
products and services have been marketed throughout the region and
beyond. The products appear to be based on a variety of needs
assessments conducted in the region. Information was provided for this
review that shows how long a product has been on the market as well as
totals sales and revenues generated by the products. Products developed
for clients include research reports and guides, instructional
materials, policy papers, handbooks, and synthesis papers. We had an
opportunity to tour the resource room which houses the collection of
research reports on assessment and were apprised of the procedures for
collecting assessment materials as well as procedures for lending
material to clients.

In discussions with the trainers and users of products and services
under the signature works, "Onward To Excellence," and "Trait-based
Assessment," information was presented that indicates Lab clients
believe these products are extremely useful and are used by customers.
The presenters represented the region served by the Lab (Washington,
Oregon, Idaho, and Alaska) and states outside the Lab's designated
region (Nevada, Iowa, Mississippi, Kansas, Florida, and Arizona).

**1. Strengths**

Strengths noted in this area include the following: (1) Staff members
involved in the research and development of signature and other works
appear to be highly qualified and capable of developing products that
are useful to clients. (2) Center and program staff appear to keep
abreast of trends in the region and beyond in efforts to develop and
provide products and services based on these trends. (3) Products and
services, in addition to having a focus on needs in the region, also
have a national focus, i.e., charter schools, and community/rural
education and school reform.

**2. Areas of needed improvement:**

While products and services are widely used throughout the region by a
variety of client groups, it appears that institutions of higher
education (IHEs) are minimally involved in using the products and
services offered by the Lab. As one of the key (perhaps even critical)
ingredients necessary to implement systemic school change in the region,
it is puzzling that the teacher education institutions have what appears
to be minimal involvement in Lab efforts toward school change.

A number of the products in the Lab's inventory have been on the market
for many months. It is generally assumed that when products no longer
generate revenues or requests, they are removed from the inventory. It
is not clear, however, to what extent this occurs or what other
assessment measures are used to determine why and when products and
services are no longer useful to the client group. Staff did state that
often products are removed from their inventory to another source such
as the Association for Supervision and Curriculum Development for
dissemination. It would seem that information regarding the "life" of a
product would be useful in looking at evolving trends in education in
the region and beyond.

**3. Recommendations for improvement**

There are no specific recommendations under this section. The Lab may
wish to consider whether more active involvement of area IHEs would
enhance Lab activities in the area of school change with regard to
teacher development.

## B. To what extent is the REL focused on customer needs?

**1. Strengths**

The Lab's focus on customer needs appears to be pervasive in the region.
In addition to Lab use of a variety of needs assessment strategies to
determine appropriate courses of action, a key Lab staff member is
assigned as a liaison to each of the states in the region. Information
provided by the state liaisons about activities in their respective
states indicates that they are very much aware of the myriad educational
needs in the states. Staff members are encouraged to spend as much time
in the field as possible. Additionally, staff members are required to
spend time in classrooms in efforts to remain "grounded" in their
primary purpose for providing educational services in the region.

Apparently a new committee on external relations has been created by the
board. This committee focuses on goals designed to impact clients
through such activities as forums and conferences, and by creating
awareness and demand for services. (This committee appears to be a new
addition to other board committees because there were no references to
it in the technical proposal, annual updates or quarterly reports.) One
of the goals of this committee is to focus on three external audiences
with priority given to clients. In support of external communication
goals, the Lab makes extensive use of Gopher and Web Internet services,
a toll free 800 number, and regular meetings with clients. These and
other strategies are used extensively to build relationships with
new/expanded client groups.

An example of Lab focus on clients includes the translation of products
and services into Spanish on behalf of the fairly large Spanish speaking
population in the region. A focus on customer needs is also evident in
the Lab's research and development efforts on topics that reflect the
characteristics of the area such as rural and urban education.

The Lab states that 10,727 individuals from local schools participated
in workshops, consultations and conferences in FY98. Participation from
local school districts was very high as well. This would suggest that
the Lab is giving careful attention to the needs of its customers.
Additionally, discussions with Lab trainers and users seem to indicate
that the Lab does focus on its customer needs. The participants in the
discussions ranged from those who are new clients to those who have been
clients for a number of years. In all cases, the discussants stressed
the need for more or continued services from the Lab whether through
direct involvement or by way of advice or follow-up support.

In providing educational services to its customers, the Lab states that
it has established 54 partner sites in which to collaborate and provide
support. According to the board members with whom we had discussions,
Lab participation in state efforts to effect school change has
contributed to school success in a variety of ways. The research-based
knowledge in the various content areas including reading, mathematics,
etc., has enabled school districts and/or individual schools to make
informed decisions about school change. The focus on classroom
assessment via the "Trait-based Assessment" and school change via the
"Onward To Excellence" programs are being used extensively in statewide
improvement efforts.

Board members stated that the Lab has been an extraordinary resource for
rural districts in the region and the resources are now more available
than ever via such technology as the Internet. Current activities
underway in the Rural Education Program include research and development
work on the charter school movement throughout the country. State
educators in this region are beginning to consider the implications of
charter schools in their respective states. Presently, Alaska is the
only state that has a charter school. In response to this move toward
charter school, the Lab is conducting meetings, developing products,
etc., in efforts to meet customer needs. In response to a question
regarding plans for "scaling up" activities with regard to the charter
school movement in the region, staff indicated that plans are already
underway for implementing training programs on the development and
implementation of charter schools.

Activities under the Mathematics and Science Education Center appear to
be focused on customer needs. Several products have been developed for
dissemination in the region and beyond. The "It's Just Good Teaching
Series" contains six publications and two videos for use by math
teachers (K-12). The materials, while based on the latest research, are
teacher friendly, i.e., easy to read and use. It was reported that over
10,000 copies of this series have been disseminated since June 1997. In
response to a question regarding the use of this series in the
preparation of teachers at IHEs, staff indicated that dissemination and
use at the IHE level is minimal. As stated earlier in this report, this
is unfortunate. It appears to me that there are a number of missed
opportunities for collaboration between the Lab and regional IHEs in the
area of teacher preparation.

One of the board members provided a most interesting comment regarding
the importance of the Lab to the region. He stated: "If we didn't have
the Lab, we would have to create one in the state of Washington." This
and other statements seem to indicate that the Lab is focused on, and
responsive to customer needs, and is striving to provide the best
services possible to myriad groups of people in an extremely diverse
region. The fact is that it is not likely that Lab services and products
would be sought if they did not address the needs and requests of its
clients. This Lab appears to receive numerous requests for its services
and products.

**2. Areas of Needed Improvement/Recommendation:**

None noted

## 

## V. Outcomes and Impact

**A. To what extent is the REL's work contributing to improved student
success,**

**particularly in intensive implementation sites?**

**1. Strengths**

The information reviewed seems to indicate that the Lab is contributing
somewhat to improved student success. The impact of general services to
clients ranged from a low of 30 percent for "student performance was
positively affected" to a high of 82 percent for "new skills and/or
knowledge acquired." Consideration must be given to the fact that there
are many variables over which the Lab has no control, particularly since
it does not provide direct services to students even in intensive
implementation sites. A statement on page 28 of the Annual Report of
Institutional Performance more aptly describes this stance: "NWREL may
provide interventions at any or all steps in this continuum (path to
improved student outcomes), depending on a client's needs, resources,
and interests. In most cases, NWREL services are requested to facilitate
just one step in the improvement process, so attempts to measure NWREL's
impact on the ultimate goal (improved student outcomes) is not realistic
or feasible."

Nevertheless, having said this, the Lab reports that it does monitor the
impact of its services with regard to client/student success. A primary
vehicle for monitoring its activities is the "Professional Activity
Reporting System" (PARS) which documents and tracks services to clients
in seven types of activities. Additional strategies for documenting Lab
contributions to student success include client surveys such as the one
conducted by the Gilmore Research Group, which was mentioned earlier in
this report.

The updated annual report for year four describes the Lab\'s efforts to
monitor the impact of its products and services with regard to student
success. This is apparently a slow and somewhat arduous process. Note
that in the Rural School Partnership, the Lab had to deal with the fact
that one of the sites dropped out of the partnership and in another
there was apparently poor communication at the site level and perhaps
with Lab staff. Additionally, at least two site liaisons resigned at
this site. It would appear that progress in rural, remote areas require
more time, ongoing communication, and constant monitoring to assure even
minor successes with regard to Lab interventions.

While the impact of services provided is generally positive as noted on
a graph depicting the Lab's visibility in the region, it seems that
teachers are the least aware of the Lab. Over 40 percent of teachers
have not heard of NWREL and only a negligible percentage has used the
web site. Approximately 18 percent have used Lab materials and 42
percent have participated in workshops/conferences. This is somewhat
disappointing given the three decades the Lab has provided educational
services in the region. It would seem that at this point, the Lab would
be a household word among educators in the region. Superintendents and
principals are much more aware of the Lab. Of course, this is desired
and necessary.

In responding to the observation about teacher awareness of the Lab,
staff stated that teachers are in fact aware of and use Lab services.
The problem is that the Lab name and logo are not always noticed. For
example, when asked about the school change model, "Onward to
Excellence," teachers [were]{.underline} aware of the model. However,
they did not attribute the model to the Lab. This is perhaps the "nature
of the beast." Lab staff stated they were trying to deal with the
discrepancies in teacher awareness of the Lab.

During its long history of providing educational services in the
northwest, the Lab has worked extensively in Alaska. Current efforts
include "Alaska Onward to Excellence," a field initiated school
improvement process. According to recent congressional findings, "Alaska
native children enter and exit school with serious educational
handicaps." These children have low performance on standardized tests.
The dropout rates are extremely and high Native Alaskans are
significantly underrepresented among holders of BA degrees. The
geographic barriers to delivering educational services in Alaskan
villages as well as other remote areas within the continental US are
challenging at best. In efforts to address this seemingly perpetual
problem, USED is again funding applications for direct grants under the
"Alaska Native Program." Activities funded under this program include
education planning, curriculum development and teacher training; home
based education for preschool children; and student enrichment.

Continuous funding of such programs suggest that efforts to improve
student success are hampered by intransigent problems that are beyond
what can be resolved by any one educational program, whether federal or
otherwise, large or small. After many years of providing educational
services in areas like Alaska, Lab staff members have developed
knowledge and skills that can be used to inform and assist others (such
as those who will be awarded grants under the Alaska Native Program) who
are attempting to improve education outcomes by, more than likely, doing
some of the same things that have already been done by the Lab. This
opportunity, if taken, would be an excellent example of networking or
collaboration between federally funded education programs as well as
with state and local education programs.

Regarding Lab efforts to increase knowledge and understanding of
educational problems, issues, and effective strategies, its inventory is
replete with research and development products and services. Many such
items have been mentioned in other parts of this report. Other examples
examined include the Catalogue of School Reform, which was developed in
collaboration with the Education Commission of the States for the
Department of Education; "Choice Matters," which provides policy
alternatives and implication for charter schools; and "Charter Schools
at the Crossroads," which presents the proceedings of a symposium on
charter school policy. These publications, as well as others are
designed to inform; not only the Lab's designated region, but also the
nation as a whole, about "hot" issues on the education scene.

Laboratory work appears to be consistent with national priorities. The
Lab, for many years has been on the forefront in contributing to the
knowledge and understanding of educational problems, issues, and
effective practices. Areas addressed by the Lab that have national
significance include such examples as the model on implementing school
change (Onward to Excellence) which is used throughout the country.
Staff members involved in this project have developed national
reputations and readily inform others of the continuing developments in
school reform.

**2. Areas of needed improvement**

The Lab has candidly provided information that shows staff awareness of
the areas of consideration or need of improvement with regard to its
overall contribution to improved student success. These recommendations,
which are a result of its own internal assessment, could be placed in
any section of this report. However, they are restated in this section
because of the overall purpose of Labs, which is to provide exemplary
educational services that result in improved student outcomes. The Lab
has stated its intent to: (1) Create a means for assessing the
understanding of the Lab as an institution. (2) Broaden surveys designed
to determine knowledge acquisition and use of services to include
community and other groups. (3) Increase evaluations of individual
programs and projects. (4) Compile and make public more rich stories
from participants in Lab work.

B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?

**1. Strengths**

The Lab appears to be meeting expectations with regard to providing
assistance in implementing comprehensive school improvement strategies.
A major assistance effort is the Lab's involvement with the
Comprehensive School Reform Demonstration (CSRD) program, which
"provides funds to help schools adopt effective, research-based
comprehensive school reform models that will help all students reach
high standards." Funds are provided to SEAs who then award competitive
grants to local school districts. The Lab appears to be providing
valuable assistance to SEAs and LEAs that will enable these entities to
"get started." Assistance includes providing information on school
change models, and working with states and school districts as they
engage in the process of selecting and implementing models. Assistance
from the Lab has enabled the five states in its region to accomplish
many of the tasks necessary to acquire funding under CSRD.

Assistance in school change is provided not only locally but nationally
as well. As noted earlier, services and products are based on sound
research and development activities. With regard to "scaling-up," the
Lab appears to have given careful consideration to how to make school
change processes accessible to its region and beyond. The two signature
works provided by the Lab for this review are examples of efforts to
assist in the implementation of school change. The works have been used
in several sites in the country.

**2. Areas of needed improvement**

A "technical question" posed to the Lab about reform assistance asked:
".... How does the offeror (Lab) intend to reach out to other schools
not yet interested in reform, particularly as the offeror appears to
select only schools already receptive to change as partners in its
development and applied research work?" The response to this question
included steps that would be taken to reach out to schools not
interested in reform including presentations, publicity about successful
school reform efforts, and outreach with other service providers such as
the Comprehensive Assistance Centers. It was further stated that the Lab
is "well known by over 90 percent of all local school leaders and over
50 percent of the region's 100,000 teachers put us in a strong position
to reach out effectively to all."

As far as the question above, there is evidence in the quarterly reports
and updated annual plans that the state liaisons are providing what
appear to be exemplary services in their respective states. What is not
clear is the extent to which the Lab has been able to reach out to
schools not interested in change initiatives. Have the proposed attempts
to reach out to such schools been successful and if so, is the Lab
providing the assistance needed? If attempts have been made and were not
successful, is there any documented evidence regarding the resistance to
change that might inform others who are trying to reach the
hard-to-reach? Such "lessons learned" would be valuable to other
institutions trying to effect institutional school change.

**3. Recommendations for improvement**

The Lab may wish to revisit technical question 4 and its response to the
question. This appears to be a critical issue that must be addressed by
those involved in educational change efforts. NWREL, with its experience
in working in areas where there is little or no interest in education
change (for a variety of reasons including lack of awareness,
isolation), has the resources to delve into this issue and so inform
others wrestling with the same problem.

C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?

**1. Strengths**

NCREL has clearly established a regional and national reputation in its
specialty area, which is school change. Efforts to facilitate school
change have been ongoing for the past 15 years. A major outcome of this
effort is the comprehensive "Onward to Excellence II: A Systemic Reform"
(OTE). This program is designed to facilitate systemic school change and
is being used in schools throughout the country. In efforts to enhance
this long term project, as well as minimize duplication, the Lab is
involved in a partnership that includes the Annenberg Institute for
School Reform at Brown University and the Consortium for Policy Research
in Education. Worthy of note is a national conference conducted by this
partnership in October 1996. This was an invitational conference that
focused on scaling-up school change. Participants at the conference
included national reform networks, other regional Labs, and state and
local education entities.

The Lab seeks to support systemic school reform via a variety of
activities including conferences, and on-site technical assistance. A
description of products and services was provided for review. The
tracking system for Lab services and products indicate that thousands of
school improvement products have been disseminated. At the same time
completed products are being disseminated, the Lab is developing and
producing new products on school change.

It is interesting to note that all of the Lab's work can be linked to
school change. For instance, the presentations on the activities in the
Mathematics and Science Education Center and the Rural Education Program
included several descriptions of services and products that contribute
to school change. The Lab appears to be addressing school change not
only at the institutional level through such programs as "Onward to
Excellence: A Systemic Reform Model" and "Trait-based Assessment," but
at the classroom level through such products as "It's Just Good
Teaching: Inquiry Strategies for Science and Mathematics Learning."
There are a number of other services and products that show the breadth
of the Lab's work in its specialty area.

A description of the Lab's strategic alliances seems to indicate that
comprehensive efforts in school change are extensive. This effort has
reached to the national level via other Labs and other education
agencies. Regional efforts are pervasive in states such as Montana and
Alaska, which are participating in the Lab's "Onward to Excellence
Program." Additionally, Lab staff have written a number of articles
which have been published or are "in press."

Customers using products and services in the specialty area include
educators from within the region and nationally. Discussions with the
panel of customers using "Onward to Excellence" indicate they are very
satisfied with this change program. They readily described the benefits
and impact attributable to the program. When asked to discuss what they
would do differently and what they saw as next steps in efforts to
effect school change, the consensus seemed to be to continue their
efforts at their respective sites with as much ongoing support from the
Lab as possible.

**2. Areas of needed improvement**

### None noted

## VI. Broad Summary

Overall, the Northwest Regional Educational Laboratory is meeting and/or
exceeding expectations in accomplishing the goals and objectives under
the OERI scope of work. The Lab has developed an exemplary track record
over a number of years. Lab operations are managed by a very capable and
committed director who is sensitive to the bottom line in all Lab
activities and that is: "Will the work we do benefit children in the
classroom." The highly qualified staff mirrors this sentiment by seeking
to produce and deliver research based products and services that have
been tediously developed before they are used in the field. It is
clearly evident that the Lab is providing services that are needed and
appreciated in its region and beyond.

I was impressed by the willingness of the director and staff to provide
answers to rather probing questions with candor and openness. At all
times during this visit, we were encouraged to ask for more information
if we believed it would aid us in doing our job. When it was necessary
to request more information, it was provided immediately or placed at
our stations by the time we arrived at the Lab the next day.

Where concerns are noted in this review it is likely information was
available, but perhaps overlooked or misunderstood. In instances where
the director and staff agree that areas of concern and/or
recommendations are "on target," it is my guess the issues in question
will be addressed in a professional and timely manner.