Archived Information

Interim Evaluation of the Mid-continent Regional Educational Laboratory

## Synthesis Report

**Introduction**

The Mid-continent Regional Educational Laboratory (McREL), one of ten
Laboratories funded by the U.S. Department of Education through a
competitive process, is currently in a five-year funding period
(December 1995--December 2000). During the third year of the current
contract, all Regional Educational Laboratories (RELs) are required to
undergo a peer-review interim evaluation conducted under the auspices of
*Standards for Conduct and Evaluation of Activities Carried Out by the
Office of Educational Research and Improvement (OERI) for Evaluation of
Recipients of Grants, Cooperative Agreements, and Contracts*, hereafter
referred to as the *Evaluation Standards*.

The panel selected for the interim evaluation of McREL consisted of
Robert Donmoyer (Director, School of Educational Policy and Leadership,
Ohio State University), Morris Lai (Director of Evaluation, Curriculum
Research & Development Group, University of Hawai'i and Panel Chair),
Diane Lassman (Director, The EXCHANGE, Minneapolis, MN), Jessie Pollack
(Educational Coordinator, Measurement and Research, Maryland State
Department of Education), and H. Wesley Smith (Superintendent, Summit
School District, Frisco, CO).

As required by the *Evaluation Standards*, panelists reviewed documents
on Laboratory operations and a sampling of materials selected
collaboratively by Decision Information Resources, Inc. (DIR), McREL,
and the OERI Program Officer assigned to the McREL contract. In
addition, the Panel conducted on-site data-collection activities at
McREL's Aurora, CO office May 3--7, 1999 (agenda attached) including (a)
attending presentations on the Laboratory's operations and management,
the two signature programs selected for review (*Moving Standards into
Practice* and *Partnerships as a Field Service Strategy*), and other
programs or components of programs funded primarily or exclusively by
the OERI contract; (b) a group telephone interview with four McREL Board
of Directors members representing three of the seven states McREL serves
under its REL contract with OERI; (c) group interviews with two groups
(from Local Educational Agencies (LEAs) and State Educational Agencies
(SEAs) with contracts with McREL in the area of standards-based
education); (d) two group interviews with individuals who work in
organizations serviced by the Laboratory's field service team, and (e)
an interview with McREL's U.S. Department of Education Program Officer.

This synthesis interim-evaluation report has been organized basically
along the lines of the format recommended for individual panelists'
reports. In a slight format modification, two sections for each of the
eight evaluation questions have been combined -- "Recommendations for
improvement" has been incorporated into "Areas of needed improvement."

Bulleted statements in bold had been used in the exit interview with
McREL held on May 7, 1999, the last day of the site visit. They
represent the Panel's collective compilation of highlights of responses
to the evaluation questions.

**I. Brief** **Overview of the Mid-continent Regional Educational
Laboratory**

The Mid-continent Regional Educational Laboratory (McREL), one of the
original Regional Educational Laboratories, has served the central
region now consisting of Colorado, Missouri, Kansas, Nebraska, South
Dakota, North Dakota, and Wyoming since 1966. The region has 20% of the
nation's land mass and 6% of its population. Major cities in the region
include Denver, CO; Kansas City, MO; Kansas City, KS; Omaha, NE; and,
St. Louis, MO. Small towns, small cities, and rural areas, however,
characterize most of the region.

Diverse and underserved populations exist throughout the region. In
rural areas there is poverty that is exacerbated by the scarcity of
services and communications. Underserved populations are often too small
to command attention, and their educational needs are hidden within the
relatively successful schools they attend.

Travel can be difficult because of long distances and weather
conditions. Where air travel

is possible, travelers are subjected to multiple connecting flights and
often left with over two hundred miles of ground travel to complete a
trip.

All of the states of the region are involved with school reform. Some of
the states are leading in the reform movement; others are slowly
overcoming inertia. Several of the states in the region are among the
most educationally successful in the country by traditional measures.
This is both a boon and an obstacle to school reform.

Over the past 33 years, the Laboratory has had four Executive Directors.
For eight years McREL has been housed in its presently leased space in
Aurora, Colorado, a suburb of Denver. McREL presently employs a staff of
88. OERI Regional Educational Laboratory monies, \$11,545,727, accounted
for 43% of the Laboratory's total funds during fiscal years 1996, 1997,
1998. Additional sources of funds during this contract period include
\$15,268,735 from other direct federal awards, pass-through, and other
contracts, royalties, publications, interest, and investments.

Opportunities to serve the region include mandates by the seven states
for school districts to align curriculum and instruction with state
standards or student assessment measures. McREL's specialty area for
this contract period is Curriculum, Learning and Instruction. Prior to
the present contract, McREL's energies had been moving toward this
focus, as stated in its

proposal summary for this competition in 1995.

The Laboratory has three major operating components. The first component
is the Mid-continent Regional Educational Laboratory, which includes the
Regional Educational Laboratory (REL) contract, the High Plains
Mathematics and Science Consortium, and related federally funded
contracts mainly from State Education Agencies (SEAs). The second
component is the Institute, which conducts not-for-profit activities
that are not federally funded and which includes non-profit contracts
with districts within the region as well as districts, state agencies,
and other groups outside the region. The third component is MCL, which
conducts for-profit activities. The Executive Director of the REL is
also the Executive Director of the Institute. A different person serves
as the Executive Director of MCL. A single person serves as president
for all three components.

**II. Implementation and Management**

**A. To what extent is the REL doing what they were approved to do
during their first three contract years?**

**1. Strengths**

> **• Has met contractual obligations**

All panelists agreed that the Laboratory has met its contractual
obligations, whether viewed in terms of (formally proposed) goals,
activities, or tasks. The partnerships that are part of Signature Work
#2 provide strong examples of the contractually required collaboration
and constituent involvement. That state liaisons and facilitation groups
are in place and functioning well is good (partial) evidence that the
Laboratory, as promised in their 1995 proposal, is implementing a
comprehensive set of field services to scale up and provide direct
assistance. McREL has been an active member of the Laboratory Network
Program (LNP) and also has provided assistance to the U.S. Department of
Education's Office of Educational Research and Improvement (OERI) in
supporting educational reform by conducting research syntheses and
sponsoring conferences. The Laboratory has expanded and built upon its
expertise in curriculum, learning, instruction, and school change
through its extensive work in the area of standards.

> **• Focus has evolved into being much more standards based**

Although the Laboratory's 1995 proposal committed to systemic reform, it
did not address standards to a notable degree. As early as 1989,
however, McREL had begun to conduct research on standards and assist the
education community as it responded to the National Governors
Association's call for developing national standards as the keystone of
school reform. By the third year of the current contract, the Laboratory
had defined itself by its connection with standards, an area of primary
focus in education today.

**• Funds and resources have been leveraged**

McREL excels in using internal resources such as staff and technology to
positively enhance the work being carried out under the REL contract.
This critical infrastructure and strategic alliances discussed under the
next bullet enable McREL to leverage funds and resources and be highly
responsive to educators and other clients in the region at all levels of
service. In general the Laboratory uses federal funds to support the
development of programs, products, and services and then obtains funds
from State Education Agencies (SEAs) and Local Education Agencies (LEAs)
to scale up and disseminate within and beyond the Laboratory's region.
In short, the Laboratory leverages funds and other resources to the
benefit of all users in its region.

> **• Extensive and successful strategic alliances**

Alliances/partnerships help keep McREL grounded in the reality of the
classroom and build long-term relationships. Such alliances allow McREL
to leverage its resources while allowing states and local districts to
leverage their resources as well. The Laboratory has established formal
long-term, field-based collaborative action-research partnerships that
promote systemic reform with six of the seven states in the region. In
addition, it has developed less formal, shorter-term relationships with
associations and schools both within the region (e.g., Adams City High
School and Skyview High School in Colorado and with the Chief State
School Officers \[CSSOs\]) and outside the region (e.g., the POLARIS
Project in Alaska).

McREL has also established multiple partnerships with university
researchers to design survey items and adapt and administer them in
college-level classrooms within the region (e.g., Colorado and North
Dakota) and outside the region (e.g., North Carolina, Florida, and
Alaska). In an effort to scale up its work in learner-centered
instruction, the Laboratory has collaborated with five other regional
laboratories and has been an active participant in the Laboratory
Network Program (LNP).

Formal collaborative partnerships that are responsive to local, state,
and regional needs have been established in various configurations such
as State Facilitation Groups, Regional Field Services Teams, and
Collaborative State Action Teams. To provide services to regional
constituents, McREL teamed with several Boards of Cooperative
Educational Services (BOCES) in Colorado and Nebraska. It also
collaborated with Central Region states to develop implementation plans
for the Comprehensive School Reform Demonstration initiative.

In addition to formal collaborations, McREL created and maintained
partnerships with a broad range of other service providers,
universities, and professional associations to leverage its resources
and scale up systemic reform. These included the Colorado Association of
School Executives, the National Center for Educational Statistics, the
National Rural Education Association, Policy Studies Associates, South
Dakota Associated School Boards, and the University of Missouri at
Kansas and at St. Louis. Strategic alliances have also been developed
with other types of entities such as the New York Times and the Jet
Propulsion Laboratory.

**2. Areas of Needed Improvement**

**• Ethnic and gender representation on the Board**

According to the September 1998 EEO Report, the gender and ethnicity
composition of McREL staff is predominately female Caucasian (about
three fourths of the staff are women). In contrast to this figure, at
most one third of the Board is female.

The Laboratory's response to technical questions (related to the
proposal it submitted for funding in 1995) noted a resolution passed by
its Board that directed its nominating committee to actively seek out
for membership, individuals who would provide additional representation
from culturally diverse populations as well as from two job-related
constituencies. In an on-site interview, however, a sample of Board
members informed the Panel that they did not know of any Board efforts
to improve ethnic or gender representation.

**• Some deliverables have been turned in late**

McREL's OERI program officer regularly submits ratings of various
aspects of the Laboratory. Most aspects received the maximum possible
rating of 4, with the most notable exception being the relatively
frequent rating of 3 for timely submissions of deliverables. That
problem is in part explained by the shortage of staff at McREL
(according to the "Jobs at McREL" link on the McREL web site, there were
currently six unfilled positions, three of which were at the senior
associate level). A remarkable 20% was reported as the turnover rate for
1998, suggesting that staff turnover may be worth studying to see if it
can be reduced. The Laboratory has recently tried to address the problem
through the use of short-term consultants. So far this approach seems to
be working well, according to the Laboratory leadership. The Panel
recommends that the Laboratory collect and analyze data on the degree to
which it is able to meet deadlines for deliverables using this type of
consultant arrangement.

B. To what extent is the REL using a self-monitoring process to plan and
adapt activities in response to feedback and customer needs?

**1. Strengths**

**• Quality assurance process in place**

McREL has a detailed, standardized Quality Assurance (QA) system in
place for the development of products and other deliverables. The
procedures specify different approaches based on the sensitivity of the
materials. According to McREL, the system was based on a review of QA
processes used at other Regional Educational Laboratories.

The Panel saw evidence that the QA procedures were indeed widely used by
McREL. An example in which the Laboratory used QA feedback to modify a
product is shown in the response to reviewers' comments on *Research
Into Practice: Implementing Standards in the Classroom*. Workshops
presented by staff regularly undergo formal evaluation.

> **• Annual evaluation reports (submitted voluntarily) to OERI provide
> specific information on actions taken based on previously identified
> problems**

Although not required by the OERI contract, McREL has produced annual
evaluation reports that report on all aspects of the Laboratory's
operations and management. The report for FY97 was comprehensive,
straightforward, and well formatted. In that report, McREL delineated
follow-up activities addressing problems uncovered in FY96. The FY98
evaluation report had not been completed as of the time of the Panel's
site visit. The Panel found it odd that such an evaluation report was
not required by OERI.

**• Frequently collect data on client satisfaction**

McREL regularly collects data on its clients' satisfaction with the
products and services produced by the Laboratory. According to the
Laboratory, the purpose of these client-satisfaction data is to ensure
the quality of deliverables and products as they move through the
development process. A good example of the Laboratory's efforts to
collect feedback from clients is the *Field Service Partners Survey*
completed in 1999.

**• Regular needs-sensing meetings with the field**

McREL uses needs sensing to monitor its work. Customer feedback is
elicited for training events, conferences, and products. There are
regular and frequent meetings in the field with Chief State School
Officers (CSSOs), LEAs, and other stakeholders. For each of the states
in the region, a McREL staff serves as a liaison, who works with the
State Facilitation Groups (SFGs), which are normally composed of a
deputy superintendent, a researcher, and a field-services person to set
the agenda for service activities in the state and Collaborative State
Action Teams (CSATs), who meet at least yearly with the Chief State
School Officer.

**• Self-reflection during meetings of McREL leadership and staff**

Annual staff retreats allow for self-reflection on a large scale. In
addition staff keep activity logs, which are shared with other staff and
recorded in a database. Regular debriefings with staff, senior leaders,
and program directors include reflection on field experiences to help
refine the client focus. The Panel found the efforts discussed under
this bullet combined with those addressed in the previous two bullets to
be a nice mix of self-monitoring methods.

> **2. Areas of Needed Improvement**

• Improve methods used for self-monitoring (e.g., ensure appropriate
sampling and adequate response rates)

Even though it may seem antithetical to Laboratory interests, it is
important to document failure as well as success. It is as important to
know what does not work in school reform as it is to know what does.

McREL informed the Panel that about 14 staff members have some degree of
formal, rigorous training in measurement, research, and statistics.
There is a suggestion in some of the products the Panel reviewed that
the Laboratory's technical expertise, particularly in the measurement,
survey, and action research arena, has not been fully called upon.

Although formal evaluation mechanisms are in place and appear to get
implemented, at least some of the formal evaluations seem to be
questionable sources of information. For example, the telephone
interview of *The Systematic Identification and Articulation of Content
and Standards*, which was designed "to assess the utility and impact of
\[the\] standards document in school districts within the McREL region,"
(Evaluation Brief No. 98-1, p. 2), had a sample size of only 11 from the
originally identified sample of 28 interviewees.

The Panel also found much of the evaluation data collected not
particularly useful for improving workshops because the questionnaires
consisted mostly of fairly generic items and participants' perceptions
of their mastery of the skills focused on during the sessions. At best,
these are rather gross indicators and can do little more than alert
presenters of extreme displeasure among participants. Just as we ask
teachers to incorporate authentic assessment into their teaching (and
strengthen their teaching in the process), workshop participants should
be given an opportunity to demonstrate their mastery of skills taught
during a session rather than simply reporting whether they believe they
have achieved mastery.

The several self-monitoring efforts could be improved by (a) increasing
the response rate (where the response rates were low, the Panel did not
see an indication of any additional effort to increase the rate); (b)
ensuring samples are representative of the population; and (c) including
the

collection of data directly addressing product or program improvement.

To more fully serve the emerging measurement and research needs that
defining and implementing standards have created, McREL should bring its
technical staff resources into the research and assessment planning and
evaluation collaborations more frequently. One panelist suggested that
if current staff are overcommitted and hiring more permanent staff with
expertise in measurement and research is difficult to carry out, then a
possible alternative is to form a long-term national technical advisory
group that convenes quarterly.

**• Use formal self-monitoring to complement informal self-monitoring**

There has been a broad array of formal (e.g., a survey) and informal
(e.g., self-reflection meetings) self-monitoring activities occurring
throughout the contract period. The Panel suggests that the two types of
activities be used to systematically complement each other.

#### III. Quality

**A. To what extent is the REL developing high quality products and
services?**

**1. Strengths**

> **• Individual users and professional organizations see products as
> being of high quality**

Evidence that the Laboratory is developing high-quality products and
services ranges from enthusiastic testimonials from many clients in the
region to the large number of requests for publications and resources.
McREL's *Compendium of Standards* is seen as a valuable resource for
school reformers. During a teleconference, an educator from a state in
the region noted that McREL had changed the culture of its SEA from
being mainly a regulatory agency to being a resource facilitator for
school improvement. Another client praised the Laboratory for its work
in helping to develop standards and matching district assessments. There
were cases of unaccredited schools becoming fully accredited after
McREL-led intervention. These clients as well as several others
expressed strong appreciation for McREL.

McREL makes good use of comprehensive literature reviews, expert panels,
and field-based experiences to guide the development of its products and
programs. The formal Quality Assurance procedure in place for all
developing products and services helps promote high quality.

**• Services are often fully tailored to clients' unique needs**

Clients interviewed by the Panel suggested that they considered the
services to be of high quality, in large part because McREL staff were
willing and able to tailor what they did to fit local needs. In some
cases, the clients felt that McREL staff went out of their way to
customize their services.

**2. Areas of Needed Improvement**

• Field testing and quality assurance are often limited to client
satisfaction and not focused on student outcomes or changes in teacher
behavior

Popularity and use are not the best measures of quality. The Panel
recognizes the fact that for many McREL programs it may be premature to
try to measure student outcomes; however, it may make sense to measure
the intermediary outcome of teacher change or improvement.

When it is appropriate, the Laboratory needs to conduct rigorous studies
to evaluate the impact its products or services have on students and
teachers. Unless there is good information on student achievement and
teacher change, one can only guess at whether the reform efforts the
Laboratory promotes are positive or negative.

> **• More reliance on external peer review using reviewers with
> technical expertise**

For the most part, research done by McREL has not gone through the
traditional quality-assurance process established by the research
community such as systematic review by external experts and publication
in quality refereed journals. Although such journals would not be
expected to be the primary publication venues for a Regional Educational
Laboratory, it seems reasonable to expect that major research studies
such as the meta-analysis of the instruction literature should be
submitted to the research community's traditional mechanism for
validation before large amounts of federal monies are spent on
development efforts based on this research.

When McREL external reviewers raised significant technical issues such
as low response rate with regard to a survey conducted by the Gallup
organization, such concerns were dismissed, unconvincingly to the Panel,
by Gallup (see advance-reading item #168). Despite the various flaws
noted by the external reviewers of the Gallup-conducted study, McREL
plans to have Gallup conduct similar surveys to obtain industry's views
and teachers' views. The Panel recommends that when external reviewers
critique future work by agencies such as Gallup, McREL should (a) allow
the reviewers to further critique the responses from the agencies and
(b) adjust the Laboratory's future work accordingly.

One panelist suggested that OERI might consider accepting drafts of
publishable research articles in lieu of technical reports, which seldom
get read by many people.

• Possible over-generalization or over-assertion from some of the data
collected

Despite concerns expressed about the quality of the method used in the
aforementioned Gallup survey, McREL published the results with what the
Panel views to be an overly bold title, *What Americans Believe Students
Should Know: A Survey of U.S. Adults.* A related concern is the two
aforementioned similarly flawed surveys that the Laboratory still plans
to conduct.

The Gallup organization's representation on the Board of Directors of
McREL was of some concern to the Panel. The fact that the Gallup
executive excuses himself when Gallup contracts are being discussed is
only mildly reassuring, because it is not clear that others on the Board
have the technical expertise needed to assess quality issues in the area
of research. It should be noted, however, that the Gallup executive is
also a former university dean and a Board member who, according to the
Laboratory, pushes McREL to follow up on research or evaluation
findings.

**IV. Utility**

A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?

**1. Strengths**

• Many local, regional, and outside-of-the-region clients indicate a
high regard for McREL products and services

McREL's products and services are used extensively; for example, the
Laboratory has contracts with 33 different states and a number of
foreign countries to work on standards-based reform, and a number of
professional associations look to the Laboratory for assistance and
materials, especially in the standards area.

The volume of work conducted by McREL is impressive. In FY96, McREL
staff were involved in more than 500 training/technical-assistance
activities that served more than 14,000 persons. Those numbers increased
in FY97. In FY98, the Laboratory received almost \$3 million for McREL
publications and non-federal contracts.

> **• McREL is responding to urgent needs of clients**

By trying to be current and poised to help schools on short notice, the
Laboratory has sometimes been able to serve as a resource much needed by
schools under pressure to respond to mandates and initiatives (e.g.,
moving to (a) standards and standards-based assessment, (b) reading
improvement, and (c) comprehensive school reform).

> **• McREL's web site shows high and increasing usage**

Many of the Laboratory's works are available by downloading from the
McREL web site. The level of use as measured by hits and time spent
visiting the site is increasing and has passed the impressive level of
more than 1 million hits per month. The McREL database is referenced on
or hot linked to more than 1,300 other Uniform Resource Locators (URLs).
The Panel was also impressed by the Laboratory's effort, through its
webmaster, to incorporate the latest standards (e.g., *Web Style Guide*
\[Yale University Press, March, 1999\] and WWW Consortium)

for ensuring that web sites are maximized for clarity and
maneuverability.

McREL sites have received more than 20 awards, including the Links2Go
Resource award for its collection of lesson plans, the Suite101.com "Top
5 Best of the Web Award," one of the U.S. Department of Education's
"Picks o' the Month for Connections+," and NetGuide's "Gold Site Award"
for being one of the best on the web.

**2. Areas of Needed Improvement**

• Too much focus on standards may lead to exclusion of other important
aspects of schooling

Standards must be embedded in a larger view of school reform to improve
student performance. McREL should reflect on its view of standards and
how they fit in with assessment, professional development, school
climate, finance equity, and other important aspects of school reform.

McREL's focus on standards may limit its ability to respond to various
other customer needs unrelated to standards. One panelist noted that
although the notion that standards is the way education should go has
considerable face validity as well as widespread support, one could also
make the case that the standards-based approach to school reform, with
its long lists of standards somewhat akin to the long lists of
behavioral objectives that appeared in the 1970s, is misguided. That
panelist also noted that for all its commonsense appeal, the current
commitment to standards is based more on faith than it is on evidence.
It may make more sense to treat the standards-based approach as one of
several different approaches that McREL can propose and facilitate.

To truly assess the quality of McREL's work in the standards area, and
at the same time validate the whole standards-based approach, OERI
should consider funding (1) external studies of the impact of the
Laboratory's services and products and (2) studies of the impact of
standards-driven reform attempts.

• Expand efforts to provide culturally and linguistically appropriate
products and services

The region has enough cultural and linguistic diversity that McREL's
products and services could realize a substantially broader
dissemination base if efforts were expanded along those lines. Such an
expansion could benefit from guidance from a more culturally diverse
Board.

**B. To what extent is the REL focused on customer needs?**

**1. Strengths**

> **• McREL is sensitive to needs of clients and of OERI**

Most of the feedback from clients as well as from OERI has been highly
positive, with several interviewees reporting that the Laboratory was
very responsive and really met their needs. By sponsoring many workshops
and conferences throughout the country, McREL has provided the level of
support needed to facilitate major school changes. By conducting regular
needs assessments, the Laboratory remains informed of the type of
support schools, LEAs, and SEAs need and want. By developing and testing
its products in real school settings, the Laboratory enhances the
likelihood that the products will be useful to teachers and other
educators. In its proposal to OERI, the Laboratory aptly noted that all
the laboratories "have struggled to align the agendas of the Department
of Education on the national level, the various agendas of states in
each region and the needs of people not traditionally well served by
existing educational systems."

> **• Several mechanisms are used to identify needs**

The Laboratory regularly collects feedback from clients through
diversified methods; for example, it commissioned the Gallup
Organization and Quality Education Data to conduct telephone polls, held
focus groups with SEA staff within the region, convened meetings with
the central region's Chief State School Officers, and internally tracked
information requests and feedback from McREL field staff. Program
directors' meetings and brown-bag self-reflection sessions are held at
regular intervals, and the Laboratory Board engages in strategic
planning every few years.

**2. Areas of Needed Improvement**

> **• Missing feedback from students**

Students' perspectives were generally missing in the evidence the Panel
reviewed. Furthermore, the needs assessments lacked data on student
performance and behavior.

• Need to use more than client satisfaction to determine whether client
needs were met

Needs assessments consist almost exclusively of responses to
questionnaires from clients or potential clients. The Laboratory seems
to be so sensitive to clients' needs or wants that it even structures
its research in its intensive research sites around the principles of
collaborative action research. This sensitivity to clients' needs is
laudable, but the Laboratory must realize that clients' interests and
concerns may not coincide with the research questions the Laboratory
needs to answer. It is important to collect data on how materials and
services are used and eventually what impact such use has on the way
teachers teach and how students learn. In short it is essential to know
whether teachers changed and whether students improved.

**V. Outcomes and Impact**

A. To what extent is the REL's work contributing to improved student
success, particularly in intensive implementation sites?

**1. Strengths**

> **• McREL's focus on standards is essentially a focus on student
> achievement**

To the extent that standards-based reform contributes to improved
student achievement, one can assume that the Laboratory's work
contributes to improved student achievement. The Laboratory continues to
stress the importance of standards-driven instruction instead of
measurement-driven instruction. Clients in places such as Iowa, North
Dakota, Wyoming, and

Missouri affirm that teachers are now teaching to standards thanks to
McREL.

> **• Starting to address diversity in relation to standards**

The Laboratory deserves kudos for linking diversity concerns to the
standards movement. The Diversity Roundtable project seems like a good
model for linking research and practice. It commissions research papers
from well known scholars and uses these as the basis for (a) interaction
between practitioners or policy makers and these scholars and (b) the
development of more practitioner-friendly materials.

**2. Areas of Needed Improvement**

> **• Lack of baseline data to allow for monitoring change/effect over
> time**

The Panel found the general absence of baseline student data
troublesome, because if there is no basis for comparison, it is
virtually impossible to determine the impact of any product, service, or
intervention. The Laboratory suggested that reconstructing the data
years later as needed was a way to resolve this concern. The Panel
regards such reconstruction of student data as an extremely difficult
process and the reconstruction of data on teacher behavior and practice
as virtually impossible.

• Evidence of student success or of intermediary indicators such as
teacher behavior is limited

Although self-report data on participants' satisfaction and perceived
benefits are valuable, they are not sufficient to inform design of
quality professional-development activities. As recommended by the
Standards on Professional Development, McREL should evaluate the impact
of its professional development on participants' achievement (an example
of such information on achievement was the data on the improvement of
12th grade writing in Minot, ND).

The Panel did not come across data that could be used as intermediary
indicators. One way to gather such data would be to conduct structured
observations by trained raters to determine how and to what degree
teachers are using newly acquired knowledge, skills, and abilities in
their classrooms. From that point, one could start evaluating how those
changes might be linked to improved student learning on the standards.

B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?

**1. Strengths**

• McREL's products and services are widely used, and it is recognized at
the local, state, and national levels as a leading expert on standards
and on comprehensive school reform

Standards are at the heart of comprehensive school reform. The work of
McREL to make standards available, understandable, and usable by SEAs
and LEAs is of tremendous assistance to the field. Comprehensive school
reform strategies begin with standards.

McREL uses its expertise and prominence in the field of standards to
help states and districts implement reform. The Laboratory has created a
reform presence in the mid-continent region that would be hard to
imagine there without McREL.

McREL's four comprehensive and collaborative applied research and
development programs, implemented in full partnership with stakeholders
in each of the seven states in the region, created tools and strategies
that facilitated systemic reform for all students. Regional Field
Service Teams developed services based on the specific needs of each
state and the cross-cutting needs of the region.

Testimonials the Panel heard from people in the field suggested that the
Laboratory provided significant technical support at the local and state
levels for implementing the Comprehensive School Reform Demonstration
(CSRD) legislation in the region. All states have now qualified for
funding. Although it is difficult to determine precisely McREL's role in
the states' success, the testimonial evidence suggests it was
substantial.

• A concern for standards-based reform permeates virtually all of
McREL's activities

By permeating the Laboratory's activities, the concern for
standards-based reform seems to be more integrated than would be the
case if standards were seen as a more discrete entity. Although the
Panel found this permeation to be a strength, at the same time, it is
concerned about over-reliance on standards as the (only) way to
successful school reform.

**2. Areas of Needed Improvement**

• Given its national prominence, McREL needs to ensure that its products
and services are solidly research based

Because the Laboratory is so influential, it needs to be especially
careful about passing off speculation as research-based knowledge. It
might be desirable to publish speculative thinking in the form of
debates or symposia to which people with different perspectives
contribute.

During the most recent Congressional debate about funding research,
there were complaints that much laboratory and research center work was
simply ideology. McREL needs to be sensitive to this issue and make sure
that its work is not speculation or ideology masquerading as
research-based findings.

**• Need to better address the issue of time as it relates to the
teaching of standards**

The issue of the time it takes to address standards for various students
versus the time available is vital to successful reform. McREL raised
the issue to a very high level of national attention and now needs to
address it explicitly. The utility of Laboratory products and services
would be improved if they reflected a greater recognition that there is
very important school life beyond standards and assessment. It is
important to not assume that all the time in schools is available for
addressing standards.

McREL has a vested interest in demonstrating the efficacy of standards
in promoting improved student achievement in general. In particular, the
Laboratory should study which standards and instructional approaches to
standards are most successful in promoting overall student achievement.

A related issue is that the Laboratory has not shown concern for what
may be the negative effects of a headlong implementation of
standards-based reform on students, teachers, and state school systems.
Many of the states, and many schools within the states, have
historically been very successful educationally. Care should be taken to
not risk what already works. McREL should consider documenting and
studying successful schools, which might not be using an approach
centered on standards.

C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?

**1. Strengths**

• McREL has a national and international reputation in the area of
standards as they apply to curriculum, learning, and instruction

To the extent that standards-based reform can be thought of as
encompassing much of curriculum, learning, and instruction, the
Laboratory has indeed developed a national and international reputation
in its specialty area. McREL is currently providing services or having
its products and programs used in places throughout the U.S. By adapting
and implementing content standards, McREL has developed a model for
tracking standards and benchmarks in the content areas identified in the
National Goals. The Laboratory has developed and disseminated print
resource materials, monitored and integrated research literature,
provided extensive professional development opportunities, sponsored
numerous conferences and workshops, presented at regional and national
forums, and developed extensive partnerships with U.S. Department of
Education-funded institutions and service providers.

Among its most widely used products are *Content Knowledge: A Compendium
of Standards and Benchmarks for K-12 Education* and *The Fall and Rise
of Standards-Based Education*. Other popular publications are
*Comprehensive Guide to Designing a Standards-Based Classroom, School
and District*; *Awash in a Sea of Standards*; *and Eight Questions You*

*Should Ask Before Implementing Standards-based Education at the Local
Level*.

The Laboratory has cooperative working relationships on specific
projects with nationally prominent institutions or agencies such as the
Jet Propulsion Laboratory, Lockheed Martin Astronautics, National
Education Association, Phi Delta Kappa, the International Reading
Association, the National Council of Teachers of English, the American
Psychological Association, and the New York Times. The Laboratory and
its partners such as ASCD have published numerous Laboratory-developed
materials. McREL is well represented in professional journals (e.g.,
*American School Board Journal*, *Journal of Science Education and
Technology, The Professional Educator*, *Technology in Education*, *Phi
Delta Kappan*, *Education Week*, and *Educational Psychologist*), books
(e.g., *School Wars* \[Jossey-Bass\], *Internet Links for Science
Education* \[Plenum Press\], and *The Learner-Centered Classroom and
School* \[Jossey-Bass\]), and presentations at professional conferences
(e.g., American Association of School Administrators, National Council
of Teachers of Mathematics, American Educational Research Association,
and American Psychological Association).

**2. Areas of Needed Improvement**

> **• Given its national prominence, McREL needs to ensure that its
> products and services are solidly research based**

So great is the reputation of McREL regarding standards that it must now
be more circumspect in its pronouncements about the subject. Being among
the most highly reputed experts on the matter gives the Laboratory great
credibility and influence. The assessment design and development work
essential to standards-based research and reform is often more
controversial and loaded with decision options that have numerous
benefits and trade-offs than the identification of the content standards
themselves.

**VI. Overall Evaluation of Total Laboratory Programs, Products, and
Services**

McREL is a successful Regional Educational Laboratory. Its cutting-edge
work on standards can potentially have tremendous positive impact on
students. The Laboratory is making a great contribution regionally and
nationally to the cause of school reform, especially with regard to
incorporating standards.

McREL is service oriented, committed to responding to customer needs.

To ensure high quality, the Laboratory needs to address concerns related
to research and development issues. It is also important for the
Laboratory to reflect on how its work fits into the larger milieu of
today's schools.

The Laboratory is being challenged to attract and retain highly skilled
staff in a highly competitive market and in a location with a relatively
high cost of living. This challenge is interfering to some degree with
the ability of the Laboratory to move ahead as quickly as it otherwise
could. Within the Laboratory, diversity of staff is an ongoing concern,
and progress has been slow.

VII\. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement

McREL's strengths are numerous. They have a multi-talented, energetic,
caring, and motivated staff with a shared goal: To be the standard on
standards and the operating system ("McREL inside") of the standards
movement. By leveraging funds from other sources, they have accomplished
an enormous amount of work. They are engaged in cutting-edge projects,
for the most part, which are on track and within expected timelines
except for a few deliverables. The two signature works positively
impressed the Panel. When modifications to the proposed work have
occurred, justifications have been convincing. The work quality is the
direct result of the talent combined with a formal quality-assurance
program and a caring involved governing board.

Based on the site presentations and interviews and a review of the
advance materials, the major areas of concern and need are (1) a more
systematic and rigorous collection of evaluation data, attending to
issues such as representativeness, sample size, and response rates; (2)
viewing the standards-based approach as not necessarily the exclusive
approach for education; (3) collecting data on student performance and
teacher change; (4) increasing cultural diversity on the Board as well
as on the Laboratory staff; and (5) the production and dissemination of
peer-reviewed work in refereed journals.

Finally, the Panel wishes to acknowledge McREL's full cooperation in the
evaluation. Without such excellent collaboration, the evaluation would
have been almost impossible to carry out within the agreed-upon
parameters.