**Archived Information**

**Interim Evaluation of the North Central Regional Educational
Laboratory**

# I. Overview

As part of the NCREL interim evaluation team, I participated in a
one-week site visit to the Laboratory on April 26-30, 1999. I reviewed
Laboratory materials extensively both prior to the visit and as a
natural part of the flow of work during the site visit, including both
materials mailed to us prior to the visit and on-site materials. I
participated in numerous focus groups and presentations, two site visits
of intensive work, and interviews with NCREL staff. I had full access to
information, and the staff readily provided additional resources and
information upon request.

**II. Implementation and Management**

A.  **To what extent is the REL doing what they were approved to do
    during their first three contract years?**

### Strengths

Overall, NCREL is meeting or exceeding expectations in performing the
work they have been contracted to do. NCREL is meeting or exceeding
expectations in their work of: (1) providing design consultation to
schools and communities; (2) identifying exemplary programs and
resources in critical content areas; (3) providing national leadership
in promoting the use of technologies to improve learning; (4)
establishing NCREL as a leading regional resource for networking and
policy; and (5) establishing NCREL as a leading regional resource for
professional development.

NCREL is completing work with focus on making connections and scaling
up. Overall,

NCREL's strength at this point in the cycle is on making connections,
although the Lab is currently meeting expectations for developing
infrastructure for scale-up. The work on scaling up is underway, with
the focus on potential to scale up evolving primarily from the context
of making connections as contracted: their work in contributing to
strategies for moving educational innovation to scale is emerging
through focus on developing products and processes amenable to roll-out.
For example, work such as Learning with Technology is developed in
context of the field and is then converted to format amenable to broad
access such as professional development packaging, software and/or or
web-based tools. This pattern fits the design of the contract plan as
outlined in the OERI contract, in which each of the four centers has a
specific subsection that outlines "contributions to scaling up."

There are three major areas of focus NCREL is using to successfully meet
or exceed expectations: (1) fluid, project-based organizational
structure that allows flow of human resources; (2) external partnerships
that make connections between research and practice and maximize
resources; and (3) using technology as an organizer and vehicle for
making connections and supporting scale-up.

First, NCREL has developed and implemented a coherent organizational
structure that focuses on internal connectedness and, as such, supports
the organization in completing work they are approved to do in context
of connections and scale-up. NCREL is completing its work through a
project-based approach that connects its work across its four main
centers as described in their OERI contract: (1) Center for Teaching,
Learning, and Curriculum (CTLC); (2) School and Community Development
Center (SCD); (3) Center for Scaling Up (CSU); and (4) Evaluation and
Policy Information Center (EPIC). Although these four centers are used
as organizers for work NCREL was approved to do during this contract, it
is clear from interviews, quarterly reports, products, and planning
documents that projects cut across centers according to the demands of
the tasks. There is also evidence of evolutionary planning, with work
flowing differently across centers and with external partners as the
demands of the project change. This flow of work is best exemplified by
the project description of how NCREL provided support for helping
states, districts, and schools build awareness of, apply for, and select
partners for Comprehensive School Reform -- NCREL functioned as a
"learning organization" in its best sense, with fluid response in a
timely, focused manner. Throughout the quarterly reports, there is
evidence of this flexibility and mutual support in completing the work
the Lab was approved to do, with cross-center work ranging from advisory
roles to full partnerships. Cross-center operation of the four-group
organizational system positively impacts the quality and the successful
completion of approved work. For example, there is strong evidence of
EPIC's evaluation input and assistance throughout the Lab's work,
regardless of which center hosted the project. There is also strong
evidence of the Center for Scaling Up's focus on building an
infrastructure of materials and processes amenable to being moved to
scale; this influence cuts across the centers and is infused throughout
the projects.

Throughout the visit, there was strong evidence of strength and
commitment in personnel, a climate of thoughtfulness, and an everyday
norm of cooperation and positive working partnerships across the
organization. This exemplary team and their joint approach to
project-completion positively impacts completion of the work NCREL has
been approved to do. The leadership team was highly effective and
dynamic in their approach to NCREL tasks and utilizing the human
resource potential of the Lab.

Second, NCREL is completing work in context of external partnerships and
connectedness outside the organization, including work on a variety of
levels that ranges from intensive site effort to national level policy
efforts. This effort helps create infrastructure to move to scale
through connections with a wide array of partners. By completing work in
partnership with a wide array of external individuals and organizations,
the Lab is developing products and processes in context of connecting
research with the reality of practice. This was particularly evident in
the intensive sites as well as in interviews with external partners.

NCREL is providing design consultation to schools and communities and
using that consultation as a Laboratory for developing materials and
processes that are both connected and supportive of scale-up, as
outlined in follow-up interviews and site interviews for intensive
intervention sites. The intensive site work provides solid grounding in
practice for the research and development work of the Lab, serving in a
sense as professional development in "the practical" for NCREL. For
example, the Lab is focusing effort on filling the research knowledge
gap in how external consultants with specific areas of focus can best
fit into the complex work of site reform with maximum impact and
intensive site work offers a Laboratory opportunity to explore and find
solutions to this important issue in context of the reality of practice.
Furthermore, NCREL is connecting their broad audience -- ranging from
classroom teachers and schools to policy-makers, states, and regions --
with exemplary programs and resources, cutting through the complexity by
maintaining a focused eye on teaching, learning, and increasing student
success as the centerpiece of their work as evidenced by the materials
and products they produce for dissemination.

Finally, NCREL is using the broad concept of "using technology to
support instruction and learning" as an organizer for making connections
and creating infrastructure with potential to move to scale as defined
in their OERI contract. Using technology both as exemplar and as medium,
the Lab is providing local, regional, and national leadership in
promoting the use of technologies to improve learning. They serve as a
strong resource for networking and policy both through direct work with
clients and indirectly through materials and processes made available on
a broad scale through print and electronic media. Using both
face-to-face work and technology, NCREL is establishing itself as a
leading regional resource for networking and policy with focus on
teaching, learning, and achievement. Professional development is
receiving top priority, with products and processes emerging from work
in partnership and through field experience then packaged, presented,
and made available to a broad spectrum of consumers.

### Areas of needed improvement

While NCREL is meeting the agreements they were approved to do during
the first three contract years, it is time to being to shift focus more
strongly to the recipients' issues of moving to scale. Scale is an issue
both of delivery and dissemination -- moving products, services, and
processes out into the field -- but it is also an issue of receiving.
The organization and work seems to be more focused on delivery and
dissemination, with less attention to issues of reducing the disconnect
between delivering and receiving, particularly as it may be an inherent
part of the materials and processes themselves. This work is hampered by
the disconnect between the broad, sweeping focus of the national Lab
work -- making connections and a deep and meaningful way and informing
the daunting issue of moving reform to scale -- and the five year cycle
of Lab work. Educational change at this level of focus -- making
connections and moving to scale \-- requires a much longer range cycle
of effort and persistent focus. NCREL is doing well within the time
parameters imposed externally. They are struggling more Labwide with the
area of scale-up in relation to issues of making products and processes
"receivable" than they are with making connections -- as would be
expected with a five-year cycle of work.

> **[Recommendations for improvement]{.underline}**

1.  Explore options for partnerships and opportunities to extend the
    focus of the Lab's work beyond the 5-year cycle, with the 5-year
    span in context of a larger strategic view of the issues of
    connectedness and scaling up.

2.  Deliberately shift focus for the last part of this contract to scale
    up issues of both delivering products and services out and issues of
    consumer use on a broad scale. Focusing deliberately on the products
    and processes already developed or firmly underway, looking in great
    depth and with a researcher's eye at how to move them to scale by
    increasing their potential to connect with consumers in context of
    the practitioner's reality. This is particularly pertinent for
    teachers, students, classroom, and school level audiences -- the
    most difficult audience to impact in moving to scale.

B. To what extent is NCREL using a self-monitoring process to plan and
adapt activities in response to feedback and customer needs?

### Strengths

NCREL uses the Four Stage Model that includes assessing needs and
setting goals, designing planning and evaluation systems, implementing
the plan, and evaluating the plan then renewing effort according to that
feedback. The Lab uses this model to design and evaluate their own
performance internally as well as to build in effective strategies into
projects across the centers. There is evidence of strong feedback
throughout the Lab's work, ranging from simple surveys as part of
professional development activities to the highly technical Gallup Polls
that were contracted out to external reviewers.

### Areas of needed improvement

NCREL has made web-based information readily available to those who have
technological capacity, which includes not only the concept of having
adequate hardware but also the human skill and motivation to access the
information. Because the use of technology in education is in itself a
cutting edge piece of the puzzle, it is difficult to separate
technology-based delivery of products and processes from their receipt
by the potential consumer -- particularly to educators in the field.
There appears to be a disconnect between the products and services being
made available through technology and the capacity and/or will of the
potential recipients, such as teachers in intensive sites who have not
accessed information electronically and the relatively low percentage of
Pathways hits that come from teachers. When focus on receipt of
information is not incorporated as a key area of focus of the work of
the Lab, any significant disconnect between NCREL dissemination and
consumer utilization threatens to limit use of high quality Lab products
and processes. There is definite impact on the rollout potential when
the plan for moving to scale is heavily linked to "scale-friendly"
packaging.

**[Recommendations for improvement]{.underline}**

Use intensive Lab sites to view the contextual perspective on consumer
utilization of electronic products and processes.

Because NCREL is directly involved in intensive sites, workers have
direct access to viewing Lab products and processes through the eye of
the consumer. While providing hardware, skill training, and motivation
to potential consumers is beyond the scope of the Lab's work, accessing
electronic products and processes in context of overall school reform
efforts offers easy opportunity to learn how the Lab might adapt its
electronic products and delivery systems so they are more readily used
and useful in context of holistic change. It also offers opportunity to
research in a more deliberate way the consumer side of the issues of
moving to scale using technology in context of practice. NCREL staff
working in intensive sites who may be less familiar with products such
as Pathways or Gateway offer golden opportunity to view Lab deliverables
with an external eye that is analytical and educated. This knowledge can
then be looped back in and may have implications for adapting products,
processes, and delivery systems of NCREL's work.

**III. Quality**

**A. To what extent is NCREL developing high quality products and
services?**

**[Strengths]{.underline}**

NCREL has developed and is in the process of developing a wide array of
quality products and services that are applicable to different
audiences, such as research articles, conferences, technical assistance,
print, web-based, and electronic formats.

In particular, NCREL has exhibited strength in developing high quality
products and services around the concept of "engaged learning" as a
powerful conceptual framework for reconceptualizing high quality
teaching and learning on a broad scale. The engaged learning concept
encapsulates changes in both the role of student as active participant
in learning and in the role of the teacher as coach and facilitator
rather than as disseminator of wisdom. This concept appears to be
strongly infused into the vernacular at the teacher, school, district,
and state (Illinois) levels, with indication that the concept is moving
throughout the service region -- a major positive factor in the scale-up
potential of the products and services being created. It was evident at
the sites visited and through focus groups and interviews that teachers
and administrators actively use the construct in rethinking learning and
teaching in their daily work and that it is serving as a powerful
organizer for core level school change. This paradigm shift is
phenomenal: it reaches directly into the classroom with a powerful
metaphor that makes major changes in conceptualization of learning and
teaching. It moves directly and strongly into the normally impervious
arena of core technology -- learning and teaching in the classroom.

Using the concept of engaged learning, NCREL has developed a number of
strong products and processes. CTLC and SCD have focused in particular
on the teachers' role in delivering instruction and in understanding how
students use that delivered instruction to learn, with products such and
the Engaged Learning Safari offering educators clear benchmarks for
understanding engaged learning and with potential for scale-up inherent
in the product. The Gateway Project currently under development offers
particular promise for supporting engaged learning in context of local,
state, and national demands for high achievement of standards as
measured by performance assessment: a rich array of resources tailored
specifically to the teachers' perspective offers exciting new
possibilities for supporting teachers in particular \-- and learners to
some degree\-- in achieving critical "gateway" standards. The site
supports both improvement of teaching talent as well as teacher content
knowledge, and it is a very user-friendly way to explore an important
construct (such as "energy" in the science area) in great detail as well
as to obtain an assortment of teaching resources. It is masterful in its
focus on prioritizing standards as a way to cut through issues of volume
and overload in the world of standards and assessment, as well as in its
approach to improving teaching and learning.

By engaging experts in development of materials, such as Hallinger and
Bridges' involvement in Problem-Based Learning, the Lab not only
harnesses expertise for informing its own products and services and for
attracting the attention of a variety of audiences but also provides
direct link to existing bodies of knowledge and collections of high
quality work outside the Lab's influence. Through these direct linkages,
the Lab is very efficient in extending its arm of influence with minimal
relative expenditure of the Lab's funds, time, and talent.

### Areas of needed improvement

First, the concept of "engaged learning" is permeating the field of
service in a positive way, but it is not as highly visible in the
products, processes, publicity, and dissemination processes that will
let the Lab move the concept to scale. The concept is there in the
materials and processes, but it is not "front and center" like it seems
to be in the field. There is opportunity lost for making this amazing
paradigm shift on a broad scale and for creating positive ethos related
to concepts of constructivist learning and radical change in teaching
delivery systems as well as student roles. Pulled to the front in a
deliberate way, the concept of engaged learning could be a powerful tool
for scaling up by creating positive context for a wide range of
supportive materials and processes, both those created by the Lab and
those created elsewhere.

Second, although the Pathways server continues to make a wide range of
information available to educators and has received numerous awards and
receives a large number of "hits", it has a number of quality issues
that need to be addressed. In attempting to serve the needs of a broad
audience, the product fails to maintain high enough quality for the
empirical research world or to be user-friendly enough to serve teachers
in their world, and it has not kept pace with changes in the technology
field as well as with issues of effective web-based communication
strategies. More specifically, for example, while there is a great deal
of research information compiled on different topics of potential
interest, the research articles in particular have definite limitations
that make them "unfriendly" both to researchers and to practitioners:

-   many contain citations in context of text but lack a reference
    section at the end of that text

```{=html}
<!-- -->
```
-   many of the literature reviews have obvious gaps and fail to cover
    the literature thoroughly or evenly

```{=html}
<!-- -->
```
-   the reviews are writing with authority and certitude rather than
    with the tentativeness and conflict that is a natural part of the
    emerging nature of knowledge construction and thus some of the
    research reviews tends to take on an editorial slant

```{=html}
<!-- -->
```
-   the overall writing styles are formal and unfriendly to a
    non-researcher audience

```{=html}
<!-- -->
```
-   best practice exemplars and empirical research is blended rather
    than clearly delineated, sending an overall message of authority and
    expertise when it is not necessarily supported empirically

Third, research findings in depth remain relatively inaccessible to
people in the field, with the exception of broad translations in the
form of reviews. Electronic research is a cutting edge part of the field
of technology, and the Lab seems to be behind in this arena. Although
there are bibliographies and reference lists available on the web site,
hand-delivered through the resource center, and incorporated as part of
written materials, the information in these listings remains virtually
inaccessible to consumers unless they expend considerable time, energy,
and resources to filter through the list and then go out to obtain the
sources. Lists of sources alone are inadequate in communicating research
findings, and there are broad web-based and electric resources readily
available that are not part of NCREL's web sites or services.

**[Recommendations for improvement]{.underline}**

1.  Use the concept of Engaged Learning as a major organizer for
    products and processes.

The concept of engaged learning is more visible in interviews and
discussions with people than it is in products and processes ready to be
disseminated on a broader scale. This is a powerful construct that has
potential to make major impact in the field, and it needs to be "pulled
front and center" and used as an organizer for the web page, as a
conceptual framework that links NCREL products and processes into a
seamless, readily-understood conceptualization of overall change in
classrooms, schools, states, and regions. It is to your credit that you
have developed such a powerful mental model, one that has already begun
to shape major conceptual changes in the field. It is effective with a
broad audience, ranging from individual teachers and schools to
policymakers and state officials. Take the next step and use it more
deliberately and extensively.

2.  Capitalize on the potential strength of Pathways, even though it is
    not a "new" product.

Pathways has strong name recognition and numerous visitors -- almost a
"brand name" strength -- and it offers strong possibilities for
maintaining cutting edge information. Capitalize on this strength.
First, clearly identify your audience and create top quality work
targeted to that group. There is some indication that the site is
preferred more by audiences other than teacher audiences (such as
curriculum directors). It might be strengthened by tailoring it to a
more narrow audience such as curriculum planners and students in
pre-service courses, curriculum planners, and others. In order to be
usable to that audience, the research reviews need to be comprehensive
and current, and citations need to be included with more information
readily available on each article. All major perspectives should be
included, even where there is disagreement and knowledge gaps -- a
natural part of the empirical field. Teachers could continue to access
and use Pathways as an information source, with a higher level of
integrity in the information available to them. If, on the other hand, a
broad teachers audience is the focus, changes in the writing styles and
layout of research should make empirical information much more
user-friendly, and research findings might be organized around
field-friendly conceptual frameworks such as the concept of "engaged
learning".

3.  Make research more readily accessible.

Rather than simply providing lists of references, focus on the next
level of quality -- supporting access to the information each of the
references holds and supporting users in being able to create their own
lists through searches rather than depending on the Lab for direct
services. Bibliographies alone offer little information, and that
information is difficult to obtain without great effort. Consider
supporting references and bibliographies with another layer of available
information. For example, the Resource Center might create annotated
bibliographies for all sources cited using summaries of findings such as
those found in ERIC, rather than simply listing sources. Or you might
provide links to library web sites where that information is available.
You might help the region organize consortia (or use existing groups) to
purchase access to full text journals and other resources normally
available only to students in universities, to set up easy access to
ERIC searches, and to make sources available electronically so users can
move beyond the list of sources on the web, perhaps even letting
consumers (schools, districts, policymakers) pay their share of the
costs.

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

### Strengths

There is strong evidence that the Lab is developing a wide array of
products and services that serve a broad constituency, ranging from
teachers in classrooms to policymakers. As mentioned in the section on
quality products, there is a wide range of products, processes, and
supports available, and this array is connected conceptually around
broad themes.

Through partnerships with existing schools and organizations, the Lab
creates these products and services in context of the field. Teachers,
administrators, university professors, central office members, state
department officials, and others spoke often and with pride of working
in partnership with NCREL to develop usable products and services that
were then disseminated to broad audiences. Through evolutionary planning
using an ongoing cycle of planning, effort, feedback, and change, NCREL
works dynamically with partners in the field to create products and
processes that merge research with practice and are useful to
practitioners. By forging partnerships with outside agencies such as
Ameritech to create products such as Parentech, NCREL takes advantage of
the energy and funding available externally to craft strong work in
context of existing demand -- a natural for scale-up.

One of the strongest delivery systems for Lab products and processes in
terms of consumer access and desirability -- and thus potential for
moving to scale \-- is the use of courses as a delivery system. NCREL
harnesses the power of the course structure to make products and
services useful and used by customers in two ways: (1) the Lab has
created, and continues to create, courses that can be used for
in-service training, and (2) the Lab has formed working partnerships
with teachers in higher education so that Lab products and processes can
be used in context of pre-service education courses. It is obvious from
the quarterly reports that NCREL is capitalizing on this vehicle and is
designing more courses as vehicles for moving products, processes, and
information sources to scale.

NCREL has developed a broad dissemination strategy for in-service
training based on rollout through train-the-trainer models, with the
idea that Lab-controlled training will hold the line on quality while
allowing scale-up that is cost-effective. While the train-the-trainer
model has definite limitations, feedback from consumers indicates that
it has been highly effective in some sites, particularly in those sites
who attached sufficient funds and supportive policies, while it has been
ineffective in others as replication of planned delivery. However, there
is evidence that the rollout process was sustained in many cases by
consumers' adapting the product and/or using parts of the product in
ways they deemed to be helpful to their overall work.

NCREL makes products and processes widely available to potential
consumers and attempts to make them aware of the availability of the
information through mass mailings, web-based links, meetings, and
partnerships at all levels.

### Areas of needed improvement

The area of needed improvement for this question is the same as the area
addressed in Section II B. There is greater focus on delivering products
and processes out, once developed in partnership, than there is on those
unfamiliar with the products receiving and using them. This greater
focus has implications for scale-up, since the plan for moving to scale
is highly related to packaging and dissemination using technology and
other processes amenable to scale.

**[Recommendations for improvement]{.underline}**

Focus on the consumer receipt and use of products and processes,
particularly those products and processes that will be linchpins in the
rollout process.

**B. To what extent is NCREL focused on customer need?**

### Strengths

The Lab provides a wide array of services in response to requests
throughout the region, and they take a wide range of opportunities to
move the products and processes that emerge from these partnerships to a
broader audience. Projects such as the Leadership for Integrating
Technology are developed through pilot testing with ongoing adaptation
in response to customer need.

On a broader scale, NCREL is responsive to large gaps in the knowledge
base that impacts customer need, even though the consumers themselves
may be unaware of its impact. By exploring gaps in knowledge \-- such as
contributing to the knowledge base on the nature of successful technical
assistance for those entering schools and districts from the outside and
ways for schools and districts to obtain best fit between their needs
and the external assistance available -- the Lab makes major
contribution to customers in a way that is, in the beginning, indirect
but powerful. Through work at intensive sites, the Lab is exploring
contextual barriers to change and the interplay between single
interventions in context of whole school and classroom context. This
serves as both Lab for knowledge construction as well as offering going
professional development for NCREL staff that keeps their intellectual
work and perspective grounded in learning and teaching at a practical
level.

### Areas of needed improvement

Focus on customer need tends to be on specific partners and projects,
with an eye to the scale-up process as part of the process of developing
products and processes amenable to scale. Focus on customers at the
"to-scale" level is not as dominant a theme and is designed to emerge
from the primary focus on immediate customers through scale-friendly
packaging and dissemination strategies. There is less focus on "the
customer" at a scale level -- what will people need when they have
absolutely no face-to-face intervention? This is, inherently, one of the
critical knowledge gaps in moving to scale and is among the stickiest of
our national problems.

### Recommendations for improvement

Shift customer focus to "customer" in its broadest sense. This
recommendation is related to the recommendation to shift focus to issues
of scale as stated in the Recommendations for Improvement in Section II
A., recommendation 2.

**V. Outcomes and Impact**

**A. To what extent is NCREL's work contributing to improved student
success, particularly in intensive implementation sites?**

**[Strengths]{.underline}**

NCREL is playing in active role in the quest to raise student
achievement at intensive implementation sites. Achievement test data is
lending support to the Lab's contribution to achievement, as well as to
improvements in other measures of student success such as the high
attendance rates at one of the sites we visited. The Lab has strong
impact on teachers and is providing focus on engaged learning and
changing instruction throughout the school.

**[Areas of needed improvement]{.underline}**

It is most difficult empirically to identify causal relationships, and
it is particularly difficult in context of practice where there are
multiple interventions that are part of the school's innovation history
as well as their current efforts to reform. It appears that NCREL staff
who are participating in the interventions are also serving as primary
documentation collectors within those intensive sites. Thus data is
being collected on the site participants themselves, but there is little
empirical evidence being collected on the researcher as subject.

**[Recommendations for improvement]{.underline}**

Study researchers themselves in context of practice. The work at
intensive sites offers perfect opportunity for the research world to
gain knowledge on researcher interventions in context of practice and
whole school change. Consider having external researchers study NCREL
staff on site to measure the impact of NCREL interventions on intensive
site improvement and changes in student success that can be attributed
to NCREL interventions.

**B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?**

### Strengths

NCREL is participating in a wide range of intensive site efforts that
include a wide variety of interventions. The work in these broad
projects is evident in the quarterly reports. By combining OERI funds
with a wide range of other externally contracted projects, the Lab
leverages opportunity to assist states and localities in implementing
improvement strategies and to learn from these efforts in the process.

**[Areas of needed improvement]{.underline}**

None

**[Recommendations for improvement]{.underline}**

None

**C. To what extent has NCREL made progress in establishing a regional
and national reputation in its field?**

### Strengths

NCREL has strong focus on the use of technology to support instruction
and learning. As well, the Lab focuses its energy on harnessing the
strength of technology for design of user-friendly products and
processes, for dissemination on a broad scale in a cost-effective
manner, and for broad and open access to information through technology.

Many of the Lab's products are exemplars that serve as benchmarks for
the public in designing technology-based educational resources and
supports, such as the Gateway web site and the Engaged Learning Safari
CD-ROM, web, and video formatted project. In both instances, the Lab is
folding a number of key issues of importance into a technical format.
The Engaged Learning Safari, for example, harnesses the power of the
concept of "engaged learning" to create a paradigm shift in the field
and to portray constructivist, research-based practice in a politically
palatable manner. At the same time, it gives benchmarks for engaged
instruction so teachers and schools can begin to get a sense of "how
good is good enough", and it is disseminated in a wide range of media so
that customers with different preferences will be served through
technology.

The Lab has extensive projects within its region, in partnership with
others outside the region, and in partnership and coordination with
other Labs. The Department of Defense Education Activity for
professional development in technology applications sends a strong
message of support for the Lab's sound national reputation.

According to ongoing customer feedback, the Gallup polls in particular,
NCREL has strong name recognition with a wide variety of audiences.
NCREL has participated in a wide range of partnerships, and the quality
of the work that emerges from those partnerships and the quality of the
individual NCREL staff members creates an overall reputation of strength

### Areas of needed improvement

The lowest level of name recognition among customers appears to be among
teachers. The teachers know the Lab's work under other terms such as
through the popular work Plugging In, but they do not necessarily link
that work with NCREL.

**[Recommendations for improvement]{.underline}**

Explore opportunities to expand name recognition among teachers. This
"brand name" associated with quality products might offer a valuable
tool in the process of scaling up, since overarching name recognition is
a factor in advertising and distribution.

#### VI. Overall Evaluation of Total Laboratory Programs, Products, and Services

Overall, NCREL is meeting or exceeding contractual obligations, serving
both the region and the nation with broad array of quality programs,
products, and services that are nested in context of the comprehensive
view of education reform and that meet contractually-defined scale-up
requirements. The Lab has created a yeasty, thoughtful culture capable
of struggling with key issues of education reform in a deep way. At the
same time, the Lab is grounded in practice, both through partnerships
and through work in intensive sites. The Lab has managed to attract and
maintain a large cadre of talented, committed individuals who work hard
-- both individually and collectively -- to find ways to improve
education through comprehensive school reform and scalability of reform
efforts.

**VII. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement**

NCREL exhibits a wide array of strengths. Overall, NCREL is making
strong contribution to educational change. The Lab is meeting or
exceeding all contractual expectations and has developed -- and
continues to develop \-- quality products and processes with structural
and partnership capacity to move to scale. The work of making
connections; creating "scalabilty" in products, processes, and
partnerships; and making major gains in the specialty area of technology
runs throughout the Lab's work. NCREL is staffed by committed, talented
personnel, and the Lab uses fluid, project-based work and evolutionary
planning to maximize human potential and flow resources and intellectual
capitol across centers. NCREL holds high quality standards for data
gathering, analysis, and use of data to make decisions, with both
internal and external reviewers and processes in place. These high
standards are evident in both the internal work of the Lab and in their
external partnerships, products, and processes. The Lab has created a
monumental paradigm shift through the concept of "engaged learning" that
is having impact both at the policy formation level and at the policy
implementation level. And, finally, NCREL maintains focus on using
technology to support school and student achievement in a deep, systemic
way.

There are a few areas for improvement, and they fall primarily within
the arena of building on current strengths and successes. First, one
area for improvement is beyond the scope of the Lab's control but bears
mention: a five year planning and implementation cycle for work of this
scope is prohibitive. Second, NCREL tends to focus including customer
voice through development of products and partnerships in strong
collaboration with practitioners in the field, using feedback processes
through field tests, then following with revisions and rollout. This
highly effective strategy is a strength, but the voice of the
"non-face-time" customer is less evident in the process than is the
customer who also has personal contact with the Lab in some way. The
silence of this distance consumer creates potential scalabilty
limitations in the work under development, since there is limited
"voice" given to the future consumer who has no "face time." Third,
NCREL has not fully developed the growth potential of areas of strong
success, such as harnessing the concept of engaged learning as a
powerful organizer for NCREL products and processes or maximizing the
Pathways server's name recognition to draw people and organizations to
cutting edge information. Fourth, the Lab is behind in the area of
supplying consumers with references, focusing on providing lists rather
than linkages to searching through technology or otherwise making
summaries of information found within references more readily available.

**[Strategies for improvement include:]{.underline}**

1.  Nest the five year contractual context within a broader and more
    long-range strategic plan through planning with external
    partnerships, since the scope of work the Lab is undergoing operates
    best in context a longer view of change.

2.  Shift focus somewhat at this point in the contract cycle to include
    greater focus on the voice of the non-face-time consumer, and insure
    products and processes are in the best condition possible to serve
    this to-scale audience.

3.  Capitalize on successes by bringing the concept of "engaged
    learning" to the front page -- it's probably more of a communication
    and marketing issue rather than a direct educational issue, so the
    Lab might need to explore opportunities for advice and support
    within those fields rather than within the usual fields of education
    change. Also, use the Pathway web site as a centerpiece for cutting
    edge information, since it has brand name recognition and gets a lot
    of traffic.

4.  Rethink the Lab's role in helping consumers access information
    through references, focusing in particular on the role of technology
    in accomplishing that work.