Archived Information

# **Interim Evaluation of the North Central Regional Educational Laboratory**

## Synthesis Report

I.  **Brief Overview**

**...Of the Laboratory**

The North Central Regional Educational Laboratory (NCREL) is described
by its director as an "adolescent Laboratory.\" First funded by OERI in
1985, it has been refunded through two additional cycles in 1990 and
1995. NCREL serves a region that includes Illinois, Indiana, Iowa,
Michigan, Minnesota, Ohio, and Wisconsin, i.e. the relatively
homogeneous midwest. The region contains 20 percent of the country\'s
population, 20 percent of the its school-aged youth, and 25 percent of
its public schools. The Laboratory began with a budget of a million and
a half dollars and a staff of 25. For FY 1998, its budget was 16 million
dollars, of which 78 percent is provided by Congress, while 22 percent
comes from state, local, and foundation/private funding. The Lab now has
a staff of about 120 people, most of whom are housed in an office
building in suburban Chicago.

NCREL is organized into four separate but interconnected centers and has
one specialty area, which is technology. (Laboratory specialties are
specified under the current RFP. For NCREL, the choice of technology Lab
was a natural outgrowth of the emphasis they have traditionally put on
the use of technology to serve the region). The centers are: the Center
for Teaching, Learning and Curriculum (CTLC); the Center for School and
Community Development (CSCD); the Center for Scaling Up; and the
Evaluation and Policy Information Center (EPIC). NCREL has six published
goals:

1\. Provide design consultation to schools and communities

2\. Identify exemplary programs and resources in critical content areas

3\. Contribute to and study strategies for moving educational
innovations scale

4.  Provide national leadership in promoting the use of technologies to
    improve learning

5\. Establish NCREL as a leading regional resource for network and
policy

6.  Establish NCREL as a leading regional resource for professional
    development

**\... Of the Activities**

The six-person peer review panel visited NCREL during the week of April
25, 1999. The NCREL Interim Evaluation site visit team was comprised of
a six-member panel: Nancy Brigham, (panel chair, Partner/Independent
Consultant, Rosenblum-Brigham Associates Weymouth, MA); Nancy
Clark-Chiarelli, (Senior Research Associate, Education Development
Center, Newton, MA); Marilyn Willis Crawford, (Research Assistant,
Peabody College of Vanderbilt University, Hickory, KY); Andrew Hayes
(Associate Professor, University of North Carolina at Wilmington);
Wilfred Johnson, (Chair, Department of Curriculum, Howard University,
Silver Spring, MD); and John Nash, (Associate Director, Stanford
Learning Laboratory, Stanford University, Stanford, CA). Team members
reviewed Laboratory materials extensively both prior to the visit and as
a natural part of the flow of work during the site visit. The agenda
consisted of a mix of presentations around the signature works, visits
to schools, discussion periods with Lab staff and time for peer review
panel interaction and reflection. The signature works considered by the
peer review panel are *Design Support for School Improvement* and
*Technology for Teaching and Learning*.

The review panel brought many perspectives to bear on the work of the
Lab and was an effective mechanism for examining its work. However, in
some ways it is analogous to a group of people in a rowboat drifting
past an iceberg and attempting to evaluate not only the eighth they can
see, but also the seven-eighths below the surface.

**II. Implementation and Management**

**A. To what extent is the REL doing what they were approved to do
during their first three contract years?**

**[Strengths]{.underline}**

NCREL has conducted the activities described in their contract, and has
reported to OERI modifications made during the contract period. These
modifications appear to be clearly justified by events outside NCREL\'s
control, shifts in priorities supported by OERI, or to take advantage of
opportunities that arose. When changes have been made, care has been
taken to provide justification, as demonstrated through the filing of
updates (such as the FY99 Scope of Work Update to the Amendment to
NCREL's Technical Proposal). Timelines appear to be generally adhered to
and the REL seems to be on a trajectory that would allow them to
complete their planned scope of work during this contract period. For
example, the activities summarized in the 1^st^ Quarter Report of FY '99
go beyond the expected accomplishments outlined in the "Activities
Planned for the Next Quarter" in the 4^th^ Quarter Report of FY '98.

Panelists felt that NCREL has done a commendable job leveraging funds
and staff to support the scope of work as reported in their report of
total funding as of November 1998. Monies from sources such as the DOE
Math and Science Grant, DOE NCRTEC Grant, Department of Defense, and the
Chicago Public Schools, complement OERI funding and contribute to a
cohesive stream of work. Further, the Lab has done a good job of
incorporating additional grants into their center structure, thus making
it possible to provide a seamless approach to getting the work done, no
matter the funding source.

The Lab appears to have an established system of strategic alliances
with bodies such as regional CSSOs, state governors and legislatures,
and advisory networks (e.g., National Rural Education Association and
Urban Education Network). NCREL has also recently partnered with
institutions of higher learning around integration of their *Learning
with Technology* course in the preparation of preservice teachers as
described during our site visit. The strength of their networks and
strategic alliances was supported by the Board of Directors during our
teleconference, many of whom represent these partnerships. Throughout
the visit, there was evidence of strength and commitment, a climate of
thoughtfulness, and an everyday norm of cooperation and positive working
partnerships across the organization.

**[Areas of Needed Improvement]{.underline}**

In general, panel members were satisfied that the Laboratory is doing
what they said they would do during the contract period. There were two
issues raised by individual panel members. Excerpts from individual
reports appear below.

> *Th(e) work is hampered by the disconnect between the broad, sweeping
> focus of the national Lab work -- making connections in a meaningful
> way and informing the daunting issue of moving reform to scale -- and
> the five-year cycle of Lab work. Educational change at this level
> requires a longer cycle of effort and persistent focus.*
>
> *The teacher voice on the Board of Directors could be stronger. It is
> unclear how the Teacher Advisory Board identified in the Response to
> Technical Questions interfaces with the Board of Directors. In
> addition, the Board of Directors appears to have a rather uncritical
> stance toward the REL\... it was surprising to hear during our
> teleconference that Board members felt that there are [no]{.underline}
> problems with the Lab scaling up their work given that the Lab has
> identified the challenges in this work.*

**[Recommendations for improvement]{.underline}**

The panelists who raised these issues also made some suggestions about
improvement.

> *Explore options for partnerships and opportunities to extend the
> focus of the Labs work beyond the five-year cycle, with the five-year
> span in context of a larger strategic view of the issues of
> connectedness and scaling up. Deliberately shift focus for the last
> part of this contract to scale up issues of both delivering products
> and services out and issues of consumer use on a broad scale. Focus
> deliberately on the products and processes already developed or firmly
> underway, looking in great depth and with a researcher's eye at how to
> move them to scale by increasing their potential to connect with
> consumers in context of the practitioner's reality.*
>
> *The Board of Directors should help NCREL to think about issues of
> scaling up. It would also seem advisable to have the teacher advisory
> board directly linked to the Board of Directors.*

**B. To what extent is the REL using a self-monitoring process to plan
and adapt activities in response to feedback and customer needs?**

**[Strengths]{.underline}**

Several topics are included under this header. First, the structure of
the Lab is discussed, then the specific responsibilities of the centers,
and finally the specific quality assurances that guide the
self-monitoring process.

The panelists were intrigued and sometimes puzzled by the Center
structure employed by NCREL. Three panel members regarded the structure
quite positively.

> *NCREL has developed and implemented a coherent organizational
> structure that focuses on internal connectedness and, as such,
> supports the organization in completing work they are approved to do
> in context of connections and scale-up. The centers have many cross
> appointments, which do not allow any of them to function as an
> independent silo. Individuals may be identified with a center, but are
> affiliated closely with the Lab.*
>
> *The Lab is organized into four Centers that correspond to its major
> functions. Given the size of the total staff and the quantity and
> complexity of projects and tasks, some form of subdivision of the
> organization almost certainly is essential. This form seems
> reasonable. Care appears to be taken to have an overarching system for
> working that engages staff from all the Centers in projects as their
> skills and services are needed. While there is an obvious identity of
> staff with their function, the Centers do not appear to be isolated in
> either their sentiments or work.*

Several reviewers spoke warmly of the work of the Evaluation and Policy
Information Center (EPIC). Of the four centers, it is the one most
directly related to this question and its work is central to planning
and managing all Lab projects. Indeed, a major determinant of the
quality of work by the Lab is feedback from the monitoring processes.
There is evidence in project evaluation reports and later versions of
products or processes that the feedback from

EPIC is actually used to make improvements.

Quality Assurance procedures are well-documented in the NCREL Quality
Assurance and Evaluation document provided for the panel's review. In
this document, the REL has identified the criteria and procedures for
both external and internal reviews of products or written materials,
including web-sites (3/18/99 Quality Assurance for Web-sites documents.)
To be considered high quality, an NCREL product must: meet an identified
need; have a clearly defined purpose and audience and a feasible
dissemination and implementation plan; be delivered in a format and
presented in a style that is useful to clients; represent the best
available knowledge drawn from research and practice; adhere to high
standards for useful, ethical, valid and reliable inquiry applied
research and evaluation studies; promote balanced and positive
portrayals of diversity in the use of photos, graphics, and textual
components, and other components; and, conform to NCREL's editorial and
technical standards.

An overall unifying theory drives the self-monitoring process. This
framework is a customized TQM cycle with four major parts: Assessing
Needs and Setting Goals; Designing Plan and Evaluating Alternatives,
Implementing Plan, Evaluating Plan and Renewing Efforts. NCREL uses this
framework to drive the collection of data and the refinement of
activities

across the spectrum of work.

Additional mechanisms are in place to obtain feedback from internal and
external parties to improve performance. In addition to documenting the
review process for individual projects, NCREL undertakes a portfolio
review with its Board of Directors. This review allows the Board members
to review NCREL's entire portfolio of products and services, organized
by the four centers. The Board evaluates the work of NCREL based on
three overarching evaluation questions: Is NCREL doing the right work?
Is NCREL doing the work it said it would do? Is NCREL doing its work
well? The quality of NCREL products is also reviewed by clients via
response cards included with all NCREL products.

NCREL surveyed its clients around issues of satisfaction of key products
and services (March 1999 document). Overall, 87 percent of clients rated
the quality of the product/service as excellent or good. About 87
percent rated the quality of planning of the product/service as
excellent or good; 89 percent rated the timeliness of the
product/service as excellent or good. Among clients, 87 percent rated
the quality of the presentation of the product/service as excellent or
good. As outlined in the NCREL Quality Assurance and Evaluation
Document, external feedback is solicited from a variety of sources:
through partnerships with R&D organizations, regional feedback from key
client groups, and regional surveys of educators and agencies.

In addition, the panel heard directly from several people who have
worked in various capacities with NCREL that the staff is receptive to
feedback and very willing to make changes once they appreciate the need
for them. For example, one higher education representative said that she
had told NCREL staff that the Lab is not sufficiently well-known in the
regional higher education community and that, as a result, she has been
given a mandate by the Laboratory, to find ways to create greater name
and reputation recognition through giving workshops, making

conference presentations, and the like.

**[Areas of Needed Improvement]{.underline}**

Comments concerned the Lab\'s structure and one panelist questioned the
data-collection process.

> *Given that projects cut across centers, it would seem important to
> have very well-established systems of communication. For example, the
> theme of engaged learning was identified in the presentation of the
> Technology Center has one of the most powerful "big ideas" to emerge
> from their work. Yet, when staff from CSCD were asked about the
> indicators of engaged learning, they were unable to respond.*
>
> *The organizational structure, while flexible may lead to stress
> within the organization as staff begin to have too many masters;
> potentially wear too many hats. Diffusion of treatment is an issue.
> Perhaps a new center devoted to training and delivery could increase
> fidelity of transfer, but this would be very expensive.*
>
> *A large part of data-collection regarding effectiveness or quality of
> products or services depends on surveys or questionnaires of various
> forms. Many of the items included in (the materials) are prone to
> distortion error from "social desirability" or "generosity" in
> responding. It is not clear that these two forms of error are taken
> into account routinely in either the design of the data-collection
> systems or in analyses and interpretation of responses. A significant
> proportion of the survey materials and approaches depend on voluntary
> response. Some of these have low response rates and are prone to
> returns that represent particular segments of the recipients --
> sometimes positive and sometimes negative, depending on the original
> relationship with the Lab.*

**[Recommendations for improvement]{.underline}**

The panelists had several ideas for changes or modifications to the
Lab\'s structures and the relationship among them.

> *Rethinking the structure. This, I think, should be based on three
> questions: (1) does the structure reflect the need to \"put the pieces
> together\"? (2) does the existence of a center called Scaling Up
> actually contribute to the success of doing scale up? (3) does the
> center structure really reflect the priorities of the Lab, including
> technology and professional development?*
>
> *The structure issue is complex because it reflects not only the
> priorities of the Lab, but the way they see themselves. My impression
> is of a relatively traditional hierarchical organization, which lacks
> the elements of cross-fertilization of ideas that create a learning
> community. Surely, if we are convinced that learning communities are
> the optimal way to undergird organizational change, it would be
> beneficial to reflect that conviction in the structure of an
> organization dedicated to facilitating organizational change.*

One panelist suggested some new approaches to the data collection
process.

> *Account for "social desirability" and "generosity" in responding for
> all measurement processes that are prone to such error\...Among
> methods might be triangulation or adoption of other forms of
> measurement\... It appears that some of the resources given to
> "inspection" of processes and "results" could be used better if the
> original designs represented principles more grounded in technical or
> theoretical models.*

**III. Quality**

**A. To what extent is the REL developing high quality products and
services?**

**[Strengths]{.underline}**

The quality of the products and services provided by NCREL is generally
high, although somewhat uneven. (One reviewer commented that the wider
the intended dissemination and the more the product focused on
technology, the higher its quality tends to be.) Certainly, there is a
wide array of printed and web-based materials addressing a broad range
of interests and target audiences. These materials include special
papers dealing with topics of immediate or critical interest, regular
publications covering general topics of interest, series publications
dealing with single topics, publications that focus on particular
audiences of policy-makers, administrators, trainers, teachers, or
scholars, training materials, and web materials. for general information
and training.

Certain products produced by the Lab seem to have \"star\" quality and
attracted particular interest among panel members. Similarly, the
process of development was seen as a real

strength by several reviewers.

> *In the Every Child Can Succeed program with Chicago Public Schools,
> the literature on literacy and language acquisition of children
> speaking English as a second language is at the fundamental level and
> is skillfully woven into the design of the implementation. Much to
> their credit, NCREL has used this research to build teacher knowledge
> and understanding of language and literacy acquisition that is a key
> piece of professional development.*
>
> *The publication Plugging In is a successful, widely disseminated
> product devoted to choosing and using educational technology. The
> indicators of engaged learning are one of the real contributions that
> NCREL is making to the challenge of integrating technology into the
> curriculum.*
>
> *In the area of print materials, NCREL has an impressive library of
> works. NCREL's print library runs the gamut from draft academic papers
> to bound training materials, to glossy, magazine style reports. The
> overall quality of these materials, as reviewed by this panelist,
> appears to be very high. In cases where I was able to obtain
> background information on the development process for certain
> materials, such as the leadership training modules for technology,
> developed in conjunction with the Chicago Academy for School
> Leadership (CASL), the Lab appears to have gone to great lengths to
> solicit key stakeholder input in the development process.*
>
> *The concept of "engaged learning" (is) a powerful conceptual
> framework for reconceptualizing high quality teaching and learning on
> a broad scale. The engaged learning concept encapsulates changes in
> both the role of student as active participant in learning and in the
> role of the teacher as coach and facilitator rather than as
> disseminator of wisdom. This concept appears to be strongly infused
> into the vernacular at the teacher, school, district, and state
> (Illinois) levels, with indication that the concept is moving
> throughout the service region -þ a major positive factor in the
> scale-up potential of the products and services being created. It was
> evident at the sites visited and through focus groups and interviews
> that teachers and administrators actively use the construct in
> rethinking learning and teaching in their daily work. It is serving as
> a powerful organizer for core level school change. This paradigm shift
> is phenomenal: it reaches directly into the classroom with a powerful
> metaphor that makes major changes in conceptualization of learning and
> teaching.*
>
> *Using the concept of engaged learning, NCREL has developed a number
> of strong products and processes. CTLC and CSCD have focused in
> particular on the teachers' role in delivering instruction and in
> understanding how students use that delivered instruction to learn,
> with products such as the Engaged Learning Safari offering educators
> clear benchmarks for understanding engaged learning and with potential
> for scale-up inherent in the product.*
>
> *The Engaged Learning Safari teaches teachers research to practice
> techniques and how to infuse these results into the curriculum þ
> mathematics, science, literacy and the inter-disciplinary learning. I
> especially want to identify one product that I thought was stellar:
> Gateway Concepts. This product is based on empirical evidence of need.
> It is standards-based. It is user friendly and is incredibly
> generative in terms of where a teacher can take it. It is clearly a
> vehicle for professional development that supports engaged learning in
> classrooms. Very impressive! I also want to recognize Learning with
> Technology for its contributions to teacher professional development
> in technology, yes, but also for its power in reshaping teachers'
> thinking about their practice. The message in the* *program is that
> technology is a tool to engage students in learning. I found it
> fascinating that teachers begin to really "get it" -- what
> problem-based learning is -- when they use a tool that is a natural
> facilitator!*
>
> *A set of "captured wisdom" has been developed and presented in CD
> format. These case demonstrations appear to have been carefully
> selected to illustrate the applications clearly and to demonstrate how
> they actually might be adopted or adapted. The production quality of
> the video and audio are good, and the application program\...makes the
> materials easy to access and to move within.*
>
> *The Captured Wisdom (cd) permits teachers to look at other learning
> situations and assist them in integrating best practices into their
> classrooms.*

**[Areas of needed improvement]{.underline}**

> Although, as the preceding paragraphs attest, the panelists thought
> that much of the work of the Lab is of very high quality, most of them
> did find some areas for improvement.
>
> *The concept of "engaged learning" is not as highly visible in the
> products, processes, publicity, and dissemination processes that will
> let the Lab move the concept to scale. The concept is there in the
> materials and processes, but it is not "front and center" as it seems
> to be in the field. There is opportunity lost for making this amazing
> paradigm shift on a broad scale and for creating positive ethos
> related to concepts of constructivist learning and radical change in
> teaching delivery systems as well as student roles. Pulled to the
> front\...the concept of engaged learning could be a powerful tool for
> scaling up\...*
>
> *\...(T)he Pathways server has a number of quality issues\...
> addressed. In attempting to serve the needs of a broad audience, the
> product fails to maintain high enough quality for the empirical
> research world or to be user-friendly enough to serve teachers in
> their world, and it has not kept pace with changes in the technology
> field as well as with issues of effective web-based communication
> strategies. Specifically, the research articles in particular have
> definite limitations that make them unfriendly both to researchers and
> to practitioners: Many contain citations in context of text but lack a
> reference section at the end of that text. Many of the literature
> reviews have obvious gaps and fail to cover the literature thoroughly
> or evenly. The reviews are written with authority and certitude rather
> than with the tentativeness and conflict that is a natural part of the
> emerging nature of knowledge construction and thus some of the
> research reviews tends to take on an editorial slant, The overall
> writing styles are formal and unfriendly to a non-researcher audience,
> and best practice exemplars and empirical research is blended rather
> than clearly delineated, sending an overall message of authority and
> expertise when it is not necessarily supported empirically.*
>
> *Research findings in depth remain relatively inaccessible to people
> in the field, with the exception of broad translations in the form of
> reviews\...Although there are bibliographies and reference lists
> available on the web-site, hand-delivered through the resource center,
> and incorporated as part of written materials, the information in
> these listings remains virtually inaccessible to consumers unless they
> expend considerable time, energy, and resources to filter through the
> list and then go out to obtain the sources.*
>
> *Making Good Choices appeared to have some limitations in terms of its
> implementation. A question such as: "How effective is your school's
> curriculum?" seems straightforward but is actually at the heart of
> systemic reform. The tool may be a wonderful springboard for
> professional development and rich conversation among school faculty
> and administration. I am not sure that it can be used that easily to
> make good choices about models of comprehensive school reform without
> a facilitator laying a lot of ground work.*
>
> *In reviewing the literature base for implementation of Every Child
> Can Succeed, I failed to see reflected important findings from seminal
> works in literacy that extend beyond those important domains described
> in the previous section. For example, the 1998 NCR publication
> Preventing Early Reading Difficulties emphasizes the need for an end
> to the "reading wars" and a polarization of approaches. Young children
> need a balanced program. The findings on phonemic awareness and the
> need for systematic and sequential opportunities for students to work
> with alphabetic principles to develop grapho-phonemic, as well as
> semantic and syntactic cueing strategies, are not included in works
> cited. This is particularly of concern when working with K-3 schools.
> Moreover, while I think the STRP has strength as a program for
> improving reading, I do not believe it is a balanced program for young
> readers, K-3.*
>
> *I heard that EPIC is looking over notes from the site work, but it
> isn't clear to me that the design for evaluation has been built in as
> a key component of the work to scale up. What is the level of
> implementation of the various interventions? Especially if multiple
> interventions are introduced, it is critical that fidelity of
> treatment be monitored to understand potential student outcomes.*
>
> *The Lab has made a heavy commitment to a process of "co-development"
> of its processes and services with its clients. While that approach
> has much to recommend it, there are some serious potential weaknesses
> that can result from implementation that does not have some deliberate
> method for introducing and assuring use of the best technical and
> theoretical information into design decision-making. This appears to
> be an important point of quality "slippage" in Lab projects. None of
> the responses made by Lab staff during the site visit revealed an
> overarching technical or theoretical approach to co-development.*
>
> *While the Lab seems very successful in responding to requests for
> services, there does not appear to be an overarching strategy for
> initiating services for those who are not inclined to request it.*
>
> *Although there are bibliographies and reference lists available on
> the web site, hand-delivered through the resource center, and
> incorporated as part of written materials, the information in these
> listings remains virtually inaccessible to consumers unless they
> expend considerable time, energy, and resources to filter through the
> list and then go out to obtain the sources. Lists of sources alone are
> inadequate in communicating research findings, and there are broad
> web-based and electric resources readily available that are not part
> of NCREL's web sites or services.*

**[Recommendations for improvement]{.underline}**

Panelists made suggestions they believe will improve the product
quality.

> *Use the concept of Engaged Learning as a major organizer for products
> and processes. This is a powerful construct that has potential to make
> major impact in the field, and it needs to be "pulled front and
> center" and used as an organizer for the web page, as a conceptual
> framework that links NCREL products and processes into a seamless,
> readily-understood conceptualization of overall change in classrooms,
> schools, states, and regions\...(U)se it more deliberately and
> extensively.*
>
> *Capitalize on the potential strength of Pathways, even though it is
> not a "new" product. Pathways has strong name recognition and numerous
> visitors almost a "brand name" strength -- and it offers strong
> possibilities for maintaining cutting edge information. First, clearly
> identify your audience and create top quality work targeted to that
> group\...It might be strengthened by tailoring it to a more narrow
> audience such as curriculum planners and students in pre-service
> courses, curriculum planners, and others\...the research reviews need
> to be comprehensive and current, and citations need to be included
> with more information readily available on each article. All major
> perspectives should be included, even where there are disagreements
> and knowledge gaps\...*
>
> *Consider supporting references and bibliographies with another layer
> of available information. For example, the Resource Center might
> create annotated bibliographies for all sources cited using summaries
> of findings such as those found in ERIC, rather than simply listing
> sources.*
>
> *While STRP is a program that I might recommend for grades three and
> above, I would strongly encourage NCREL to revisit the decision to
> broker (this) program to K-3 schools\...A balanced program for early
> reading instruction must be more inclusive than the scope of this
> program.*
>
> *Develop the initial designs for Lab processes that represent more
> powerful technical or theoretical models than ones now used, and adopt
> a strategy for "scaling up" that actually represents what is already
> known in that field. Related bodies of information are well
> established in the areas of, among others, innovative and incremental
> change, instructional-systems development, small-group structures and
> change, role theory, motivation theory, some economics theories,
> "critical mass" theory, and culture and climate.*
>
> *NCREL is moving from print documents to audio-video as shown in
> utilization statistics compiled by NCREL. For example, in 1997, the
> number of print products distributed to clients was 131,093; in 1998,
> that number went down to 107,686. These statistics were matched by a
> corresponding increase in the distribution of audio/video/cd products.
> However, documentation is still not informative enough\...while they
> can document the number of hits on the web-site, they cannot document
> the number of people these hits represent, nor the usefulness of the
> web-site\--except through individual testimony.*

**[Recommendations for Improvement]{.underline}**

The Lab is already taking steps through an on-line survey to address
issues around their web-sites. As to their surveys and assessment
instruments, they are hamstrung in a sense by needing to provide some
kind of help for schools in their region without having the resources to
do the job in any depth. The panelists suggest they revisit some of
their tools and match them more closely to the capacity of the people
who will be using them.

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

**[Strengths]{.underline}**

In the 1995 Gallup Survey of a random sample of educators in the NCREL
region, respondents were asked to rate the usefulness of services and
products received from NCREL. A little less than half (45 percent) rated
violence and substance abuse prevention and early childhood (45 percent)
as very useful. Other areas rated as very useful include professional
development (32 percent), curriculum (37 percent), and assessment (33
percent).

Lab staff appears to do everything they can to ensure that their
products are useful, ranging from careful needs-sensing as the basis for
developing products and services to modifying and customizing products
and services based on feedback from the field. Panelists are convinced
that the work of the Lab is very useful to the field and, to the extent
it could be

judged, they were positive about evidence of use.

> *One major external partnership, with which the panel became quite
> familiar is that with Chicago Public Schools. Replying to an RFP in
> 1998, NCREL became an external partner with 11 elementary schools,
> which were on probation for low achievement. The usefulness of the
> work done by NCREL staff in this role is attested to by the fact that
> six of the schools are now off probation because of improved
> performance on standardized tests. Two of the partnered schools were
> recognized by the district for the \"most improved\" scores \-- one in
> writing and one in science. NCREL also supported five of the schools
> in developing CSRD grant proposals and all were selected as grant
> recipients. NCREL is helping to develop a learning framework at
> Brentwood, another of its CPS partners in which technology provides
> support to instruction, and is not merely an end in itself; NCREL's
> presence in the school has fostered an attitude of learning and
> exploration with technology among the staff and students. Also, at
> Brentwood, email is used as a questioning method in lieu of classroom
> visitations, due to time constraints teachers have. Evidence suggests
> this is an effective way to dialogue with teachers regarding
> practice.*
>
> *A key feature of the efforts to improve instruction in the region is
> the focus on "engaged learning". For that purpose, the Lab has
> developed training programs and materials that are extremely
> "accessible" to teachers -- making big changes in instructional
> practice but in ways that do not appear to be so difficult. The
> programs are efficient in amount of time required, yet still prepare
> teachers to begin using the approaches in important ways.*
>
> *The Lab has developed several web-sites that are emerging as key
> sources of information on topics important to the field of education,
> generally, not just in the region. These sites are cross-linked among
> themselves and with a number of other related sites. They are
> intuitively appealing and easy to access and use for information
> access and for more in-depth study of content.*
>
> One of the strongest delivery systems for Lab products and processes
> in terms of consumer access and desirability -- and thus potential for
> moving to scale \-- is the use of courses as a delivery system. NCREL
> harnesses the power of the course structure to make products and
> services useful and used by customers in two ways: 1) the Lab has
> created, and continues to create, courses that can be used for
> in-service training and, and 2) the Lab has formed working
> partnerships with teachers in higher education so that Lab products
> and processes can be used in context of pre-service education courses.
> It is obvious from the quarterly reports that NCREL is capitalizing on
> this vehicle and is designing more courses as vehicles for moving
> products, processes, and information sources to scale.

**[Areas of needed improvement]{.underline}**

The work with CPS was an area in which panelists saw need for
improvement. The panelists also identified name recognition, gaps in
service, fidelity of implementation, sustainability, and the specter of
obsolescence. Individual reports are cited below.

> *With the Chicago Public Schools, example, NCREL is providing a
> package of services to 11 sites. The extent to which they can actually
> bundle services is dependent on: (1) the requirements of the contract
> they signed with the Chicago Public Schools, (2) the extent to which
> the frontline NCREL staff is familiar with all the services from all
> parts of the Lab, and (3) the extent to which the schools are
> receptive to a variety of services.*
>
> *I have two questions about the intensive sites. Are the lessons they
> are learning generalizable or are they getting stuck in a school by
> school approach to systemic reform? Are the intensive sites black
> holes into which they will continue to pour resources without
> concomitant reward in adding to the knowledge base?*
>
> *The Lab has been successful in serving people and systems in which
> conditions support change. It has not been so successful in developing
> that readiness throughout the region (i.e., in states other than
> Illinois and Ohio) and strategically providing services to develop
> readiness and then the support needed to follow through to adoption.
> There remain significant "gaps" in the region with sparse service and
> intensive impact from the Lab.*
>
> *An area of concern is the sustainability of NCREL's impact after it
> pulls out of a project. NCREL staff spoke about the challenges
> associated with the movement of products from the Lab to the field.
> They can point to specific cases where the product was introduced to a
> region through an SEA and "took off." They also pointed to other
> states or districts that had undergone the same training, but bore "no
> fruit."*
>
> *An even greater challenge is the issue of fidelity. Again, this is an
> issue with which the Lab wrestles and has tried to maintain some
> control of the fidelity through the trainer of trainer model.*
>
> *There is a need to specifically identify those schools that have
> probationary status\...but may not have the necessary funding to pay
> for\...services or materials to improve their status.*

**[Recommendations for Improvement]{.underline}**

Individual panelists identified some ways to improve areas they had
targeted. Suggestions appear below.

> *As for some other issues addressed herein, there is a need for
> overall strategic view of how the mission of the Lab will be served in
> all areas of the region, especially to the level already apparent in
> some states.*
>
> *There needs to be a more intense effort targeting schools in need of
> intensive implementation. These schools should definitely be targeted
> by including and assessing the building administrators regarding their
> impressions as to what they need to engage their teachers and students
> in teaching and learning. Each school staff should be involved in the
> specifics of what is involved in curriculum development and teaching.
> Training, wherever it is needed should include all members as a team.
> Mentors may be needed to model best practices.*
>
> *As the Lab ponders scaling up, issues of utility, fidelity, and
> sustainability naturally arise. Are clients using the products? How
> are they using the products? Will they continue to use them when NCREL
> is no longer involved? I think the lessons learned from the Training
> of Trainers model in Learning with Technology are important. True,
> NCREL wants to get products out the door. But, unless the products
> tied to structures to which teachers have some affinity and there
> is\...shared investment in improving schools, the products probably
> remain under-utilized. NCREL has many strategic partnerships with
> solid track records that are clearly focused on reform. I would
> encourage NCREL to capitalize on and exploit those relationships
> (e.g., urban systemic alliance; rural systemic) and to be "selectively
> strategic" in their dissemination efforts.*

**B. To what extent is the NCREL focused on customer needs?**

**[Strengths]{.underline}**

NCREL places a clear emphasis on establishing need for their product or
service and has developed an issues scanning system (Panel Presentation,
April 28, 1999; Issue Scanning Development Document, December 16, 1998)
that surveys needs in the field from a variety of sources, including:
internal system, Resource Center requests, web-site hits, requests for
proposals in the field, and specific requests for services. Individual
reviewers chose to focus on different elements of the NCREL apparatus
for focusing on customer needs.

> *The NCREL Resource Center handles all incoming field requests for
> information. As a special library within the DuPage Library System,
> the Resource Center is equipped to direct callers to appropriate
> materials, either developed by the Lab or by others. The Resource
> Center currently contains over 9,000 holdings.*
>
> *In terms of setting priorities in accordance with customer needs,
> NCREL is adept at this, particularly in the development and delivery
> of custom, field-based interventions. At Cardenas Elementary School,
> the principal noted that NCREL staff work hard with the school and are
> considered more "coaches" than "consultants." Further, NCREL worked
> with Cardenas to determine the best program to address their reading
> and math deficiencies. Surveying the field, the school looked at the
> "Success for All" model, among others. Through their dialogue with
> NCREL, however, the teachers at Cardenas determined that the materials
> they had in place were fine. They decided that perhaps they weren't
> using the materials properly. It was around this decision that NCREL
> began the process of crafting an intervention.*
>
> *NCREL is explicit in its mandate to co-develop projects with field
> sites. Given hat is known about the glacial pace at which
> organizations change, early buy-in form stakeholders in the field is
> crucial. Some may argue that the co-development process dilutes the
> impact that a "purer" research based, clinical model may bring.
> However, any intervention that is "layered" upon existing
> organizational structures may fail without some level of
> co-development.*

**[Areas of needed improvement]{.underline}**

In this area, panelists identified some complex and deeply-rooted
issues. These are reported below via excerpts from their reports.

> *A difficult task in determining needs from expressions is determining
> the need from the expression (Yes, that says what I meant.).
> Expressions usually are symptoms of something that may, or may not,
> actually be a need. The critical responsibility of the service
> provider is to determine the "actual" need that triggered the
> expressions. Here the issue is similar to the task of determining
> "root cause" of poor process performance in efforts for process
> improvement.*
>
> *Outreach is often the fulcrum for promoting desire in a client who
> knows that there is a need, but who does not know or have the time for
> analyzing what the specific or general need(s) are.*
>
> *The "lessons learned" from intensive sites have to be used to inform
> the work on scale up. Therefore, a major recommendation focuses on the
> need to make the link between the intensive work and scale up. While
> the intensive work does respond to needs from the field, I believe
> that those doing it have already learned some important lessons. One,
> often where a school or outside agency [thinks]{.underline} the need
> is, isn't really where the need is. As in Senge's model of
> organizational change, where we think we need to "tweak" the system is
> often not the starting point. In work with CPS, NCREL learned that in
> order to address achievement in reading and mathematics, often other
> systemic issues also have to be addressed (e.g., supervision of staff;
> professional development; home-school relationships; classroom
> management; school climate).*

**[Recommendations for improvement]{.underline}**

The panelists urged the Lab to rethink some of its activities, as shown
below.

> *Use some of the experiences in the schools where there is intensive
> work to assess organizational\...and other conditions that reveal
> needs at a more "fine-grained" level than the broad categories usually
> expressed in the surveys.*
>
> *There needs to on-going "soul-searching" within the organization
> about the costs and benefits associated with responding to the many
> requests for services and products they receive. Clearly, NCREL must
> be responsive, but it must also be strategic.*
>
> *NCREL wants to get products out the door. But, unless the products
> are tied to structures with which teachers have close affinity and
> some basis for shared investment in improving schools, the products
> probably remain under-utilized. NCREL has many strategic partnerships
> with solid track records that are clearly focused on reform. I would
> encourage NCREL to capitalize on and exploit those relationships
> (e.g., urban systemic alliance; rural systemic) and to be "selectively
> strategic" in their dissemination efforts.*
>
> *(Be more) specific about how change will be sustained after field
> projects are over.*

**V. Outcomes and Impact**

A.  **To what extent is the REL\'s work contributing to improved student
    success, particularly in intensive implementation sites?**

> **[Strengths]{.underline}**

As part of the overall REL Performance Indicator process, NCREL surveyed
its clients about the impact of key products and services on their
instructional practice (March 1999 document). Overall, 74 percent said
that the product/service increased their awareness of important new
skills and knowledge, 53 percent reported they used the product/service
to inform decision making and planning, and 50 percent said they used
the product/service to change or enhance the quality of professional
practices. Panelists noted the difficulty the Lab, or any such external
partner, has in documenting attributable \"student success\".

> *At the intensive sites in the CPS system, the work of the Laboratory
> as an external partner was certainly one of the factors (and likely
> the major one) that resulted in six of the 11 schools being taken off
> probation because of the improvement in their test scores. This is one
> of few areas in which NCREL is working where they have the opportunity
> to demonstrate student success.*
>
> *While student outcomes (learning and achievement) are the ultimate
> (or distal) outcomes, it is often also helpful within the context of
> school reform to consider proximal outcomes: change associated with
> the practice of teachers and administrators. We heard from teachers
> and administrators that work with NCREL has changed how teachers teach
> and how instructional leaders do their job. The impact of NCREL's work
> in terms of student, or distal, outcomes is emerging. Achievement data
> supports positive change in student achievement in six of eleven
> Chicago Public Schools with whom NCREL worked (1997-98). Other
> outcomes are noted anecdotally by teachers in areas such as student
> engagement in learning and improvement in student attitude toward
> learning.*
>
> *At one school, the principal has become an instructional leader and
> is visiting classes. She herself credits NCREL for showing her that
> she is the instructional leader. Teachers talked about the benefits of
> the professional development they are receiving and how teaching
> Everyday Math has transformed their instruction in mathematics.*
>
> *At another school, both the teachers and principal enthusiastically
> described how work with NCREL has helped teachers reflect on
> instructional practices and collaboration. Teachers are designing
> units that engage students in inquiry and exploration of a topic to
> develop deep understanding of key concepts. Teachers are\...asking
> students to take an active role in their own learning. One district
> administrator spoke very positively about NCREL's influence on his
> thinking and the district's stance toward engaged learning and
> technology.*
>
> *(T)he bulk of the products and services created by the Lab are
> designed for consumption by trainers, or by teachers. These services
> and products, in the form of training, technical assistance, and print
> products (including web-sites and CD-ROMs) are often deployed in a
> setting that is rich with complementary (and sometimes competitive)
> reform efforts. This makes assessing impact more difficult, let alone
> understanding the relationship of NCREL products to increased student
> learning.*

**[Areas of Needed Improvement]{.underline}**

The panel was disappointed, although somewhat sympathetic, given the
circumstances cited above, to read that on a survey (March 1999) of
clients\' perceptions of the impact of key services/products on student
learning, only 23 percent of respondents said that the service/product
positively affected student performance. Some specific areas that
panelists felt might contribute to this concern are reported below.

> *I wondered about the unevenness of technology integrated into various
> intensive sites. If NCREL's specialty area is technology, shouldn\'t
> it be integrated into work across sites? When asked about this, NCREL
> staff replied that they felt it less needed since they had staff
> on-site. This, however, seems to beg the question. If technology is a
> powerful learning tool (which I believe it is) and NCREL has
> considerable expertise in this area, shouldn't it be a "value added"
> in schools working with the Lab?*
>
> *Within the area of impact of student learning, the Lab must be able
> to demonstrate this on a wider scale.*
>
> *It appears that NCREL staff who are participating in the
> interventions (at the intensive sites) are also serving as primary
> documentation collectors within those intensive sites. Thus, data is
> being collected on the site participants themselves, but there is
> little empirical evidence being collected on the researcher as
> subject.*

**[Recommendations for improvement]{.underline}**

Some panelists made specific recommendations. One made a plea for more
and enhanced services.

1.  *Reevaluate the integration of technology into services provided to
    schools*

2.  *Design professional development services and products around the
    mentoring process for administrators\...*

3.  *(Commission) third-party evaluation of student achievement in sites
    where NCREL has made significant contributions to reform*

> *This is a tricky area for the Lab to address. In order to assess
> student learning on a broad scale, the Lab will no doubt be forced to
> analyze state-wide standardized test scores. The types of teaching and
> learning that the Lab promotes may indeed raise scores, but the direct
> link is nebulous, and literature suggests that standardized tests are
> a poor measure of the kind of learning occurring in engaged learning
> environments.*
>
> *Study researchers themselves in context of practice. The work at
> intensive sites offers perfect opportunity for the research world to
> gain knowledge on researcher interventions in context of practice and
> whole school change. Consider having external researchers study NCREL
> staff on site to measure the impact of NCREL interventions on
> intensive site improvement and changes in student success that can be
> attributed to NCREL interventions.*
>
> *Assist those underserved schools in writing proposals for funding or
> for partnerships. Direct attention to them so as to make them healthy
> institutions for children to learn. There should also be other
> resources to support collaboration in the schools currently receiving
> services, in the form of professional libraries and planning session
> along with all of the school staff.*

**B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?**

**[Strengths]{.underline}**

NCREL has been asked by its regional constituencies to play a major role
in comprehensive school reform. The Center for Scaling Up in conjunction
with other centers at the Lab responded to the first request from
Wisconsin upon passage of the Obey-Porter Bill and worked with
localities to obtain CSRD funds. Panelists referred to several instances
in which the

work of NCREL was instrumental in the implementation process.

> *NCREL worked with Illinois to try and develop a set of
> recommendations to address the following problems: (1) a drop in
> reading scores while math scores remained fair and consistent, (2)
> identification of the characteristics and the quality and consistency
> of the reading program, and (3) extent to which instruction,
> curriculum, and reading assessment is aligned. From this project,
> NCREL developed a core set of processes to guide districts and states
> in monitoring and changing a reading program.*
>
> *(T)he Center for Scaling Up delivers many products and services in
> ways that make them accessible to a wide audience. Other items of note
> in this area: The Lab serves as an information resource for states and
> localities through it's Resource Center and through it's policy
> documents; The Lab's Pathways web-site includes very useful
> information about comprehensive school improvement strategies; The Lab
> demonstrates a tendency to work "with" partners rather than "for"
> partners as evidenced by its wide network of partnerships across its
> seven state region.*
>
> *The Center for Scaling Up provides an explicit avenue for bringing
> successful projects to scale.*

**[Areas of Needed Improvement]{.underline}**

Panelists primarily targeted the Center for Scale Up \-- one commenting
that the very name of it suggests that they have the answers about
scaling up successfully, when in fact they are seeking the answers
through multiple approaches, as described by one panelist.

> *Various obstacles in rolling out services on a larger scale have been
> identified by NCREL and their constituents whom the panel interviewed.
> NCREL staff spoke about the difficulties of working with intermediate
> state agencies in disseminating training to their teachers. In the
> case of Learning with Technology, the intent was to train these
> intermediaries to be the trainer of trainers. Unfortunately, a
> significant portion of those trained have not gone on to do subsequent
> trainings.*

**[Recommendations for Improvement]{.underline}**

Panelists had some recommendations around the scaling up issue.

> *I would like to see the Lab staff look again at the model they have
> devised. Would it be useful to back off from trying to do scale up and
> spend more time trying to understand it theoretically or through some
> very small studies of it?*
>
> *Critical analysis of what happens to the effectiveness of project
> work once it is brought to scale and training is conducted by other
> agencies.*
>
> *Follow-up studies (perhaps several case studies) on the "life" of a
> product once it is brought to scale and "set free" in the educational
> marketplace*

**C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?**

**[Strengths]{.underline}**

NCREL has very high visibility within its region, at least among
administrators. An estimated 80 percent of district superintendents and
67 percent of principals in NCREL's region have heard of NCREL (Gallup
Survey, 1995). One reviewer commented:

> *NCREL has a strong focus on the use of technology to support
> instruction and learning\...the Lab focuses its energy on harnessing
> the strength of technology for design of user-friendly products and
> processes, for dissemination on a broad scale in a cost-effective
> manner, and for broad, open access to information through technology.*
>
> *The Lab has extensive projects within its region, in partnership with
> others outside the region, and in partnership and coordination with
> other Labs. The DOD Education Activity for professional development in
> technology applications sends a strong message of support for the
> Lab's sound national reputation.*
>
> **[Areas of Needed Improvement]{.underline}**

Based on the results of the 1995 Gallup Survey of a random sample in the
region, only 21 percent of teachers said they had heard of NCREL. The
teachers may know the Lab's work under other terms such as through the
popular work Plugging In, but they do not necessarily link that work
with NCREL. The panel also heard from interviews that there is some
tendency for states not contiguous to NCREL to feel somewhat
marginalized.

**[Recommendations for Improvement]{.underline}**

Reviewers focused on the visibility issue.

> *NCREL appears to work hard to gain visibility. As the use of
> technology increases in schools, I am optimistic that NCREL's
> viability will increase as well.*
>
> *Explore opportunities to expand name recognition among teachers. This
> "brand name" associated with quality products might offer a valuable
> tool in the process of scaling up, since name recognition is a factor
> in advertising and distribution.*

VI. **Overall Evaluation of Total Laboratory Programs, Products, and
    Services**

The panel\'s overall evaluation of NCREL\'s Laboratory programs,
products, and services is that is an extremely hard-working, creative,
and capable organization that is generally healthy and quite productive.
As in any organization, however, the question must be asked, is the
organization able to leverage its human resources to the maximum? In
terms of products and services, the Lab produces some outstanding work
and some that deserves some careful reassessment. Based on our
interviews with Lab staff, they are mindful of trying to understand what
works in their products and services and what doesn't. For example, they
have unpacked their successful product Plugging In to figure out why it
works so well (apparently because it is not just about technology, but
about the context and placement of technology) and to learn from it.
Similarly, when they have experimented with the train the trainer
approach, they have been attentive to what makes it work and why. In
sum, the Laboratory does a lot of quality work and when they meet with
setbacks, they work diligently to understand why and to do better next
time.

**VII. Broad Summary of Strengths, Challenges, and Recommendations**

**[Strengths]{.underline}**

NCREL is an organization that appears to be able to attract staff member
who are articulate, enthusiastic, and very committed to their work. In
fact, the present and "pending" executive directors very clearly
identified the staff as a major resource of the organization. This is no
minor accomplishment and I think NCREL and its leadership should be
congratulated. Since many of its strengths have been documented earlier
in this report, the bulleted list below is at attempt to reiterate those
that particularly impressed panelists.

-   They have forged strong partnerships with external reform agencies
    and leverage monies very effectively from a variety of sources to
    supplement the OERI contract to build a cohesive and sustained
    stream of work.

-   NCREL's willingness to "get into the trenches" speaks volumes to its
    commitment to understand and improve schools.

```{=html}
<!-- -->
```
-   Intensive field sites are a real-life context for research on
    practice.

```{=html}
<!-- -->
```
-   NCREL produces many high quality publications and products.

```{=html}
<!-- -->
```
-   The QA cycles in place are paramount and clearly evident.

-   NCREL has built an impressive reputation among leaders at state and
    district both as a school reform agent, and a leader in technology.

-   Their links with schools of education in the region and the inroads
    they have made into preservice teacher preparation are exciting.

**[Challenges and Recommendations]{.underline}**

Many of the most salient of these have been discussed earlier in this
paper. Here, the focus on a few issues that come with strong feelings
from various panelists.

> *The issue of scale up is a thorny one. NCREL can make a tremendous
> contribution to this literature and to the systemic school reform
> movement in this area. After spending several days on site, I am
> wondering if the "collective wisdom" of the organization could be more
> utilized in regard to scale up. From the outset I have questioned why
> NCREL chose to create a center called "Scale Up" rather than have that
> function embedded within other structures. I think the Lab is making
> efforts to ensure connection between centers, but I question if the
> current organizational structure contributes to the understanding of
> this issue.*
>
> *Revisit the organizational structure of the Lab. The 4-phases of
> implementing projects require major interdepartmental efforts; NCREL
> itself has described how different departments interface around one
> project. What I heard from the presentations, however, was a process
> by which centers and departments became involved in projects in a
> sequential, rather than an interactive, manner.*
>
> *As one contemplates bringing services and products to scale, it would
> seem critical that evaluation be ongoing and it seems this linear
> sequence may not harness the collective brain power of the
> organization. As a panelist I would not presume to prescribe an
> organizational framework, but I do recommend a fresh look at how NCREL
> operates as a learning community.*
>
> *Intensive sites are wonderful opportunities for developing
> understandings of how change occurs. On the other hand, they can be
> incredibly labor, resource-intensive sites that bear very few results
> that can be used to scale up or to develop procedural knowledge about
> change. I believe that there is a danger for coaches and those working
> closely with the schools to become immersed in the technical
> assistance aspects of the work and have little time for the
> documentation, evaluation, and reflection that is critical. This is
> more of a caution than a deficiency.*
>
> *The overall focus question I would pose is "To what extent are we
> using all the knowledge that is extant in this Laboratory to inform
> the direct services we provide to clients?\"*

The panel thanks the staff of NCREL for its hospitality and its cheerful
tolerance of the disruption of regular work to meet our needs. We
particularly appreciate the willingness of every staff member to face
thorny issues and discuss difficult questions honestly and thoughtfully.