Archived Information

**Interim Evaluation of the Southeastern Regional Vision for Education**

**Synthesis Report**

**I. Brief Overview of the Laboratory**

SERVE (Southeastern Regional Vision for Education) was established in
1990 to improve educational opportunities and offerings in the states of
North Carolina, South Carolina, Georgia, Florida, Alabama, and
Mississippi. In the national competition conducted by the U.S.
Department of Education that year, it replaced an organization named
SEIL that formerly was the educational Laboratory for the region. Just
under a decade old, SERVE is thus the third youngest regional
educational Laboratory in the system, just behind those Labs first
funded in this current cycle (1995). SERVE is in its second contract
period, and a mid-contract evaluation is legislatively required.

The evaluation visit to SERVE's Greensboro, N.C. offices took place May
10-14, 1999. The members of the peer review panel engaged by the
evaluation contractor, DIR (Decision Information Resources of Houston,
TX), were Joyce Stern (Chair), Barbara Clements, Marilyn Willis
Crawford, Kerry Davidson, Robert Egbert and Nancy Karweit. U.S.
Department of Education (USED) representatives at the site were Carol
Chelemer (Office of Educational Research and Improvement \[OERI\], the
unit that funds and administers the Laboratory program) and Deborah
Williams (Office of Educational Research and Improvement \[OERI\]).
Extensive background materials produced by SERVE and whose selection was
determined jointly by SERVE and DIR were supplied to the panel prior to
the site visit and were reviewed using the DIR rating sheets.

For four days, the panel as a group heard presentations, interviewed
staff, met recipients

of SERVE programs, and further examined materials. Two unscheduled
meetings took place involving panel representatives. The first was with
Elizabeth Byrom, Director of the Technology in Learning program. Panel
members Barbara Clements and Nancy Karweit requested the meeting
(Chelemer and Williams also attended). The second concerned data
gathering issues. Clements and Karweit, joined by Joyce Stern, directed
questions to Charles Ahearn, head of the Publications and Quality
Assurance unit and Jerry Natkin, SERVE's Director of Evaluation. In sum,
sufficient documentation, information, and answers were provided to
illuminate the major initiatives conducted by the Lab and allow the
panel to address the key evaluation questions developed by DIR. On the
fifth day of its visit, the Panel summarized its findings at an exit
interview addressed to SERVE staff.

SERVE, the regional educational Laboratory (REL), is a part of the
umbrella organization SERVE, Inc., a non-profit corporation. Of this
year's nearly \$12 million budget for the corporation, the REL makes up
\$6.8 million, or about 58 percent. The Lab is considered to be the
"core business of SERVE" so that all the projects within the
organization are intended to complement the work of the REL. Other
initiatives and projects represent activities SERVE is authorized to
perform under various contracts. These include USED awards to conduct
the SouthEast and Islands Regional Technology in Education Consortium
(SEIR\*TEC), the National Center for Homeless Education, the Eisenhower
Consortium for Mathematics and Science Education, the Comprehensive
School Reform Development (CSRD) project, the Charter Schools Institute,
and other activities.

Lab and non-Lab projects are combined administratively into six program
offices: assessment, accountability and standards; children, families,
and communities; education policy; improvement in science and
mathematics; school development and reform; and technology in learning;
plus one initiative (teachers and teaching). These are headed by the
deputy executive director for programs. The technology services,
publishing and quality assurance, evaluation, and new projects
"greenhouse," will shortly be overseen by a newly hired deputy executive
director for planning and development. Both deputies report to the
executive director who in turn is responsible to the Board of Directors.
In the present REL contract, an Executive Management Team (the executive
director, the two deputy executives, the director of operations, and the
executive assistant) now leads the organization. This organizational
structure was created by the current director's (interim) predecessor,
who replaced a "flat" structure in which the executive director oversaw
all activities. The present executive director, John Sanders, filled
several key vacancies, including the two deputy positions, since he
joined SERVE less than a year ago.

The SERVE Board of Directors, which meets twice annually and has
conference calls at least quarterly, has 40 members---six from each
state plus three former teachers of the year, and a representative of
the Native American Education Council. All expected constituencies have
a place on the Board as well as on its advisory committees. Each state
is represented by its governor (or designee), chief state school officer
(or designee), a state legislator, a corporate executive, a regional- or
state-based corporation representative, and an educational researcher.
The term is for three years. Current Chair is Alabama corporate
executive, Al Knight. The panel's telephone interview with the Executive
Committee of the Board revealed not only its members' knowledge of Lab
operations, but that they had an intimate understanding of SERVE
programs and were committed to its success. Committee members uniformly
praised SERVE for the sensitive manner in which regional needs were
reflected in finely tailored educational initiatives, e.g., introducing
"best practices" from other parts of the nation, and for serving as an
objective and neutral observer for policymakers on controversial issues.
As one Board member phrased it, "SERVE is the expert, not the advocate."

SERVE is a part of the University of North Carolina higher educational
system and its principle offices are located on the UNC Greensboro
campus. It is the only Lab associated with a public institution of
higher learning. It is also the first Lab to house a policy analyst in
each of the state capitals, and the first to utilize a decentralized
("distributed") administrative approach. SERVE now has a staff of 90
located in three major offices in Atlanta, Greensboro and Tallahassee
and in programmatic offices spread throughout the six states. Specific
projects augment staff by adding consultants or subcontracting work to
training and marketing firms. SERVE's decentralized structure was
designed to permit the organization to quickly respond to schools,
policymakers, and other stakeholders in the region. Interactive
technology, from conference calls to desktop video, keeps the far-flung
locales connected. SERVE maintains an internet site in which each
program can be accessed from the home page. Links to other sites, such
as other regional Labs and federal research centers conducting work in
related areas, also are provided on the web page.

Approximately 6.6 million students are enrolled in the elementary and
secondary schools of the region. At \$32,100, teacher salaries in the
region fall below the U.S. average of \$37,600. Georgia has the highest
at \$34,000, while Mississippi has the low of \$27,600. Compared with
the nation as a whole, the Southeast has a higher percent of 5-17 year
olds below the poverty line (24 percent vs. 19 percent), a higher
percentage of minority students (41 percent vs. 35 percent), a higher
percent of the population living in rural areas (25 percent vs. 20
percent), a lower percent of the population with four years of high
school (77 percent vs. 81 percent), and lower annual expenditures per
pupil in K-12 public schools (\$4,900 vs. \$5, 700) ranging from \$5,300
in

Florida to \$4,000 in Mississippi. (Source: SERVE/NCES data handout
revised 5/11/99).

Regional information is important to understand as it affects the way
SERVE approaches its work and determines the issues it focuses on. The
Laboratory's articulated mission is to "promote and support the
continuous improvement of education opportunities for all learners in
the Southeast." Further, SERVE is guided by a vision of "a world in
which all persons are members of productive learning communities that
contribute to the continuous improvement of the quality of life." To
this end, SERVE hopes to foster a culture of learning in those schools
with which it partners. The SERVE technical proposal argued that any
work in the Southeast must address issues of regional attitudes, poverty
and demographic isolation. SERVE's work of service to the field and
applied research focuses on strategies to address these concerns. In
particular, extended collaborations were proposed to overcome the
effects of isolation. Building local capacity to define and address
problems in education through systematic reform and scaling up of
effective practices is further seen as a way to overcome economic and
attitudinal constraints.

The remainder of this report uses a format designed by DIR to describe
the panel's observation of SERVE's relative success in addressing its
stated plans. It also points out areas that the panel felt needed
improvement or that otherwise could be strengthened as the Lab pursues
its mission, and suggests appropriate changes.

**II. Implementation and Management**

**A. To what extent is the REL doing what it was approved to do during
its first three contract years?**

**1. Strengths**

Except for certain documented modifications, SERVE has followed the
programmatic plan put forth to OERI in its original technical proposal
for the current contract period. It has done so in the face of sometimes
extreme management difficulties including an unwieldy administrative
structure and turnover of key personnel, including the resignation of
its original executive director (one of its founders) two years ago.
These challenges sometimes impeded progress (as was evident in the 1996
and 1997 quarterly reports), but most activities seem currently on
track. This success in the face of adversity is a testament to staff
dedication to an ethos of service, to the commitment of its Governing
Board, which is actively involved in the affairs of the REL, and to the
unwavering support of the University of North Carolina whose provost,
Edward Uprichard, one of the founders of SERVE, participated in the
panel's visit.

SERVE is clearly a client-oriented organization, responding to as many
requests as possible. One particular area where this orientation can be
seen is in the already noted placement of a senior policy analyst in
each of the state education agencies. Hired with the approval of the
chief state school officer (CSSO), each policy analyst assists the chief
and other state officials by doing research, developing policy briefs,
and advising on cross-state issues, often with the assistance of the
other policy analysts. All have doctorates and research backgrounds. As
they network with each other, interact with SERVE program officers, and
participate in the Lab's quality assurance (QA) process, their views and
influence pervade the organization. Their role is reactive, responding
to requests from policymakers; and also proactive, as they serve on
pivotal committees and maintain key relationships that allow them to
help shape policy agendas. This arrangement has afforded SERVE a
continuous presence in state policy circles, while providing a conduit
of information on evolving policy priorities within and across the
states. Many positive statements were made to the panel, e.g., by Board
members, concerning the important role of these policy analysts, such as
that they are "neutral" and "objective."

Another strength of the Lab is its evident capacity to "establish
networks, strategic alliances, and partnerships with other RELs, other
institutions, and key individuals and other organizations in the region"
(DIR Indicator). Indeed, collaboration primarily for conducting special
projects of interest to individual states, is characteristic of SERVE;
there are few activities that the Lab seems to carry out solo. This is
the hallmark of a strong institution seeking to maximize its
effectiveness and stretch its dollars. A few examples must suffice:

The SERVE/University of Southern Florida Dropout Prevention
Collaborative yielded a report of successful dropout prevention
programs.

-   SERVE organized a consortium of superintendents and school
    leadership teams to share ideas on how to build school cultures for
    long-term school improvement efforts (SERVE-Leads).

-   With the North Carolina Public School Forum, SERVE seeks to extend
    the concept of school/business partnerships to more and more
    communities.

-   Class size emerged as hot issue on both coasts last year. Both the
    WestEd REL and SERVE responded\--the former with an issue paper for
    the California legislature and the latter with a publication for
    general audiences, Does Class Size Make a Difference?\--Recent
    Findings from State and District Initiatives. It proved very
    popular. Information the Labs exchanged (as documented in briefing
    materials) contributed to the quality and usefulness of both
    products.

```{=html}
<!-- -->
```
-   The LNP Assessment Toolkit for Professional Developers is a major
    example of cross-Lab collaboration nationally. Initiated in the last
    contract period by the NCREL Lab, this instrument is used to train
    trainers of teachers in developing assessments pegged to standards.
    A mature product and part of SERVE's Signature Work #1, it is
    employed by the Laboratory in a number of venues.

```{=html}
<!-- -->
```
-   Each school/teacher/principal/district/official involved with the
    REL can come to regard SERVE as a partner in its endeavors (if those
    interviewed on site or those quoted in REL publications are
    representative). It is commendable that SERVE manages to bring its
    considerable expertise onto a given site and yet consistently not
    impose its views, but instead gain the trust of clients there. Thus
    the Lab is permitted to stay for extended periods, enabling an
    intervention to take hold and mature, thereby increasing the
    likelihood of effectiveness and sustainability.

Such examples illustrate yet another Laboratory achievement. In a very
short time, SERVE has created a powerful infrastructure for wide-scale
impact across its service region. With highly trained policy analysts
advising top state officials in every state, SERVE has the ability to
impact policy formulation, while with its strong professional
development and training infrastructure created by program-level
personnel, the organization also can impact implementation in a major
way. It does the latter by hosting conferences, providing training,
developing and disseminating materials, and otherwise supporting state
education departments, districts, schools and teachers in their work.
Moreover, the organizational structure allows the Lab to be responsive
to different states, and facilitates information dissemination across
states.

> **2. Areas of needed improvement**

Several panel members felt that the strengths just noted could be
considerably enhanced with greater internal (a) substantive
communication and (b) coordination/collaboration between projects,
within programs, and across programs. After meetings with staff, some
members felt that obvious "content connections" are not being made, nor
are there structures in place to encourage staff to discuss their
activities so as to promote collaboration and coordination. For
instance, while there would seem to be a content connection between the
alternative assessment school projects and the Senior Project, a
distinct alternative assessment, the panel heard no evidence that one
informs the other. Yet experience from the assessment project could feed
into training on the Senior Project, particularly as a district looks to
widely adopting Senior Project-type activities.

Similarly, the Assessment, Accountability and Standards Program seems to
operate in relative isolation from the School Development and Reform
Program, except for joint planning of a conference for low-performing
schools. Clients of school development projects who the panel
interviewed said they did not know about the Lab's alternative
assessment activities. In short, obvious linkages do not seem to be
readily and adequately acknowledged. Linkages also seem to be missing
between REL and non-REL funded activities, such as between the
Technology in Learning project and the SEIR\*TEC program. Finally,
various activities do not benefit from collaboration around the most
important issue---student achievement (see B.2.below).

> **3. Recommendations for improvement**

The Executive Management Team should set aside staff time directed
toward determining where logical linkages occur and where collaboration
could be beneficial. Seminars, monthly luncheons, or other low-key
mechanisms for instituting information exchange of a programmatic and
substantive nature could start the process. Upgraded technology should
permit staff in Atlanta and Tallahassee to participate fully. Indeed,
events in this activity could emanate from these two cities as well as
from Greensboro. This exercise also could help shape evaluation
parameters of how well the Lab is serving its clients plus guide
long-term planning. A parallel approach would be to hire an outside
facilitator to help identify ways to collaborate that are consistent
with palatable organizational change.

**B. To what extent is the REL using a self-monitoring process to plan
and adapt activities in response to feedback and customer needs?**

(Note: The needs assessment aspect of this issue is dealt with below in
Question IV. B.)

> **1. Strengths**

The Lab is strongly conscious of customer satisfaction and engages in a
continuous process of monitoring in order to more effectively serve its
diverse customers. The REL incorporates evaluation in its customer
service projects on a regular and ongoing basis with the help of the
evaluation unit. Single event monitoring routinely takes place as a
source of immediate feedback to on-site project staff. And staff
debriefings focus on how well the clients were served and what could be
done to improve. Indeed, responsiveness to customers is one of the major
overall strengths of this Laboratory. That is probably one reason why
SERVE's work is so well received and why in its short life it has won
the praise and confidence of state department officials, its Board, and
its clients. Data from the evaluation unit provided to the panel ahead
of time, as well as data exhibited at the briefing on site illustrated a
generally high level of overall customer satisfaction across all
programs. The distributed organization and the roles of the policy
analysts also contribute to continuous self-monitoring.

Quality assurance (QA) for publications prior to completion and
dissemination is highly evolved. The publishing unit provided flowcharts
and stepwise summaries illustrating the close attention paid to ensuring
quality. This is evidenced in the products: Most REL publications and
the two videos the panel saw on teacher assessment and on the use of
technology in assessment were generally thoughtful and well
written/scripted and reflected the Lab's effort invested on the front
end. The dissemination data also revealed that most products were in
demand and had received wide distribution. Reflective of the quality
standard SERVE strives for and often achieves is that the publication,
Achieving Your Vision of Professional Development was named "Book of the
Year" in 1998 by the National Council on Staff Development (NCSD).This
publication is one example, among others, of high quality use of
research, involvement of strong expertise from the research community
working with practitioners, and is a high quality exemplar for the
region.

> **2. Areas of needed improvement**

-   Several panel members felt the evaluation process needed to become
    "more formal and more rigorous." Most feedback comes from clients
    who use the Lab's services, participate in development of Lab
    products, or attend its training sessions and seminars. This
    feedback focuses on satisfaction and the design often falls short of
    identifying ways to improve a product or strategy. Most
    significantly, the panel found little evidence of external
    monitoring and evaluation.

-   Nor is there evidence of consistent, rigorous monitoring of student
    achievement and student success data. If incorporated at all, these
    outcome measures are added late in the process, once the product or
    process is in a final or close-to-final stage, or once it has been
    in use. For example, the Senior Project which has been adopted and
    dispersed in a number of schools and districts, is not undergoing
    rigorous scrutiny in relation to student test score changes. Nor is
    an evaluative effort underway to identify distinguishing
    characteristics and issues of students who are successful in
    completing the project versus those who fail or do poorly.

```{=html}
<!-- -->
```
-   In this context but viewed from a wider angle, SERVE did not appear
    to have analytical processes in place to learn from its own
    programmatic failures and the limitations of what it has been
    promoting. The focus throughout the site visit was on successes,
    with little reference to how staff must continuously be struggling
    with the complexity of teaching "all students" to achieve at high
    levels.

-   Finally, several members of the panel felt that the Lab used the
    term "research" in a very broad, unrestricted sense. Yet people in
    the field are making major choices based on the assumption that what
    the Lab tells them to do and use has an inherent guarantee of
    success because it is "research-based." Under present circumstances,
    this is risky. One panel member spoke for the others when she wrote:

> There was a wide range of understanding \[at SERVE\] of what
> constitutes "research": the term was applied to experts used to give
> advice, to the act of searching to find sources of information, to
> staff observations gleaned informally from projects in the field, and
> to practical projects in the field that seemed to be working. The
> complication with this approach is that the research support base
> becomes naturally overstated, implying effects well beyond the scope
> of evidence (emphasis added). Quality control processes that would
> allow the Lab to distinguish products, processes, and practices that
> have strong empirical support from those that have little--or are
> appropriate only in particular instances with particular groups, for
> example\--are missing.
>
> **3. Recommendations for improvement**

-   Institute a rigorous, external and totally independent quality
    control process, linked with increased internal QA.

-   Build ways to define and gather student and school success data at
    the construction phase rather than at the end of development, and/or
    utilization of Lab products and services. See it as a goal (with
    steps toward its attainment), rather than a by-product (of teacher
    training or other reforms).

-   Build in critical analyses that might allow adaptations, changes,
    and growth of a given program to make it better or to learn how to
    target and adapt the existing program for particular populations.

-   When screening materials and programs that might shape or be used in
    Lab projects, limit the term "research-based" to empirically
    supported programs and those grounded in data-driven, demonstrated
    student success. Add understanding of their strengths and
    limitations, the limitations of the literature-based knowledge, and
    the levels of tentativeness that are inherent in the term
    "research-based." There is a need to be vigilant because of the
    scalability factor in the infrastructure---one must refrain from
    disseminating anything with less than full honesty about its
    potential for success. The scope of adding this level of rigor is
    obviously beyond the reach of current staffing and organizational
    mechanisms and will need to be done through an external system of
    networks or evaluators, or through a combination of
    internal-external controls.

**III. Quality**

**To what extent is the REL developing high quality products and
services?**

**1. Strengths**

The SERVE Laboratory deserves high marks for carrying out a "coherent
and sustained program of work" (DIR indicator). With a comparatively
small staff, augmented as needed by consultants, SERVE has generated
many useful products, conducted pilots, held conferences of critical
importance to the region, and gone into schools that needed or agreed to
be part of a sustained intervention effort. This is evidenced by the
range of quality products and services that make up its two Signature
Works, the first relating to the topical areas of assessment,
accountability, and standards, and the second to broader comprehensive
school improvement thrusts. Since there is no "one way" to approach
either assessment or school reform, the Lab correctly has adopted a
multi-prong strategy. The panel had the opportunity to examine a large
sample of the publications and to read about the services as well as
listen to formal presentations by program directors and a dozen or so
satisfied "clients."

All states in the region in recent years have focused on developing
content standards. Signature Work #1 addresses this development on a
number of fronts within the Office of Assessment, Accountability, and
Standards. First, the Assessment Toolkit, presented at conferences and
in sessions for trainers, shows how assessments may be constructed. Its
popularity reflects the high need for such a "cookbook." Using this and
other materials developed in-house, SERVE trained 3,500 teachers in a
four-day session on how to develop an assessment aligned with a
standard. For a much more in depth, R&D effort, SERVE has partnered with
nine "intensive site" school districts to demonstrate how assessment can
be an instrument for reform, using assessments as a teaching device to
help a child learn and equally to help a teacher teach better. While
focused on professional development, the whole-school approach also
features the essential corollaries of developing local leadership,
principal training, and school-based assessment teams. The strategy is
spelled out in the user-friendly SERVE publication, Using Accountability
as a Lever for Changing the Culture of Schools, done in collaboration
with one of its R&D school sites.

The panel met several clients representative of those participating as
partners in this effort, all of whom felt that this challenging approach
was transforming their teaching and the way they even thought about
teaching. A mature SERVE collaboration with the Bay County Schools of
Panama City, Florida was invited by the Association for Supervision and
Curriculum Development (ASCD) to be part of its Assessment Consortium, a
singular honor. In yet another example, SERVE has promoted the use of
the aforementioned Senior Project, a kind of hands-on culminating
assessment developed and proven elsewhere. The panel interviewed
enthusiastic teachers and students who were part of this effort at pilot
sites. In yet one more variant, the panel received a SERVE video,
Technology as a Tool for Student Assessment, which portrayed three
dramatically different and innovative ways in which schools in the
region used assessment as a learning device.

Signature Work #2 addresses whole school reform and operates out of the
Office of School Development and Reform. Here, too, several strands make
up the effort: short-term training programs, publications, e.g.,
Resources for School Improvement, conferences, and work with
low-performing schools. Major SERVE resources have gone into its three
(annual) successful School Improvement Forums, its Southern States
Seminar on Low-Performing Schools, its assistance to Title I schools
needing to design a "whole school" approach, and its programmatic
assistance to schools seeking to benefit from the CSRD legislation.
Again, the panel got the opportunity to meet enthusiastic school staff
who had been assisted in some way by one of SERVE's school improvement
strategies. (See also relevant comments under II. A. and B., above.)

**2. Areas of needed improvement**

The extent to which SERVE-assisted school reform efforts (#2) reflected
awareness of standards, assessment, and accountability (#1) was absent
in what the panel heard or read. It may be a "given," but then the
connection needs to be spelled out. See the critique on collaboration
(II. A.), above. Panel members with certain content expertise felt that
some Signature Work materials did not reflect familiarity with the most
current relevant research. The panel was concerned that a given
intervention/initiative/strategy be based on the most robust research
possible. This issue is elaborated in the recommendations under II. B.,
above. Finally, demands placed on Lab resources by the Title I
requirements and by the CSRD suggest that these efforts may be
understaffed.

**3. Recommendations for improvement**

-   Expand collaboration as described earlier, to enhance programmatic
    quality.

-   For planned interventions, reach beyond the region to ensure they
    reflect the most current thinking and research in the field. Use of
    outside content experts, not necessarily from the region (depending
    on the issue), is critical to ensure accuracy and timeliness of
    information being imparted. Also, by consulting more outside
    reviewers SERVE might be better guided to build on work done
    elsewhere rather than create duplicative products.

-   For publications, add to an already robust QA system the best
    content experts in a given field to ensure the validity of
    information in all documents, videos, etc. bearing SERVE's name.

```{=html}
<!-- -->
```
-   Add staff to ensure the Lab can continue to meet its obligations
    under the contract, particularly in the area of Title I and CSRD
    activities.

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

**1. Strengths**

The evidence abundantly points to Lab products and services that are
"useful to and used by customers in appropriate settings" (DIR
indicator). There is no lack of examples from the materials sent for
panel review, for this REL is not only prolific, but evidently knows its
audience (see B., below). The Lab regularly seeks the reactions of its
customers and clients in post-intervention surveys and questionnaires.
Reactions are uniformly positive to both publications and services.
Written feedback from Signature Work \# 1 and #2 users and partners was
glowing---as were testimonials from representative clients invited for
the panel's on-site visit. They repeatedly said that they did not know
how they could have done this \[assessment or reform\] work without
SERVE's guidance and assistance. Regarding the Senior Project, for
example, one said, "... \[It\] has changed the culture of the school
system by focusing attention on the process by which all children can
leave public education as critical thinkers, collaborative workers, and
productive citizens.... This starts in Kindergarten." There were praises
also for the organization's reliability. As another client observed,
"SERVE is here every year---it doesn't go away."

Utilization statistics (materials disseminated, web-site "hits,"
conference/training attendance) testify to a high degree of regional
awareness of the Laboratory and the popularity of the full range of its
products in all their variety and scope. Regarding the latter, in
addition to abundant print material, the panel was given examples of
CD-ROM and of videos, and was directed to the website, all of which was
useful.

**2. Areas of needed improvement**

-   While SERVE has become visible in its region, little has been done
    to assert its presence in national research and development
    settings, e.g. AERA, nor has much been published in refereed
    journals or more popular vehicles like the Phi Delta Kappan.

-   Regarding the videos, one panel member felt that while the scripts
    were good, the products could have been enhanced with "bullets" of
    major points made, and that the camerawork left something to be
    desired.

**3. Recommendations for improvement**

-   Several panelists expressed the wish that SERVE staff would carry
    its best products to a national audience. Where it has done so,
    e.g., in the case of the ASCD and NCSD awards, SERVE has received
    well-deserved recognition. More to the point, it has made valuable
    contributions to the field, bringing to a wider audience critical
    insights and strategies\--the results of hard work and serious
    commitment to a search for solutions.

-   The same panelist from 2., above, would like to see the Lab do more
    videos. She suggested they be used not only in workshops, but be
    aired on public and or local television (or at least clips) as part
    of education human interest stories.

**B. To what extent is the REL focused on customer needs?**

**1. Strengths**

Despite distance and diversity of state contexts, the customer's voice
is heard at SERVE thanks to the remarkable infrastructure the REL has
created. Effective connections operate from the state policy level, to
the governance level, to the implementation level. Its state-level
analysts offer the Lab strong partnerships with potential for direct
impact on policy. Their networks are both formal and informal, and
through them, the Lab's influence stretches across the region. SERVE's
Board ensures consultation from the range of constituencies. Here,
replacement of a higher education presence with representatives from the
world of education research made for a more appropriate connection given
SERVE's clients and mission. Furthermore, the Lab has augmented these
strengths with a Teacher Advisory Board. At the program and project
levels, SERVE identifies customers and "extends the warm hand of
Southern hospitality," to quote one reviewer, to schools and districts
in all its six states. The Lab uses customer feedback such as Delphi
techniques and focus groups (at, for example its regional Forum),
surveys, and informal networks to determine customer priorities and
responds accordingly, tailoring products, training, and other services
to the intended audience, and refining them as needed. The REL has
documented a close match between what the Lab thus determines its actual
and potential clients need and want and what it does programmatically.

**2. Areas of needed improvement**

-   Although the SERVE Technical Proposal gave prominence to regional
    needs stemming from its demographics, actual products and
    descriptions of activities are relatively silent on who is being
    served. This is unfortunate given the population's comparative
    poverty and the presence of a large number and proportion of
    minorities and isolated rural residents (See data in I., above). To
    be sure, "small, rural" does receive mention and several programs
    are in fact most likely addressing the needs of minorities (Title I,
    CSRD, and the Delta project). Still the Lab's documents do not
    reveal the extent of service to those most in need. Its data
    gathering deficiency in this regard does not permit it.

**3. Recommendations for improvement**

-   The Lab should carefully examine the extent to which there is
    minority representation in all its programs, and quickly correct
    identified deficiencies. In any case, data should be collected to
    disaggregate this information. So for example, "teachers trained,"
    and "students impacted," should be displayed by race, and for that
    matter, gender. While this disaggregation need not appear in every
    data display, there are settings in which its absence is a glaring
    omission. Hopefully, the data bases can be redesigned, for example,
    to reveal the numbers of minorities trained/impacted/served and the
    proportion of, e.g., schools/students/teachers in a given
    intervention site that are minority. Similarly, urban, rural and
    suburban sites, etc. should be identified in the data.

**V. Outcomes and Impact**

**A. To what extent is the REL's work contributing to improved student
success, particularly in intensive implementation sites?**

**1. Strengths**

Anecdotal information suggests SERVE programs are uniformly doing good
things and accomplishing much for children. But hard data regarding
student success are generally absent in the REL's various programs. For
example, in the school improvement effort where there is no direct SERVE
intervention in the learning process, schools nevertheless reported that
where they had incorporated the planning strategies taught by SERVE,
reading scores went up, e.g., from 33 percent to 53 percent in one third
grade class. What appears to have been changed is the culture of the
school and thus a myriad of elements. Likewise, the Senior Project is
credited with revitalizing the senior year for many students and for
bringing out gifts and sharpening skills in many participants. In other
assessment interventions where SERVE data document student improvement,
the numbers are either too few to be able to draw firm conclusions, or
what is available also is anecdotal.

**2. Areas of needed improvement**

The panel could not find sufficient demonstration of impact across the
programs because student outcomes did not receive a high enough priority
in the program design stage. This is unfortunate because student results
are the proverbial "bottom line." Issues related to what succeeded and
what did not are especially relevant in applied research such as is
typically done by the Labs. This should be considered not only for
standardized tests but for alternative assessments, too. Here, student
work must be analyzed as it is essential to anchor "how good is good
enough." Without benchmarks, there are no standards in place to drive
instruction or to measure levels of achievement in the classroom.

**3. Recommendations for improvement**

SERVE needs to have program effectiveness demonstrated in terms of
student impact, including academic outcomes. More pre-post testing needs
to be done to document outcomes that can be clearly linked to the
program being implemented. This would best be achieved by having this
incorporated in the design of the program so that training would be
consciously directed toward achieving the best possible student
outcomes. Specifically, SERVE needs to shift gears in its professional
development strategy, including that for alternative assessments, to
focus more on student work.

**B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?**

**1. Strengths**

The REL excels at working with schools to implement comprehensive school
reform. School improvement is one of the Signature Works of SERVE and it
has addressed this mission with dedication. Since 1996, training for
school improvement teams has involved over 2,000 participants in 4-day
sessions. A large number of publications have been produced to assist
these teams in research and decision making. The REL's annual School
Improvement Forum, which draws around 450 attendees, features SERVE
programs and numerous other ideas from practitioners and others about
ways to improve schools. SERVE's current work with Title I schools and
with the CSRD moved the Lab even more visibly into the position of
advising others on school reform options.

**2. Areas of needed improvement**

-   Most of the work completed by SERVE is relevant to the national
    debate about reform. However, the REL does not have high visibility
    beyond the Southeast. Its products do not always appear among the
    references cited in training manuals and reports. By framing its
    work in the context of research-based "best practice" and
    demonstrating how it contributes to the field, SERVE could increase
    its reputation and gain more widespread recognition. To do so,
    however, its programs must be drawn together more closely, its work
    must exhibit more up-to-date sets of references, and more careful
    research must be done.

-   There was some concern on the panel about uniformity of Lab effort
    among the SERVE states. Right now, certain efforts are concentrated
    in one or a few states, while other efforts are concentrated
    elsewhere. There is insufficient coherence across the region and
    variety among its target sites was not demonstrated.

```{=html}
<!-- -->
```
-   Likewise there was disappointment that the collaborative effort of
    RELs in the Delta region was not continued from the last contract
    and that SERVE's continuing efforts here were not stronger, given
    the great need.

**3. Recommendations for improvement**

The points below relate together to the concerns just expressed about
deeper research and wider implementation.

-   For its various client groups, SERVE should develop comprehensive
    research summaries related to their various activities and show how
    these activities contribute to school reform. Other activities in
    Labs and centers around the U.S. should be referenced if relevant.
    Increased collaboration with researchers could help to improve study
    design and ensure effective outcomes. In addition, working with
    well-known researchers can lend credibility to SERVE work and help
    to ensure that it gets published. SERVE staff needs to write some
    articles about their well-done projects and seek collaborators or
    other pilot sites in the region and outside of the region (working
    with the relevant Lab).

```{=html}
<!-- -->
```
-   The panel is not advocating a formula approach to resource
    allocation, but does suggest that the Lab think about how its
    programs and resources are distributed among the several states, so
    there is more visibility throughout the region.

```{=html}
<!-- -->
```
-   More attention should be given to the purposeful selection of
    participating school and district sites to ensure variation in key
    demographic and contextual factors within SERVE implementation
    sites. It is clear that in many cases schools and districts were
    selected because of their individual requests. One advantage of a
    more proactive recruitment effort within identified political and
    demographic categories is that this could enhance understanding of
    those interventions which work and those which do not under varying
    local circumstances.

```{=html}
<!-- -->
```
-   Given the extent of poverty and isolation in the Delta, the panel
    hopes SERVE would take the lead in reviving collaborations of the
    last contract and support the panel in its plan to encourage
    Secretary Riley to forge a multi-agency effort there.

**C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?**

**1. Strengths**

Under the current contract, SERVE has the leadership role in Early
Childhood Education among the Laboratories. It has enlisted the
collaboration of several other Labs, the National Center on Early
Development and Learning and the National Association for the Education
of Young Children. These organizations can help to disseminate the
information SERVE has developed and will be developing. SERVE also
presents its specialty well on the web and links it to other sites with
related information. There is evidence that it is establishing a
reputation in those sections of the region in which the Early Childhood
unit has intervention programs, especially North Carolina and Florida.
The Lab also has contributed respectable publications to the field,
notably Terrific Transitions, and disseminated them, but mostly in the
region.

**2. Areas of needed improvement**

The program is not sufficiently strong to be a true specialty area yet
of the Lab, though the foundation is being laid. There is a sense that
it is still searching for its identity and direction---probably because
of staff turnover and vacancies in the first years of the contract. Even
so, the Lab began to make its mark. With a full-time director and a full
complement of staff, the Lab should be able to better develop its
specialty and achieve its place. Once this is begun, SERVE should strive
to do more seminal work, worthy of being published in national refereed
journals.

> **3. Recommendations for improvement**

Though it is of course suitable and desirable to popularize and extend
knowledge, e.g., through pilots, the Lab should take care not to
duplicate work being done by others. Rather, SERVE should find a voice
of is own among those organizations that are working in this field and
seek to make a unique contribution. It could begin by pursuing more
partners and alliances, e.g. through contacts of the National Education
Goals Panel among others. Organizations such as the Council for
Exceptional Children can help in dissemination and also suggest how
documents can be used. Finally, the Lab should develop a dissemination
plan and consider placing documents on-line.

**VI. Overall Evaluation of Total Laboratory Programs, Products and
Services**

Launched to serve its regional constituency just nine years ago, the
SERVE Lab has grown into a full-fledged, contributing member of the
national Laboratory fraternity and is steadily and conscientiously
fulfilling its self-proclaimed mission. Its programs are consistent with
the broad education focus of the states of its region and of the nation
as a whole---school improvement and assessment. At the program and
policy levels, its cadre of able and experienced educators has made
excellent connections across the region and, in some cases, across the
nation. They have brought training and materials to hundreds of
principals and thousands of teachers who otherwise would not have known
about key ways to organize, teach, and assess, or about strategies to
restructure faltering schools and turn around students.

Regarding its customers, SERVE is committed to fully engaging them in a
continuous, interactive manner. At the state, district, or individual
participant level, a question posed is likely to be met by a prompt
SERVE response in an ambience of intimacy and concern that has, in turn,
generated genuine client support. The highly committed Board, along with
other key stakeholders, likewise has a high regard for staff and the
work they do. Lab products and publications have reached hundreds, and
in some cases thousands who undoubtedly benefit from the information
transmitted concerning, e.g., alternative assessment, school leadership,
and early childhood education, to cite just three examples. While from
the start some of the information produced was strong, in recent years
the quality has definitely improved, witness the awards and other
recognition the REL has received.

Leadership difficulties that slowed progress in the early years of the
contract, notably in the specialty area, seem to have been largely
overcome. The Lab has many strong programs, products, and services in
place, is evolving organizationally, and is now positioned to perform at
an even higher level.

This potential can only be achieved, however, if steps are taken to
address programmatic

inadequacies noted in the body of this report. For example,

1.  Improve the research base from which the REL derives its
    programmatic thrusts and seek exposure in the field\--both to get
    input from researchers and from content experts currently working
    elsewhere on relevant topics, and to share with the wider audience
    of peers and practitioners what SERVE is doing.

2.  Aiming for greater programmatic cohesion in SERVE, internal
    collaboration must be considerably expanded, both within the REL and
    between REL and non-REL activities.

3.  Student impact must receive a much greater priority and be featured
    in the initial program design. For alternative assessments,
    benchmarks must be established to distinguish qualitatively among
    outcomes.

4.  External programmatic evaluations should be employed and become part
    of the QA system.

5.  Extra care must be taken to comb the literature so that SERVE
    uses/adapts what has been already produced, freeing up staff and
    resources to make original contributions and extend the frontiers of
    applied knowledge.

6.  Improve the data base capabilities so that there can be better
    demographic profiles of who exactly is getting assisted, trained, or
    otherwise is benefiting from SERVE intervention and service.

7.  Take care that given interventions are not concentrated in one or
    two states, but that there is distribution across the service area.
    Build the variables into the study design so that outcomes can be
    pegged to these variables, e.g., gender, race and location.

In sum, SERVE has achieved much. It is now time for it to move to the
next level of quality by demanding even more of itself as an
institution.

**VII. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement**

**1. Strengths**

SERVE's primary strength is the quality and commitment of its staff.
This cadre of professionals operates in an apparently congenial
environment led by an experienced executive director, heading a tight
management team of top executives. Operations are overseen by an
involved Board of Directors and supported with materials and
encouragement by the University of North Carolina. SERVE has enhanced
its own strength immeasurably by establishing numerous alliances and
partnerships outside the Laboratory, documenting many successes as a
result of these collaborations.

The Laboratory has focused its programmatic efforts on areas of great
need in the region---assessment, school improvement, and early childhood
education, and held conferences and training sessions to reach many
educators with information on programs and strategies of potential
effectiveness. It has been prolific in the generation of numerous and
varied publications whose worth was enhanced by an effective quality
assurance process. But it also has proven to be effective users and
disseminators of available products and service. It has been wise, as
well, in positioning policy analysts to mutually serve the state
education agencies and the REL. It has demonstrated a keen understanding
of the political dynamics and social protocols respected in the region,
which has opened many doors and kept them open for SERVE to both
contribute and benefit. Its needs sensing and evaluative capabilities
are strong and these tasks are carried out conscientiously.

In the process, SERVE has developed a unique and highly effective
infrastructure. It has the capacity for disseminating research-based
practice and moving research to scale by impacting both policy formation
at the state level and policy implementation at the district and school
level. The potential for this system to effect positive change in the
region is truly profound. The recent reorganization and staff expansion,
plus the promise of stable leadership now positions SERVE for an
explosive impact on the region.

> **2. Areas and Strategies for Improvement**

Intra-Lab communication needs to be facilitated to enhance the potential
for synergistic effects. While the recent reorganization was a
tremendous improvement, the panel feels that presently there are too few
structural and managerial mechanisms available to promote collaboration
within the Lab to the optimum degree.

Regarding products and materials, the panel would like to see the Lab
use more nationally available information and resources. One purpose
would be to avoid any duplication of effort and thus employ Lab
resources to add to the field. A second purpose would be to ensure staff
and customer access to the most current research information and
effective strategies for school improvement.

A dissemination strategy devised jointly by program personnel with the
publications unit needs to be developed for REL products. Information
should reach the maximum audience in the region and beyond, and in a
variety of ways. Publications should generate presentations in schools,
workshops, regional and national conferences, and translate into
offerings in journals, magazines and TV to heighten public and
professional awareness. This recommendation applies to all program
areas, but particularly to the specialty topic of early childhood
education.

Evaluation parameters need to be far more finely tuned to get at program
impact and prepare for long-range planning. This will allow the
organization as a whole to raise the quality of work with consistency.
For example, the Lab should be able to report programmatically by state,
impact by SES, race, and locale. And in this regard, probably the
greatest need for programmatic improvement lies in determining desirable
student outcomes before programs are launched and gearing program
strategies to obtain these outcomes accordingly.

Several of these recommendations speak to enhancing quality control. In
addition, the panel wishes to emphasize the necessity of review or
consultation by external content experts in the QA process. The panel is
looking for greater rigor in assessing that a given strategy was "proven
or effective."

Finally, the panel observed that two of its concerns could be more fully
addressed by involving others beyond SERVE. The Delta region with its
chronic poverty and lack of educational and employment opportunities has
been mentioned already. The panel recommends that SERVE explore with the
other Labs that participated in the Delta project under the former
contract reviving joint efforts there again. But the panel realizes that
even this effort would be small compared to the great need. For this
reason, it plans to propose to Secretary Riley that an integrated,
multi-Lab, multi federal agency initiative be launched to impact the
area through greater coordination and collaboration.

Second, while the panel has recommended changes to data gathering design
for SERVE, it also suggests that its recommendations be taken to the
other Labs for possible consideration by their Evaluation Coordinators.
These coordinators have been developing cross-Lab performance indicators
under the present contract. This might be an opportune time to ask them
to consider our proposal for richer demographic data so that data
gathering would be comparable across the Lab regions.

SERVE has made tremendous strides since its founding. It has overcome
serious obstacles and produced valuable work despite these obstacles.
From a solid foundation, it is now in a position to better realize its
original vision for the region. The panel hopes that this report will be
accepted as a guide for future planning and will enable SERVE to advance
with confidence in its future.