Archived Information

Interim Evaluation of the Southeastern Regional Vision for Education

I. Brief Overview of Laboratory

The site visit for the interim evaluation of the Southeast Regional
Vision for Education (SERVE) Laboratory was held May 10-14, 1999 at the
SERVE headquarters on the campus of the University of North Carolina
Greensboro. Some staff members from SERVE offices in Florida and Georgia
joined the headquarters staff for the site visit. Panel members were
sent many documents to review prior to the site visit. I reviewed these
documents prior to conducting the site visit, and reviewed some of them
again during the site visit. In addition to the group interviews
scheduled for all members of the Panel, I also participated in a brief
conversation with the Director of the Technology in Learning Program and
another panel member. I also was involved in a conversation around the
topic of quality and evaluation with SERVE staff and a couple of other
panel members. Despite four very intensive days and the review of the
comprehensive set of materials provided, I still feel there are areas of
Lab activities about which I should know more in order to evaluate them.
Many of the areas that go unmentioned in this report are not here
because I think they are just fine or because I did not feel informed
enough to comment. Lack of discussion should not be construed as lack of
support.

II\. Implementation and Management

A.  To what extent is the REL doing what they were approved to do during
    the first three contract years?

SERVE, now in its second 5-year contract, has undergone substantial
changes since it was created in 1990. Approximately 1.5 years ago, there
was a change in leadership in the Lab. With the approval of the Board of
Directors and the Lab monitor in the Office of Educational Research and
Improvement of the U.S. Department of Education, major changes were made
in the structure and organization of the Lab. The new structure has
program areas reflecting key content areas of interest to the Lab,
selected based on the needs of the region. This new organization seems
to make it easier for Lab clients to identify where in the organization
to go for assistance. In addition, it allows for newly funded activities
to be organized with existing activities for better coordination. The
changes included in the Fiscal Year 99 Annual Plan reflect what appear
to be appropriate changes in the activities from those stated in the Lab
Proposal written in 1995. Some of the activities proposed were vague and
were eventually found to be of little interest to the educators in the
region. Other activities evolved over the course of the contract, and
the Annual Plan describes these changes. As a result, comparing the
performance of SERVE to the original proposal is not only difficult but
also inappropriate. My comments about the quality of the work, thus,
will reflect the scope of work as it is now being carried out for the
most part.

The restructuring of SERVE also included some changes in staffing. What
was previously a flat organizational structure at SERVE, now has an
administrative layer that has the potential to promote coordination and
quality in programming. Staff in the two new Deputy Director positions
are new to their positions (one has not even started yet), so it is
difficult to evaluate how well the new structure will work. However, it
is clear that the new Executive Director brings strong organizational
skills to the position. For instance, in the brief time he has been at
the Lab, he has improved the quality and timeliness of quarterly
reports. Key staff members who have been with the Lab for a number of
years are to be commended for continuing to do excellent work despite
leadership problems at the beginning of this contract. It is time,
however, for the Lab to move ahead to bring more cohesiveness to its
activities under the new leadership.

1\. Strengths

Despite its youth, SERVE appears to be well-established and functioning
efficiently. SERVE benefits from the support of the University of North
Carolina Greensboro, and especially the Provost, who was one of the
original founders of SERVE. Many of SERVE's policies for salaries,
hiring practices, and other general organizational policies are in place
because they are required by the University. SERVE also has a very
supportive Executive Board. The Executive Board seems to represent the
many types of users in the region, and their comments were very helpful
to the Panel.

SERVE is clearly a client-oriented organization, responding to all
requests, apparently. One particular area where this client-orientation
can be seen is in the placement of Senior Policy Analysts in the state
education agencies in SERVE states. Hired with the assistance of the
chief state school officer, the state's policy analyst assists the chief
and other state officials in doing research, developing policy briefs,
and advising on cross-state issues, often with the assistance of the
other policy analysts. Many positive statements have been heard
concerning the importance of the role of these policy analysts, such as
that they are neutral and objective. While the diffusion of staff can
present many problems with communications, this unique way of
distributing personnel appears to offer many advantages such as having
the policy analysts working within the context from which the issues
arise and having staff close to conduct training.

SERVE obviously has shown its worth and the support of the U.S.
Department of Education (USED) as it has received numerous additional
grants and contracts from USED, including the SouthEast and Islands
Regional Technology in Education Consortium (SEIR\*TEC), the National
Center for Homeless Education, the Eisenhower Consortium for Mathematics
and Science Education, the Comprehensive School Reform Development
project and other activities. Arranged as they are under the seven SERVE
program areas, there is much opportunity for the Lab to benefit from
joint or supportive activities.

SERVE has also developed useful relationships with other organizations
in the region, primarily for conducting special projects of interest to
individual states. Among the groups with which SERVE has worked are the
Public School Forum of North Carolina, Delta State University, the
Georgia Partnership for Early Childhood Education, the South Carolina
School Boards Association, and the National Dropout Prevention Center.
SERVE has also begun work with the National Association for the
Education of Young Children and the National Center for Early
Development and Learning on its specialty area of Early Childhood
Education.

2\. Areas of Needed Improvement

One of the major concerns I have about the functioning of SERVE is the
lack of substantive communication and collaboration between projects
within programs and across programs. The distribution of personnel in
several offices does not appear to be a major cause of this lack of
communication, although it must surely contribute. Discussions with
program and project staff have shown that obvious content connections
are not made, nor are staff encouraged to discuss their activities in
such depth as to promote collaboration and coordination.

For instance, there is an obvious content connection between the
Alternative Assessment project and the Senior Project activity. I sensed
a lack of coordination among the two projects which may be, in part,
because SERVE uses consultants who have developed the activity and
provide the training, while SERVE is primarily a convener and
facilitator. The Senior Project is an example of an alternative
assessment, and could be discussed (and in fact may be) as such in the
training on alternative assessment. Likewise, there are things that can
be learned from the Alternative Assessment project that could feed into
the training on the Senior Project, particularly as a district or school
looks to infuse Senior Project type of activities into the overall
curriculum.

Similarly, the Assessment, Accountability, and Standards Program seems
to work in isolation of the School Development and Reform Program,
except for the joint planning of a conference for low performing
schools. Clients of the School Development and Reform Program who were
asked if they had heard of SERVE's Alternative Assessment project
indicated they had not heard about it. Assessment, accountability and
standards are essential components of the school reform movement, as is
Math and Science Education. Still, the linkages do not seem to be
acknowledged in program activities, and the activities do not benefit
from the collaboration around the most important issue -- student
achievement. Other linkages seem to be missing between REL and non-REL
funded activities, such as between the Technology in Learning and the
SEIR\*TEC activities. One logical linkage came up during our discussions
that had not been previously identified. Thus, an opportunity is lost to
show the linkages to educators in the region so that they could see
other ways they could benefit from SERVE activities.

SERVE appears to be behind in the efforts to build its specialty area
into a national

powerhouse. It is difficult to tell exactly why problems arose in moving
the activities along on time. It could be that the focus on completing a
seminal work for policy makers was secondary to providing training in
areas where SERVE had expertise. In any case, SERVE is hard at work
moving along the revision of the Framework related to early childhood
education.

3\. Recommendations for Improvement

Schools and districts are bombarded with programs and professional
development activities, and they often do not have the where-with-all to
determine how they fit together. The same may be true for educators who
see the vast array of disparate offerings from SERVE. I recommend that
SERVE staff set aside time to step back and look at the overall area of
School Development and Reform (which is the overarching idea of all
activities within SERVE) and discuss the content of their activities and
programs in depth so that they could determine where logical linkages
occur and where projects can benefit from collaboration. Such an
activity would assist in determining collaboration activities possible
in current Lab activities, evaluating how well the Lab is serving its
clients, as well as for long term planning.

B.  To what extent is the REL using a self-monitoring process to plan
    and adapt activities in response to feedback and customer needs?

1\. Strengths

SERVE described a Quality Assurance (QA) process that is supposed to be
used for publications and activities of the Lab. From what I can see,
the publications seem to go through the rigorous QA process before being
published. There were several instances of where publications were
supposed to be out, but were withheld for revisions due to the QA
process.

A major component of the QA process is feedback from the field, and
there was evidence provided of studies and surveys done to assess the
satisfaction of clients with regard to SERVE

publications and activities.

With regard to the School Development and Reform Program there was
mention of a staff debriefing focused on how well the clients were
served and what could be done to improve the services provided.

2\. Areas of Needed Improvement

The SERVE program areas could benefit from more scrutiny from outside of
the states in the region. Sometimes the focus seemed to be
state-specific. By using more outside reviewers SERVE might be better
able to identify areas of their research that are repetitive of other
research already done or areas where the research does not clearly show
effects.

3\. Recommendations for Improvement

SERVE should continue to use outside reviewers of documents and bring in
folks to provide evaluative information about seminars and training
sessions. In particular, publications which purport to have the best
research on a particular topic should have an outside review by someone
who is knowledgeable about the research and who could offer more
informed suggestions for improvement. Division H of the American
Educational Research Association does evaluations of its sessions at the
annual meeting. Such an evaluation process could be implemented to
promote high quality sessions at SERVE meetings.

III\. Quality

To what extent is the REL developing high quality products and services?

1\. Strengths

SERVE has a number of interesting and high quality activities. For
instance, the Senior Project is a very exciting activity that SERVE is
sponsoring. This is a good example of SERVE brokering the assistance of
the program developers. In addition, it is nice to see SERVE take the
very interesting Toolkit on Alternative Assessment and provide training
to the field. This is an area of much interest in the instructional
improvement area, and SERVE seems to be filling a critical need in this
area. Helping the client to exploit these materials for the benefit of
preservice education is a particularly enticing area.

SERVE staff obviously benefit from a skilled publications office capable
of putting out slick, easy to read documents. This is important because
educators are reluctant to pick up dreary looking documents. In addition
to being colorful and nicely formatted, they are interesting to read and
informative. We learned that two of SERVE documents have received
national recognition, namely *Achieving Your Vision of Professional
Development*, honored by the National Council on Staff Development, and
*Study Guide for Classroom Assessment: Linking Instruction and
Assessment*, honored by Division H of AERA. Staff are to be commended
for these honors.

It is clear from the presentations we have observed that SERVE staff
conduct well-organized and interesting seminars, workshops and training
sessions. Participants are given many opportunities to interact during
the sessions, something they rate highly in their evaluations. Materials
are attractive and nicely put together. There is some evidence of
attempts to make information more widely available and in an attractive
format on the SERVE web site.

2\. Areas of Needed Improvement

A major concern of mine is the lack of complete and current research
documentation of the products and services provided to help the clients.
Many of the products reviewed for the site visit contained few if any
research citations, and many of the citations were dated or from
questionable resources. For instance, the Assessment HotSpots document
had no listing of references that I could find, although one reference
was cited. We were not provided copies of the "researches" provided by
DISC to clients. It is highly possible that these documents are simply
listings of research citations, but not carefully reviewed lists that
sift through what is available for the clients.

I feel that many of the ideas "researched" by SERVE staff are "good
ideas" that a school or district is trying, and that SERVE thinks would
be valuable to watch, rather than field-based implementation of proven
models. Often these studies rely on observations of a very few schools.
Based on these activities, the idea is written up and shared with other
schools. Doing work with a few schools would be okay if carefully
designed research components were included that could be linked to other
studies. For instance, surveys and other documents based on research on
the change process could be used to help evaluate the implementation of
a program by a school. This baseline data is important in looking at how
the program is implemented and what types of student outcomes could be
expected. While there was some mention of such a research-type activity,
a commitment to careful collection of data was not very apparent. Some
mention was made of looking at student achievement next. A better plan
would be to incorporate these types of outcomes into the activity from
the beginning rather than identify these needs at the end. The
activities associated with Alternative Assessment and Senior Project
need to be linked more closely with student achievement. State
assessments are available in all of the states in the region (although
some are new). Most districts also do testing. These testing programs
could also be used. A useful focus would be determine why programs are
not successful in some places or with some types of teachers or
students.

SERVE does not yet have the national reputation on its specialty area
that it is seeking. In part, this is due to problems seen in the Lab
over the past few years. On the other hand, I think SERVE and its
partners in the LNP have had a difficult time determining what products
make the most sense to provide. Some of the originally planned
activities appear to have been abandoned.

3\. Recommendations for Improvement

At the beginning of each contract year, training materials, in
particular, should be reviewed for the relevance of the contents and the
quality of the references. Periodic reviews of the literature should be
done to see newer references or references that suggest some materials
should be removed from training materials.

It would be useful if SERVE staff would provide carefully put together
annotated bibliographies for their clients on a set of topics closely
related to school improvement and reform. SERVE should be careful to
include only references to programs that have been shown by research to
be effective.

SERVE could take it further, however, by working with early implementers
to identify ways that the project has had a positive effect on other
facets of schools' curriculum and instruction. For instance, one client
we saw talked about how the skills required for Senior Project are now
being promoted in earlier grades so that students are better prepared to
succeed by the time they reach 12^th^ grade. Many of the issues in the
Senior Project and Alternative Assessment project are related and more
connections could be established between them.

IV\. Utility

A.  To what extent are the products and services provided by the
    Laboratory useful to and used by customers?

1\. Strengths

Testimony by the clients over the course of the site visit and documents
we received indicated that the products and services provided by SERVE
are extremely well-received and used often. Reviews from training
sessions complimented SERVE on the quality and usefulness of the
information presented, and the thanks felt for SERVE providing the
opportunity for school and district folks to come together for training
and collegiality.

SERVE has been strategic in its commitment to the "train the trainers"
concept, since it builds both capacity at the local level and leverages
the ability of the SERVE staff to meet the needs of their many clients.
SERVE has obtained training for its staff in areas of interest to the
region, such as Conflict Resolution. Building this capability in staff
is an excellent way to serve the community. Using outside resources
where internal resources are not available is also a strength of SERVE
in some areas.

There is some evidence that SERVE customizes training for various
clients. Adjustments in training sessions are made based on the
suggestions of the site. For instance, one client mentioned that SERVE
was asked to do the beginning and ending sessions of a training series
but the district folks provided the middle part. It appears from client
testimony that SERVE staff and very responsive to requests for
assistance, and provide follow-up as requested.

Showcases were designed and presented according to the wishes of the
state education agencies. Other projects were done for entire states
rather than a cross-section of states.

SERVE also seems to provide some services to the needy areas in rural
locations and

those with high numbers of low performing students.

2\. Areas of Needed Improvement

SERVE clients, at least the ones we have seen, read about, or heard
from, indicate complete faith in the materials provided by SERVE. This
means that the products must be of the highest quality possible. It also
means that the viability of the research presented in the documents is
essential, as a program described in a SERVE publication may be adopted
wholesale as it is presented.

Another critical area where SERVE products and services could use some
improvement is in the area of evaluation. While it would be impossible
for SERVE to follow all of its clients to the end of program
implementation to see what is the impact, it is essential that SERVE
work with clients to establish benchmarks for appropriate implementation
and concrete outcomes that schools and districts can use to see if the
innovation is successful. This is particularly true with Title I and
CSRD projects, as evaluation is an essential component. Many SERVE
documents and training modules have customer satisfaction as a desired
outcome. I think this is shortsighted. The major outcome should be
student achievement, and some mention of this goal should be present in
all of the SERVE materials.

I do not feel I have a good sense of how promising programs are scaled
up from one location to another, other than by spreading the word
through seminars. In some instances SERVE has done some excellent work
in conjunction with a state (e.g., North Carolina and, to some extent,
with Alabama). It was not clear if the development work done for these
sites was then offered to other states within the region.

3\. Recommendations for Improvement

As mentioned in question one, SERVE should look at each activity from
the standpoint

of what it offers to achieving school reform in the service of student
achievement. Since SERVE projects are not formal research activities,
the link to student achievement may not be direct. However, clients
should be encouraged to look at both how the process of educating
students is changed through input of SERVE and how student achievement
may be affected for the better. For SERVE applied research activities,
student achievement or other student outcomes (e.g., increased
attendance, continued attendance for potential dropouts, satisfactory
completion of projects) should be evaluated to the extent possible. This
may include reviewing the unacceptable performance of some students and
looking at what pieces might be missing from the training or the
implementation of the project. This type of thing was discussed in the
session with regard to the Senior Project.

One issue that came up was the development of a database containing
information about participants in SERVE activities. The plan is to link
participants to information about their sites so that SERVE can analyze
the types of clients being served. When asked if this type of thing was
being coordinated with other Labs, the answer was no, not at this time.
I recommend that SERVE take the leadership in bringing together the Labs
around the issue of developing commonalties among their databases so
that data from the Labs can be analyzed together in a more meaningful
way than if all Labs "do their own thing." This is not a complicated
process. It requires identifying the information needs that are common
to the Labs (most especially the data needed for reporting indicators),
specifying what data elements and definitions should be used, and
discussing logical formats for collecting and reporting the data. Lab
staff charged with the responsibility for analyzing data (probably folks
in the evaluation unit) could use this activity to ensure that their
data systems contain the common data elements and other elements unique
to the Lab. I recommend that this activity be conducted with the
assistance of a person knowledgeable about Labs and information systems
from outside the Lab who can help Lab staff identify potential
connections in areas which they might otherwise miss.

B. To what extent is the REL focused on customer needs?

1\. Strengths

With the assistance of the Evaluation Program, SERVE conducts DELPHI
needs assessments, customer satisfaction surveys, and other activities
focused on learning how pleased the clients are with SERVE services and
activities. These activities are done both in general at the beginning
of each contract year, as well as for specific projects and activities.
SERVE staff indicated that changes are made in their activities based on
this feedback. For instance, some originally planned activities, such as
the pilot approach to high school self-assessment, were dropped "due to
lack of school interest."[^1] Others were refined in response to
suggestions from the field.

Many specific products have been mentioned that were produced
specifically for a state or other group. For instance, special training
materials were developed for Alabama. SERVE is now looking at grading
and reporting, an area of much interest, in response to work being done
with a district.

SERVE has begun to provide materials in a variety of formats. They are
producing a series of video magazines providing "staff development"
around a topic raised in the field. Materials have also been produced on
CD-ROM, instead of just paper format.

2\. Areas of Needed Improvement

SERVE staff seem too willing to provide something in response to every
request from the field. This means they feel they must develop expertise
and produce something when there may already be something available.
Some products may be less than fully-researched and provide guidance
based on faulty conclusions or dated ideas.

SERVE could use a more concentrated effort to meet the needs of schools
with special needs in the community, such as rural schools and schools
with large numbers of disadvantaged students. While many of the
activities conducted by SERVE are used by Title I and rural schools,
there is no special outreach to the ones who do not choose to
participate, at least that I could see.

I also think SERVE should find more and better ways to use other
technologies.

3\. Recommendations for Improvement

SERVE should look at its overall structure, as mentioned above in
question IA. As a result of this review, SERVE staff should be able to
articulate exactly what components of school reform they feel they have
materials and services to cover. Where there are holes in SERVE
capabilities, the organization should identify other sources (e.g.,
Labs, centers, other research-based information) to fill particular
educator needs. Careful consideration should be made as to what
resources are cited in response to local questions. Recommended
resources should be fully research-based, have undergone public
scrutiny, have been implemented in schools, districts or states, and
have been evaluated according to whether they were implemented correctly
and whether there were concrete outcomes delivered. Only after a careful
review of existing literature, should a new program of work be begun. In
addition, SERVE should undertake new work in collaboration both with
experts from inside the organization and with experts outside the
organization.

I think SERVE can do some more scaling up with other rural schools but
involving them in collaborative activities with other rural schools.
SERVE should promote continued discussions via listservs and exchanges
of promising practices. I hesitate to tell SERVE what they should do,
because they probably can come up with better ideas. But, I think more
focused attention on rural schools would be effective.

I would like to see SERVE use technology more for disseminating
materials and promoting communication. Clients who have participated in
SERVE activities should have listservs to promote discussion and problem
solving between meetings. The CD-ROM Toolkit is a good example of what
can be done. Providing materials on CD-ROM allows for easier
duplication, revisions to meet local needs, etc. For training materials
on school-level planning and evaluation, Internet links to other
resources can be made on CD-ROMS. Videos can now be put on CD-ROMS and
accompanied by training materials. Distribution of CD-ROMS is much
easier than large notebooks and videotapes. I would like to see more
information included in the SERVE web site, such as up-to-date
bibliographies, training for how to do literature searches, etc. The web
site is very attractive and there are many links to useful information.
However, there is more that can be done.

V. Outcomes and Impact

A.  To what extent is the REL's work contributing to improve student
    success, particularly in intensive implementation sites?

1\. Strengths

There was some evidence regarding the Student Project that student
performance is positively effected, although it appears to be mostly
anecdotal and after-the-fact. Similarly, there is a sense that the
Alternative Assessment materials may have an effect on student
performance. The School Development and Reform program had some
achievement data provided by clients,

but it seemed to be data identified and collected after the program was
over.

SERVE has produced numerous publications extolling the virtues of the
various programs. The distribution of these documents appears to be
wide, with many materials reaching the desk of school principals in the
region.

2\. Areas of Needed Improvement

There was a lack of research basis for some of the SERVE work; thus
links to student achievement may have been difficult to assess. From
questions to staff, we learned that there is a desire to look at the
effects of their activities on student achievement. However, they are
talking about looking at achievement long after the implementation
process has begun. Evaluation planning should be done before the project
is begun. Issues related to what succeeded and what did not are
especially relevant in applied research such as is typically done by
Labs.

Although many of the materials developed by SERVE are widely
distributed, they may have limited applicability if they are not
addressing a cohesive set of activities toward school reform. Thus the
publications may not make if off the principal's desk and into the hands
of innovators.

Most of the work completed by SERVE is relevant to the national debate
about reform. However, the SERVE work does not always appear among the
references cited in training manuals and reports. By framing its work in
the context of research-based "best practice" and how it contributes to
the field, SERVE could most likely gain a better reputation and more
widespread recognition if the programs were drawn together more closely,
had a more up-to-date set of references, and were researched in a more
careful fashion.

3\. Recommendations for Improvement

SERVE should develop comprehensive research summaries related to their
various

activities and show how these activities contribute to school reform.
Other activities in Labs and centers around the US should be referenced
if relevant. Increased collaboration with researchers could help to
improve study design and ensure effective outcomes. In addition, working
with well-known researchers can lend credibility to SERVE work and help
to ensure that it gets published. SERVE needs to write some articles
about their well-done projects and seek collaborators or other pilots
sites in the region and outside of the region (working with the relevant
Lab).

B.  To what extent does the Laboratory assist states and localities to
    implement comprehensive school improvement strategies?

1\. Strengths

SERVE has provided numerous opportunities for states and localities to
get assistance on the implementation of comprehensive school improvement
strategies. In addition to various annual workshops, SERVE staff have
worked directly with states, districts, and schools to provide technical
assistance and training related to school improvement. For instance,
SERVE staff provides workshops on how to develop a comprehensive school
reform plan for Title I schools developing schoolwide programs,
workshops on how to apply for federal funding under CSRD, and showcases
of recommended research-based activities that can be adopted as a part
of CSRD. In addition to providing training and technical assistance, it
appears that they provide opportunities for schools, LEAs, and/or states
to present information about the programs they have adopted and get
feedback from their peers. It appears that SERVE staff help CSRD schools
think through the evidence they will need concerning the impact of the
program implemented. One document, in particular, *Catalog of School
Reform Models* provides specific resources about research-based
programs. In addition, information is available on the SERVE web site.
SERVE seems to have a broad array of materials that are relevant to
schools undergoing reform that are readily available and disseminated
somewhat efficiently.

I think the training materials developed by SERVE are very attractive
and probably very useful. I noticed activities in the training
materials, and they appear to be appropriate to the content and useful
for keeping the interest and promoting involvement of the audience.

It appears that SERVE is very responsive to the states in terms of
preparing specialized training activities. I notice several notebooks in
which SERVE materials appeared with locally produced materials in an
activity for a specific state. Apparently SERVE has a number of
time-tested training materials in certain areas that it can pull off the
shelf and use to customize a workshop according to the needs of the
field.

While certain SERVE training activities are finite in length and do not
lead directly into another SERVE activity, it appears that SERVE staff
are responsive to requests for additional assistance and readily
available to answer questions.

With regard to policies issues, SERVE's network of Policy Analysts
provides excellent assistance to policy makers on site at the SEAs. It
is good to see that the chief state school officers have such trust and
confidence in these folks. This network of policy analysts both within
SERVE and across Labs is a particularly useful group, I would imagine.
The extent to which they can help state level policy makers understand
how all these reform efforts fit together only makes the Labs more
essential to the states.

2\. Areas of Needed Improvement

This is one area where linkages between the programs would seem logical.
The materials prepared for schools by the School Development and Reform
Program do not include information about alternative assessment and
accountability prepared by the Assessment, Accountability and Standards
Program. I did not see them even referenced in the training materials.

The availability of research references differs across the program area
documents. While some have complete listings, others have no references
at all. For instance, I really did not see research references in the
Forum training materials. Granted the various presenters should provide
references to their own work. On the other hand, it would seem useful
for SERVE to provide up-to-date references on topics key to the
discussions at the forums.

3\. Recommendations for Improvements

I think one of the most useful roles of the Labs is the dissemination of
up-to-date information on research-based programs. With the use of the
Internet, this information can be disseminated more easily and can be
updated as needed. One issue that organizations must face, however, is
whether they rely heavily upon sales of printed documents or if they are
willing to give away the contents of their documents. In my experience,
I have found that many folks will still want the printed document even
if it is available on the Internet, so providing access via the Internet
should not reduce revenues substantially. I would like to encourage
SERVE to provide its documents online, not just sell them. Education
policy is moving so rapidly that the Internet has become a major source
of information for decision making. If SERVE's excellent documents are
not available online, then their value will be missed. I am particularly
interested in having SERVE provide Internet access to up-to-date
bibliographies, particularly if the items in the list can be linked via
the Internet. Better yet would be syntheses of research placed on the
Internet, or linked to via the Internet. North Carolina State
Superintendent of Public Instruction Mike Ward said in his testimony
before the U.S. Senate Committee on Health, Education, Labor, and
Pensions that while basic research about teaching and learning is
helpful, "the research activity that is often most helpful to policy
makers is the synthesis and publication in a single, understandable
report of multiple studies examining a given topic" (p. 2). SERVE is
currently providing some of these policy documents, and they are
excellent. If other Labs are doing similar documents, it would be useful
to have links to them to serve your clients. I suspect this will be a
part of the Lab web site. In the meantime, making the links directly to
other Labs should be possible, rather than making your clients have to
search for them.

C.  To what extent has the REL made progress in establishing a regional
    and national reputation in its specialty area?

1\. Strengths

SERVE entered into the current contract with some expertise and
experience in the area of Early Childhood Education. As a result, it was
easy for SERVE staff to begin immediately responding to requests for
help in the area of Early Childhood Education. SERVE has produced some
very nice documents concerning transitions, training of teachers and
caregivers, policies related to providing good child care experiences
for children, brain research, and linking services for children. The
documents have been distributed widely within the region. The training
materials we saw on transitions looked very nice and useful.

SERVE has taken the lead in the area of Early Childhood Education and
has enlisted the collaboration of several other Labs, the National
Center on Early Development and Learning, and the National Association
for the Education of Young Children. These are some of the logical
groups to include, although there are probably others that should be
included. These organizations can help to disseminate the information
that SERVE has developed and will be developing. SERVE has done a nice
job of presenting its specialty area on the web and linking the SERVE
web site to other sites with related information.

SERVE is being called upon by a couple of its client states to provide
specific assistance in the area of Early Childhood Education (North
Carolina and Florida). This focus by these two states may help to
identify areas and services not presently met by SERVE or other
organizations, and lead to more interesting activities in this area.

2\. Areas of Needed Improvement

The ramping up of this set of activities by SERVE was apparently
hampered during the first three years of the contract. As a result, we
see only a few limited, but nice documents, and not the seminal work one
would expect in a specialty area. The Framework under development was
actually developed by another Lab during the previous contract period.
Thus the work being done on the framework is apparently not original to
SERVE. I do not wish to take away from the importance of the revision
work and pilot testing that is being done. I only hope that SERVE will
be recognized for the importance of its portion of the work. As a
result, it is imperative that SERVE look for ways to more widely
disseminate the materials it has developed about early childhood.

We do not have evidence from this site visit or the materials about the
distribution of the early childhood publications other than what was
sent out originally to the mailing list. We were also not provided
information about hits on the web site either, so it is difficult to
look at distribution of the information to interested parties outside of
the SERVE network. This is true of all the SERVE publications.

3\. Recommendations for Improvements

The area of Early Childhood Education is a hot area, both from a policy
perspective and from a practical perspective. I saw a link to the first
National Education Goal in one document. I think there are many more
partners and alliances that are possible for SERVE to pursue that could
help dissemination information and provide input into how documents can
be used. I think it would be good to look at the contacts of the
National Education Goals Panel. I suggest you consider adding the
Council for Exceptional Children. Although their focus is more on
children with special needs, there is a lot going on in Special
Education and services to children ages 0-2 and 3-5. There is much to be
learned from Special Education in this area.

SERVE staff already know that they have a need to do a better job of
writing up its work for journals. And, I think there is work to be done
on a dissemination plan. Placing the documents online would probably
bring much more attention to SERVE in this area.

VI\. Overall Evaluation of Total Laboratory Programs, Products and
Services

As a relatively new Lab, I do not have the same expectations for SERVE
as I would for a more experienced Lab. In addition, the changes in
organization and staff have yet to be fully implemented. Thus I am
making my comments in the spirit of what SERVE has the potential to
become.

SERVE clearly has an excellent reputation as a responsive and
service-oriented Lab. The staff is clearly devoted to the vision of the
Lab and what the Lab can bring to the communities in the Southeast. The
focus of the Lab on school improvement and student outcomes is right on
target both for the region and for the country as a whole. The work
SERVE is doing to assist states with Title I and CSRD implementation is
very good, and the districts and schools appear to be very grateful. The
small, decentralized staff has been sorely taxed to provide a wide array
of services to the region, and it has been able to provide an amazing
amount of training to the field. Probably as a result of this diffusion
of effort, there has been limited progress in the development of the
specialty area.

I think that SERVE has the potential to provide even better assistance
to the Southeast if it can build a more cohesive programmatic focus and
a clearer picture of how the work that it is doing relates to student
success. More rigor is needed in identifying effective programs and
researching the effects on educators, school, and, most importantly,
students. SERVE needs to look outside the Southeastern boundaries for
expert help to assist in program design, development, and
implementation. In addition, SERVE needs to look for better ways to
disseminate SERVE work outside of the Southeast.

VII\. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement

SERVE has an excellent staff and an interesting set of programs and
activities. SERVE has an excellent reputation in the region among those
who know about the Laboratory, and the comments about the responsiveness
and helpfulness of the staff are impressive. In its short life, SERVE
has managed to develop a set of programs that meet the needs of
educators both in the Southeast, but also across the United States.

I refer to the programs as a set, such as a group living under one roof,
or in this case many roofs. Unfortunately I do not see them yet as a
cohesive set even though they all address the same mission of promoting
and supporting "the continuous improvement of education opportunities
for all learners in the Southeast."

All of the SERVE programs relate to essential elements of school reform
and each offers an important contribution to overall school improvement
and student achievement. As it appears today, the programs seem to be
operating independently; that is, while there may be conversations
between staff of the programs, there is no concerted effort for staff to
collaborate on research or products. With the reorganization of the
programs and the full staffing of positions in the organization, now is
the opportunity to pull together the disparate pieces of the
organization to make it a cohesive whole. By conceptualizing the
organization around school improvement and student achievement, you can
identify where the SERVE strengths are in providing assistance to your
clients. You can also identify where you have no relevant program, and
then you can identify resources outside of the organization that could
be brokered for use by your clients. Such a plan could result in your
clients having the very best assistance possible.

Specifically, I have seen little real collaboration between programs and
even within some programs, other than pulling together symposiums. By
collaboration I mean meeting together to discuss an idea generated by
one program and identifying ways to work together in such a way that
both (or all) programs get something that adds to their component of the
whole. In other words, there would be synergy, where the sum is greater
than the parts. There are many logical links between the Assessment,
Accountability and Standards Program and the School Development & Reform
Program. One of the documents I reviewed in the first area is *Using
Accountability as a Lever for Changing the Culture of Schools*. This
document expands the definition of accountability beyond assessment and
suggests the use of indicators to monitor improvements in the school.
Another document, *Total Quality Management: Passing Fad or the Read
Thing*, also speaks to the issues of continuous school improvement.
These documents provide evidence of important knowledge and resources
within that Program. Comments from SERVE clients show an understanding
of how important the Assessment Program activities are to overall school
reform and student achievement. But there is no evidence of
collaboration by these two programs around school improvement. We were
surprised to hear on Wednesday that clients of the School Development
and Reform Program had not heard about the Alternative Assessment and
Senior Project activities. It seems like if you are trying to give your
clients the benefit of SERVE's vast experience and knowledge base that
you would share information about other programs within the
organization.

Another bit of evidence in this area is that there is a project planned
by the School Development and Reform Program around "Becoming an Expert
Teacher of Reading (Training Program)." The staff of the Assessment
Program has developed several documents related to instruction in
reading. In addition, there is a new program on Teachers and Teaching.
Yet we heard no mention of collaboration among these programs, merely a
mention of talking between two staff members.

I have seen a few documents and activities that have joined concepts
from across programs, such as the Choctaw Project involving early
childhood education and technology, but I don't see much scaling up, and
there are many more possible connections. I have not seen very many
connections between non-REL activities and REL activities, although
having persons working in dual roles by necessity means there is some
connection. Again, I think collaborative activities could be designed
that address both REL and non-REL requirements, and benefit from the
joint effort. I think this is what USED hoped would happen.

The notion of projects/programs working together in Labs and centers is
not a new one. In the early 1980's, when I was working at the Research
and Development Center for Teacher Education, USED was urging (maybe
even demanding) that projects within centers and Labs collaborate, not
just tolerate each other. My project thought we were way ahead of
everyone because we were spending a lot of time on the road doing
workshops on classroom management under the sponsorship of the Labs. But
then we were told to make it work within the Center. I remember how hard
it was, as some of the projects were very different, unlike the
complimentary nature that we see here among SERVE programs. Specifically
I remember getting trained in the Concerns-Based Adoption Model (CBAM)
and other related activities of that project so that their work could
inform the work we were doing on the implementation of research-based
classroom management and we could share our findings back with them. Our
project worked closely with another project that focused on providing
research-based professional development and support to beginning
teachers. We provided classroom management training to the teachers in
their project, they monitored implementation, and together we looked at
the results of the study to add to the knowledge base of both our
projects. It can be done, but it takes work, dedication, and support
from your leadership. And you have distance issues to be addressed.

I suggest you invite someone such as Jennifer O'Day to facilitate your
discussions about how the SERVE programs fit together and where
opportunities to collaborate exist. She worked with us at the Council of
Chief State School Officers and provided great insight and leadership in
helping us work through how our different activities fit into the
overall goal of improving education organizations to promote student
learning.

By rethinking how your programs fit together and developing more
collaborative activities, you should find that the organization is more
cohesive, and you may find yourselves better positioned to compete for
the next cycle of funding. Your clients will benefit from seeing how the
pieces fit together, and you will be better able to help them find
assistance in areas where you do not currently have activities. You have
a capable and enthusiastic staff and many excellent activities and
ideas. You have laid the groundwork for a better organization. I would
like to see you get there and have continued success.

[^1]: Supplement to the *FY '99 Updated Annual Plan*, p. 16.