Archived Information

**Interim Evaluation of the Southwest Educational Development
Laboratory**

### 

# I. Brief Overview of the Laboratory

The Southwest Educational Development Laboratory (SEDL) has been in
existence since June, 1966. During the past 33 years, it has undergone a
series of transformations, both in terms of the states it serves and its
mission. In November, 1995, SEDL was awarded a five-year, \$22.2 million
Regional Educational Laboratory (REL) contract. Six programmatic goals
were identified:

1.  Enhancing family and community involvement in education;

2.  Addressing diversity in language and culture;

3.  Aligning and supporting policy development;

4.  Promoting instructional coherence;

5.  Applying technology to restructuring and learning; and

6.  Changing the organization and management of schooling.

Within the current contract, SEDL has identified "addressing diversity
in language and culture" (Goal 2) as a "specialty area." In addition,
for the purposes of the interim evaluation, two "signature programs"
were identified. The Technical Assistance Program (TAP), Signature
Program 1, is associated with Goal 5. The Collaborative Action Team
(CAT) Program, Signature Program 2, is associated with Goal 1.

At present, SEDL employs 100 staff members: 20 in clerical, secretarial,
and technical positions; 38 in professional positions; and 41 in
supervisory or director positions. Seventy percent are White (not
Hispanic), 19 percent are Hispanic; eight percent are Black (not
Hispanic), and the remaining three percent are Asian, Pacific Islander,
Native American Indian, or Alaskan native. Seventy-one percent are
female.

This interim evaluation is based on several sources of data. Prior to
visiting SEDL, I skimmed through a box of material provided by the SEDL
staff. I also had participated in a conference call during which Drs.
Wesley Hoover and Joan Buttram. During my week long visit to SEDL I had
opportunities to listen to a variety of presentations, talk with several
SEDL staff members, and examine additional materials. The comments that
follow are based on my reactions to what I have read, observed, and
heard. I have organized my reactions around the eight research questions
provided by Decision Information Resources (DIR), Inc. For each question
I make a few comments (supported by the sources of data which engendered
them), then discuss a few strengths, and finally offer a few
recommendations.

##### II. Implementation and Management

A. To what extent is SEDL doing what it was supposed to do during their
first three contract years?

Either the program of work outlined in the initial proposal has been
executed or the modifications submitted to and approved by OERI have
been executed (or are in the process of being executed).

SEDL lost about 1.5 years because of a change in CEO and the
implementation of an internal review process (letter of resignation of
former CEO sent to Board on April 25, 1996). On April 10-11, 1997, the
Board gave "full support and confidence" for the internal review process
\[Selected Board Minutes\]. By March, 1998, SEDL had in place an
extensive compensation framework. The length of the internal review is
open to question. Data supplied by SEDL in the Briefing Booklet says
March-April 1997. In his 5/10/99 interview Gil Garcia indicated the time
frame was much longer, Spring, 1997, to early 1998. Joan Buttram
(interview, 5/10/99) indicated that SEDL was meeting deadlines for
deliverables 70-75 percent of the time. Gil Garcia (interview, 5/10/99)
stated that there had been substantial improvement in the past seven to
nine months. Prior to Joan Buttram's arrival (about 18 months into the
current contract) not only were there delays in deliverables but the
quality of the deliverables was questionable.

About 62 percent of the total SEDL funding is REL funding (Laboratory
Background Data). Of the remaining 38 percent, the vast majority is
federal money. Monies from REL and other sources are combined to fund
three programs: Program for the Improvement of Teaching and Learning,
Language and Diversity Program, and Technology Assistance Program. Two
SEDL programs lie outside REL funding: Comprehensive Assistance Program
and Special Education & Rehabilitation Services Program.

There are few external (that is, non-governmental) resources. The Board
voiced a concern for diversification of funding. Their goal for 2000 was
70 percent government (federal and state), 15 percent philanthropy, and
15 percent market (e.g., LEAs) (Selected Board Minutes, 1995-98.
Currently, more than 99 percent comes from government funding. Gil
Garcia (telephone interview, 5/10) indicated that diversification of
funding has not been a federal priority. He did say, however, that those
Labs that have diversified have become stronger organizations. He also
suggested that the language and cultural diversity goal may need
additional external funding if it is to be accomplished. Furthermore, he
asserted that SEDL is the "biggest player in the region" in this regard.
\[Note. The lack of addition resources are intriguing in light of the
fact that several program noted their connections with various

foundations (e.g., Pew, Mott).\]

All five CSSOs are on the Board. Some partnerships have been established
with other RELs (Lab Network Program). Alliance with Regional Technical
Assistance Center (RTAC) and various other "assistance centers" (e.g.,
technology, special education, desegregation). One difficulty noted by
Joan Buttram (May 13, 1999, interview) is that not all these "assistance
centers" serve the same region as SEDL serves. Finally, there are
numerous informal partnership (interview with Joan Buttram and Wesley
Hoover, May 13, 1999).

# 1. Strengths

1.  The modifications made in original contract are consistent with
    reorganization and re-direction of SEDL following the internal
    review process.

2.  Timeline and quality of deliverables have improved since the
    completion of the internal review process.

3.  Good use of REL and non-REL funds to support SEDL's programs.

# 2. Recommendations for Improvement

1.  Other sources of funding beyond the Federal Government should be
    vigorously pursued.

2.  The new organizational framework should be complemented with a
    conceptual framework that integrates work across programs. Our
    interview with Shirley Hord (May 13, 1999) suggests that discussions
    about such framework have taken place. At present there are many
    opportunities for collaboration across programs that are not being
    taken advantage of. One such framework is simply a nesting of
    classrooms within schools within communities. Within this framework,
    CAT focuses on community-school relations, FIRST focuses on schools,
    and TAP and Organizing for Diversity focuses on classrooms. It is
    likely that all four programs, therefore, can combine to improve
    student achievement. In contrast, no single program is likely to
    have a substantial impact.

B. To what extent is SEDL using a self-monitoring process to plan and
adapt activities in response to feedback and customer needs?

From January, 1997, through April, 1997, SEDL conducted an comprehensive
program review (aka "internal review). \[See "Plan for SEDL's
Comprehensive Program Review, January 15, 1997; "Laboratory Background"
in SEDL Brief Booklet, May 10-14, 1999.\] As a result of that review,
from September through November, 1997, changes were made in the programs
and program offices. Concurrently and then subsequently, from August,
1997, through March, 1998, a new compensation framework was in place and
staff were assigned to new positions. \[Note. Although I believe that
the long-term result of this self-assessment will be extremely positive
(based on presentations by and discussions with staff members, the
short-term impact is negative in terms of the quality and quantity of
work done during this 15 month period. An example is Signature Program
#1, the TAP program, where their first year's work on Technology
Planning was delayed by the comprehensive program review in the second
year and refocused on Constructive Learning Environments in year three.
Consequently, it was not until Spring, 1998, that their work in schools
actually began. Nonetheless, the change was clearly needed. The Board
(interview on May 10, 1999) gave the following rationale for the change.
There was a need to improve SEDL's image. The organization as too "top
heavy." The compensation package was "out of kilter." There was a need
to address new and emerging issues "out in the community."

SEDL's QA process, described in its original proposal, was fully
implemented in March, 1996. QA activities have been documented in each
of SEDL's quarterly progress reports since that time. As of February
1999 a total of 98 reviews had been conducted of 77 unique products. The
types of products reviewed include workshop and conference plans and
agendas, invitations to schools and communities to apply for
participation in specific research designs, videos, and web-based
products. \[Taken from SEDL's response to questions about their Year 4
plan and documented during site visit, May 10-14, 1999; also extended
example given during May 10, 1999, oral presentation; also, SEDL "Needs
Assessment Procedures" document.\]

QA process "allows for" external review. Whether such a review takes
place depends on whether the program manager seems a need for it
(interview with Wesley Hoover, May 13, 1999). There are several examples
of external reviews being conducted (e.g., CAT modules, Organizing for
Diversity modules). The Institutional Review Team has been reduced from
three to one over time (with one departure and one death).

# 1. Strengths

1.  The internal review process resulted in substantial and important
    changes.

2.  The QA system is used to produce high quality products and services.
    Importantly, the QA system extends beyond SEDL to include clients
    and experts in the field.

# 2. Recommendations for Improvement

1.  At present, external review is an option within the QA system to be
    used at the discretion of program managers. I would suggest that
    some form of external review be used for all products and services
    to which clients have access (e.g., modules, training sessions).

2.  There is a need for a coherent, comprehensive database that enables
    SEDL to monitor the quality of work across programs and projects.
    This would provide opportunities to share work across programs and
    projects as well as to determine when integrating programs and
    projects at various sites may be useful or advantageous.

##### III. Quality

**To what extent is SEDL developing high quality products and
services?**

For both "signature works" the development process began with a
literature review (oral presentations, May 11-12, as well as copies of
literature reviews in supplementary files). Both underwent QA review,
complete with external review. The TAP program included a pilot test in
its development. In this way, input from practitioners was incorporated
into module design.

There is a nice balance of quantitative and qualitative research
methodology. Examples of qualitative work are: "Selected Field Feedback
on Process and Materials" \[Signature Work #2\] and "Pilot and Field
Test Feedback \[Signature Work #1\]. There also are examples of straight
quantitative work (e.g., "Selected Participant Evaluations of Start-Up
Orientation" \[Signature Work #2.\] Finally, there are examples that
combine quantitative and qualitative work (e.g., "State of Sites Report"
\[Signature Work #1.\]

The development approaches used are consistent with what is known in the
area of staff and professional development as well as school change. For
example, the need for a "critical mass" of committed teachers is
recognized in Signature Work #1, TAP (oral presentation, May 11, 1999,
and supportive documentation). Similarly, the need to adapt the CAT
program to the needs of specific communities was recognized and acted
upon (oral presentation, May 12, 1999, and supportive documentation).

Finally, surveys of clients (n = 797) suggest that 84 percent or more of
the clients rated the quality of products and services to be excellent
or good. \[Unfortunately, the response rate was only 36 percent.\]

# 1. Strengths

1.  The quality of products and services is generally quite high.

2.  State-of-the art knowledge, processes, and tools are used to design
    and implement products and services.

# 2. Recommendations for Improvement

1.  Each product and service seems to be developed "from scratch" based
    on a literature review. A greater coordination among others working
    the field may enable the SEDL staff to adapt existing products and
    services, rather than constantly producing their own.

2.  More attention should be paid to the importance of the client
    survey, with greater effort made to increase the response rate to at
    least 50 percent.

IV\. Utility

A. To what extent are the products and services provided by SEDL useful
to and used by customers?

In combination, the two "Signature Works" are being delivered to 21
sites (6 for TAP, 15 for CAT). The sites for both "Signature Works" are
in all five SEDL states. The duration differs according to the
"Signature Works." Signature Work #1 has been in the field for only one
year. In contrast, Signature Work #2 has been in the field for three
years. In addition, Signature Work #2 was based on work done during the
prior five-year SEDL contract. Finally, "Organizing for Diversity" is in
one site, "Promoting Instructional Coherence" is in five sites, and
"Creating Communities of Continuous Inquiry and Improvement" is in one
site.

Based on client interviews on May 11-12, 1999, SEDL conducts ongoing
interaction with their clients/users. Also based on client interviews on
May 11-12, 1999, SEDL's products and services are both useful to and
used by customers. In addition, client surveys (n = 797) suggest that
SEDL products and services increased awareness and/or acquisition of the
important new skills (77 percent) and informed decision making or
planning (68 percent).

There are several publications that are available (e.g., Public
Deliberations, Native Education Resources, the policy group's Occasional
Paper Series). In addition, there is a website featuring all programs
and services. The website appears to be functional (in terms of the
number of "hits"). Finally, e-mail and LISTSERVs are used to communicate
with clients, board members, etc.

# 1. Strengths

1.  Products and services are well received by the clients being served.

2.  Responsiveness to clients is a major strength of the SEDL operation.

3.  The combination of traditional publications, web pages, LISTSERVs,
    etc., increases the opportunities of others to access SEDL's
    products and services.

# 2. Recommendations for Improvement

> 1\. As projects move toward dissemination, SEDL should explicitly
> address their role in dissemination. Questions such as the following
> should be answered: Are co-developers the key to dissemination? Is
> there a need for SEDL to engage in research on dissemination? Is there
> a need for SEDL to maintain some degree of involvement with their
> developmental sites beyond their "use" as intensive implementation
> sites?

##### B. To what extent is the REL focused on customer needs?

Efforts are focused on "critical concentrations based on regional
diversity" (REL Planning Process Document). Five critical concentrations
are (1) rural delta, (2) border with Mexico, (3) Indian nations, (4)
rural, and (5) urban (oral presentation, May 10, 1999, plus other
sources).

An extensive "needs sensing" process is in place. It includes trend
analysis, focus groups, surveys. (REL Planning Process Document). Also,
with respect to Signature Program #1, TAP, schools were they first
classified into one of four cells: low technology, low constructive
learning environment; low technology, high constructive learning
environment; high technology, low constructive learning environment; and
high on both. Efforts were generally focused on schools in the low-low
cell (oral presentation, May 11, 1999).

SEDL programs were determined using National Education Goals, REL
Specialty Area, Comprehensive School Reform elements, Future Conditions
of Education (FCE) survey, Board Provisionals, Communication Survey).
Also, there are nine criteria to be met by each program initiative
(e.g., strong contribution to the knowledge base, promise to make a
positive difference

in lives of children, sensible and doable).

Pilot tests of each program are conducted (e.g., G5-Technology and
Restructuring -- Pilot Participant Feedback, June 8-9, 1998; oral
presentations on Signature Work #1 and Signature Work #2, May 11-12,
1999). Also, two client-focused criteria are included in the QA. All
products, publications, and services state must:

1.  Meet an identified need and be likely to have a significant,
    positive impact on the intended audience, and

2.  Be readily understandable and use for those for whom it is intended.

Finally, there is multi-year commitment to "intensive implementation
sites" (oral presentation, May 10, 1999).

#### 1. Strengths

1.  SEDL is definitely focused on and responsive to customer needs.

2.  There is in place an extensive "needs sensing" process (including
    trend analysis, focus groups, and surveys).

# 2. Recommendations for Improvement

1.  Some discussion should be held concerning the number of sites that
    can be properly served with current staff and budget. The intricate
    balance between quality and quantity must be addressed.

2.  A matrix of programs by critical concentrations should be prepared
    so that one can see (1) that all critical concentrations are being
    served and (2) that the programs are being matched properly with the
    critical concentrations.

V. Outcomes and Impact

A. To what extent is the REL's work contributing to improved student
success, particularly in intensive implementation sites?

Concerns about lack of student data were voiced by several people/groups
(e.g., Selected Board Minutes, 1995-98; Gil Garcia (COTR) during
telephone interview on May 10, 1999; CAT client interviews, May 12,
1999). In addition, client survey data (n = 797) indicate that only 23
percent of those surveyed said that SEDL products and services
positively affected student performance. Achievement test data are
ultimately needed, but other student data, appropriate to specific
programs, are also needed (e.g., increase in school attendance or
decrease in drug use for CAT program). From client perspective, such
data are needed if they are to secure additional funding.

While he agreed that student success was important, Wesley Hoover
suggested that the primary emphasis of SEDL work has been and will
continue to be community and teacher professional development (several
interviews conducted from May 10-14, 1999). In this regard, it is
instructive to note the responses of 797 clients to one of the client
survey questions. Only 40 percent indicated that SEDL products and
services had changed or enhanced the quality of their professional
practices.

CAT (Signature Work #2) has data provided by the schools that can be
used to monitor impact over time. TAP (Signature Work #1) has narrative
data on changes in teacher practices as well as data on student
engagement. Perhaps the best example is "Organizing for Diversity" which
gathers baseline data on four components (i.e., cultural knowledge,
self-awareness, communication skills, and beliefs), the implements a
year-long series of professional development seminars, and then collects
data on the same four components at the end of that year.

Several clients interviewed on May 11-12, 1999, mentioned the importance
of SEDL as a "catalyst for change." In this regard, SEDL programs are
extremely important to the populations being served. Specific examples,
include the importance of the CAT program in "giving voice to
traditional voiceless people" (client interview, May 11, 1999) and
helping primarily white, female teachers deal with students from vastly
different cultures (interview with Pat Guerra, May13, 1999).

## 1. Strengths

1.  The programs being operated by SEDL clearly address issues of
    regional and national significance.

2.  There is a clear focus on increasing community and parental
    engagement in schools and improving the practice of teaching.

3.  Evaluation plans have been written for each of the six primary SEDL
    goals.

# 2. Recommendations for Improvement

1.  An applied research design should be built into every program and
    project. In this way, research questions will guide the effort,
    **baseline data will be collected**, and decisions ultimately will
    be made on the basis of effectiveness data.

2.  Because of the importance of students in any effort to change or
    improve education, some type of student data should be gathered with
    respect to each program and project (e.g., behavior, attitude,
    achievement). Stated somewhat differently, each research agenda
    should include at least one research question pertaining to
    students. In this regard, it may be possible to prepare a menu of
    student variables (and, where necessary, instruments) from which
    project developers, co-developers, and clients can select.

B. To what extent does the REL assist states and localities to implement
comprehensive school improvement strategies?

Concerns for and examples of "scaling up" are evident in the vast
majority of SEDL's work. Examples include the CAT in Albuquerque, NM,
where they moved from 3 schools in a single "feeder system" to multiple
"clusters" in three years. In addition, the use of co-developers in
virtually every program ensures attention is paid to issues of scaling
up.

Project FIRST and the "Professional Learning Community" Project, both
headed up by Shirley Hord, are approaches to comprehensive school
improvement. In addition, SEDL is part of the "School Change
Collaborative" (an inter-Lab effort led by the Northwest Regional
Educational Laboratory). Finally, according to Shirley Hord, SEDL plans
to be involved with the more than 250 schools in their region that will
receive CSRD awards.

The State Policy Planning Service is SEDL's main thrust in influencing
policy. The general approach is to influence policy by (1) bringing
together policy makers across the five states (the Regional Policy
Analysis and Advisors Network, (2) maintaining regular communication
with them, and (3) through them, attempting to influence policy by
providing them with appropriate, accurate, and "research-based"
information (interview with Joyce Pollard, May 13, 1999).

# 1. Strengths

1.  Concerns for "scaling up" are apparent in virtually every program
    and project. Efforts at building capacity (such as the CAT project)
    and efforts are using co-developers are very important in this
    regard.

2.  Having Shirley Hord on the staff is central to the success of
    comprehensive school improvement strategies. She has both a national
    and international reputation; her work on school change and
    comprehensive school improvement is excellent.

3.  SEDL's State Policy Planning Service provides useful information to
    policymakers, thus approaching comprehensive school improvement from
    a political/legislative perspective.

# 2. Recommendations for Improvement

1.  At present, Goal 6 operates in isolation. A conceptual framework
    such as the one mentioned earlier would enable the efforts made with
    respect to Goal 6 to be integrated in an overall plan for community,
    school, teacher, and, ultimately, student improvement. Stated
    somewhat differently, a comprehensive effort at SEDL is needed if
    comprehensive school improvement in the field is to be possible.

2.  There needs to be more sharing of expertise across goal areas.

C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?

This is the weakest area in SEDL. There seems to be only one of two
functional projects. There have been few publications and presentations
over the past three years.

Much of SEDL's reputation rests with programs developed in the 1970s and
1980s and with individual staff members. There are several reasons for
this state of affairs. First, the specialty area has been leaderless for
some time. Second, some of the early projects were either completed or
seen by management as unproductive. Third, concerns for dissemination of
products designed and services provided during the current contract is
premature at this time (interview with Wesley Hoover, May 10, 1999).

# 1. Strengths

1.  The staff members assigned to the specialty area have expertise, are
    dedicated, and are conscientious.

2.  The quality of the work produced is quite good, particularly since
    it was produced under less than optimal conditions (e.g., lack of
    leadership, one or two staff members per project).

3.  The decisions to stop unproductive activities and efforts (e.g.,
    Educator Exchange, the creation of a bi-national school) were good
    ones.

# 2. Recommendations for Improvement

1.  It is absolutely imperative that a qualified, full-time leader for
    this Goal be employed.

2.  Every effort should be made to develop this Goal in such a way that
    it becomes synonymous with SEDL at the national level. This may mean
    eliminating it as a Goal and having it as the "glue" that holds all
    the other goals together. \[In this regard, there already is some
    evidence that this is in place.\] This also means publishing papers
    and reports relative to this goal in national outlets.

3.  It may be advantageous to separate cultural diversity from language
    diversity. In this way, it may be possible to separate Goal 2 from
    Task 7.

4.  Finally, the short phrase currently used to denote this specialty
    area should be changed. At present it is written very passively. The
    new phrase should indicate its emphasis on improving the quality of
    school life for students whose language and cultural backgrounds
    differ from the language of instruction and the traditional white
    middle-class culture that dominates American public schools.

# VI. Overall Evaluation of Total Laboratory Programs, Products, and Services

SEDL came out of the internal review process much stronger and more
focused than they were going into it. As a consequence, they have a
unique opportunity to redirect their time and efforts to accomplish
great things in the region. The expertise and dedication of their staff
are very high. The quality of their work is quite good. Their
responsiveness to their clients and customers is a real strength.

At the same time, however, the internal review process and the changes
in organization and personnel that resulted because of it took a year or
more out of the first three years of this contract period. Consequently,
they have a long way to go in a very short period of time.

# VII. Broad Summary of Strengths and Recommendations for Improvement

SEDL has numerous strengths. Among the most important are:

1.  The quality of their leadership team;

2.  The quality of their staff, in general;

3.  The importance of their work;

4.  The quality of their work;

5.  Their responsiveness to their clients and customers;

6.  Their relationship with their Board; and

7.  The quality of their decision making process.

There are also some noteworthy areas of improvement. In this regard, I
offer the following recommendations.

1.  The vacant leadership positions in the specialty area and in
    evaluation must be filled immediately.

2.  Despite the emphasis on community and teacher development, every
    effort must be made to place students at the heart of SEDL's work
    with communities and schools.

3.  Sources of funding other than governmental funding should be
    vigorously pursued.

4.  The new organizational framework should be completed with a
    conceptual framework that allows coordination and integration of
    efforts and expertise across programs.

5.  During the next two years, SEDL must seriously address the issue of
    dissemination and their role in it. The transition from development
    to dissemination must be made very soon.

6.  Every effort should be made to develop the "specialty area" in such
    a way that it becomes synonymous with SEDL at the regional and
    national level.