Archived Information

**Interim Evaluation of the WestEd Regional Educational Laboratory**

# 

## I. Brief Overview of the Laboratory

As an educational Laboratory contracted to the US Department of
Education to serve the states of Arizona, California, Nevada, and Utah,
WestEd is both old and new. "Old" in that it derives from two regional
entities of long standing---SWERL and FWL; "new" in that it was recently
constituted from a merger of the two. One member of this merger, FWL,
had received federal funding as an educational Laboratory for some time.
But meanwhile, SWERL remained a viable educational service operation.
The merger of these two vibrant organizations has brought to the field a
large, intellectually strong, and experienced enterprise, whose
potential to positively impact the world of education was well
articulated in the technical proposal and operationalized in the
activities and products observed by the evaluation panel.

The Department's RFP required that Labs provide broad-based
comprehensive school improvement strategies; conduct programs of
development and applied research; promote widespread access to
information about educational research and best practices; promote
excellence and equity; enhance collaboration between schools and
parents; involve practicing educators; provide access to services; and
use advanced technologies in their operations. Priorities were for the
Labs to "put the pieces together" to enhance systemic reform and to
"scale up" reform for wider impact. Other requirements are that models
be constructed in which technology is integral; ways to scale up are
identified or produced; communities of learners are helped; Labs partner
with others to further a nationwide education information and assistance
system; and that links be forged with the research community. The status
of Lab progress in

addressing these charges is presented below in the appropriate sections.

#### Implementation and Management

A. To what extent is WestEd doing what it was approved to do during its
first three contract years?

**1. Strengths**

Administratively, WestEd is moving conscientiously to carry out the
missions addressed in its technical proposal to the RFP. One
administrative element that deserves to be highlighted in this report is
the Governing Board, composed of prominent educators and other
stakeholders in each of the four service states. The WestEd Board
members who we met during our site visit were committed to the success
of the Lab in carrying out its missions and to guarding its integrity. I
observed that not only is it an active board, but one that, through a
committee system, exercises both serious oversight and involvement with
setting policy. Panel interviews with Board members also revealed strong
confidence in WestEd administration and operations. New and veteran
members expressed this. It was also evident that WestEd sees itself
operating as partners with the Board, reporting to them fully and
responsively. Collaboration as a goal thus begins at the top---between
WestEd senior staff and WestEd's Governing Board. One key example of
Board influence on policy is its call for sustained Lab intervention in
Arizona, Nevada, and Utah. This contributed to the creation of State
Alliance Projects whereby WestEd works with state education leaders to
further the capacity for reform. For example, WestEd helped the Nevada
Alliance apply (successfully) for funding to carry out an extensive and
long-term technology initiative in several rural schools in that state.

Impressions of high quality implementation achievements follow. The
"Signature Work"

of the Nevada State Strategy is an example of Lab intervention at its
finest. WestEd was uniquely positioned to assist the state at a critical
hour when the state legislature passed the major reform bill of 1997. By
"uniquely positioned," I mean the professional link between the Lab and
the state had a long history. Trust had been established. Members of the
legislature and the SDE expressed full confidence in WestEd overall---as
a resource for policy guidance and technical expertise\--and in the
Lab's State Liaison in particular. For its part, expertise in the Lab
permitted it to serve the state in terms of helping to draft
legislation, and in participating in the standards setting required by
the new law and the assessments to be crafted. Board members repeatedly
voiced the opinion that "it \[the law and SDE responsiveness to it\]
could not have happened without WestEd." The story of WestEd's role in
helping to transform education in Nevada at this time---without
infringing on the prerogatives of Nevadans---is one of "winners." The
legislature got what it sought, the SDE could meet the deadlines set by
law, and WestEd further enhanced its professional reputation. Ultimate
winners will be the children who will benefit from having a better
education as a result. In the context of this evaluation, it must be
said that the Nevada experience illustrates the full potential of what
the federal government must have had in mind when it conceived the
concept of "scaling up." WestEd, with its in-house expertise, was able
to translate a decade of relevant research and experience nationwide
about standards and assessment into new operations for a state that some
would say had generally not been in the forefront of education reform
initiatives. In this instance, it can truthfully and with some wit be
said that the Nevada strategy is "at scale."

**2. Areas of needed improvement**

Despite numerous stellar examples of success, there were instances where
WestEd progress did not go according to plan. This was most noticeable
to me regarding what in the proposal was a flagship program---the
mission to expand in this region the Baltimore-based program, *Success
for All*. It is clear from the quarterly reports that WestEd was having
difficulty keeping pace to hire trainers for the number of schools that
wished to adopt the program. But in the documents supplied to the review
panel, the demise of the program received little mention, and then
"disappeared," leaving this reviewer to wonder what had happened to the
program as well as to the personnel and federal dollars associated with
it. While these concerns were dealt with responsively and responsibly on
site, it would have been preferable had this (and other programmatic
alterations) been noted in the printed material we received prior to our
visit.

**3. Recommendations**

This recommendation is directed as much to the evaluation contractor as
to the Laboratory. For evaluation purposes, I suggest the status of
programs be tracked or presented in some way that makes it clear to the
reader what has happened in the case of programs that have either been
radically overhauled or have dropped by the wayside for what ever
reason. A long discussion is not necessary, but it is important for
reviewers to know some specifics about program status, especially when
one is dealing with the fate of a flagship program. Interestingly, the
outside reviewers who preceded this panel to WestEd, made a similar
observation.

**B. To what extent is WestEd using a self-monitoring process to plan
and adapt activities in response to feedback and customer needs?**

**1. Strengths**

WestEd operates in a culture of "reflection." During the first day's
briefing session, the panel was frequently reminded that self-monitoring
was not an "event" but was on-going. Staff members continuously ask
about their program's progress, success in carrying out its mission, and
its effects or what the Lab terms, its "impact." Emerging from this
culture of reflection, and strongly influenced by the Board asking "Tell
us what difference you have made" were five new framing questions that
are to inform subsequent work of the Lab. They are:

-   What do students have to know and be able to do?

-   What evidence will show that they know it?

-   What opportunities to learn are needed to assure that each student
    achieves?

-   What professional and community capacities and systems supports are
    needed?

-   What system policies and resources are needed?

Further, the Data Sources binder contained a section specifically
addressing Quality Assurance and Evaluation. The "Update on WestEd
Quality Assurance and Evaluation" was particularly useful, outlining the
array of QA strategies:

-   periodic assessment of client views (telephone survey)

-   program portfolio reviews with the Board of Directors

-   project/program-specific QA:

> Some examples are (a) formal advisory group; (b) content review
> panels;
>
> \(c\) external reviews by partners; (d) external reviews by peers; and
> (e) internal review by peers.

**2. Areas of needed improvement**

Having read the extensive materials provided, I still wondered how
consistently the review process is actually applied. So I was glad to
read that the Lab plans to have the programs "learn from one another"
(p. 4), which should contribute to systemic improvement and the
sought-for "coherence". As a former editor, I can only underscore the
need for rigorous prepublication review by content specialists.

Then there is the related issue of the kind of feedback WestEd gets on
its products, e.g., print materials, videos, website---beyond orders
placed and "hits." The Director of Evaluation told us that all users are
surveyed as to the quality and utility of the document they ordered. In
this connection, The Quality Assurance and Evaluation tab included (3.7)
the recent Program Evaluation and Quality Assurance Report submitted
only last month to the Board. It reported a high level of customer
satisfaction. However, the low response rate might call into question
the validity of the findings.

**3. Recommendations**

The "Product Quality Assurance and Marketability Review" now in process,
may be addressing my concerns. Had there not been such a review
underway, I would have recommended that an outside evaluator might look
at some of these issues like front-end QA by content experts to ensure
the most up-to-date thinking on a topic, and the question of customer
assessments.

**III. Quality**

**To what extent is WestEd producing high quality products and
services?**

**1. Strengths**

With the caveat that the evaluation panel was exposed to only a
selective sample of products and was introduced to highlights of the
sorts of services WestEd provides, the overall impression is quite
positive. In the words of one Board member, "WestEd means 'quality,'"
and I saw nothing to challenge that view. A small number of examples
will illustrate my point.

a\) Print material: The monographs I personally examined---"Standards:
From Documents to Dialogue", the one addressing Proposition 227, and
"Educating Limited-English Proficient Students: a Review of the Research
on School Programs and Classroom Practices," are of high intellectual
caliber and reflect the state of the art. (Indeed, the standards work of
the Lab is cutting edge, to say the least.) For that matter, the
capacity of WestEd staff generally to articulate clearly and present
information in a tidy, readable way was reflected in the briefing
materials --both in the materials prepared for that purpose and in
documents prepared for other purposes and used illustratively in the
review process.

b\) Programs: One indicator of success in this category that I would
like to address is "implements a coherent and sustained program of
work." I think WestEd seeks to excel in this regard, and largely
succeeds in this ambition. While a few efforts seemed to have lagged,
WestEd has been consistent in supporting the range of activities it set
out to do with this federal contract. Also to the point, and I think
this is critically important, the Lab strives for "coherence"\--meaning,
I understand, that each piece should inform the work of other pieces.
The program assessment component, operating across all programs, is just
the most obvious illustration. This point was stressed often in our
briefings on site. In another illustration of coherence, it is
noteworthy that WestEd has chosen to address, at some level, work in
every pupil age group---from birth (e.g., Marin City Families First) to
high school (e.g., Talent Development), and to address the interests of
the range of critical stakeholders, from state officeholders (e.g.,
crafting legislation on prospective professional development centers in
Nevada) to parents (e.g., Marin City Families First and the Western
Assessment Collaborative\--WAC), to limited English speakers (e.g.,
Bridging Cultures and across several programs). The result is that the
Lab has something significant to contribute to the knowledge base for a
range of student groups.

**2. Areas of needed improvement**

WestEd, from the testimonies of its users, is known and trusted in the
region. However, my impression is that, except in academic circles, the
work of the Lab is not well known otherwise. There does not seem to be a
significant push to extend its knowledge resources beyond its regional
boundaries and that is unfortunate because WestEd has much to teach
others in national forums other than academic conferences. The list of
publications in which staff members have published was useful to the
review panel, though only half of the items were printed during this
contract period. It showed the caliber of journals that accept REL work.
However, only one *Kappan* piece was listed, and that is a
representative vehicle that would be suitable to present information on
much of the Lab's interventions.

**3. Recommendations**

Because of the timeliness, relevance and quality of its work, I would
like to see WestEd reach a larger audience. Perhaps this will be
achieved with the push for cross-Lab collaboration and the charge to
avoid duplication of effort. But what I am really talking about is
having the work of the Lab get exposure in national magazines and
journals---academic or popular, as appropriate---not merely for the sake
of exposure, but so that what WestEd has learned can be imparted to
others.

**IV. Utility**

A.  **To what extent are the products and services provided by WestEd
    useful to and used by the customers?**

Refer to IV. B.

**B. To what extent is the REL focused on customer needs?**

**1. Strengths**

WestEd revealed in its RFP an acute sensitivity to the needs of its
region\--in particular, the political focus on student assessment issues
in all four states. Of course, the nature of this focus is very
different in each state. In terms of on-going needs assessment, what
emerged from my reading and visit was a sense of strong reliance by the
Lab on the Governing Board and on the Lab's State Liaisons, as well as
on partner interviews. For example, the updated plan for 1997 (p.15)
noted the creation of these positions as "an institution-wide point of
contact with every state in the region" with responsibilities including
"contributing state-level data to WestEd's overall assessment of
regional needs." The Operations Briefing book documented in considerable
detail the process of Board input into the needs assessment process
(2.4) in 1998, as well as WestEd responsiveness to that input. Finally,
in talking to WAC clients brought to the Lab office for the evaluation
site visit, it was clear that they were continually helping to shape and
refine the work of that cutting edge initiative.

In this context, I would like to use this opportunity to revisit the
role of the State Liaisons. It was abundantly clear from the written
material and from the site visit, that these REL individuals play a
central role in alerting the Lab to emerging issues in the states and in
involving WestEd meaningfully in state education activities. State
liaisons are pro-active, are known in their states, are a "presence"
there, and are politically astute, knowing for example, if, when and how
WestEd should seek to become involved. (While I am basing this
assessment largely on my study of the Nevada project briefing book and
on the Nevada liaison's presentation to the panel, it also came through
for the other three states elsewhere in the materials supplied the
panel, but most noticeably for Arizona.)

Some relevant observations to these two questions appear under Question
2, above, so I shall only make some additional ones here. Over the
course of the contract, WestEd has sought to better serve its clients in
a variety of ways. The Lab obtains feedback, e.g., from questionnaires
after conferences and meetings, and on a continuing basis in its
intervention activities. It is continuously asking if its strategy is
working. Customer/client/partner input revises the intervention
accordingly. I will give three examples:

-   In the Marin City Families First project, the REL shifted from using
    paraprofessionals as family advocates to using professional
    caseworkers because higher skills were needed;

-   In the Career Preparation Assessment project, the pilot resulted in
    program modification to obtain teacher buy-in. Now, most sites are
    using portfolio constructs altered to the site, rather than the
    original model. This has resulted in teacher commitment to the
    effort.

-   This final example addresses a product prepared on request, rather
    than one in the original plan that had to be modified. The Lab at
    the request of state education officials in CA, prepared a briefing
    document on class size issues to prepare educators to deal with new
    legislation limiting class size in that state. This is a prime
    example of responsiveness to customer need and reflects as well the
    depth of in-house expertise that can be readily drawn upon in the
    REL.

V.  **Outcomes and Impact**

**A. To what extent is WestEd's work contributing to improved student
success, particularly in intensive implementation sites?**

**1. Strengths**

At this point, the programs are of different maturity and so the answers
must vary accordingly. The Marin City program, for example, has resulted
in transforming the lives of those families affected. Marriages have
been saved, client employment is up, youngsters are getting better
parenting. Anecdotal evidence in the Career Preparation Assessment
suggests the process engages the pupils involved. Likewise, the
professionals involved in the WAC project speak of transformed
classrooms and children similarly engaged in learning, advances they
attribute to the *Kyosei* Assessment Dialogues.

In addition, each of the REL programs to one degree, or another,
"contributes to increased knowledge or understanding of educational
problems, issues or effective strategies." Again, some illustrations
will suffice:

-   Marin City has documented a successful strategy to positively impact
    families afflicted by drug abuse and unemployment, while at the same
    time identifying ways to address the stressed government agencies
    that are supposed to be helping such families.

-   *Kyosei* is meaningfully addressing how to change the school culture
    to benefit the entire student body through school and community
    involvement in standards and assessment.

-   The REL papers on language instruction and on class size, both
    topics of "national significance," are helping administrators and
    officials better grapple with these issues.

**2. Areas of needed improvement**

At this point, there is no student outcome data for the larger programs.
This could possibly be a problem later on when the Lab program is being
re-authorized. Happily, it seems that it is being addressed in the
programs we examined. I do wonder in some cases, if it will be truly
possible to attribute change to the intervention strategy.

**3. Recommendations**

The Board calls for evidence of program impact, and "impact" has become
a watchword in the Lab. Similarly, I urge the Lab to examine how to
obtain outcome data for [all]{.underline} its programs. In the context
of the sought for "coherence," perhaps a cross-project evaluation design
could be developed.

A good example of where impact data was presented to the review panel
during its site visit, is the Marin City handout that reported relevant
results of this nature, e.g., number of adult clients going to school.
Likewise, the 1997 performance plan report for all the Laboratories
identified eight student impact indicators (Indicator 1.4). Most of
these are applicable to the various WestEd interventions and could be
considered, though I am sure there are many more that would be
program-specific, as was the case with Marin City and the *Kyosei*
Accountability Dialogues reflections in the Sources briefing book.

**B. To what extent does WestEd assist states and localities to
implement comprehensive school improvement strategies?**

**1. Strengths**

The Lab recently received additional funds to help schools that received
grants under the Comprehensive School Reform Demonstration program. It
held a conference for schools in Arizona, Nevada, and Utah that had
received funding under the new law to help them understand what was
expected and to help them choose among program options. Staff depth on
this issue makes WestEd uniquely positioned to carry out this mission.
It is a prime example of the indicator of "facilitating widespread
access to information on best practices."

The WAC initiative is concentrated in a small number of CA and AZ sites
and districts, but the potential for this model of democratic thinking
about standards as a vehicle for profound reform is great. I personally
hope it will be allowed to incubate, during which time measures of
change can be designed so that its effectiveness on teachers and
students can be measured and fully reflect impact of this dynamic
approach to whole school reform.

The State Alliances, are very effective mechanisms for promoting whole
school change. The Nevada example, cited in another context above, will
bring technology into a number of isolated and poor schools. Here WestEd
not only "developed an appropriate alliance," but also built the
capacity of its partner sites.

CA's backlash against bilingual education, threatening to spill over
into AZ, resulted in WestEd publishing a guide on the issue, an example
of service to policymakers already noted above.

This reviewer is cognizant of a conscious effort by WestEd to address
the needs of rural schools and students, in its larger programs. This is
most appropriate given the demographics of the state.

Finally, the recent work in Nevada, discussed at length above, is a
prime example of the indicators of "influencing policy" and "scaling
up."

**2. Areas of needed improvement**

All four states do not seem to have received uniform attention in the
scope of things. Utah particularly seems to be less impacted. Also, I
was surprised that the CSRD participant evaluation comments were not
more uniformly laudatory.

**3. Recommendations**

Examine each state's demographics and devise a formula for more funding
equity across the region.

**C. To what extent has WestEd made progress in establishing a regional
and national reputation in its specialty area?**

**1. Strengths**

The Lab specialty is Assessment and Accountability. Its director is a
nationally known figure with experience as a state director of
assessment and with experience in relevant national

initiatives, notably SCANS. In terms of indicators of reputation, he has
recent publications in

academic journals.

As noted elsewhere in this report, assessment informs the work of a
number of Lab programs. The specialty specifically receives funding
under WAC. The Nevada State Strategy and the Arizona State Alliance
project, as well as an independent allocation. The review panel was
briefed on the A&A work with the Career Preparation Assessment project,
and the cross-Lab project (Laboratory Network Program-LNP), both
research based development projects with high potential for utility. I
have discussed the former above, but will note in this context that the
Lab is focusing on the "tailored" portfolio scoring process and the
potential for whole-school reform.

The Assessment Toolkit is an expansion of a vehicle for assessment
design that was originally conceived and produced at the northwest Lab.
Its further evolution is taking place under the leadership of WestEd as
the lead Lab for assessment and accountability, and involves active
input from all 10 Labs. In fact one of its strengths is the commitment
evidence by all Lab participants to this effort.

**2. Areas of needed improvement**

I would like to have learned more about the 1997 change in the WAC
change agent seminar from "performance assessment to looking at student
work" (Updated Plans for 1997). The implications for Lab work in
assessment and accountability overall are not clear. Also unclear is the
potential for CPA to affect whole school reform given that the effort is
so small. It seems too ambitious a goal.

The Toolkit, though a popular vehicle for training teachers on the
development and use of alternative forms of classroom assessments, has
grown to unwieldy size in my opinion. I believe I heard mention that
another Lab had boxed the Toolkit differently (three volumes instead of
one, for different subjects). Finally, while the Lab is active on the
assessment and accountability stage at the micro (CPA) and more macro
(Nevada) levels and there is potential for national visibility and
influence, more linkages need to be developed to give WestEd even
greater prominence in the national arena.

**3. Recommendations**

Regarding CPA, whole school reform potential may be out of reach.
Rather, I would like to see a focus on the results of more technical
work on scoring, e.g., validity, that could inform the field, together
with a realistic assessment (small "a") for reformers and school
officials of the "proper" place of portfolios in the overall evaluation
of student work. For example, what proportion of the portfolio should
"count" toward high stakes determinations?

The Toolkit could benefit from repackaging. I strongly suggest
preparation of a small version, of true "toolkit" scale, that could
serve to introduce practitioners to the ideas of the larger product.
Those "signing on" could then "graduate" to the larger opus and begin
designing assessments.

**VI. Overall Evaluation of Total Laboratory Programs, Products and
Services**

I would say the Lab is definitely on target with its declared mission
and its making a significant, positive difference in the lives of its
intended audiences both in terms of intervention and in terms of policy
guidance and shifts.

**VII. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement**

Rather than talk about which programs seem stronger, I would like to
look at the bigger picture. WestEd's true strength is the quality of its
leadership (including the Board) and its staff. Another quality I sense
is its willingness to step out of the box and engage in risky ventures
like Marin City and WAC. (The former is "risky" in that it crosses the
border into social work, but the Lab is bold enough to say the
obvious\--that home and family is where education begins and we have a
construct of what it takes to turn families around in a particularly
blighted area. WAC, and *Kyosei* in particular, is risky because it
challenges conventional thinking about the fundamental purpose of
assessment.)

In terms of improvement, I would suggest the following:

-   Perhaps WestEd's efforts are too fragmented to achieve desired
    coherence; the Lab should reflect on this and confront some hard
    focus choices.

-   Dissemination should reach wider audiences, i.e. WestEd stories
    should be told in a greater variety of venues.

-   I applaud the effort at improving QA

-   I also applaud the shift of focus toward professional development as
    central to the success of any accountability reform. The hoped-for
    Nevada centers are a case in point, which Lab work in WAC, the
    Toolkit, and CPA should inform.