Archived Information

Interim Evaluation of the Mid-Atlantic Laboratory for Student Success

**I. Brief Overview of Laboratory**

The Laboratory for Student Success (LSS) was established in December of
1995 at Temple University. Under the umbrella of Temple's Center for
Research in Human Development and Education (CRHDE), the LSS operates
under contract awarded by the Office of Educational Research and
Improvement to operate the Mid-Atlantic Regional Educational Laboratory,
serving the states of Delaware, Maryland, New Jersey, Pennsylvania, and
the District of Columbia.

**II. Implementation and Management**

A.  **To what extent is the REL doing what they were approved to do
    during their first three contract years?**

**1. Strengths**

Given the data reviewed by the panel and the stakeholders interviewed,
it appears that LSS has executed the program of work outlined in the
technical proposal submitted to ED. The work, overall, has been done in
a timely manner. In terms of use of other internal organizational
resources, LSS is supported Temple University in several ways. In
addition to in-kind contributions, Temple supports LSS by paying the
entire salary and benefit cost of the Executive Director. Further,
Temple only charges 8 percent of indirect cost to the LSS contract (a
reduction of 45 percent).

LSS states that it utilizes a two-pronged approach to meeting its
mission mandates and

program goals. The first is a long-term program of applied research and
development that focuses on ways to effectively and efficiently bring
research-based knowledge and practical know-how to bear on efforts to
improve student learning. The second is a broad-based program of
outreach and services to the field which connects to and builds on the
work of grassroots professionals and resources and the expertise of
standing structures, including schools, professional organizations, and
other educational and community resources throughout the mid-Atlantic
region, focusing particularly on those in the most isolated rural areas
and the region's inner cities.

In terms of staffing structure, the Lab has approximately 80 FTE
employees, comprised of an interdisciplinary team of researchers from
varied fields and support staff. LSS tries to recruit people
strategically, drawing from cognitive sciences, developmental
psychology, educational administration, learning technology, program
evaluation, speech pathology, and teacher education, among others.

Outside of its structure, LSS strives to create a network of
collaborating schools and related service provider agencies. Many of the
schools are considered "co-developers," with implementation specialists
involved at the schools sites. These individuals include regular and
special education teacher, principals, and district level personnel.

LSS, building upon the strengths of the Center for Research on Human
Development and Education, has created a dynamic, talented team to carry
out the arduous task of their lengthy and complex technical proposal.
That they have kept up with the scope of work proposed is admirable.

**2. Areas of needed improvement**

One of the strongest assets of the Lab may lead to one of its greatest
weaknesses in the

future. The leadership of the executive director, which drives much of
the success of the Lab in both its programmatic work and its research,
constitutes a lion's share of the variance of the Labs tenor with its
constituents.

**3. Recommendations for improvement**

I would suggest that explicit conversations begin soon on how internal
capacity can be built around strong, strategic staffing choices so that
the dynamic nature of the leadership can be carried out at all levels of
the organization.

**B. To what extent is the REL using a self-monitoring process to plan
and adapt activities in response to feedback and customer needs?**

**1. Strengths**

LSS has in place a Quality Assurance system for its products and
services. This system allows it to engage in self-assessment, utilizing
the results to improve overall Lab performance. The Evaluation Unit is
responsible for quality assurance and evaluation of the work of LSS. In
addition to conducting annual needs assessment studies to ensure the
responsiveness of the services LSS provides in the region, the
Evaluation Unit is responsible for several other QA processes:

-   maintaining the LSS database for documenting the implementation and
    evaluation of the outreach and services to the field work;

-   conducting external reviews of the work of the LSS;

-   maintaining an ongoing program of tracer studies to assess the
    quality and utility of the various professional development and
    dissemination activities conducted by the LSS staff; and

-   conducting product surveys to assess the extent to which the LSS
    products and tools are use and found useful by its customers.

The Lab's quality assurance plan helps define strategic planning needs.
Since the Evaluation

Unit is responsible for maintaining a self-correcting process for the
Lab, both formative and summative processes are uses. These components
include:

-   Product and services evaluation

> Product satisfaction studies, event evaluations, and tracer studies:
> demonstration of practicality and utility, along with needs assessment
> questions

-   Needs assessments

> Surveys of teachers, principals, superintendents

-   Internal and external evaluation

> Informal staff feedback, as well as formal reviews conducted by the
> advisory boards and OERI. The technical review board helps determine
> the quality and utility of all the LSS work.

-   Outcomes and impact

> The strength in this area surrounds the clearly articulated plans of
> action made by the evaluation unit for collection and analysis of this
> important feedback data. Another strength in this area is that Lab
> members, in particular the Field Services Unit, use the results
> derived from the data to improve their programming.

**2. Areas of needed improvement**

None noted.

**3. Recommendations for improvement**

None noted.

**III. Quality**

# To what extent is the REL developing high quality products and services?

**1. Strengths**

There is evidence that LSS is developing high quality products and
services. All of LSS\'s products and field service work is based on the
results of an applied research model. This model of utilizing applied
research to determine best practice for fieldwork allows the Lab to
minimize duplication with similar efforts.

One example of the applied research work of the Lab (and an example of
how the Lab

engages outsiders and experts) is the national study on the effects of
school reform in schools with high concentrations of low-income
students. This national study involves a multi-disciplinary team of
evaluators, and sociologists, psychologists and educational policy
scholars. This study is designed to build on the established procedural
knowledge base on what the steps are that schools are adopting that seem
to be effective in reform. Four other regional Labs are working in this
project (AEL, NWREL, SERVE, NCREL). In time, the Lab hopes to provide
the resources from the study to all Title I School-Wide (TISW) schools.
There are plans to link a new database of all 15,804 schools TISW
schools to other national databases, such as NAEP. LSS has created a
forum for discussion as a result of this work. LSS organized a number of
national invitational conferences, along with publications and reports
surrounding the project. Many groups and peer organizations participated
in the conference, including RAND, CPRE, NAACP Legal Defense Fund, AEL,
NWREL, SERVE.

The LSS network of governing boards provides a safety net for ensuring
the quality of the work going out of the Lab. Of note is the mission of
the technical review board, which provides quality control over the
research and other products created by the Lab. The LSS technical review
board is comprised of researchers and practitioners whose fields of
expertise include curriculum design, program evaluation, and research
design and methodology. Board members review documents related to their
own areas of expertise and gauge the potential impact of the work for
the target audience of researchers and policymakers.

The Lab does a superb job of collecting data and regularly provides data
to the advisory boards and constituents on a number of fronts. All the
stakeholders we talked to indicated that the Lab is responsive. Schools,
in particular, are able to turn to the Lab for assistance on an ad hoc
basis for nearly any pressing reform need or question. This rapid
response strategy has been attributed to the tenor set for the Lab by
the executive director.

**2. Areas of needed improvement**

None noted.

**3. Recommendations for improvement**

None noted.

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

**1. Strengths**

Based upon the data reviewed by the panel, it is clear that the Lab
provides products and services that are of sufficient size, scope,
duration, and intensity to produce sound guidance toward student success
efforts. The body of work and associated products known as Community for
Learning (CFL) is one example of this. The CFL program is a development
project focused on the design and implementation of comprehensive school
reform. The panel engaged "customers" of the CFL model, both at school
sites and during interview sessions at the Laboratory. All of the
feedback suggests that, through CFL, LSS is conducting ongoing
interaction with the users of the program, and that the program is
useful to its intended audience.

Products and services are deemed useful through the comprehensive
efforts of the evaluation unit. A key approaches used by the evaluation
unit is the Tracer Study LSS uses Tracer Studies to look at programs
that have been in the field for a few months. These Tracer Studies,
conducted via telephone, have been conducted in service area
concentrations (in large geographic areas, like Washington, D.C.).
Loosely speaking, LSS would ask it's constituents, "We've done an awful
lot of work in your area lately. What do you think of our work?" These
open ended questions are placed into the questionnaires on field based
work and this data is fed back to Jo Ann Manning for use in her
division.

# Areas of needed improvement

None noted.

**3. Recommendations for improvement**

None noted.

**B. To what extent is the REL focused on customer needs?**

**1. Strengths**

LSS attempts to conduct need finding studies to identify customers and
potential customers. Annual needs assessment surveys constitute one of
the LSS' primary strategies for listening to its constituents. Drawing
upon the report "Spotlight on Student Success" #202, one finds that the
goal of the LSS needs assessment surveys is to obtain input on field
practitioners need in terms of topics and technical assistance. In the
case of the needs assessment for teachers and administrators, the survey
consisted of 36 items covering topics from the overall quality of
current programs, facilities and practices in the region as well as
current and future priorities for school improvement. The data from this
and other surveys has been used to inform the Lab in the development of
priorities and refinement of the scope of work for the Lab.

Through the standing boards of the Lab, LSS is able to obtain guidance
to ensure that it sets priorities in accordance with customer needs,
tailors its products and services to intended audience. The Stakeholder
Advisory Board provides much input in this area. The Stakeholder
Advisory Board is comprised of 16 members, superintendents, teachers,
parents and a cross section of individuals that potentially would be
served by the Lab in some way. Meeting four times per year (increased
from two meetings per year) this board informs the Lab of needs, issues,
and problems of the groups that the board members represent. This board
strives to inform LSS of what they can do to be more relevant. As one
board member noted, "We inform the Lab of what 's out there, and how
best to follow through."

In the area of identification of potential customers, interviews with
the stakeholder board suggest that this is an area where the Lab has
grown recently. One board member noted that several months ago they
determined that there wasn't enough communication to the regional
constituents about what the Lab does. The board member stated that the
Lab came to board at a subsequent meeting with an analysis of their
current communication plan and behavior, coupled with a new plan to
increase communication to the constituent groups, detailing how the Lab
would change and improve.

2.  **Areas of needed improvement**

None noted.

**3. Recommendations for improvement**

None noted.

**V. Outcomes and Impact**

A.  **To what extent is the REL's work contributing to improved student
    success, particularly in intensive implementation sites?**

**1. Strengths**

The bulk of the data related to improved student success that was
available to the panel surrounded the impact of CFL. With regard to the
implementation of CFL in the Washington D.C. schools, progress patterns
across the four Stanford 9 performance levels for math and reading
increased where CFL was in place. In general, findings on program
implementation of CFL and student achievement show a continuing pattern
of improvement for a two years in a row. On average, CFL schools showed
great overall positive change in achievement that other schools
receiving targeted assistance from other programs. This is impressive
given the high mobility level of the students and the high turnover rate
of the teachers in the targeted schools.

Other data generated in the Philadelphia area suggest that the
implementation of CFL is a significant contributor to the rate of high
school completion among students who participated in the program. In a
follow-up study of middle schools students that were in a CFL component,
81 percent remained in high school after leaving the CFL middle school,
this in contrast to 40 percent who were in the same middle school, but
not exposed to the CFL model.

LSS utilizes a sound model to monitor impact over time, both short and
long term. Their work contributes to increased knowledge and
understanding of the impact of school wide comprehensive reform, an
issue of national significance.

LSS also works to place the foundations for student success within its
implementation sites. Quoting one member of the Stakeholder Advisory
Board:

> The Lab I know is doing an excellent job in New Jersey. As a result of
> LSS' involvement at the district and state level, at least 4 districts
> are using the CFL model. Before the state mandate for equity, our test
> scores were so low that we looked to the Lab as a vehicle to bring
> change in our middle schools. Beginning in the summer of 1997 we
> worked with principals and had training for teachers. It hasn't
> happened overnight, but in 1999 I see changes in school climate and
> professionalism that wasn't there before. The jury is still out on the
> test scores, but the alignment of the curriculum is in place. Getting
> it off the paper and into the classroom is where the Lab comes in.
> They have very well-trained personnel to bring about change. Jo Ann
> Manning has done a fantastic job with the principals and stressing the
> importance of instructional leadership.

2.  **Areas of needed improvement**

> The area of documenting the effect of reform strategies on student
> achievement is a difficult proposition, particularly in areas where
> many competing initiatives are at work to improve schools. The CFL
> model, LSS' flagship reform strategy, is in itself a multifaceted
> approach which deserves the attention of more comprehensive applied
> research and program evaluation studies on its impact.

3.  **Recommendations for improvement**

I recommend that the work of the evaluation unit be augmented to collect
an agreed upon set of student outcomes data that are common to all CFL
implementation sites. This data would begin to represent a common
framework for reform accountability, yet not infringe upon the research
agendas of the individual Principal Investigators who are generally
charged with developing the research questions on CFL's student
effectiveness. This broader approach (a meta approach, if you will)
could serve the Lab well in its attempts to describe its true impact on
student achievement.

**B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?**

**1. Strengths**

LSS seeks out stakeholder group involvement in the LSS program of work,
and recruits expertise and resources to compliment and support the LSS
program of work. They reach out to connect and establish collaborating
networks with practitioner and researcher organizations. The Lab is a
leader in national and regional discussions on improvement issues of
national significance and on the implementation of next-step strategies
to achieve student success. The Lab also attempts to demonstrate ways to
target resources and expertise to implement reforms for those faced with
most adverse circumstances. One of the goals of the Lab is to figure out
how one can take what works in one community and make it work in
another. It's this "procedural knowledge" that they are trying to
capture and bring to states and localities.

An excellent example of the Lab's ability to assist stakeholders at the
state level involved its work in bringing together state level
policymakers for a conference on the issue of teacher recruitment. As a
board member stated, "it was good that the Lab was the Ombudsman in this
area, and looked at ways in which we should work together on this scarce
resource (teachers). Not only on how to recruit (teachers), but what
skills they should have, and what should universities be doing in
certification and licensure preparation. Now I don't know how you
measure in dollars what that conference did, but the value added process
of the Lab calling together people who would go separate ways and
eventually bump heads, thus making separate state systems; we can now
begin to look at collaborative recruitment efforts, using an developing
electronic database for recruitment. It was thoughtful, research based,
but you got something out of it. When you leave a meeting and you have
an action plan and you can do something, that's a success."

2.  **Areas of needed improvement**

None noted.

3.  **Recommendations for improvement**

None noted.

**C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?**

**1. Strengths**

It is clear that the Lab has made great progress in establishing both a
regional and a national reputation in the area of urban school reform.
The Lab disseminates research based information, establish links in the
field, assesses reform needs, respond to feedback, targets professional
development to support implementation of reform, assist states to
sustain reform, and demonstrate effective scale up strategies. All these
activities are done a high quality level and have contributed to their
reputation.

The Lab is a prolific publishing house, and many of its staff have
published work in well-respected outlets. These include, since the Lab's
inception, 44 articles and 50 book chapters, 10 books a year, 70
conference papers, and 30 articles in journals or magazines. Not
settling only with traditional academic outlets, the Lab distributes
videos, planning guides, and manuals as well. They estimate that they
have made presentations at 150 sites in the last two years. As for the
use of the Internet and World Wide Web, LSS provides downloadable
information from their website, hotlinks to relevant sites, and listserv
hosting for discussions. An interactive online tool is called "Achieving
Student Success" is a popular online reform tool.

In terms of outreach, the Lab sponsors professional development programs
through the "State-of-the-Art Seminars," "What Works Workshops,"
"Advances Study Institutes," and "Principal Forums."

2.  **Areas of needed improvement**

The Lab admittedly needs to improve its reach to grass roots
organizations. While connecting to ASCD or NEA is relatively easy for
the Lab (via the placement of articles within [Educational
Leadership]{.underline} or through their special strand at ASCD), there
are groups that remain elusive to LSS. An increasing priority is
reaching the business community. They have not done as well in that area
as in other stakeholder arenas. Their governing board is aware of this
and directed the Lab to act in this direction.

3.  **Recommendations for improvement**

Seek strategies to reach out to the business community in meaningful
ways.

**VI. Overall Evaluation of Total Laboratory Programs, Products and
Services**

After three years the Lab has come to be known for long-term ground
breaking research with rigor and sensitivity to practice. They maintain
national and regional reputation through urban education, comprehensive
school improvement, and resilience promoting innovations. They also
conduct national/regional/local implementation studies of significance
to policymakers.

LSS is known for state of the art research syntheses on national issues
and these help facilitate informed decision making for their
constituents. Their data based planning and implementation tools, 20/20
analysis, continues to be cited as a useful tool to building level,
district level, and state level school administrators. 20/20 analysis
helps schools understand the relationship between demographics and
equity, among other things. Importantly, it underscoring the ability of
schools to use their own data, not new, fancy information, to facilitate
diagnosis and planning.

**VII. Broad Summary of Strengths, Broad Concerns**

LSS is able to work with schools and sustain the efforts of change. They
do this by developing and implementing strategies to significantly
improve the region\'s capacity for bringing effective, research-based
practice to scale. The Lab works intensively with schools in
demonstrating feasible, effective and sustainable strategies for turning
around low performing schools. These strategies manifest themselves in
the form of collaborative school reform demonstration projects,
technical assistance for databased planning and monitoring
implementations, and professional development programs focusing on (a)
implementation and (b) on-going support. The Lab is lauded for serving a
high concentration of students from economically and educationally
disadvantaged urban and rural communities.