Archived Information

Interim Evaluation of the Mid-Atlantic Laboratory for Student Success

# I. Brief Overview of Laboratory 

The Laboratory for Student Success (LSS) was established in December
1995 at Temple University in Philadelphia. Managed by Temple's Center
for Research in Human Development and Education, the LSS operates under
a contract awarded by OERI to operate the Mid-Atlantic Regional
Educational Laboratory, which serves the states of Delaware, Maryland,
New Jersey, Pennsylvania, and the District of Columbia. This portion of
the mid-Atlantic region includes suburbs with some of the wealthiest and
best educated people in the nation, inner-city ghettos with the very
poorest and least educated, and large rural areas. The specialty area of
LSS is urban education.

The interim review and evaluation visit to LSS took place on the Temple
University campus from May 17-21, 1999. This reviewer examined written
background materials provided by LSS staff approximately two weeks prior
to the on-site visit, and additional documentation was provided by the
Lab staff during the visit. Interviews were conducted on-site with the
Executive Director and senior staff members, as well as with
representatives from the Governing Board and the Stakeholder Board.
Discussions were also held at Lab headquarters with representatives from
regional schools implementing the Lab's Community for Learning (CFL)
program---including teachers, principals, students, parents, and central
office staff. Panelists also made a visit to Stetson Middle School, a
demonstration site for CFL, and interviews were conducted there with the
principal, LSS support staff, students, and teachers. Sufficient
documentation and testimony were provided during the site visit to allow
panelists to address the

key evaluation questions presented by OERI.

**II. Implementation and Management**

**A. To what extent is the REL doing what they were approved to do
during their first three contract years?**

## 1. Strengths

LSS is dedicated to serving its Region. Its Executive Director,
professional staff, and Governing Board are highly qualified,
hard-working, and clearly focused on serving the needs of the Region.

Responding thoughtfully and decisively to regional needs, LSS has taken
on the daunting task of helping turn around some of the lowest-achieving
urban and rural schools in the Region (including those in Washington,
D.C. and in rural Pennsylvania). In their Technical Proposal, LSS
clarified this focus for the contract period: "*The top priority must be
the most troubled schools and communities where many students suffer
adverse life situations compounded by inadequate schools and learning.
The LSS is committed to giving priority attention to children and youth
of the inner cities and the most neglected rural areas in all of its
work*" (ii) It was abundantly clear to this reviewer during the site
visit that in terms of following through on this top priority work
during the first three years of funding, LSS is doing what it was
approved to do.

The Lab has also committed itself to developing collaborative
relationships with other entities to achieve its goals: "*The LSS will
seek to develop institutional linkages with the mid-Atlantic region's
four state departments of education and the Washington, D.C. school
system, as well as collaborations with education and related service
delivery agencies and other R&D organizations throughout the region and
across the nation to increase the effectiveness and efficiency of the
Laboratory's outreach and work."* (v) Again, it was made clear during
the site visit that in this instance LSS is clearly doing what it was
approved to do, although admittedly it has a stronger relationship and
more leverage with some state DOEs than with others.

Another broad goal of the Lab as articulated in its Technical Proposal
involves widely sharing and disseminating the results of its work: "*To
achieve the educational vision of a schooling success for all students
in the mid-Atlantic region, the LSS will initiate a broad-based program
of dissemination and application of demonstrably effective practices in
systemic ways over time. The overall goal of this program is to ensure
that findings from the work of the Laboratory and other regional and
national R&D resources are known and used to systematically improve
teaching and learning in schools, and to build an increased capacity for
education in the broader community in which students and their families
live."* (iv) The challenge for LSS has been to determine how to
translate research-based knowledge into tools and products (and
"procedural knowledge") that are useful to schools. LSS has successfully
met this challenge in a variety of ways.

Professional development plays a central role in LSS's role as a
resource and facilitator of systemic educational reform in the
mid-Atlantic region, and is one of two broad-based outreach programs
within LSS's Services to the Field. The major component in the LSS
strategy for capacity-building in the Region and the centerpiece of the
LSS professional development model is a series of forums and seminars
that focus on how to bridge the gap between the state-of-the-art and the
state of practice. These include: Principal Leadership Forums, which
provide opportunity for the discussion of leadership roles in
implementing strategies for student success; State-of-the-Art Seminars,
which are one-day seminars that provide information on recent advances
in research and practical knowledge on what works in advancing student
success; What Works Workshops, which provide participants with hands-on
experience in designing and implementing innovative programs and
practices that are known to be effective; Advanced Study Institutes; and
site-specific professional development and technical assistance.

Community for Learning (CFL) is a comprehensive
school/family/community-coordinated approach to improving student
learning that has evolved over the past 20 years as the centerpiece of
the Executive Director's personal research agenda, and it is currently
in full operation in 46 rural and urban schools. The classroom
instructional component of CFL, the Adaptive Learning Environments Model
(ALEM), was developed under the aegis of the National Follow Through
Project and has been implemented in over 200 schools in 22 states. In
the opinion of several of the teachers and administrators interviewed
during the site visit, "ALEM is just good teaching." Selected components
of the CFL model are in place in 37 schools, and 43 schools are in the
planning stage. Implementation of the CFL components is supported by a
delivery system provided by the Services to the Field Unit of LSS that
provides organizational and professional development support at the
school and classroom levels. Data collected from participating schools
show a positive pattern of changes in math and reading scores, and
demonstrate that CFL students outperformed comparison school students in
both subjects. Although a range of teaching strategies are prescribed by
the model, CFL does not provide or promote specific curricula or
frameworks. Instead, the developer attempts to align school curricula
and instruction with district or state standards.

LSS has been successful in leveraging additional resources to support
Lab programs and activities through its connection with Temple
University and because LSS is a major program at the University's Center
for Research in Human Development and Education. In addition to
contributing to LSS operations an in-kind contribution in terms of
space, fiscal, and administrative support through the University's
grants, accounting, and personnel offices, LSS is also support by Temple
in real dollars. Not only does it pay the entire salary and benefit cost
of the Executive Director, but Temple only charges 8 percent of indirect
costs to the LSS contract. This reduced overhead provides additional
spending money for the LSS operations.

As a new regional Lab, LSS focused much of its efforts during the first
two years in recruiting high-quality staff to expand the LSS expertise
and resource base for accomplishing its goals. The forging of strategic
national, regional, and local alliances has been one of the most
effective ways of reaching a large number of LSS stakeholder groups. One
outcome of these alliances is the co-sponsorship of national and
regional events, which have helped put LSS on the map. During Year 3,
for example, 6,608 people attended various LSS cosponsored professional
development events. LSS has also played an important catalytic regional
role in terms of creating new alliances that support common reform
needs. The formation of the Mid-Atlantic Superintendents and Deans
Network and the establishment of the Mid-Atlantic Regional Task Force
for Teacher Quality, Certification, and Recruitment are significant
illustrations of the Lab's networking and brokering role within the
Region. In addition, LSS researchers are highly sought-after speakers at
major national and regional meetings.

**2. Areas of needed improvement**

> None

**3. Recommendations for improvement**

> None

**B. To what extent is the REL using a self-monitoring process to plan
and adapt activities in response to feedback and customer needs?**

**1. Strengths**

Quality assurance issues are addressed through ongoing procedures
involving feedback from the Governing Board, Stakeholder Advisory Board,
Technical Review Board, field-based collaborators, LSS clients, and
self-assessments of Lab staff. The Governing Board of Directors has been
chaired for the past three years by Joseph Dominic, who is the Education
Program Director at the Heinz Endowments in Pittsburgh. In addition to
the chief education officers (or their representatives) of the five
jurisdictions served by LSS, the Board consists of representatives from
professional organizations, teachers, principals, and the Executive Vice
President of Temple University. The Governing Board has the overall
responsibility for establishing the programmatic goals of the Lab and
for directing and overseeing the quality of the work of LSS to ensure
its timely completion in fulfilling the terms and conditions of the
contract in operating the Mid-Atlantic Regional Educational Laboratory.
The Board also conducts an annual evaluation of the work of the
Executive Director and the leadership team of the Lab. The Steering
Committee of the Board has the quality assurance responsibility. The
Governing Board is full of praise for the reputation and regional
credibility established by LSS in just three short years. As one Board
member puts it, "The Lab forges a strong link between the research side
and the practice side."

In addition to receiving regular feedback and input from the Governing
Board of Directors and the Executive Directorate, two other Boards of
Advisors provide additional expertise and guidance to the LSS staff.
Members of the Stakeholder Board are drawn from a cross section of
regional stakeholder groups that fully reflect the various socioeconomic
status, ethnic, and cultural backgrounds of the student population in
the mid-Atlantic region. Dr. Win Tillery, County Superintendent of
Schools for Camden County, New Jersey, has served as the Chair of the
Stakeholder Advisory Board since its creation. Extensive conversations
with several members of the Stakeholder Board during the site visit made
it clear to the review team that LSS is making a significant impact upon
schools within the Region.

The Technical Review Board performs several important functions related
to quality assurance review to ensure that the work of LSS is not only
technically sound, but also relevant and useful for improving policy and
practice. Its primary role is to serve as external reviewer of the work
of the LSS. LSS senior staff members shared specific examples during the
site visit of how the Technical Review Board provides detailed, critical
feedback on LSS manuscripts prior to submission for publication as a
form of quality control. Board membership includes researchers with a
wide variety of expertise in fields related to the work of LSS,
including both researchers and practitioners, such as teachers and
school administrators, with a wide range of complimentary fields of
expertise. Although Technical Review Board members were not present
during the site visit, an examination of notes from the Boards yearly
meeting with LSS staff reveals feedback and suggestions that are
insightful, constructively critical and fair-minded, and of the highest
quality. In the Technical Review Board minutes from the Lab's most
recent quarterly report, there was general consensus among Board members
that the quality and intellectual value of LSS products is high.

Community for Learning has developed a built-in monitoring and
evaluation process through which participating schools regularly assess
the degree of program implementation and use a systematic degree of
implementation assessment instrument. LSS staff assigned to the school
observe the classroom teaching and learning process regularly to assess
the extent to which the degree of program implementation results in
improved classroom process. Student achievement is monitored daily
through a diagnostic-prescription process that teachers use in
developing individualized learning plans for each student.

Annual two-day retreats of the LSS senior staff are held at off-campus
meeting facilities to review feedback from OERI, the three LSS boards,
and collaborators of the Lab. The Lab also conducts product satisfaction
surveys that are specific to a particular product or publication. The
purpose of these surveys is to determine user-satisfaction levels and to
obtain other evaluative information relative to specific products. Needs
assessment surveys are also conducted on a semi-annual basis to
ascertain the degree of satisfaction with the work of the Lab and to
help define areas of priority for meeting client needs. These surveys
are sent to a random sample of stakeholders and clients in the
mid-Atlantic region including state department officials, school
district personnel, school staff, and research and development partners.
The results from the past year indicate that the level of satisfaction
with the work of LSS remains very high.

**2. Areas of needed improvement**

None

**3. Recommendations for improvement**

None

**III. Quality**

**To what extent is the REL developing high quality products and
services?**

**1. Strengths**

At each professional development event where LSS staff play a role
(including State of the Art Seminars, What Works Workshops, Advanced
Study Institutes, and Principal Leadership Forums), participants are
asked to complete an evaluation of LSS services and programs based on a
5-point scale (1=lowest, 5=highest). Data from years 1 and 2 reveal an
average aggregate rating on this scale of 4.4 for all such events in the
areas of program content, presenters, materials, and overall evaluation.

Community for Learning was cited as an exemplary program in the November
1997 Congressional Report on the CSRD legislation. Indeed, high praise
for the CFL model was a constant theme during many conversations with
representatives from regional schools during the site visit. One
principal said that "We used it \[CFL\] as a framework to build a
focused, cohesive school. We monitor student achievement closely and
we've seen gains." A school facilitator at a CFL implementation site
explained that "For years it was dull and dreary, with no camaraderie.
When the new principal came and we decided to use CFL, everything
changed. We're now working with each other and sharing---not so
isolated." A teacher, when asked about problems at her school with staff
"buy-in" to the CFL model, had this to say: "I don't know of any teacher
in the building who hasn't bought into the program." A veteran principal
explained why she has become a CFL supporter: "I've seen teachers who
have embraced CFL and it has renewed their teaching in important ways."

It was noted by several Lab collaborators consulted during the site
visit that LSS has assembled a world-class team of scholars and
researchers that distinguishes LSS from other Regional Educational
Laboratories. Indeed, the opinion was often expressed that LSS sets the
standard within the Laboratory network for research rigor and quality.

It is useful here to include excerpts from an LSS
collaborator---Professor Edmund Gordon from Yale University and The
College Board's National Task Force on Minority High Achievement---on
the effectiveness of the Invitational Conferences:

*"As an observer and/or participant observer at several of the
conferences conducted by LSS, I am continually impressed by the quality
of this program of conferences. Consistently they are well-conceived,
excellently run, extremely informative, and provocative of discussion
and thought. I find especially interesting the combination of persons
who attend these conferences. These meetings are one place where one is
certain to encounter scholars of practice, scholars of theory, and
scholars involved in empirical work talking to each other with
appreciation and respect. The issues discussed and the works which are
reported are frequently on the cutting edges of the field. LSS seems to
have a knack for being on top of and relevant to the critical issues of
the day. Not only do LSS conferences make for excellent intellectual
exchange among the participants, but the publications which result from
these deliberations are invaluable assets to those of us working in the
field."*

This high regard for the exceptionally high quality of the invitational
conferences was universally shared by all interviewees with a direct
connection to those gatherings.

Because of its high visibility, the LSS National Invitational Conference
Series program is under close scrutiny by its stakeholder groups. The
publication of the proceedings in refereed journals and the review
processes of the widely respected publishers, such as Laurence Erlbaum
Publisher, is another indicator of the quality of the series. Another
indicator of quality is the request for copies of conference proceedings
by LSS stakeholder groups and requests for follow-up activities. For
example, because of the relevance of LSS's Conference on "Education in
the Cities: What Works and What Doesn't," LSS was asked by the
Congressional Education Committee to provide a briefing on the major
findings from the commissioned papers and the recommendations from the
workgroups. A significant factor in the Lab's skill in organizing these
conferences is the Executive Director's professional connections and
national reputation as a scholar. A member of the Stakeholder Board
interviewed during the site visit attested to the Executive Director's
"ability to connect with power sources to pull off these invitational
conferences."

The publications of the Lab and also the videos produced (for example,
the three-video series created as an implementation tool for
practitioners working with CFL) that were examined prior to and during
the site visit were all of the highest quality.

Selected LSS products have been widely used, providing schools with
intensive on-site professional development and technical assistance.
During Year 3 of operations, for example, 215 schools are involved in
implementing the 20/20 Analysis as a planning and program monitoring
tool, and 173 schools are implementing or planning to implement CFL as a
comprehensive reform strategy.

**2. Areas of needed improvement**

None

**3. Recommendations for improvement**

None

**IV. Utility**

**A. To what extent are the products and services provided by the
Laboratory useful to and used by customers?**

### 1. Strengths

Data regularly collected from CFL implementation sites indicate that it
is feasible for rural and urban schools to achieve a high degree of
implementation of CFL within a year, and to demonstrate student learning
progress. In the words of the Executive Director, "We're doing a
disservice to the country by saying that change takes a long time."
Because of the immediate results that CFL schools are able to achieve,
LSS has been asked by school districts to train principals and
facilitators of other comprehensive reform models to use the LSS
approach to data-based staff development. For example, LSS was requested
by the Superintendent of Washington, D.C. schools to provide staff
development training to all of the facilitators the district has
assigned to more than 50 low-performing schools---not just those
implementing CFL---to provide teacher support. The D.C. Superintendent
has also asked LSS to provide training to all middle school principals,
even though there is no expectation that CFL will be one of the reform
approaches for their middle school initiative. The point is that the LSS
model for staff development has been recognized as a useful approach as
part of the gearing-up process for comprehensive reform efforts of all
kinds.

An important part of the Lab's usefulness for schools within the Region
has been its growing visibility. As a member of the Stakeholder Board
stated during the site visit, "They've done a good job of making schools
aware they're here and that services are available." According to
several sources interviewed on-site, the previous Lab was "not
particularly responsive: you had to come to them---they wouldn't come to
you." Teacher and administrator satisfaction with CFL products and the
model was apparent from conversations during the site visit. The
Executive Director summarized the Lab's usefulness quite succinctly: "We
help schools use their own data to plot their own strategies." Although
the ALEM component of the CFL model may appear at first glance to be
rigidly prescriptive, there is enough flexibility built into the model
to allow for teacher autonomy, and student choice and
self-expression---coupled with a tight focus on accountability, careful
record-keeping, and skills development in math and reading.

In CFL, program implementation staff from LSS provide 10-15 days of
on-site professional development and technical assistance to teachers
and related services staff on an as-needed basis. This assistance is
custom-designed for each school based on needs identified by teachers,
observations by principals, and implementation assessment data gathered
by program staff. Additionally, the project coordinator and facilitators
and principals from participating schools assist with professional
development, and successful CFL teachers provide peer coaching and
mentoring. The facilitators are trained by the developer on site and at
three to four regional one-day meetings each year, and twice a year, the
developer visits the school to evaluate implementation, using indicators
of implementation. Implementation is supported by a team of program
implementation specialists from LSS, and commitment by consensus of a
school's staff is required for whole-school implementation. LSS holds an
annual seminar for the network of CFL schools, and all school
facilitators meet periodically for planning. A listserv has been created
for CFL teachers to share ideas, and school staff receive research
briefs and publications from LSS on a regular basis.

It is important to note the obvious effectiveness of LSS personnel in
the Services to the Field Unit in providing capacity-building support in
participating CFL schools and how often LSS staff are in the building
and classroom to impact both teachers and students. The follow-up
support is both necessary and greatly appreciated. A principal from a
CFL demonstration site explained what most schools are accustomed to
prior to their relationship with LSS: "We've had many years at our
school of people coming in with new training, and two weeks later,
nothing has changed. There was no follow-up."

#### 2. Areas of needed improvement

LSS research publications are not written in a user-friendly style or
readily accessible to practitioners. A greater effort could be made to
involve teachers as an important part of the intended audience for
research publications without "dumbing down" the content through a
reductionistic translation process.

#### 3. Recommendations for improvement

Regional Lab publications provide fertile ground for attempting to
overcome the exclusion of teachers as an intended audience, which helps
account for their absence from high-stakes policy discussions.
Imaginative and innovative ways of sharing research data and insights
with the broader public including teachers must be explored.

**B. To what extent is the REL focused on customer needs?**

In all of the program activities reviewed during the site visit, it was
clear to this reviewer that the LSS is clearly focused on responding to
customer needs as the touchstone of its work. In the words of the
Executive Director, "You meet whatever needs are there."

**1. Strengths**

The entire CFL model is organized around the accurate and ongoing
collection of school data as the basis for identifying student and
teacher \[students and teachers are the basic CFL "customers"\] needs in
the areas of instruction, skills development, and professional
development. The individualized instructional program for each student
as organized in the prescription sheet and the staff profile for
professional development planning---both made possible by CFL's
procedures for data collection---provide an accurate snapshot at any
given time of student and teacher needs within a participating school.

One of the most important design features of the National Invitational
Conferences is the

next-step planning at each conference that emphasizes finding solutions
through policy development and program implementation processes that are
responsive to the needs of schools and other stakeholder groups. These
next-step recommendations also become the source for topic
identification for subsequent conferences in the Series.

In addition to ongoing informal feedback and evaluation obtained from
sites where LSS plays a professional development role, a series of
"tracer studies" to assess the impact of LSS professional development
events in the Region is regularly carried out by telephone six months
after the event by the LSS quality control and evaluation unit using a
50 percent random sampling of attendees. Results of the tracer studies
serve as a barometer of regional needs and inform the planning of future
events.

Still another aspect of LSS's responsiveness to customer needs is
grounded in its ability to help schools build their own on-site capacity
to improve teaching and learning, and to help them avoid dependence on
outside support as the primary engine for school improvement. As the
Associate Director put it, "From the moment we're walking into a
building, we're walking out." \[i.e., we help them build their own
capacity for change and leave critical support structures in place\].

**2. Areas of needed improvement**

None

**3. Recommendations for improvement**

None

V.  **Outcomes and Impact**

**A. To what extent is the REL's work contributing to improved student
success, particularly in intensive implementation sites?**

##### 1. Strengths

The LSS 20/20 analysis tool is an example of how LSS assists schools to
improve student achievement. Using student achievement data (and other
related outcome measures) collected by the school and/or district, the
procedure identifies which students fall in the top and bottom 20
percent of the district's test scores. Data provided by LSS from schools
using 20/20 analysis show marked student improvement in outcomes
including achievement, attendance, and discipline referrals.

In its 1999 compendium of promising comprehensive school reform models
entitled [An Educator's Guide to Schoolwide Reform]{.underline}, the
Institutes for Research says the following about the positive effects of
CFL on student achievement, which corresponds to data collected for the
CFL demonstration site school visited during the site visit:

*"Evidence of positive effects on student achievement is promising,
based on the outcomes from five studies. One study found that students
using the approach had higher reading and mathematics achievement than
students who were not using the approach; and that test scores in
schools using the approach improved at the same time that scores in
similar schools declined. A second study found that math and reading
standardized test gains were significantly higher for students in CFL
schools compared to national norms. This study also found that
mainstreamed special education students in CFL schools scored
significantly higher than similar students not using the approach."*

Implementation of CFL has consistently been shown to contribute to
improvement---

though not necessarily in a direct cause-and-effect relationship---of
student achievement and related student and school outcomes, such as
higher attendance rates, parent involvement in school activities, fewer
disciplinary referrals, and more positive student attitudes towards
school. These additional outcomes which appear to be related to CFL
interventions were consistently mentioned by all school representatives
interviewed during the site visit.

Schools in some of the nation's most impoverished inner city areas have
achieved positive results following CFL implementation. A study of the
first year of implementation of five CFL schools in the District of
Columbia (schools identified as among the lowest performing in the
district) found that teachers were making significant changes in
classroom practice. The study also examined changes in student reading
scores on the Stanford 9 and found that scores improved at all five
schools, that program schools improved more than other elementary
schools in the district, and that the districtwide ranking of program
schools climbed considerably (one school jumped from 119^th^ to 46^th^,
for example). In terms of student achievement gains between fall and
spring testing for the 1997-98 academic year for reading and math, all
the CFL demonstration schools exceeded the District's improvement
standard of a 10 percent achievement gain. Another noteworthy finding in
the pattern of progress in student achievement in math and reading is
the gains made by students who scored at the top 20 percent across all
D.C. CFL demonstration schools, which is reflective of the design focus
of the CFL program. Although most efforts to increase student
achievement in urban schools like those implementing CFL in Washington,
D.C. tend to focus on the lower levels of student achievement, a key
design principle of CFL is the ability of staff to adaptively respond to
the diverse learning needs of all students, including those at both the
bottom and the top range of the achievement distribution.

The Degree of Program Implementation Assessment Battery was used to
collect

information on the extent to which the 12 critical dimensions of the
instructional component of CFL (known as the Adaptive Leaning
Environments Model or ALEM) were implemented in each classroom in all
six CFL demonstration schools in the D.C. district. This Degree of
Program Implementation Assessment Battery is routinely used in all CFL
implementation schools to assess program implementation progress.

At a middle school in inner city Philadelphia (the one visited by the
Review Panelists during their site visit) where 78 percent of students
are Latino and 93 percent live below the poverty line, students have
shown significantly higher academic progress than students at a control
school. A follow-up study of students who had attended this middle
school reported that they had a significantly lower dropout rate than
their high school peers (19 percent vs. 60 percent) and that 48 percent
of them were performing at grade level in the eleventh grade compared to
26 percent of their peers. A similarly situated elementary school in
Houston also witnessed improvements in student achievement, along with
positive changes in students' and teachers' attitudes about their
school.

#### 2. Areas of needed improvement

LSS is developing a sophisticated understanding of what comprehensive
school reform means that goes way beyond raising student achievement
test scores to include changing the organizational and intellectual
climate and culture of schools and their communities. Yet, much of the
LSS written emphasis in describing and promoting CFL emphasizes its role
in raising test scores. Because LSS is quite successful and has
established credibility in both areas---in raising test scores and in
changing the larger school culture---it is in a position to help schools
and their communities to think in more comprehensive and global ways
about what school improvement and student success mean. Without
deviating from the Lab's commitment to data-driven and research-based
approaches, LSS can begin to raise the level of discussion about school
reform and keep that discussion of larger issues on the front burner
wherever the Lab is involved.

Most interviewees involved in school reform efforts talked about a wide
variety of positive changes in teacher, student, and parental behaviors
within their schools that are not as easily measurable or quantifiable
as student scores on achievement tests, yet they are no lesser
indicators of school improvement and student "success," broadly
conceived. LSS can make an important contribution to the school change
literature and to the "procedural knowledge base" by attending to those
"other successes" and learning how to document and give credibility to
them. For example, developing ways to carefully document the evolution
of school/community partnerships, the move toward more positive
attitudes about school on the part of everyone involved, or the
difficult transition experienced by teacher who are attempting to change
their practice would provide other schools with the tools they need to
convince their communities of the benefits of continuing with
comprehensive reform efforts even if dramatic increases in student test
scores are not able to be sustained. There are a multitude of other good
reasons besides raising test scores for schools to begin the change
process, and LSS can contribute to this larger understanding of what a
"good school" is.

#### 3. Recommendations for improvement

"Collecting data" at CFL demonstration sites doesn't need to preclude
more contextualized narrative, ethnographic, and multi-voiced attempts
to "tell the story" of change and student achievement---broadly
conceived---within a particular school. It involves undertaking a more
comprehensive, labor intensive, and multi-faceted approach to "research"
and involving a wider range of participants more directly in the
research process.

**B. To what extent does the Laboratory assist states and localities to
implement comprehensive school improvement strategies?**

####### 1. Strengths

In their Technical Proposal, LSS stated that it would *"provide
assistance to state and local education agencies in their systemic
reform initiatives to "put the pieces together" to achieve a systemic
whole in improving student outcomes and to extend their successes to
scale*." The Lab was quite well-positioned to respond quickly and
supportively to requests from schools in the region when the CSRD
legislation was passed. A state Department of Education official from
the Region interviewed during the site visit made it clear that the Lab
played a critical role at that time: "Schools in the Region had no way
of knowing what these comprehensive programs were. There was only one
place to go to be able to respond to the RFP: the Lab." As the Executive
Director described the Lab's situation, "We're ahead of the game: we've
been doing comprehensive school reform for a while." A member of the
Stakeholder Board made it clear that LSS provided prompt and valuable
support to schools and districts related to knowing more about
comprehensive school reform precisely when it was needed: "The Executive
Director responded quickly and gave relevant information to the schools.
As a result, Philadelphia schools did well in the CSRD competition."
Clearly, LSS exceeds its contractual obligations in this area.

LSS excels at working with states, schools, and localities to implement
comprehensive school reform. To do this the Lab identifies timely and
emerging issues related to comprehensive reform and launches regional
conversations on topics of importance. For example, LSS disseminated
information related to the Third International Math and Science Survey
(TIMSS) to professionals and other stakeholders in the mid-Atlantic
region through its co-sponsorship of five forums. At these forums
participants were charged with examining what is known about effective
practices for student achievement in math and science and for
identifying next-step actions to meet the improvement needs of schools
through networking and regional collaborative activities.

Also noteworthy in the Region is LSS's role in promoting needed
discussion regarding teacher licensing and recruitment, another area
closely tied in with comprehensive school reform. The National
Invitational Conference Series held meetings in January 1998 on issues
of standards and accountability and in February 1999 on teachers and the
reform movement. As a result of these two conferences, LSS, in
collaboration with the Council for Basic Education, has established a
major regional initiative to work together to carry out the resolutions
reached at the conference, which are largely concerned with making sure
that the Region agrees upon high standards for teachers while
aggressively addressing teacher supply needs. Participants at these
conferences were asked to consider a common approach to graduating,
licensing, recruiting, and hiring new teachers to ensure highly
qualified teachers through teacher entry/exit and licensure standards
and high standards for teacher preparation programs throughout the
mid-Atlantic region. Related considerations were related to ensuring an
adequate supply of qualified teachers, and to helping employers and
teachers find each other. As a result of these LSS initiatives, these
issues are now being discussed at the policy level in the Region, and
several LSS collaborators questioned during the site visit made it clear
that LSS plays a unique role in terms of being able to bring all the
relevant stakeholders to the table for this kind of regional
collaboration.

An obvious LSS connection to assisting schools in the Region with
comprehensive school reform is related to the support it provides in
promoting the Lab's own preferred model of reform as embodied in the
Community for Learning (CFL) model. The transition from ALEM to CFL has
meant the scaling up of the LSS reform model from the classroom level to
the school level, and there is a good 'fit' between the CFL model and
the comprehensive school reform legislation and mandates at both state
and federal levels. Currently LSS is expanding this successful
schoolwide model to the district level (as, for example at the Penn
Hills district in rural Pennsylvania), and simultaneously participating
in state-level and regional policy conversations that coincide with the
alignment of state policies with the recent federal CSRD legislation.
LSS has also expanded the CFL approach that was originally designed for
low-achieving urban schools to be adaptable to and effective in
low-achieving rural districts.

**2. Areas of needed improvement**

One critical area for schools committed to comprehensive school reform
and working with outside support such as LSS is the issue of
sustainability: can a sufficient capacity for change be created on-site
to allow significant improvement efforts to continue when the outside
support is no longer there? As one regional principal put it during the
site visit: "It's like a baby that you need to wean. You can get so
comfortable with all this support from the Lab that you can't run with
it by yourself." Although the Lab remains in a position to continue to
offer CFL demonstration schools ongoing implementation support for the
near future, such will not always be the case. It is unrealistic to
assume that LSS will have the resources to continue indefinitely being a
regional provider of technical support for a growing network of CFL
schools, and plans should be made that address that contingency.

**3. Recommendations for improvement**

LSS should begin to strategize now within the CFL network about the
sustainability issue. How can a strong network of CFL schools support
each other and continue to provide the capacity for ongoing
implementation even if LSS personnel no longer have a direct
relationship with the schools? How can LSS help participating schools to
continue to build local capacity for sustaining comprehensive change by
leveraging additional sources of funding? Many previous reform efforts
have floundered on precisely this issue, and LSS can learn from the
mistakes made earlier.

**C. To what extent has the REL made progress in establishing a regional
and national reputation in its specialty area?**

There is overwhelming evidence from all sources examined that LSS has
indeed established a regional and national reputation in its specialty
area of urban education, and continues to build on this reputation.
Additionally, nothing was encountered during the site visit to indicate
that LSS will not continue to enjoy its well-deserved reputation. A
significant part of that reputation is related to the focus of the
Executive Director's personal research agenda for the past 20 years, and
her ability to bring together a remarkable collection of urban education
researchers of national caliber. As a member of the Governing Board put
it, "The Executive Director's work in inner cities is world-class and
groundbreaking in every way."

**1. Strengths**

LSS's exemplary work in urban education is vividly illustrated in the
two Signature\
Works examined in detail by the Review Panel, the Community for Learning
model of reform and the National Invitational Conferences. Because of
LSS's reputation, LSS-sponsored professional development programs
connected with CFL are attended by urban school staff and policymakers
from across the country, and senior staff from the Lab are often invited
as keynote speakers at major professional meetings to discuss the
replication of their work on successful research-based reform strategies
in urban communities.

In terms of the specialty area work, LSS proposed the following in their
Technical Proposal: that the Lab would "*create new vehicles of
professional literature on urban education to bring together reports of
research and syntheses of research and theory from all disciplines and
professions."* And further, the proposal stated that LSS would
"*establish a community of learners for discussing urban education
practices and policies and take on the responsibility to inform
policymakers and decision makers about urban education to bring about
more coherent and accountable local-based approaches to special
problems."* It was made abundantly clear to the review panel during the
site visit that LSS embraces its specialty area responsibility fully and
serves as a recognized national leader in the area of urban education.

Also noteworthy in this area is LSS's collaboration with the Council of
Chief State School Officers---which shares a programmatic commitment
with LSS in terms of a focus on improving student success in high
poverty urban schools\-- in cosponsoring the January 1999 invitational
conference on "Improving Results for Children and Families by Connecting
Collaborative Services with School Reform Efforts."

Another important LSS collaboration in the specialty area is related to
the U.S. Department of Education's Empowerment Zone/Enterprise Community
Task Force (EZ/EC) under the federal Office of Vocational and Adult
Education. Once an initial contact was made with LSS staff by EZ/EC
leadership, common ground was quickly identified and LSS leaders and
members of the President's Community Empowerment Board sat together as
strategic partners and planned an invitational conference cosponsored by
the Lab, the Department of Education's EZ/EC Task Force, and the
Department of Health and Human Service's EZ/EC Task Force focused on
"Development and Learning of Children and Youth in Urban America."
According to testimony heard during the site visit, this conference
proved to be "timely, comprehensive, and extremely successful," and the
post-conference publication was even more productive. Copies of the
conference proceedings were shared with all fifty State Commissioners of
Education, the Superintendents of the 105 EZ/EC school districts
nationwide, and with the local Community Empowerment Boards at every
EZ/EC site. In addition, the U.S. Department of Agriculture has agreed
to use the conference model for organizing a similar conference on Rural
America. It was repeatedly mentioned during the site visit that LSS
leaders have the capacity and political savvy to respond quickly to
emerging policy issues that intersect with already-identified LSS
interests and expertise. It is clear that LSS leaders are exceptionally
adept at collaborating with the appropriate federal-level and
state-level leadership to ensure that the resulting discussions occur
with all the important regional and national players present at the
table. The Lab's leadership role in the area of urban education is a
clear example of this expertise.

Finding ways to turn around low-performing schools in Washington, D.C.
has been a top priority of that School District and of Congress. The
school staff there faces the challenge of implementing reform mandates
with little professional development and implementation support
provided. Building the school-level and district-level capacity to
implement innovative reforms is a major challenge faced by D.C. schools.
LSS has been working intensively with D.C. schools on several fronts. In
addition to providing intensive support for the six lowest-performing
schools as a demonstration of how even the most challenged schools can
achieve significant improvements within relatively short turnaround
time, LSS was asked by the District to provide professional development
for all 58 facilitators the District has assigned to schools that
require extra implementation support, and leadership training for all
middle school principals within the District. LSS is to be commended for
these courageous efforts.

**2. Areas of needed improvement**

None

**3. Recommendations for improvement**

None

**VI. Broad Summary of Strengths, Areas for Improvement, and Strategies
for Improvement**

**1. Strengths**

The Lab is to be commended for building a strong data-driven and
research-based foundation for all their work with schools, and for
maintaining a consistent and coherent vision for applying that
foundation in addressing the needs of some of the most challenging and
low-achieving schools in the Region.

The Lab is to be commended for the high quality, commitment, and
diversity of its leadership and professional staff, and for serving as
an invaluable incubator for minority scholarship and leadership through
the exceptional and supportive network of mentoring relationships that
exist within the staff. Other Regional Labs and other organizations have
much to learn from LSS in this important area.

The Lab quite successfully "grounds" its rigorous research efforts in
field-based implementation work, and these two components continually
enrich each other. Part of the balance between these two
poles---research and practice\--is a healthy organizational tension
between a sophisticated and ambitious research agenda focused on a
particular approach to comprehensive school reform which incorporates an
inclusive special education component, and a much-in-demand "generic"
technical support capacity provided generously and promptly to the
neediest schools in the Region.

There is great integrity within LSS. The vision, mission, and goals of
the organization are congruent, and they pervade the Lab, its
activities, and its products. LSS has developed and continues to enjoy a
broad base of support and respect from the educational community and
policymakers in the four-state region it serves. The ability to
effectively collaborate with and network with various stakeholders and
to form strategic alliances is a great strength of LSS and of its
leadership.

**2. Areas of needed improvement/concerns**

The creation and ongoing effectiveness of LSS is quite heavily dependent
upon the vision, energy, research productivity, national reputation,
networking prowess, and professional connections of its Executive
Director. While her many gifts and talents help to define the Lab and
are themselves significant strengths of the organization, her absence
would create a leadership vacuum that could affect the Lab's ability to
sustain itself. This creates an organizational vulnerability that makes
the Lab unduly dependent upon one person. How can LSS "institutionalize"
the many contributions of its Executive Director so that it could
sustain its exemplary work without her?

LSS claims to be developing a "procedural knowledge base" that can
provide guidance and insight for those undertaking comprehensive school
reform in their schools, districts, and states. It is unclear, however,
just what this knowledge base looks like or what it consists of. This
reviewer is convinced that the Lab is in a strong position to be able to
articulate this "procedural knowledge base" based on its extensive and
intensive involvement in whole-school reform efforts. However, there is
a need for a clearer and more focused understanding and articulation of
what a procedural knowledge base is before the knowledge base can be
codified and shared. In particular, there is a need for understanding
how the direct participants in comprehensive school reform efforts at
the local school level---that is, teachers, administrators, students,
implementation specialists, facilitators, parents, and community
members---can contribute to articulating and documenting that knowledge
base before it can be adequately codified and shared. My sense is that
valuable and detailed portions of this procedural knowledge base will be
more "story-driven" than data-driven, more process-oriented than
results-oriented, and more related to qualitative, narrative, and
ethnographic research approaches than to quantitative and statistical
approaches. Participants in whole-school change efforts at the "micro"
level---that is, the school and district levels\--will need support and
training in documenting and sharing what they are doing in order for
this to happen, and LSS can help facilitate that process. There was some
evidence provided during the site visit that efforts are being made to
address these issues, and LSS should be encouraged to continue to move,
thoughtfully and supportively, in this direction.

The Lab should consider committing more of its resources to the support
and training of its field workers: the quality, consistency, and
thoughtfulness of their work will determine future LSS relationships
with schools within the Region. LSS should beware of stretching these
critical boundary-spanners too thin, and should build in time and
support for reflection and writing into their job descriptions. In
particular, implementation specialists and school facilitators play key
roles in terms of sustaining schoolwide and district-wide reform efforts
once they've begun. They are the true choreographers of the
capacity-building process, and they have much to contribute to the
emerging procedural knowledge base related to school reform. Beyond
that, more opportunities can be provided for interested teachers at
implementation sites to participate in LSS-sponsored activities (all the
teachers, not just the teacher-leaders). When network teachers are
"ready" to change, LSS should be prepared to support them in a variety
of ways as they move rapidly forward.

**3. Recommendations for improvement**

LSS should consider adding special incentives for network teachers who
want to become more involved in the process of comprehensive school
reform: for example, providing mini-grants for action research or
teacher research projects and for teacher writing and publication;
invitations for commissioned papers; opportunities to co-publish and
co-present at national conferences, etc. CFL clearly supports and
respects teachers as implementers and instructors within the change
process, but tends to ignore their potential for reflecting upon,
articulating, and documenting the change process within their schools.
LSS support for these enlarged teacher roles can serve the long-range
best interests of the LSS agenda and of comprehensive school reform
within the Region.