PLANNING AND EVALUATION SERVICE

THE LONGITUDINAL EVALUATION

OF SCHOOL CHANGE AND

PERFORMANCE (LESCP)

IN TITLE I SCHOOLS

FINAL REPORT

VOLUME 1: EXECUTIVE SUMMARY

2001

U.S. DEPARTMENT OF EDUCATION ~ OFFICE OF THE DEPUTY SECRETARY

DOC #2001-20
THE LONGITUDINAL EVALUATION OF SCHOOL CHANGE

AND PERFORMANCE (LESCP)

IN TITLE I SCHOOLS

FINAL REPORT

VOLUME I: EXECUTIVE SUMMARY

Prepared for:

U.S. Department of Education

Office of the Deputy Secretary

Contract No. EA 96008001

Prepared by:

Westat, Rockville, Md.

and

Policy Studies Associates, Washington, D.C.

2001
This report was prepared for the U.S. Department of Education under Contract No. EA96008001. The project

monitor was Daphne Hardcastle in the Planning and Evaluation Service. The views expressed herein are those

of the contractor. No official endorsement by the U.S. Department of Education is intended or should be

inferred.

U.S. Department of Education

Rod Paige

Secretary

Office of the Deputy Secretary

William D. Hansen

Deputy Secretary

Planning and Evaluation Service

Alan L. Ginsburg

Director

Elementary and Secondary Education Division

Ricky Takai

Director

July 2001

This report is in the public domain. Authorization to produce it in whole or in part is granted. While permission

to reprint this publication is not necessary, the citation should be the following: U.S. Department of Education,

Office of the Deputy Secretary, Planning and Evaluation Service, The Longitudinal Evaluation of School

Change and Performance in Title I Schools, Volume 1: Executive Summary, Washington, D.C., 2001.

To order copies of this report, write

ED Pubs

Editorial Publications Center

U.S. Department of Education

P.O. Box 1398

Jessup, MD 20794-1398;

via fax, dial (301) 470-1244;

or via electronic mail, send your request to edpubs@inet.ed.gov.

You may also call toll-free 1-877-433-7827 (1-877-4-ED-PUBS). If 877 service is not yet available in your area,

call 1-800-872-5327 (1-800-USA-LEARN); Those who use a telecommunications device for the deaf (TDD) or

a teletypewriter (TTY), should call 8-800-437-0833.

To order online, point your Internet browser to www.ed.gov/pubs/edpubs.html.

This report is also available on the department’s web site at www.ed.gov/offices/OUS/PES/eval.html.

On request, this publication is available in alternative formats, such as Braille, large print, audiotape, or

computer diskette. For more information, please contact the Department’s Alternate Format Center at

(202) 260-9895 or (202) 205-8113.
iii

CONTENTS

ACKNOWLEDGMENTS ....................................................................................... v

OVERVIEW........................................................................................................ 1

BACKGROUND .................................................................................................. 3

KEY FINDINGS .................................................................................................. 8

CONCLUSIONS................................................................................................... 16

Figures

1 Poverty level of schools in the LESCP sample (n=71) .................................... 4

2 LESCP scores relative to national and urban norms for closed-ended

reading.............................................................................................................. 8

3 Effect of poverty on LESCP student achievement on the SAT-9 in

reading, third to fifth grade .............................................................................. 9

4 Effect of poverty on LESCP student achievement on the SAT-9 in

mathematics, third to fifth grade ...................................................................... 10

5 LESCP student achievement gains on the SAT-9 closed-ended reading

test, third to fifth grade..................................................................................... 11

6 LESCP-predicted score on the SAT-9 closed-ended reading test,

third grade ........................................................................................................ 12

7 LESCP student achievement gains on the SAT-9 closed-ended

mathematics test, third to fifth grade................................................................ 14

8 LESCP-predicted score on SAT-9 closed-ended mathematics test,

third grade ........................................................................................................ 15

9 Relationship between policy environments and favorable instructional

practices............................................................................................................ 17

v

ACKNOWLEDGMENTS

We are indebted to many individuals whose contributions made the Longitudinal Evaluation of

School Change and Performance (LESCP) possible. The project extended over 5 years and had many

components and phases. Here we can only mention a few of the individuals who had a role in the design,

conduct, and reporting of LESCP. We are especially grateful to our federal Project Officers Elois Scott and

Daphne Hardcastle and to Audrey Pendleton and Jeffery Rodamar who served in an acting capacity. They

provided invaluable substantive guidance, as well as support on the administrative and operational side. We

wish to thank Alan Ginsburg, director of the Planning and Evaluation Service (PES), whose ideas and

questions helped us formulate the research design, study questions, and analyses for LESCP. Ricky Takai,

director of the Elementary and Secondary Division of PES, asked the hard questions as the study and

analyses progressed and kept us on course. Other staff of PES who contributed in various capacities over the

life of LESCP are Stephanie Stullich, Joanne Bogart, and Barbara Coates. From the Compensatory

Education Programs office, Mary Jean LeTendre and Susan Wilhelm provided thoughtful advice. We also

thank Valena Plisko and Michael Ross of the National Center for Education Statistics for their helpful

expertise.

Members of the LESCP Technical Work Group were active contributors to the study effort from

the very beginning. They provided advice on the conceptual framework and research design, reviewed and

advised on plans for analysis, and ultimately on the analysis itself. We wish especially to thank them for

their reviews and thoughtful comments and recommendations during the development of this report.

Members of the Technical Work Group are identified below. Each member served for the entire study

period except where noted.

Technical Work Group Members

Dr. David Cordray, Department of Psychology and Human Development, Vanderbilt University

Dr. Judith McDonald, (1998–2000) Team Leader, School Support/Title I, Indian Education, Oklahoma State

Department of Education

Dr. Andrew Porter, Wisconsin Center for Education Research, School of Education, University of

Wisconsin-Madison

Dr. Margaret Goertz, Consortium for Policy Research in Education

Dr. Mary Ann Millsap, Vice President, Abt Associates

Dr. Jim Simmons, Program Evaluator for the Office of Student Academic Education, Mississippi Department

of Education
vi

Dr. Joseph F. Johnson, Charles A. Dana Center, University of Texas at Austin

Ms. Virginia Plunkett, (1997–98) Colorado Department of Education

Dr. Dorothy Strickland, Graduate School of Education Rutgers University

A large number of Westat and subcontractor staff contributed to LESCP. Linda LeBlanc of

Westat served as project director and Brenda Turnbull of Policy Studies Associates (PSA) as co-project

director. From Westat, Alexander Ratnofsky, Patricia Troppe, William Davis, Ann Webber, and Camilla

Heid served on the analysis and reporting team. Raymond Olsen, Stephanie Huang, Bahn Cheah, Alan

Atkins, Sandra Daley, and Cyril Ravindra provided systems support for analysis and survey operations.

Juanita Lucas-McLean, Therese Koraganie, and Dawn Thomas handled all survey data collection and

processing. Heather Steadman provided secretarial support, Carolyn Gatling provided word processing

support, and Arna Lane edited this report.

At PSA, Megan Welsh played a major role in the study’s analysis and reporting. Other PSA

staff members who also made substantial contributions to the study are Joelle Gruber, Ellen Pechman, Ullik

Rouk, Christina Russell, and Jessica Wodatch.

Jane Hannaway of the Urban Institute coordinated the teams of site visitors from the Urban

Institute and along with Nancy Sharkey assisted with analyses of some of the early study data.

Everett Barnes and Allen Schenck organized and oversaw the work of RMC Research

Corporation staff from that company’s Portsmouth, Denver, and Portland offices over three cycles of site

visits and data collection for LESCP.

The analyses of student achievement benefited from the help of Aline Sayer, of the Radcliffe

Institute for Advanced Study, Harvard University, who lent her expertise in the statistical technique of

hierarchical linear modeling.

The Longitudinal Evaluation of School Change and Performance would not have been possible

without the support and participation of school principals and teachers who welcomed us into their schools

and provided the heart of the information on which this report is based. We are particularly indebted to the

school administrators who took on the task of coordinating all LESCP activities at their schools. Our

greatest thanks go to the 12,000 students who took part in the study and did their best on the assessments we

administered each spring and to their parents for allowing their children to contribute to the body of

knowledge on school reform.
1

THE LONGITUDINAL EVALUATION OF SCHOOL CHANGE

AND PERFORMANCE (LESCP)

IN TITLE I SCHOOLS

OVERVIEW

The Longitudinal Evaluation of School Change and Performance (LESCP) followed the

progress of students in 71 high-poverty schools as they moved from third to fifth grade. The study was

designed to investigate the impact on student achievement of specific classroom practices fostered by

school-, district-, and state-level policies. This longitudinal analysis, conducted between 1996 and 1999

as part of the National Assessment of Title I, especially was intended to test the effects of changes in

curriculum and instruction called for by advocates of standards-based reform. Many of these changes

were being made in the wake of the 1994 Amendments to Title I of the Elementary and Secondary

Education Act, which called on states to adopt challenging academic standards and assessments aligned

with these standards by 2001.

Like previous research on Title I, this study clearly demonstrated that student and school

poverty adversely affected student achievement in both reading and mathematics. Although most Title I

research has sought to identify school practices that can improve student achievement, this was the first

major study to examine the impact of standards-based reform practices on student achievement. Students in

the LESCP schools, on average, did not catch up with national norms during the course of the study. The

data analysis, however, revealed certain school practices and standards-based policies that were more likely

to result in student achievement gains as students moved from the third grade through the fourth and fifth

grades. Not all school practices and standards-based policies studied were clearly linked with student

achievement gains.

The study found that reading achievement improved faster when two factors were present:

• Teachers gave high ratings to their professional development in reading. The

growth in student test scores between grades three and five was about 20 percent

greater when teachers rated their professional development high than when they gave

it a low rating.

• Third-grade teachers were especially active in outreach to parents of lowachieving students. Growth in test scores between third and fifth grade was 50

percent higher for those students whose teachers and schools reported high levels of
2

parental outreach early than students whose teachers and schools reported low levels

of parent outreach activities for the third grade.

Conversely, reading achievement was less likely to improve when fifth-grade teachers spent

considerable time engaged in basic instruction such as filling out worksheets or reading aloud. Growth in

test scores was 10 percent lower when teachers spent a lot of time on basic instruction than when they

spent little time engaged in these activities.

Factors that positively affected mathematics achievement gains were the following:

• Teachers who highly rated their professional development in mathematics.

Growth in test scores between grades three and five was 50 percent higher for those

students whose teachers and schools rated their professional development high than

when they gave it a low rating.

• Early teacher outreach to parents of students who initially showed low

achievement. Test scores in mathematics grew between the third and fifth grade at a

40 percent higher rate for students in schools whose teachers reported high levels of

parental outreach than students in schools whose teachers reported low levels of

parental outreach activities

• Instructional practices that involved students in more exploration in upper

grades. Growth in test scores between the third and fifth grades was about 17 percent

greater for students whose fifth grade teachers reported relatively very high usage of

exploration in instruction versus students whose fifth grade teachers reported

relatively very low usage.

On the other hand, students’ mathematics scores fell further behind when their schools had

disproportionately more low-achieving students and teachers were relatively satisfied with their own

instruction skills. Teachers' use of standards and assessments had inconsistent effects on mathematics

achievement. Students’ gain scores were lower than the LESCP average in schools where teachers

reported knowing and using standards and assessments. One interpretation of these findings is that

schools that initially paid the highest level of attention to standards and assessments might have been the

schools where poor student performance was a problem.

Achieving higher levels of implementation on the significant variables can have important

impacts for student achievement growth. For example, in mathematics, the 2-year gain would be 18

points higher if both the student's own teacher and the average of all teachers in the school gave very high

ratings to professional development (at the 90th percentile for all teachers and schools in the sample) as

opposed to very low ratings (at the 10th percentile). An 18-point differential is sizable compared with a
3

46-point average, 2-year gain in mathematics for all schools. For the variable of outreach to the parents

of low achievers, the 90th to 10th percentile differential would translate into a 17-point gain for students

who were initially low achieving.

BACKGROUND

Title I is the major federal program designed to improve learning and achievement among

elementary and secondary students in high-poverty schools. Where poverty is at least 50 percent, the

school may use Title I funds for schoolwide improvements benefiting its at-risk students. The 1994

amendments to Title I were enacted to help students who are failing, or most at risk of failing, to meet

challenging academic standards. The amendments charged states with developing or adopting challenging

standards for what students need to learn and at what level of proficiency. To measure how well students

and schools are meeting these standards, states also must administer assessments aligned with these

standards. Like other federal education policies, Title I does not prescribe how to raise student

achievement; rather, its impact depends on how states and districts reform curriculum and instructional

practices to meet their academic standards. When this study began in 1996, few states, districts, or

schools had begun to implement standards-based reform programs, much less demonstrate their

effectiveness. Consequently, measurable effects on student achievement could only be expected in

schools and classrooms that had instituted reforms at least several years earlier.

Because of this, the schools selected for this study were not intended to be statistically

representative of high-poverty schools at the national or the local level. The 71 schools, all of which

received Title I funds as high-poverty schools, were in 18 school districts in 7 states where standardsbased reforms, including assessments and accountability provisions, either were under way before 1996

or began while the study was in progress. Although all of the schools were affected by reform policies

involving standards, assessments, and accountability, the policies and the speed and thoroughness of

implementation varied. Nonetheless, because these schools had been putting reforms into practice for

some time, their experiences provided a good picture of how standards-based reform has played out in

high-poverty schools and offer valuable lessons about standards-based practices that can improve student

achievement.

Of the 71 schools, 59 were operating schoolwide programs in 1998–99, in which Title I

funds can be used for the benefit of all students rather than being targeted to selected students. This
4

reflected, in part, the high-poverty levels of participating schools (see Figure 1): 15 schools had more

than 90 percent of their students living in poverty, 25 schools had between 75 percent and 90 percent,

21 schools between 50 percent and 75 percent, and 10 schools fewer than 50 percent. In all schools, the

poverty rate was higher than 35 percent. 

Figure 1. Poverty level of schools in the LESCP sample (n=71)

0

5

10

15

20

25

30

>90% 75-90% 50-75% <50%

Number of schools

*

Percentage of students eligible for free or reduced-price lunch

_______________

*In all LESCP schools, the poverty rate was higher than 35 percent.

School and Classroom Variables

This study drew on several sources of data: standardized reading and mathematics

achievement test scores,1 teacher surveys, district administrator and principal interviews, classroom

observations, focus groups of school staff and parents, and documents regarding school districts’ policies

related to standards-based reform. It traced the students’ achievement scores and examined the effect of a

number of student and school-level variables involving school practices, teacher preparation, and reform

 

1

 The SAT-9 scores provided a uniform measure of student scores and gains across the whole sample, enabling the study team to make

comparisons among all the schools and classrooms. Because Title I emphasizes attainment of state standards, however, the SAT-9 would not be

a good choice for this purpose if it measured content that was not closely related to the states’ own expectations for students. Therefore, the

study team used data from state assessments to gauge whether student performance on the SAT-9 had a reasonably good relationship to

performance on the knowledge and skills that the states were measuring. The researchers correlated schools' performance on the SAT-9 with

their performance on the state assessment by comparing how the school did on each measure in relation to the other LESCP schools in the state.

In five of the seven states, there was good correlation between relative rankings by SAT-9 and relative rankings by state assessments. Thus,

while SAT-9 does not map perfectly onto the content and skills measured by all states, it appeared to provide a good substitute for state

assessments in the analysis of student achievement. Number of schools
5

policies on both initial achievement and changes over time.2 Data were analyzed for both individual

teachers and entire schools. Information about teachers’ curriculum and instructional practices can be

held up against student achievement to show which practices may have an effect on achievement gains.

School-level data are important because some potentially effective instructional influences on learning

may come from aspects of the whole school environment or from the broader district- or state-level policy

environment. Students’ achievement, when examined against particular practices in their classrooms and

schools, provides valuable evidence about the impact of these practices.

Most variables were derived from teachers’ survey responses about their familiarity, beliefs,

practices, and preparation related to standards-based reform. The questions addressed specific parts of an

overall vision of standards-based reform: a framework of content and performance standards, together

with assessments and curriculum keyed to those standards, that would command attention and guide

classroom practice; curriculum and instruction designed to engage students in relatively advanced

academic tasks rather than on rote drill and practice; teachers prepared to teach in new ways, having

participated in professional development geared to the standards and assessments; and active

communication between school and home.3 The researchers organized the responses into index

variables—combinations of the teacher’s answers to survey questions that were closely related to each

other, statistically as well as conceptually. (Appendixes B and D in Volume 2 of the report describe the

statistical properties of each index.)

Several variables that were intended to measure the level of activity in standards-based

reform were based on an analysis of district and school policy documents. Although the study was not

designed to investigate policy environments intensively, and no data were collected at the state level, it

was still possible to use this documentary evidence to identify significant differences among the districts.

Based on the policies described in these documents and data published in Quality Counts, the annual

 

2

 The study did not focus on other kinds of school variables such as very small class size (because the sample did not include many very small

classes) or the use of comprehensive models for school reform (because only a few schools in the sample were using them).

3

 The interrelated parts of a standards-based reform vision are described in J. O’Day and M.S. Smith. (1993). “Systemic reform and educational

opportunity,” in S. Fuhrman (Ed.) Designing coherent education policy. San Francisco: Jossey-Bass, pp. 250-312, and in L.B. Resnick and K.J.

Nolan. (1995). “Standards for education,” in D. Ravitch (Ed.), Debating the future of American education: Do we need national standards and

assessments? Washington, D.C.: Brookings Institution. Other sources for the variables measured in this study included work on curriculum and

instruction, such as A. Porter. (1998). A curriculum out of balance: The case of elementary school mathematics. East Lansing, MI.: Institute for

Research on Teaching, College of Education, Michigan State University; and M.S. Knapp and P.M. Shields, (Eds.), (1990). Better schooling for

the children of poverty: Alternatives to conventional wisdom, Volume II: Commissioned papers and literature review. Washington, D.C.: U.S.

Department of Education; and work on parent and family involvement such as S.L. Dauber and J.L. Epstein. (1993). “Parent attitudes and

practices of involvement in inner-city elementary and middle schools,” in A.T. Henderson and N. Berla (Eds.), A new generation of evidence:

The family is crucial to student achievement, Washington, D.C.: Center for Law and Education; and U.S. Department of Education. (1994).

Strong families, strong schools: Building community partnerships for learning. Washington, D.C.: Author.
6

Education Week status report on the implementation of standards, districts could be classified as falling at

the high or low end on specific aspects of standards-based reform.

The study used a statistical technique called hierarchical linear modeling (HLM), which

makes it possible to isolate and measure the effect of each independent variable on an outcome.

(Appendix A in Volume 2 describes the HLM analysis.) This statistical technique enabled researchers to

identify whether each variable had a significant, independent relationship to achievement. The technique

also allowed them to estimate how many points higher or lower a student would score if a variable were

present to an especially great degree in a classroom or school, relative to the rest of the LESCP sample.

Although the effect of each variable was isolated for the study, in actual schools and classrooms, the

different variables occur together, creating overall instructional conditions in which students learn. The

bottom line for students, then, is a combination of all the positive or negative influences on achievement.

The school and classroom practice variables tested as possible contributors to students’ test

scores and gains were the following:

• Visibility of Standards and Assessments. This variable was based on responses

regarding teachers’ familiarity with their state or district content standards, curriculum

frameworks, student assessments, and performance standards for mathematics and

reading, as well as their perceptions about how their curriculum reflected each of

these elements.

• Basic Instruction in Upper Elementary School Grades. This variable drew on

responses regarding how much time students spent in more basic, rather than

advanced, activities in reading instruction, such as reading aloud, completing

workbooks or skill-sheet assignments, practicing phonics, and practicing word attack.

• Preparation for Instruction. This variable drew on responses about how well

prepared teachers considered themselves to use small-group reading and mathematics

instruction; take into account students’ existing skill levels when planning curriculum

and instruction; integrate reading or mathematics instruction into other content areas;

use a variety of assessment strategies; and teach groups that are heterogeneous in

ability.

• Exploration in Instruction. This variable, specific to mathematics, was developed

from teachers’ responses about how much they used manipulatives to demonstrate

concepts in mathematics and discuss multiple approaches to solve a problem. The

variable also covered the frequency with which their students worked on problems in

small groups, had whole-class discussions about solutions developed in small groups,

had student-led whole group discussions, represent and analyze relationships using

tables and graphs, responded to questions or assignments that required writing at least
7

a paragraph, worked with manipulatives, and worked on projects and assignments that

take a week or more to finish.

• Presentation and Practice in Instruction. This variable, also specific to

mathematics, was based on responses about the extent to which teachers used

relatively conventional instructional activities, such as lectures or presentations,

whole-group discussions, demonstrations of exercises on the board, and tests. This

variable also addressed how often teachers had students respond orally to questions,

work individually on written assignments or worksheets in class, and practice or drill

on computational skills.

• Rating of Professional Development. This variable drew on teachers’ opinions

about the quality of their professional development in reading and mathematics

content and in instructional strategies. This variable also covered teachers’ opinions

about how well their professional development reflected their school’s or

department’s plans to change practice; supported reform efforts in their school;

supported state or district standards or curriculum frameworks; supported state or

district assessments; bolstered their confidence in using new pedagogical approaches;

and enhanced their ability to adapt their teaching to state assessment requirements,

state standards, or curriculum framework requirements.

• Outreach to Low-Achievers’ Parents. This variable measured the extent to which

teachers communicated with parents of low-achieving students through face-to-face

meetings, sent them materials on ways to help their child at home, and telephoned

them when their child was having problems and, more routinely, when there were no

problems.

• Focus on Assessment and Accountability. This variable, which characterized

districts that were especially active in measurement-related reforms, combined several

items concerning the reporting and the consequences of assessments: whether the

district reported assessment data in terms of proficiency levels; whether the district

publicly reported school and district achievement status in readable and

understandable formats using data disaggregated for at least two demographic groups

by 1998; and whether schools in the state or district were subject to sanctions or

rewards based on student performance as of 1997.4

• Standards, Consequences, and Plans. This somewhat broader variable combined

the enactment of standards and accountability policies with the citation of these

policies in school and district improvement plans. It summarized a number of districtlevel items: whether the state or district had enacted content standards in reading and

mathematics by 1997; whether schools were subject to state or district sanctions or

rewards based on student performance as of 1997; whether district and school

improvement plans consistently referred to content standards in all 3 years of the

 

4

 A total of 34 schools were in “high” districts, and 20 were in “low” districts.
8

study; and whether district and school improvement plans consistently set specific

goals linked to required assessments in all 3 years.5

KEY FINDINGS

The findings of this study are presented here in four sections. The first section reviews

findings having to do with the effects of poverty on students' mathematics and reading achievement. The

next two sections summarize the effects of school and classroom practices on reading and mathematics

achievement. A final section presents findings on the effects of the policy environment of instructional

practices.

Effects of Poverty on Students’ Reading and Mathematics Achievement

Individual and school poverty had a clear, negative effect on student achievement.

• On average, students in the LESCP sample scored below national norms and urban

norms in all years and grades tested (see Figure 2).

Figure 2. LESCP scores relative to national and urban norms for closed-ended reading*

602

607

614

607

621

628

637

634

640

646

654

647

570

580

590

600

610

620

630

640

650

660

Third grade (1997) Fourth grade (1998) Fifth grade (1999)

Full LESCP

Longitudinal LESCP

National norms

Urban norms

* This figure presents two different groups of students who were tested as part of LESCP. The “full LESCP” refers to all students tested in a

particular grade for a particular year (e.g., third graders in 1997). The “longitudinal LESCP” refers to only those students who were tested

three times: as third graders in 1997, fourth graders in 1998, and fifth graders in 1999.

 

5

 Twenty-six schools were in high districts, and eight were in low districts.
9

 ̧ Students in the LESCP schools had average reading scores of 602 in third grade

and 640 in fifth grade, compared with national norms of 614 in third grade and

654 in fifth grade.6

 ̧ Students in the LESCP schools had average mathematics scores of 593 in third

grade and 640 in fifth grade, compared with national norms of 600 and 646.

• Students who lived in poverty (i.e., were eligible for free or reduced-price lunch) did

significantly worse initially on the reading and mathematics tests than other students

in this sample.7 The analysis showed that a student whose teachers and school were at

the average for every variable measured, but who came from a poor family, would

 ̧ Have a third-grade reading score 6.1 points below the average for the sample.

Such a student then would make gains at an average pace (if no other variable

departed from the average value), neither closing the gap nor falling further

behind in reading (see Figure 3).

Figure 3. Effect of poverty on LESCP student achievement on the SAT-9 in reading, third to fifth grade

596.8

602.5

608.6

+38.3

+38.3

+38.3

500 525 550 575 600 625 650 675

100% of students in school 

are poor

Student is eligible for free or

reduced-price lunch

Student experiences average

conditions on all variables

Scale score points

Third-grade score

Third- to fifth-grade gain

Reading scores under 

average conditions:

Effect of poverty variables 

on reading scores:

Fifth-grade national norm (654 points)

Third-grade national norm (614 points)

640.8

635.1

646.9

Figure reads: A student who experiences average conditions on all variables has a predicted third-grade score of

608.6 and a predicted fifth-grade score of 646.9 (gain of 38.3). Both scores are below the national

norms for that grade. A student who is eligible for free or reduced-price lunch has a predicted thirdgrade score of 602.5 and a predicted fifth-grade score of 640.8 (gain of 38.3).

 

6

 Because these are scale scores, which generally rise with each year that a student advances, a typical student would start out with a score of

around 614 and gain about 40 points over 2 years.

7

 This finding is consistent with other research on Title I schools. See, for example, M.J. Puma, N. Karweit, C. Price, A. Ricciuti, W. Thomspon,

and M. Vaden-Kiernan. (1997). Prospects: Final report on student outcomes. Washington, D.C.: U.S. Department of Education.
10

 ̧ Score 5.8 points below the average third-grade student in this sample on the

mathematics test, with all other variables held constant (see Figure 4).

• Students who attended schools with the highest percentages of poor students

performed

 ̧ Worse initially on the reading test than their peers.8 In a school where 100

percent of students were eligible for free or reduced-price lunch, a student’s

third-grade reading score would be 11.8 points below the average for the

sample, holding all the other variables constant. However, school poverty by

itself did not affect the pace at which students gained in reading. These

students would neither close the gap nor fall farther behind if all other variables

were at average values for the sample.

 ̧ 8.9 points below the average on the mathematics test in third grade. However,

the students in these schools partially closed this initial gap, gaining 5.8 more

points than an average LESCP student by the end of fifth grade.

Figure 4. Effect of poverty on LESCP student achievement on the SAT-9 in mathematics,

third to fifth grade

588.2

591.3

597.1 +46.2

+46.2

+52

500 525 550 575 600 625 650 675

100% of students in school 

are poor

Student is eligible for free or

reduced-price lunch

Student experiences average

conditions on all variables

Scale score points

Third-grade score

Fifth-grade national norm (646 points) Third- to fifth-grade gain

Third-grade national norm (600 points)

Mathematics scores 

under average 

conditions:

Effect of poverty variables 

on mathematics scores:

643.3

637.0

640.2

Figure reads: A student who experiences average conditions on all variables has a predicted third-grade score of

597.1 and a predicted fifth-grade score of 643.3 (gain of 46.2). Both scores are below the national

norms for that grade. A student who is eligible for free or reduced-price lunch has a predicted thirdgrade score of 591.3 and a predicted fifth-grade score of 637.5 (gain of 46.2).

 

8

 Again, this finding is consistent with the earlier work of Puma and colleagues (1997), among others.
11

Effects of School and Classroom Practices on Reading Achievement

When third-grade teachers were especially active in outreach to low achievers’ parents,

students made faster gains in reading over the next 2 years, gaining 4.6 points more than students

whose teachers made only an average amount of outreach.9 In addition, in schools where all thirdgrade teachers were especially active in outreach to low achievers’ parents, students gained an

additional 3.7 points by fifth grade. Third-grade classrooms with generally low achievement tended

to be the ones in which teachers reported contacting more parents of low achievers.

When teachers gave higher ratings to their professional development in reading, students

gained 3.1 points more than average during 2 years in school (see Figure 5). This occurred when

teachers described their professional development as matching the school’s reform plan, focusing

on standards and assessments, and boosting their confidence in using new approaches.

Figure 5. LESCP student achievement gains on the SAT-9 closed-ended reading test, third to fifth grade

+38.3

+36.4

+41.4

+46.6

500 525 550 575 600 625 650 675

Teacher and school report 

high outreach to low

achievers' parent2

School reports high rating of

professional development

Teacher emphasizes basic

instruction in upper grades1

Student experiences average

conditions on all variables

Scale score points

Average third-grade score (608.6)

Third- to fifth-grade gain Fifth-grade national norm (654 points)

Third-grade national norm (614 points)

Reading scores under 

average conditions:

Effect of each variable

 on reading gains:

646.9

645.0

650.0

650.1

1 Teachers emphasizing basic instruction spend more time with worksheets, reading aloud, and other types of relatively routine skill practice.

2 Outreach to low achievers' parents measured in third grade was associated with both the predicted third-grade score and the score gain from

third to fifth grade. For that variable, the figure reflects a significant difference in the predicted third-grade score (603.5) rather than the average

third-grade score (608.6).

 

9

 We use the 90th percentile of teacher responses to illustrate the size of the effect associated with each instructional variable.
12

Students whose fifth-grade teachers reported spending more time instructing the class at a

basic level in reading with worksheets, reading aloud, and other types of relatively routine

skill practice gained 1.9 points less on average than those whose teachers spent an average

amount of time working at that level. This result appears to support the theory that continuous

instruction in basics at the upper elementary level impedes students’ progress. Another possible

explanation for the result is that teachers whose students showed weaker reading skills in fifth

grade felt the need to reinforce some basic skills such as phonics.

Students’ initial reading scores tended to be higher in classrooms where teachers reported

they were aware of and implementing the policies of standards-based reform (see Figure 6).

When third-grade teachers reported very high visibility of standards and assessments and said that

they believed their curriculum reflected these policies, their students scored 2.8 points above the

LESCP average in the third grade.

Figure 6. LESCP-predicted score on the SAT-9 closed-ended reading test, third grade

603.5

611.4

608.6

500 525 550 575 600 625 650 675

Teacher and school report high

outreach to low achievers' parents

Teacher reports high visibility of

standards and assessments

Student experiences average

conditions on all variables

Scale score points

Third-grade national norm (614 points)

Reading scores under 

average conditions:

Effect of each variable

on reading scores:

Figure reads: A student who experiences average conditions on all variables has a predicted third-grade score of

608.6, below the national norm. For a student whose teacher reports high visibility of standards and

assessments, the HLM model predicts a third-grade score of 611.4 (2.8 points higher).

The best combination of circumstances for reading achievement gains was the following:

• Less use of basic instruction in upper elementary school grades,

• Higher teacher ratings of professional development,

• More intense outreach to parents of low achievers, and

• Higher visibility of standards and assessments in third grade.
13

The sample included 110 students whose classrooms and schools offered exactly this combination

of circumstances. These students, who started out 10.6 points behind their peers and 18 points

below national norms, narrowed the gap to end up 4.9 points behind their peers and 12 points

below national norms.

Effects of School and Classroom Practices on Mathematics Achievement

Students of fifth-grade teachers who reported higher levels of exploration in mathematics

instruction gained 3.6 points more than the average on mathematics tests over 2 years.

10

Implementation of reforms that call for more student-initiated activities and more complicated

assignments in mathematics appeared to have a positive relationship to student gains. However,

instruction that involved more exploration may have been offered to classes that showed greater

skill in mathematics (see Figure 7).

Teachers’ ratings of their professional development consistently were related to greater

student gains in mathematics. When all fourth-grade teachers gave high ratings to professional

development in mathematics (saying it was well matched to reform plans, standards, and

assessments and that they had learned from it), students’ gains exceeded the average by 3.9 points.

A similarly high rating from a fifth-grade teacher was associated with an additional boost of 3

points.

Outreach to parents was related to mathematics achievement gains for one subgroup of

students, those who initially showed low achievement. It is possible that when teachers involved

low-achieving students’ parents early on, students were able to resolve their learning difficulties

before they multiplied and went on to raise their test scores.

Schools with more low-achieving third graders relative to the rest of the sample fell further

behind in mathematics over the course of the study. With all other variables held constant, the

schools at the low end on student performance in third grade had gains that were 10.8 points below

the average for the sample. Apparently, the presence of large numbers of low-achieving students

either depressed gains for all students or was an indicator of generally unfavorable educational

conditions in the school.

Students of teachers who expressed high confidence in their preparation for specific kinds of

mathematics instruction gained 5.2 points less than average. To explain this surprising finding,

one can speculate that the teachers who had not changed their practices in some years would report

greater levels of comfort about their own preparation, whereas ambitious changes in instruction

might have given teachers doubts about their own skills at the same time as they provided good

environments for student learning.

 

10 Again, we use the 90th percentile of teacher responses to illustrate the size of the effect associated with each instructional variable.
14

Figure 7. LESCP student achievement gains on the SAT-9 closed-ended mathematics test,

third to fifth grade

+41.0

+53.1

+49.8

+38.6

+35.4

+46.2

500 525 550 575 600 625 650 675

School reports high self-confidence

in instruction

Teacher and school report high

rating of professional development

Teacher reports large amount of

exploration in instruction

School reports high visibility of

standards and assessments

Many low-scoring third-grade

students in school

Student experiences average

conditions on all variables

Scale score points

Average third-grade score (597.1)

Third- to fifth-grade gain

Effect of each variable on 

mathematics gains:

Mathematics scores under 

average conditions:

Fifth-grade national norm (646 points)

Third-grade national norm (600 points)

643.3

635.7

646.9

650.2

638.1

632.5

Figure reads: A student who experiences average conditions on all variables has a predicted gain of 46.2 points

from third to fifth grade, resulting in a predicted fifth-grade score of 643.3. A student whose teacher

reports a large amount of exploration in instruction has a predicted gain of 49.8 points, with a

predicted fifth-grade score of 646.9.

The visibility of standards and assessments had an inconclusive effect on students’

mathematics achievement. On the one hand, students in schools where teachers reported

knowing and using standards and assessments showed student gains that were 7.6 points less than

average. On the other hand, third-grade mathematics scores were 6.5 points higher in those schools

where third-grade teachers reported the highest visibility of standards and assessments in 1997 (see

Figure 8). This puzzling combination of findings might reflect changes in the policy climate

during the late 1990s. In 1997, the schools adhering most closely to standards and assessments

might have included those that were especially alert to professional trends. As standards-based

reform became more prominent, schools with the highest levels of attention to standards and

assessments might have been, increasingly, the schools where poor student performance was a

problem.
15

Figure 8. LESCP-predicted score on SAT-9 closed-ended mathematics test, third grade

603.6

597.1

500 525 550 575 600 625 650 675

School reports high visibility of

standards and assessments

Student experiences average

conditions on all variables

Scale score points

Mathematics scores under 

average conditions:

Effect of variable on 

mathematics score:

Third-grade national norm (600 points)

Figure reads: A student who experiences average conditions on all variables has a predicted third-grade score of

597.1, below the national norm. For a student whose school reports high visibility of standards and

assessments, the HLM model predicts a third-grade score of 603.6 (6.5 points higher), above the

national norm.

The best combination of circumstances for mathematics gains was the following:

• Relatively higher amounts of exploration in instruction,

• A teacher who believed he or she had more to learn in mathematics instruction, and

• Higher teacher ratings of professional development.

Students who experienced this combination of instructional variables, starting out 18 points behind

their counterparts in the LESCP sample and 20 points below national norms, would close 40

percent of the gap between themselves and the longitudinal sample and about 50 percent of the gap

between their scores and national norms.11

 

11 For simplicity, the variable of visibility of standards and assessments is left out of this analysis because its effects were mixed. Strictly

interpreted, the data suggest that students would do better if their schools were initially very attuned to standards and assessments (relative to

the rest of the sample) but then did not keep pace with the other schools in intensifying their attention to these policies.
16

Effects of the Policy Environment on Instructional Practices

In general, instructional practices associated with student achievement gains were equally

accessible to all students in the LESCP sample, including those from relatively lower income

families, lower achieving students, and students in schools serving comparatively worse-off

families. Thus, within the limited range of poverty and achievement found in this sample, there

were no systematic differences in equity of access to these instructional practices.

Teachers in districts and states that emphasized assessment and accountability were more

likely to report having received high-quality professional development in mathematics, which

was associated with greater student gains in mathematics (see Figure 9).

Teachers in districts and states that combined reforms in standards, consequences, and plans

reported more visible standards and assessments in mathematics and rated professional

development in both reading and mathematics highly.

• In reading, the favorable rating for professional development worked in favor of

student achievement gains.

• In mathematics, the visibility of standards and assessments had a mixed effect on

achievement, but the rating of professional development was associated with greater

gains.

CONCLUSIONS

This study identified school practices and state or local policies that contributed to higher

student achievement and greater achievement gains in Title I elementary schools in states and districts

that had been engaged in standards-based reform for some time. Several school practicesranging from

high-quality professional development and highly visible standards and assessments to less basic

instruction and more intense outreach to low-achievers’ parentshad positive effects on student

achievement in reading and mathematics.

The larger education policy environment appeared to contribute to the ways in which

teachers experienced elements of reform. In those states and districts that combined several standardsbased reformsthe enactment of standards, accountability for schools, and plans that referred specifically

to standardsteachers described their professional development as well matched to standards and

assessments, and said that they had learned from that professional development. Their students, in turn,

made greater gains in reading and mathematics.
17

Figure 9. Relationship between policy environments and favorable instructional practices

Districts and states

emphasize assessment and

accountability

Policy environment: Teachers more likely to report:

Effect of instructional practice

on student achievement:

High rating of professional

development for mathematics

Positive effect on mathematics

achievement gains

Districts and states

emphasize standards,

consequences, and plans

High visibility of standards and

assessments for mathematics

High rating of professional

development for reading and

mathematics

Negative effect on mathematics

achievement gains

Positive effect on reading

and mathematics

achievement gains
18

The study’s findings lend some support to the policy position that a framework, including

standards, assessments, and professional development, can improve student achievement when teachers

are engaged with that framework. In particular, students did better when teachers reported that their

professional development was coordinated with standards and assessments. Teachers’ reports that they

knew and used standards and assessments had a mixed relationship to student performance. Outreach to

the parents of low-achieving students was of long-term benefit to reading achievement for all students and

to mathematics achievement for low-achieving students. More challenging curriculum and instruction

were associated, in general, with greater student gains in both subjects. All these instructional conditions

could combine to help mitigate the serious negative effects of poverty, at both the student and school

levels, on achievement. An additional investigation along these lines is being conducted through the

development of intensive case studies of high-performing LESCP schools. These cases studies will focus

on teachers’ curriculum and instruction in two sets of schools: high-poverty schools that were consistently

high performing throughout the LESCP study and high-poverty schools that showed significant gains in

student performance during the LESCP study.

It bears repeating that the data were drawn from a select sample of high-poverty schools that

was neither representative nor random. However, because the schools in this sample were among the

earliest to begin implementing elements of standards-based reform, they offer a robust set of data for

policymakers considering how to advance standards-based reform on a national level. Drawn from several

different states, the sampled schools reflect a range of conditions and are likely to illustrate relationships

that would be found around the country. As such, this study’s findings can help suggest ways that all

students can achieve high standards.