NCEE STUDY SNAPSHOT June 2010

The Evaluation of 

Charter School Impacts

Charter middle schools popular enough to 

hold admissions lotteries are, on average, no 

more successful than nearby traditional public 

schools in boosting student achievement, 

behavior, and attendance. However, charter 

schools vary widely; some are more effective 

and some are less effective than nearby 

traditional public schools. Those located 

in large urban areas and those serving 

disadvantaged students are the most successful. 

Overall, parents and students who win entry 

into a charter are more likely to be satisfied 

with their school than those who do not win 

and must attend another school.

The policy context

Charter schools are an important and growing 

component of the public school system in the 

United States. There are now more than 5,000 

of these schools, which first appeared in the 

early 1990s, and they are expected to play a role 

in federal school improvement. However, there 

remains considerable debate as to whether, how, 

and under what circumstances charter schools 

improve the outcomes of students who attend 

them. 

The study approach 

This study—the first large-scale randomized trial 

of the effectiveness of charter schools in multiple 

states and types of communities—was conducted 

with 2,330 students who applied to 36 charter 

middle schools that held lotteries for admission. 

The study was funded by the Institute of Education 

Sciences (IES) and conducted by Mathematica 

Policy Research and its partners.

The study focused on students who applied to 

charter schools that met three criteria:

• They were middle schools, with entry usually 

in grades 5 to 8. Studying middle schools 

allowed researchers to use state-mandated 

achievement test results in this study.

• The schools were open 2 years or more. Thus, 

the results would not be influenced by the 

challenges of start-up schools, which are less 

stable in their operations.

• They had substantially more eligible students 

than they could admit. Choosing schools with 

lotteries created an ideal study sample—two 

statistically similar groups of students, one 

offered admission to the charter they applied to 

and one not offered admission.

The charter middle schools that were eligible and 

agreed to participate in the study were somewhat 

different from other charter middle schools in 

the United States. Participating charter middle 

schools served more advantaged students than the 

typical charter middle school. For example, the 

study charter schools had smaller percentages of 

students who were eligible for free or reduced-
2 THE EVALUATION OF CHARTER SCHOOL IMPACTS

NCEE STUDY SNAPSHOT

price school meals (44 percent versus 62 percent 

nationally) and smaller percentages of students 

who scored below proficiency on their state 

assessments at the time they applied to the charter 

school (34 percent versus 49 percent, in math). In 

addition, fewer African American students attended 

the study schools than attended typical charter 

schools (16 percent versus 29 percent).

In each charter school, impacts were estimated by 

comparing average achievement outcomes among 

lottery winners with those of lottery losers over 

the 2 years following the lottery, controlling for 

students’ background characteristics. Researchers 

estimated overall impacts by averaging across 

all sites. They also examined which school 

characteristics could be possible predictors of 

higher or lower impacts on students’ test scores in 

reading and math.

The primary outcome measure was student 

performance on state math and reading tests. 

Because the tests varied by grade and from state to 

state, the scores were converted to a comparable 

scale. The scale is based on how each student’s 

score measured up to all other students within 

the same state, so that the average score for all 

students in the same state and grade is set to 0, 

and about two-thirds of students have scores that 

range from -1.0 to +1.0 (which represents one 

standard deviation). The study also used school 

records and student and parent surveys to examine 

other outcomes—including student effort in 

school, behavior, and attitudes, as well as parental 

involvement and satisfaction.

The findings

•	 On average, study charter schools did not have 

a statistically significant impact on student 

achievement. However, the averages mask wide 

variation across the charter schools in how 

well their lottery winners performed compared 

to the lottery losers, who typically went back 

to their neighborhood schools. Based on the 

comparable scale, math achievement impacts 

ranged from a minus .78 for one charter to a 

plus .53 for another. (There also is statistically 

significant variation in reading impacts, 

although not as wide). “Impacts” represent 

the difference between the charter school 

and the other schooling options available to 

students. Charter schools with high average 

achievement may have negative achievement 

impacts if the other nearby schools provide 

more benefit to students. Similarly, charter 

schools with low average test scores may have 

higher impacts because they provide a better 

educational experience for their students than 

the neighborhood schools. 

•	 Study charter schools did not significantly 

affect most other outcomes examined, except 

for parent and student satisfaction. These 

outcomes included absences, suspensions, and 

other measures of student performance, as well 

as survey-based measures of student effort in 

school, student well-being, student behavior 

and attitudes, and parental involvement. 

However, lottery winners and their parents 

were more satisfied with their schools. For 

instance, lottery winners were 12 percentage 

points more likely to report that they liked 

school a lot than were lottery losers. Parents 

of lottery winners were 33 percentage points 

more likely to rate their child’s school as 

“excellent” than were parents of lottery losers.

Figure 1. Average Year 2 test scores of charter lottery 

winners and losers are not significantly 

different

NOTE: Test scores shown here are based on students’ performance 

on state assessments across multiple states. These scores were 

standardized using a scale in which the average score for all students 

in the same state and grade is set to 0, and about 68 percent of all 

students have scores that range from -1.0 to +1.0 (which represents 

one standard deviation).

Reading Math

0.00

0.10

0.20

0.30

0.40

0.50

z score

0.31

0.37

0.33

0.39

Lottery winners Lottery losers
NCEE STUDY SNAPSHOT

THE EVALUATION OF CHARTER SCHOOL IMPACTS 3

•	 Study charter schools were more effective for 

lower income and lower achieving students 

and less effective for higher income and higher 

achieving students. 

On average, two subgroups of lottery 

winners—those with initial low test scores and 

those from low-income families—benefited 

academically from admission to charter schools 

(in math but not reading) while their more 

advantaged counterparts did not (lottery 

winners had lower achievement than lottery 

losers in both reading and math). However, 

there were no significant differences in charter 

school impacts for other student subgroups—

such as those defined by race, ethnicity, and 

gender.

•	 The variation in student achievement impacts 

among charter schools may be related to certain 

school characteristics. Charter schools in large 

urban areas, those serving more lower income 

or more lower achieving students produced 

positive impacts on student math scores 

relative to other nearby school options. Charter 

schools outside of large urban areas, those 

serving fewer low-income students, and those 

Figure 2. Study charter schools show wide variation on Year 2 math score impact

z score

Year 2 math scores

1.0

0.8

0.6

0.4

0.2

0.0

0.2

-0.4

-0.6

-0.8

-1.0

NOTE: Each bar represents a study charter school. The red bars show a negative impact on math scores, while the blue bars represent a positive impact. 

Not all of these impacts are statistically significant.

Figure 3. Impacts on Year 2 test scores vary significantly 

by family income status

Estimated impact

Reading Math

-0.20

-0.10

0.00

0.10

0.20

-0.12

0.05

-0.14

0.17

Higher income Lower income

NOTE: The estimated impact on reading test scores is statistically 

significant for students from higher income families; in math, it is 

statistically significant for students from both higher and lower income 

families. The difference between the impacts for the lower income and 

higher income groups also is statistically significant in both math and 

reading.
NCEE 2010-4031

U . S . D e p a r t m e n t o f E d u c a t i o n

4 THE EVALUATION OF CHARTER SCHOOL IMPACTS

serving higher achieving students had negative 

impacts on both reading and math scores. Less 

negative impacts were found in smaller charter 

schools and those more likely to use ability 

grouping. 

The study in context with other charter 

school research

The Evaluation of Charter School Impacts adds to 

the growing body of evidence on this important 

topic. Although the literature on charter school 

impacts covers a broad range of charter schools 

and methodological strategies, there remain 

gaps in the research that this study was designed 

to address. The study used an experimental 

lottery-based methodology, but—unlike previous 

experimental studies—included charter schools in 

multiple states and in suburban, rural, and urban 

areas. Most previous studies of charter school 

impacts focused almost entirely on impacts on test 

scores and provided little evidence on the impacts 

on other student outcomes. Common themes 

include the following:

• Studies that covered a wide span of states 

and/or districts, like this study, also found 

that charter schools did not positively impact 

student achievement on average. 

• Previous lottery-based studies—which 

focused on charter schools in large urban 

areas, primarily serving large populations of 

disadvantaged students—have found positive 

impacts. Such findings are consistent with the 

findings of this new study.

• A key finding from this study—that individual 

charter schools varied in their impacts on 

students’ reading and math achievement—also 

was found in several other studies.

• While this study found significantly 

different impacts for students from different 

socioeconomic backgrounds, findings from 

previous charter school studies concerning 

variation in charter school impacts across 

student subgroups have been mixed.

Looking ahead 

This study was not able to determine why some 

charter schools—particularly those in urban areas, 

serving more disadvantaged students—appear 

to be more effective than the traditional public 

schools these students would have attended if 

they had not enrolled in the charter lottery. Nor 

could the study appropriately investigate why 

other charter schools, serving more advantaged 

students, appear to provide less academic benefit 

than the nearby alternatives. Studies focused on 

better understanding these patterns would provide 

useful information for policymakers and educators 

seeking to improve student achievement through 

the expansion and adaptation of charter schools.

IES develops these briefs to offer short, accessible summaries of complex technical 

evaluation reports. For the full report with technical details, see: 

http://ies.ed.gov/ncee/pubs/20104029.

Gleason, P., Clark, M., Tuttle, C., and Dwoyer, E. (2010). The Evaluation of Charter School Impacts 

(NCEE 2010-4029). Washington, DC: U.S. Department of Education, Institute of Education Sciences, 

National Center for Education Evaluation and Regional Assistance.