NCEE 2010-4030 U.S. DEPARTMENT OF EDUCATION 

The Evaluation of 

Charter School Impacts 

Executive Summary 
This page is intentionally left blank.
June 2010 

Philip Gleason 

Melissa Clark 

Christina Clark Tuttle 

Emily Dwoyer 

Mathematica Policy Research 

Marsha Silverberg 

Project Officer 

Institute of Education Sciences 

NCEE 2010-4030 

U.S. DEPARTMENT OF EDUCATION 

The Evaluation of 

Charter School Impacts 

Executive Summary 
U.S. Department of Education 

Arne Duncan 

Secretary 

Institute of Education Sciences 

John Q. Easton 

Director 

National Center for Education Evaluation and Regional Assistance 

John Q. Easton 

Acting Commissioner 

June 2010 

The report was prepared for the Institute of Education Sciences under Contract No. ED-01-C0039/0010. 

The project officer is Marsha Silverberg in the National Center for Education Evaluation and Regional 

Assistance. 

IES evaluation reports present objective information on the conditions of implementation and impacts of 

the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or 

views with regard to actions policymakers or practitioners should take in light of the findings in the 

reports. 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While 

permission to reprint this publication is not necessary, the citation should be: Gleason, P., Clark, M., 

Tuttle, C. C., and Dwoyer, E. (2010). The Evaluation of Charter School Impacts: Executive Summary 

(NCEE 2010-4030). Washington, DC: National Center for Education Evaluation and Regional 

Assistance, Institute of Education Sciences, U.S. Department of Education. 

To order copies of this report, 

• Write to ED Pubs, U.S. Department of Education, P.O. Box 22207, Alexandria, VA 22304. 

• Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, 

call 800-872-5327 (800-USA-LEARN). Those who use a telecommunications device for the deaf 

(TDD) or a teletypewriter (TTY) should call 800-437-0833. 

• Fax your request to 703-605-6794. 

• Order online at www.edpubs.gov. 

This report also is available on the IES website at http://ies.ed.gov/ncee. 

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or 

computer diskette. For more information, please contact the Department’s Alternate Format Center at 

202-260-9895 or 202-205-8113.
v 

ACKNOWLEDGMENTS 

This study was made possible by the efforts of many individuals and organizations over a sixyear period. We greatly appreciate the willingness of study charter school principals and other 

administrative staff to participate in the evaluation, as well as their contributions in providing us 

access to their lotteries and admissions processes and endless patience in answering questions about 

how these processes worked. Without their active involvement, this study would not have been 

possible. 

The study and this report benefitted greatly from input at various stages from Paul Hill at the 

University of Washington’s Center on Reinventing Public Education. We also thank the study’s 

technical working group for their contributions. The group included Tom Cook, Jay Greene, Laura 

Hamilton, Rick Hanushek, Helen Ladd, Rebecca Maynard, Robert Meyer, and Larry Orr. At 

Mathematica Policy Research, Brian Gill and Peter Schochet provided critical technical review and 

comments. We also thank two anonymous reviewers for their thoughtful comments and 

suggestions. 

The study would not have been possible without contributions from many other individuals at 

Mathematica as well as our research partner, Optimal Solutions Group (OSG). Rita Stapulonis led all 

aspects of the data collection effort for the beginning years of the study and Robert Olsen provided 

input to the study design and school recruiting efforts. Monica Leal Priddy oversaw the evaluation 

activities at OSG and made valuable contributions to the school recruiting and data collection 

efforts. We also thank other OSG staff for their contributions, including Mark Turner, Ja-Nee 

Jackson, Mark Partridge, Veronika Peleschuk, and Amy Peterson. At Mathematica, Alex Bogin, 

Stacie Feldman, Annalee Kelly, and Chris Rafferty served as liaisons to participating charter schools, 

as well as assisted with other aspects of data collection. Expert research assistance was provided by 

Jeff Holt, Dominic Harris, Rita Zota, Laura Guy, Patricia Seunarine, Kathleen Paganelli, Bonnie 

Hart, Jeff Grigg, and Jill Bowdon. Jennifer Baskwell led the production of the report, with assistance 

from Donna Dorsey, Susie Moore, and Lisa Walls. Last but not least, we would like to thank the 

staff at Mathematica’s Princeton Survey Operations Center for collecting the data used for the 

analysis, and the team of charter school recruiters, who worked diligently to secure the participation 

of the study’s charter schools. 
This page is intentionally left blank. 
vii 

DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST 

The research team for this evaluation consists of a prime contractor, Mathematica Policy 

Research; its subcontractor, Optimal Solutions Group; and a consultant, Paul Hill at the University 

of Washington’s Center on Reinventing Public Education. None of these organizations or their key 

staff members have financial interests that could be affected by findings from the study. None of the 

members of the Technical Working Group, convened by the research team to provide advice and 

guidance, have financial interests that could be affected by findings from the study. 
This page is intentionally left blank. 
1

 

EXECUTIVE SUMMARY 

Charter schools, first launched in the 1990s, are an important and growing component of the 

public school system in the United States. As of November 2009, more than 5,000 charter schools 

served over 1.5 million students—approximately three percent of all public school students—in 40 

states and the District of Columbia (Center for Education Reform 2009). Charter schools are 

intended to play a key role in school improvement under the existing Elementary and Secondary 

Education Act (No Child Left Behind) as well as the programs established under the American 

Recovery and Reinvestment Act of 2009. However, there remains considerable debate as to 

whether, how, and under what circumstances charter schools improve the outcomes of students 

who attend them. This report summarizes the results of a new study: the Evaluation of Charter 

School Impacts, a large-scale randomized trial of the effectiveness of charter schools funded by the 

Institute of Education Sciences and conducted by Mathematica Policy Research and its partners.1

The evaluation, which we conducted in 36 charter middle schools across 15 states, compares 

outcomes of students who applied and were admitted to these schools through randomized 

admissions lotteries (lottery winners) with the outcomes of students who also applied to these 

schools and participated in the lotteries but were not admitted (lottery losers). This analytic approach 

produces the most reliable impact estimates. But because the study could only include charter 

middle schools that held lotteries, the results do not necessarily apply to the full set of charter 

middle schools in the U.S. 

Key findings from the evaluation include: 

• On average, charter middle schools that hold lotteries are neither more nor less 

successful than traditional public schools in improving student achievement, 

behavior, and school progress. Participating schools had no significant impacts on 

math or reading test scores either a year or two years after students applied, other 

measures of academic progress (such as attendance or grade promotion), or student 

conduct within or outside of school. Being admitted to a study charter school did 

significantly and consistently improve both students’ and parents’ satisfaction with 

school. 

• The impact of charter middle schools on student achievement varies 

significantly across schools. Across 28 sites (covering 32 schools), the effects on 

reading scores after two years were estimated to be greater than zero in 11 sites and less 

than zero in 17 sites (with magnitudes ranging from -0.43 to +0.33 standard deviation 

units), with 4 of the individual site estimates statistically significant. The estimated 

effects on math scores were greater than zero in 10 sites and less than zero in 18 of the 

28 sites (-0.78 to +0.65 standard deviation units), with 10 of the site estimates 

statistically significant. 

• In our exploratory analysis, for example, we found that study charter schools 

serving more low income or low achieving students had statistically significant 

positive effects on math test scores, while charter schools serving more 

1 The evaluation team also included Optimal Solutions Group and Paul Hill of the University of Washington’s 

Center on Reinventing Public Education. 
2

 

advantaged students—those with higher income and prior achievement—had 

significant negative effects on math test scores. Charter middle schools in large 

urban areas also had significant positive impacts on math achievement compared to 

negative impacts in other locales, although urbanicity was no longer an influential factor 

once such characteristics as students’ demographics and income levels were controlled 

for. There were also differential effects on reading achievement, with negative and 

significant impacts for study charter schools serving more advantaged students and no 

impacts for study charter schools serving fewer advantaged students. 

• Some operational features of charter middle schools are associated with more 

positive (or less negative) impacts on achievement. These features include smaller 

enrollments and the use of ability grouping in math or English classes. Although impacts 

differed for study charter schools with longer- versus shorter- hours of operations or 

higher versus lower revenue per student, these features were no longer significant once 

other school and student characteristics were controlled for. We found no statistically 

significant relationships between achievement impacts and the charter schools’ policy 

environment, including the extent of its decision-making autonomy, the type of 

authorizer and how the authorizer held the school accountable, and whether it was 

operated by a private organization. 

What Kinds of Charter Schools Participated in the Study? 

To be eligible for the study, charter middle schools had to meet two important criteria.2 First, 

they had to have been in operation for at least two years, on the grounds that these schools were 

likely to be relatively stable in their organization and procedures. Second, each participating charter 

middle school had to have more applicants to their entry grade (4-7) in the year of the study than 

they could accommodate and to hold an admissions lottery to determine which students would be 

admitted. Not all charter schools met these criteria or agreed to participate in the study, and those 

that did participate differed in some ways from other charter middle schools nationally. In addition 

to operating longer than other charter middle schools in the country (7.0 versus 5.9 years), based on 

study data, participating schools served more advantaged students. For example, a smaller 

proportion of students in study charter schools were eligible for free or reduced-price school meals 

(44 versus 62 percent), came from minority racial or ethnic groups (47 versus 62 percent), or scored 

below the proficient level on their state assessment at the time they applied to the charter school (for 

example, 34 versus 49 percent in math). On the other hand, study charter schools were statistically 

similar to other charter middle schools in their location (urban versus suburban or rural), size, per 

student revenues, and teacher qualifications. 

Study charter schools—attended by 78 percent of the lottery winners—were different from the 

schools their students would have attended if they had not won entrance to the charter school 

through the admissions lottery; in other words, schools attended by lottery losers (Table 1). 

Compared to the schools that lottery losers attended, lottery winners attended schools that: 

2 Middle schools were chosen primarily because of the likely availability of test score data from school records for 

this group. By relying on school records for test scores, we were able to avoid administering a test to sample members, 

reducing evaluation costs and the burden on sample members. 
3

• were smaller (484 versus 736 students) 

• had longer school days (7.2 versus 6.7 hours) 

• were less likely to have a library (64 versus 88 percent) or gym (53 versus 79 percent) 

• operated more autonomously 

There were no statistically significant differences in the racial/ethnic distribution of students, 

percentage of students eligible for free or reduced-price school meals, student-teacher ratios, or 

length of the school year at the two sets of institutions. However, there were differences of 

10 percentage points or more for some student characteristics, and some of these differences nearly 

reached the level of statistical significance: the average percentage of white, non-Hispanic students at 

schools attended by lottery winners was 56 versus 46 percent at schools attended by lottery losers 

(p-value = 0.191); the average percentage of students eligible for free or reduced-price meals was 

33 versus 45 percent (p-value = 0.055) respectively. 

Table ES.1. Characteristics of Schools Attended by Lottery Winners and Lottery Losers 

Schools Lottery 

Winners Attended 

Schools Lottery 

Losers Attended Difference 

Enrollment (Means) 

Total enrollment 484 736 -252†† 

Student-teacher ratio 15.1 15.9 -0.8 

Time in School (Means) 

Length of school day, in hours 7.2 6.7 0.4† 

Length of school year, in days 181.2 179.9 1.3 

School Facilities (Percentages) 

Library 64% 88% -25%†† 

Gym 53% 79% -27%†† 

Counselor 81% 88% -7% 

Nurse’s office 72% 85% -13% 

Method of Organizing Classes (Percentages) 

Some/all math classes grouped by ability level 39% 56% -17% 

Some/all English classes grouped by ability level 36% 46% -9% 

Characteristics of Students at School (Means) 

Percentage of Hispanic students 26% 32% -5% 

Percentage of white students 56% 46% 10% 

Percentage of black students 12% 16% -3% 

Percentage of students receiving free or reduced-price lunches 33% 45% -12% 

Autonomy Indexa

 (Mean) 4.1 1.4 2.6†† 

a The autonomy index measures the extent to which principals report that they have control over decisions relating to 

staffing, budgetary matters, curriculum/instruction, or other school policies. The index has possible values ranging from 1 to 

5, with higher values reflecting greater control over these decisions. 

 †Difference is statistically significant at the 0.05 level, two-tailed test. 

††Difference is statistically significant at the 0.01 level, two-tailed test. 

What Kinds of Students Applied to Study Schools and Participated in the Study? 

On average, 7 out of 10 students met their state proficiency level in reading (71 percent) and 

math (66 percent) in the year they applied (Figure 1). Fifty-eight percent were white, 

11 percent black, and more than one quarter (28 percent) were of Hispanic origins. In terms of 

eligibility for special programs and services, one third of participating applicants qualified for the 
federal free or reduced-price lunch program, 17 percent had an individualized education plan (IEP), 

and 9 percent were limited English proficient or English language learners. 

By comparing the characteristics of the students not offered admission to study charter schools 

(that is, the lottery losers) with the full populations of students at the schools they attended during 

the follow-up period, we were able to examine how applicants to the charter schools in the study 

differed from other students living in the area who did not apply to the charter schools. The charter 

school applicants were more likely to have achieved proficiency on their state reading tests 

(73 versus 57 percent) as well as their state math test (58 versus 45 percent). 

Figure ES.1. Characteristics of Students in the Sample 

0.71

0.66

0.58

0.11

0.28

0.34

0.17

0.09

0.00

0.20

0.40

0.60

0.80

Proficient: 

Reading

Proficient: 

Math

White Black Hispanic FRP IEP LEP

Proportion 

FRP = Free or reduced-price school meals. 

IEP = Individualized education plan. 

LEP = Limited English proficient or English language learner. 

How Were the Impacts Estimated? 

To estimate charter schools impacts, we compared outcomes of lottery winners to those of 

lottery losers in each participating site, controlling for students’ background characteristics. We then 

averaged impacts over all the sites to produce an overall estimate of the impact of being admitted to a 

study charter school. 

4
Study Design 

Participating Schools: Charter middle schools that held admissions lotteries in their entry grade were 

recruited to participate in the study during the 2004-2005 and 2005-2006 school years. The evaluation 

included 36 schools in 15 states. 

Participating Students: Applicants to the schools’ entry grade who participated in the schools’ 

admissions lotteries and for whom we obtained parental consent formed the student sample. Students 

admitted to the school outside the lottery process were excluded. The primary analysis sample was also 

restricted to students who took the state assessment in the previous year (typically those who attended 

traditional public schools). This sample included 2,330 students. 

Research Design: Lottery winners who were offered admission to a participating charter school—either 

at the time of the lottery or through the beginning of the school year—formed the study’s treatment 

group. The control group comprised lottery losers who were not offered admission. Study team members 

observed participating schools’ lotteries and monitored the entire admissions process at each school. 

Analysis: In each participating charter school, impacts were estimated by comparing average outcomes 

among lottery winners with those of lottery losers over the two years following the lottery, controlling for 

students’ background characteristics. An average impact was calculated over all participating schools. We 

conducted a variety of tests to determine the sensitivity of the results to the specific methods used to 

define the analysis sample and estimate impacts. For an exploratory analysis examining the relationship 

between achievement impacts and schools’ characteristics and policy environments, we used the main 

impact model to estimate the correlation between a site’s impact and the school characteristic or policy of 

interest. 

Outcomes: The primary outcome of interest was student achievement, as measured by students’ 

performance on their state tests in reading and math. Because the tests varied from state to state, we 

converted scores to a comparable scale (z-scores) for the analysis. In addition, we conducted an 

exploratory analysis of charter schools’ impacts on several other outcomes, including student effort in 

school, behavior, and attitudes, as well as parental involvement and satisfaction. These additional 

outcomes came from school administrative records and student and parent surveys. 

We also estimated the impact of actually attending a charter school, as opposed to simply being offered 

admission to the school. Most of the students offered admission to study charter schools as lottery 

winners ultimately attended those schools (78 percent), and another 3 percent attended a nonparticipating charter school nearby. The remaining lottery winners chose to attend a traditional 

public school (15 percent), a private school (3 percent), or were either home schooled or the school 

type was unknown (1 percent). Most students who did not win the charter school lotteries attended 

a traditional public school (78 percent). However 15 percent attended a charter school (6 percent 

attended one participating in the study and 9 percent attended a non-study charter school)3

, 

4 percent attended a private school, and the remaining 3 percent were either home schooled or their 

school type was unknown. To take into account the fact that not all lottery winners attended a 

charter school and that some lottery losers did attend a charter school, we used a standard statistical 

modeling approach in which the lotteries act as an “instrumental variable” for charter school 

5

 3 Lottery losers who attended a participating charter middle school were students who were either mistakenly 

admitted to the school despite a losing lottery draw or who were admitted to the school after the first half of the school 

year following the school lottery. Because these students were admitted after we had made a final determination of their 

treatment status and they did not receive the full charter school treatment, they remained in the control group. 
6

 

attendance. The results of our analysis were similar regardless of whether we estimated the impact of 

being admitted to a study charter school or the impact of actually attending a charter school. 

The primary outcome of interest for this study was student achievement, as reflected by 

students’ performance on state tests in reading and math. Because the tests varied from state to 

state, we converted scores to a comparable scale for the analysis. The converted scores, or z-scores, 

reflect students’ performance on their state test relative to the typical student in that state and grade. 

This is a standard approach in examining assessment data across multiple states. 

What Was the Average Impact of Study Charter Schools? 

The main goal of the study was to estimate charter school impacts on student achievement. In 

addition, to further our understanding of charter schools’ influence, we conducted an exploratory 

analysis of charter schools’ impacts on several other outcomes, including student effort in school, 

behavior, and attitudes, as well as parental involvement and satisfaction.4

 Estimating impacts on all 

these outcomes, the study found the following: 

• On average, study charter schools did not have a statistically significant impact 

on student achievement. Although students admitted to charter middle schools 

through lotteries scored lower on state reading and math assessments (by 0.06 to 0.07 

standard deviations in Year 2—the second year after the lottery) than students who 

applied but were not admitted (lottery losers), these differences were not statistically 

significant after adjusting for multiple treatment-control comparisons; thus these 

findings may be false discoveries (Figure 2).5

• Study charter schools positively affected parent and student satisfaction with and 

perceptions of school. Lottery winners and their parents were significantly more 

satisfied with their schools than lottery losers according to all 11 measures of student and 

parent satisfaction and perceptions examined by the study, after adjustment for multiple 

hypothesis testing. For instance, lottery winners were 13 percentage points more likely to 

report they “like school a lot” than lottery losers (Figure 3). Similarly, the parents of 

lottery winners were 33 percentage points more likely to rate their child’s school as 

“excellent” than parents of lottery losers.

4 Multiple treatment-control group comparisons (e.g, to estimate impacts on varied educational outcomes or time 

periods) may yield misleading estimates or “false discoveries.” In order to separate possible false discoveries from more 

reliable findings, we followed the framework recommended by the National Center for Education Evaluation (NCEE) 

(Schochet 2008). For our key outcomes, we applied formal adjustments for our multiple comparisons, or what is known 

as multiple hypothesis testing. 

5 The magnitude of these effects—which are cumulative over the two year follow-up period—are equal to 

approximately one-quarter-year less instruction for students in charter schools than what they would have received had 

they not been admitted. This is based on estimates from Hill et al. (2007), who found that the average annual test score 

gains across a sample of seven nationally normed tests in grades 5 through 8 were, on average, 0.26 standard deviations 

in reading and 0.31 standard deviations in math. 
Figure ES.2. Average Year 2 Test Scores of Lottery Winners and Losers 

0.31 0.33

0.37 0.39

0.00

0.10

0.20

0.30

0.40

0.50

Reading Math

Lottery Winners

Lottery Losers

z-Score 

Note: Test scores shown here are based on students’ performance on state assessments across multiple 

states. To make scores comparable across states, they were standardized into effect size or z-score 

units, in which a one unit change represents one standard deviation among all of the students in a state 

 +Difference in means is statistically significant at the 0.05 level after adjusting for multiple hypothesis testing. 

++Difference in means is statistically significant at the 0.01 level after adjusting for multiple hypothesis testing. 

• Study charter schools did not significantly affect most other outcomes examined.

The study estimated charter schools’ impacts on 35 other outcomes, including absences, 

suspensions, and other measures of student performance, as well as survey-based 

measures of student effort in school, student well-being, student behavior and attitudes, 

and parental involvement. There was no evidence that study charter schools had any 

impact on the majority of these outcomes.6 For instance, there were no significant 

differences between lottery winners and losers in the proportion reporting that they 

worked hard in school or expected to attend college (Figure 4). Similarly, there were no 

significant differences between the groups in the index reflecting student-reported bad 

behavior outside of school. 

7

 6 There were three exceptions. Parents of lottery winners reported their children to be better adjusted than parents 

of lottery losers, according to an index measuring this outcome. Parents of lottery winners were more likely than parents 

of lottery losers to attend events or volunteer at their child’s school, according to an index measuring parents’ presence 

at the child’s school, but were less likely to belong to the school’s parent-teacher association (PTA) or similar 

organization. 
Figure ES.3. Student and Parent Satisfaction with School Among Lottery Winners and Losers 

75

70

62**

37**

0

20

40

60

80

100

Student likes school 

a lot

Parent rates school 

as "excellent"

Lottery Winners

Lottery Losers

Percent 

 *Difference in means is statistically significant at the 0.05 level after adjusting for multiple hypothesis testing. 

**Difference in means is statistically significant at the 0.01 level after adjusting for multiple hypothesis testing. 

Figure ES.4. Measures of Student Effort and Well-Being, Lottery Winners and Losers 

92 91 90 88

0

20

40

60

80

100

Student works hard 

in school

Student expects to 

attend college

Lottery Winners

Lottery Losers

Percent 

 *Difference in means is statistically significant at the 0.05 level after adjusting for multiple hypothesis testing. 

**Difference in means is statistically significant at the 0.01 level after adjusting for multiple hypothesis testing. 

 

8
• Study charter schools’ impacts on student achievement were inversely related to 

students’ income levels. Study charter schools had a negative and statistically 

significant impact on test scores of higher income students (those not certified for free 

or reduced-price school lunches), but a positive and significant impact on Year 2 math 

scores among lower income students (Figure 5). The impact on Year 2 reading scores 

among lower income students was not statistically significant. However, the difference 

in impacts between the two groups was statistically significant for both reading and 

math. 

• There was some evidence of an inverse relationship between students’ baseline 

achievement levels and charter school impacts on achievement. We found a 

strong and statistically significant negative association between students’ baseline test 

scores and charter schools impacts on their subsequent reading and math scores. The 

higher the achievement scores of their incoming students, the more negative were the 

estimated impacts of study charter schools. On the other hand, when we split students 

evenly into two groups—those with higher versus lower baseline achievement levels—

differences in impacts between the two groups were not statistically significant after 

adjusting for multiple treatment-control comparisons. 

• There were no significant differences in charter school impacts for other student 

subgroups. Charter school impacts were statistically similar for student subgroups 

defined by race and ethnicity and gender. 

Figure ES.5. Impacts on Year 2 Test Scores, by Income Status (as determined by eligibility for the free and 

reduced-price lunch program) 

-0.12**

-0.14*

0.05

0.17*

-0.2

-0.1

0

0.1

0.2

Reading^ Math^^

Higher Income

Lower Income

Estimated Impact 

 *Impact estimate is statistically significant at the 0.05 level after adjusting for multiple hypothesis testing. 

**Impact estimate is statistically significant at the 0.01 level after adjusting for multiple hypothesis testing. 

 ^Difference between estimates for subgroups is statistically significant at the 0.05 level after adjusting for multiple 

hypothesis testing. 

^^Difference between estimates for subgroups is statistically significant at the 0.01 level after adjusting for multiple 

hypothesis testing. 

 

9
Did Some Charter Schools Improve Student Achievement? 

Although study charter schools neither positively nor negatively affected most student 

outcomes on average, these averages mask variation across the schools in their impacts on students. 

The schools’ impacts on students’ math and reading scores after two years varied widely (Figure 6). 

In math, for example, the lowest performing charter school led to a decline of more than half a 

standard deviation in students’ test scores while the highest performing school led to an increase of 

more than half a standard deviation; 10 of the 28 site-level estimates were statistically significant. 

Figure ES.6. Distribution of Site-Level Impact Estimates 

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

z-score

p-value < 0.001 ** 

Year 2 Reading Scores

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

z-score

p-value < 0.001 **

Year 2 Math Scores

Note: p-values are from tests of homogeneity of impacts. 

 *Variation in impacts is statistically significant at the 0.05 level, two-tailed test. 

**Variation in impacts is statistically significant at the 0.01 level, two-tailed test. 

Shaded bars are statistically significant impacts at the 0.05 level, two-tailed test. 

10
We examined whether there were identifiable characteristics of the study charter schools, their 

students, or the conditions under which they operated that were associated with impacts on student 

achievement. Because this analysis was correlational, we could not determine whether the school 

characteristics themselves directly influenced charter school effectiveness, or whether relationships 

between the characteristics and impacts were driven by some other factor we did not measure that 

was associated both with the characteristics we examined and with charter school effectiveness. 

Nonetheless, these results may suggest avenues for more rigorous research in the future. 

We found that the characteristics of the students served by study charter schools were strongly 

related to the schools’ impacts on student achievement, particularly in math. Schools serving the 

largest proportions of disadvantaged and lower achieving students had more positive and statistically 

significant impacts; schools serving the smallest proportions of these students had negative and 

statistically significant impacts. The differences between the two groups of schools were also 

statistically significant. 

• On average, the study schools with the highest proportions of disadvantaged students 

(based on their eligibility for free or reduced-price school meals) had a positive impact 

on Year 2 math scores of 0.18 (Figure 7). Schools with the lowest proportions of 

disadvantaged students had, on average, a negative impact on math scores of -0.24. 

• Study schools with average student baseline achievement above the median had a 

negative impact on Year 2 math scores of -0.21 standard deviations, while schools with 

average student achievement below the median had a positive impact of 0.12 standard 

deviations (Figure 8). 

Figure ES.7. Impacts on Year 2 Test Scores, by Characteristics of Charter Schools’ Student Populations: 

Economic Status 

0.00

0.18††

-0.11†

-0.24†† -0.30

-0.20

-0.10

0.00

0.10

0.20

Reading Math

Schools Serving More 

Economically Disadvantaged 

Students

Schools Serving Fewer 

Economically Disadvantaged 

Students

##

Estimated Impact 

 †Impact estimate is statistically significant at the 0.05 level. 

††Impact estimate is statistically significant at the 0.01 level. 

 #Difference between estimates in two sets of sites is statistically significant at the 0.05 level. 

##Difference between estimates in two sets of sites is statistically significant at the 0.01 level. 

 

11
Figure ES.8. Impacts on Year 2 Test Scores, by Characteristics of Charter Schools’ Student Populations: 

Mean Baseline Test Scores 

0.03

0.12†

-0.15††

-0.21††

-0.30

-0.20

-0.10

0.00

0.10

0.20

Reading Math

Schools Serving Lower 

Achieving Students

Schools Serving Higher 

Achieving Students

## ##

Estimated Impact 

 †Impact estimate is statistically significant at the 0.05 level. 

††Impact estimate is statistically significant at the 0.01 level. 

 #Difference between estimates in two sets of sites is statistically significant at the 0.05 level. 

##Difference between estimates in two sets of sites is statistically significant at the 0.01 level. 

Study schools located in large urban areas had a significant positive impact on Year 2 math 

scores of 0.16 standard deviations, compared with a significant negative impact of -0.14 standard 

deviations for schools outside of large urban areas. The difference between impacts for urban versus 

non-urban schools was also significant. However, once we accounted for both student and school 

characteristics, there were no significant differences between the impacts of study charter schools 

located in versus outside of large urban areas. 

We also examined whether achievement impacts were associated with selected school policies 

and practices. Because impacts were estimated by comparing outcomes for lottery winners versus 

lottery losers, we also measured the school characteristics as the difference between schools attended 

by lottery winners (the study charter school in most cases) and the schools attended by lottery losers 

(typically traditional public schools). For example, the school enrollment measure reflected how 

much larger or smaller the study charter school was than nearby traditional public schools that the 

lottery losers attended. 

Key characteristics associated with charter school impacts on student achievement included: 

• Enrollment: Smaller charter schools had significantly less negative impacts than larger 

charter schools. For each 100 student increase in enrollment, the estimated impact on 

Year 2 mathematics was -0.06 standard deviation units more negative (p-value of the 

correlation<=0.001). 

• Ability Grouping: Charter schools more likely to use ability grouping for math (relative 

to schools attended by lottery losers) had significantly less negative impacts than charter 

schools less likely to use ability grouping. The same was not the case for reading. 

We examined several other aspects of school operations, including the length of the school day 

and year (hours of operation), the student-teacher ratio, and the experience level of teachers. When 

the “hours of operation” measure was examined in isolation, we found that charter school impacts 

12
13

on math achievement were significantly different for schools with longer versus shorter hours of 

operation. Impacts were also significantly related to the student-teacher ratio, when that 

characteristic was examined in isolation. However, we did not find evidence of a significant 

relationship between these characteristics and charter school impacts once we controlled for other 

aspects of charter school operations and student characteristics. 

Finally, we examined a variety of policy-related characteristics of charter schools, including 

measures of their number of years operating, autonomy, accountability, revenues, type of authorizer, 

and management structure. With one exception (revenues per student), we found no evidence that 

these measures of the policy environment in which charter schools operated were related to their 

impacts on student achievement. In the case of per student revenues, we found a significant 

relationship between revenues and impacts on math achievement in models that did not account for 

other charter school characteristics, but this relationship was no longer statistically significant once 

we controlled for these other charter school characteristics. 

Looking Ahead 

The estimated relationships between charter school characteristics and impacts described above 

reinforce the notion that not all charter schools are the same—some are more effective than nearby 

traditional public schools, and others are less effective. Among charter schools popular enough to 

hold lotteries, overall, our results suggest that they are no more successful than nearby traditional 

public schools in boosting student achievement. However, those located in large urban areas and 

serving disadvantaged students are the most successful in doing so, a finding consistent with other 

recent lottery-based studies on charter schools in large urban areas (Abdulkadiroglu et al. 2009; 

Hoxby et al. 2009). Our study was not able to determine why these charter schools appear to be 

more effective than others, but further investigation focused on better understanding the reasons for 

this relationship would provide useful information for policymakers and educators seeking to 

improve student achievement through the expansion and adaptation of charter schools. 
14

REFERENCES 

Abdulkadiroglu, Atila, Josh Angrist, Sarah Cohodes, Susan Dynarski, Jon Fullerton, Thomas Kane, 

and Parag Pathak. “Informing the Debate: Comparing Boston’s Charter, Pilot and Traditional 

Schools.” Boston: Boston Foundation, January 2009. 

Center for Education Reform. “K-12 Facts.” Available at 

[http://www.edreform.com/Fast_Facts/K12_Facts/]. Accessed November 2009. 

Hill, Carolyn J., Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey. “Empirical 

Benchmarks for Interpreting Effect Sizes in Research.” MDRC Working Paper on Research 

Methodology. New York: MDRC, 2007. 

Hoxby, Caroline M., Sonali Murarka, and Jenny Kang. “How New York City’s Charter Schools 

Affect Student Achievement: August 2009 Report.” Second report in series. Cambridge, MA: 

New York City Charter Schools Evaluation Project, September 2009. 

Schochet, Peter Z. “Guidelines for Multiple Testing in Impact Evaluations of Educational 

Interventions: Final Report.” Princeton, NJ: Mathematica Policy Research, 2008.