January 2018
 

Making Connections 

An exploratory analysis of 

features of New Orleans charter 

schools associated with 

student achievement growth 

Patrick J. Wolf 

University of Arkansas 

Shannon Lasserre-Cortez 

American Institutes for Research 

Key findings 

Exploratory analyses showed that school-level student 

achievement growth during the 2012/13 school year in New 

Orleans open-enrollment public charter schools was positively 

associated with having: 

• Kindergarten as an entry grade. 

• An extended school year. 

• More experienced teachers. 

It was negatively associated with having: 

• A higher percentage of teachers with a graduate degree. 

• A higher student–teacher ratio. 

• More student supports offered. 

 
U.S. Department of Education 

Betsy DeVos, Secretary 

Institute of Education Sciences 

Thomas W. Brock, Commissioner for Education Research 

Delegated the Duties of Director 

National Center for Education Evaluation and Regional Assistance 

Ricky Takai, Acting Commissioner 

Elizabeth Eisner, Associate Commissioner 

Amy Johnson, Action Editor 

Chris Boccanfuso, Project Officer 

REL 2018–287 

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts 

unbiased large-scale evaluations of education programs and practices supported by federal 

funds; provides research-based technical assistance to educators and policymakers; and 

supports the synthesis and the widespread dissemination of the results of research and 

evaluation throughout the United States. 

January 2018 

This report was prepared for the Institute of Education Sciences (IES) under Contract 

ED-IES-12-C-0012 by Regional Educational Laboratory Southwest administered by SEDL. 

The content of the publication does not necessarily reflect the views or policies of IES or 

the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. 

This REL report is in the public domain. While permission to reprint this publication is 

not necessary, it should be cited as: 

Wolf, P. J., & Lasserre-Cortez, S. (2018). An exploratory analysis of features of New Orleans 

charter schools associated with student achievement growth (REL 2018–287). Washington, 

DC: U.S. Department of Education, Institute of Education Sciences, National Center for 

Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved from http://ies.ed.gov/ncee/edlabs. 

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/

ncee/edlabs. 
Summary 

This exploratory study examined the associations of certain organizational, operational, and instructional features (“potential indicators”) of open-enrollment public charter 

schools in New Orleans, Louisiana, with school-level value-added measures of student 

achievement growth in the 2012/13 school year. The achievement domains included were 

English language arts, math, and science. 

Of the 85 public schools in the city during the 2012/13 school year, 75 were charter schools, 

supporting 92 different campuses and enrolling more than 84  percent of public school 

students in New Orleans. This predominance of charter schools makes New Orleans a 

particularly promising location for exploring the potential indicators of charter school 

effectiveness. 

Six features emerged from this study as potential indicators of New Orleans charter school 

effectiveness, particularly for school-level measures of student achievement growth in 

English language arts. School-level student achievement growth was positively associated 

with having kindergarten as an entry grade, an extended school year, and more experienced teachers. It was negatively associated with having a higher percentage of teachers 

with a graduate degree, a higher student–teacher ratio, and more student supports offered. 

This exploratory descriptive analysis is limited in important ways. Because it is nonexperimental, the associations identified should not be interpreted as causal. Because the 

study had to be conducted at the school level and only 50 schools were in the potential 

indicator analysis sample, the study had a limited ability to identify systematic associations 

between school organizational, operational, and instructional features and school-level 

value-added measures of student achievement growth. 

Despite these limitations, the exploratory results are a first step in identifying charter 

school conditions that could be studied through more rigorous research to determine 

whether they are valid indicators of student success. 

i 
Contents 

Summary i 

Why this study? 1 

What the study examined 3 

What the study found 6 

Implications of the study findings 7 

Limitations of the study 9 

Appendix A. Literature review A-1 

Appendix B. Literature review process, data, and methodology B-1 

Appendix C. Characteristics of New Orleans charter schools and Louisiana public schools C-1 

Appendix D. Detailed results D-1 

Notes Notes-1 

References Ref-1 

Boxes 

1 

2 

3 

Potential indicators of charter school effectiveness identified in the research and used in 

the current study 2 

Key terms 4 

Data sources and research methods 5 

Figure 

B1 The most common methodology applied in the 58 studies reviewed was 

nonexperimental quantitative analysis B-2 

Tables 

1 

B1 

Association of potential indicators of effectiveness of New Orleans public charter 

schools with English language arts, math, and science value-added measure scores, 

2012/13 (unstandardized regression coefficient estimates) 

Variables included in the current study of potential indicators of the effectiveness of 

New Orleans charter schools, 2012/13 

6 

B-4 

B2 Descriptive statistics for analytic sample of open-enrollment charter schools in New 

Orleans with at least some value-added measure data in 2012/13 B-6 

B3 Size and composition of study samples of New Orleans charter schools in 2012/13 B-7 

C1 Descriptive statistics for the census of 2012/13 New Orleans public charter school 

campuses, 2012/13 C-1 

C2 Comparison of New Orleans public charter schools and state public schools on selected 

characteristics, 2012/13 C-3 

ii 
D1 Association of potential indicators of effectiveness in New Orleans public charter 

schools for English language arts value-added measure, 2012/13 (unstandardized 

regression coefficient estimates) D-1 

D2 Association of potential indicators of effectiveness in New Orleans public charter 

schools for math value-added measure, 2012/13 (unstandardized regression coefficient 

estimates) D-2 

D3 Association of potential indicators of effectiveness in New Orleans public charter 

schools for science value-added measure, 2012/13 (unstandardized regression coefficient 

estimates) D-3 

iii 
Why this study? 

For decades education policymakers and practitioners have been concerned about improving student achievement in New Orleans, Louisiana. After the devastation of Hurricane 

Katrina in 2005, the public school system in New Orleans, already one of the lowest performing in the United States, was reorganized, and most schools became charter schools 

(Harris & Larsen, 2016). As a result, the number of charter schools in New Orleans 

expanded rapidly. Charter schooling is now the primary governance structure for public 

education in the city, making New Orleans unique among U.S. school districts. During 

the 2012/13 school year (the period covered by this study), 75 of the 85 public schools in 

New Orleans were chartered, enrolling more than 84 percent of public school students in 

the city in 92 school campuses. 

Public charter schools operate outside the traditional public school district system according to provisions of their organizational charter. The charter typically sets performance 

goals for the school but leaves school leaders free to determine the best means to achieve 

those goals. This arrangement provides charter schools with the flexibility to innovate in 

both their features and their operation. Therefore, charter schools provide an opportunity to explore what school characteristics are potential indicators of school effectiveness 

(Maranto, Milliman, Hess, & Gresham, 2001). 

The Louisiana Charter Schools Research Alliance of the Regional Educational Laboratory (REL) Southwest requested an exploratory analysis of the distinctive features and potential indicators of charter school effectiveness in New Orleans. The alliance was organized 

as a source of information on projects and studies that would be useful to charter school 

operators and overseers in the state.1 

Structural features of schools, such as entry grade and grade span, long have been thought 

to affect student achievement (for example, Purkey & Smith 1983), by either facilitating or 

hindering student learning. This study explored the association between various organizational, operational, and instructional features of charter schools in New Orleans and their 

students’ achievement growth. Charter school features correlated with student achievement gains are described in this study as “potential indicators” of charter school effectiveness. This study builds on prior research identifying a variety of school features that are 

potential indicators of charter school effectiveness (box 1). 

This study explored 

the association 

between various 

organizational, 

operational, and 

instructional 

features of charter 

schools in New 

Orleans and 

their students’ 

achievement 

growth 

1 
Box 1. Potential indicators of charter school effectiveness identified in the 

research and used in the current study 

A small number of studies of charter schools have examined school characteristics that are 

potential indicators, or correlates, of charter school effectiveness. The following list of potential indicators of effectiveness, identified in prior studies, includes characteristics for which 

reliable measures were available for the current study’s analysis: 

•	 Authorizer type (government entity or nonprofit organization; Witte, Weimer, Shober, & 

Schlomer, 2007; Zimmer, Gill, Attridge, & Obenauf, 2014). 

•	 School age (Bifulco & Ladd, 2006; Buddin & Zimmer, 2005; Hanushek, Kain, Rivkin, & 

Branch, 2007; Kelly & Loveless, 2012; Zimmer, Blanc, Gill, & Christman, 2008). 

•	 School size (Berends, Goldring, Stein, & Cravens, 2010; Gleason et al., 2010). 

•	 Grades served, including number of grades and kindergarten as the entry grade (Abdulkadiroglu et al., 2009; Angrist, Cohodes, Dynarski, Pathak, & Walters, 2013; Jacob & Rockoff, 

2011; Zimmer et al., 2009). 

•	 Extended learning day (Angrist, Pathak, & Walters, 2011; Dobbie & Fryer, 2011; Gleason, 

Tuttle, Gill, Nichols-Barrer, & Teh, 2014; Macey, Decker, & Eckes, 2009; Musher et al., 

2005). 

•	 Extended school year (Hoxby, Murarka, & Kang, 2009; Musher, Musher, Graviss, & Strudler, 2005). 

•	 Percentage of teachers traditionally certified (Steele, Vernez, Gottfried, & Schwam-Baird, 

2011). 

•	 Teacher experience in years (Steele et al., 2011). 

Other studies of traditional public schools rather than charter schools have suggested 

that certain school features might be potential indicators of effectiveness. Those for which 

reliable measures were available for the current study’s analysis include: 

•	 Percentage of teachers traditionally certified (Darling-Hammond, Holtzman, Gatlin, & 

Heilig, 2005). 

•	 Percentage of teachers with a graduate degree (Betts, Zau, & Rice, 2003; Dee, 2004). 

•	 Teacher experience in years (Staiger & Rockoff, 2010). 

•	 Teacher quality, represented in this analysis by the percentage of teachers rated highly 

effective (Rockoff, 2004; Rivkin, Hanushek & Kain, 2005). 

•	 Class size, represented in this analysis by student–teacher ratio (Mosteller, 1997). 

•	 Availability of student support staff, such as counselors, speech therapists, and occupational therapists (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 2000; Rockoff, 

2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). 

RAND researchers, in partnership with the Cowen Institute for Public Education Initiatives 

at Tulane University, conducted an exploratory study of the features that distinguish New 

Orleans public charter schools from traditional public schools in the city (Steele et al., 2011). 

Two features identified by that study are included in the current study’s analysis: 

•	 Higher percentage of traditionally certified teachers. 

•	 Teachers with less experience. 

Additional information on each of these potential indicators of charter school effectiveness is in appendix A. The review process that yielded the list is described in appendix B. 

2 
What the study examined 

This exploratory analysis addressed the following research question: 

•	 What organizational, operational, and instructional features of New Orleans 

charter schools serving grades 3–8 are potential indicators of student achievement 

growth in English language arts, math, and science? 

The analysis focused on three types of potential indicators: organizational features, operational features, and instructional features. Organizational features are related to a school 

as an organization, irrespective of its mission. Such features include age, size, scope, and 

source of authority and oversight, features that are relevant to all organizations regardless 

of what they do. Organizational features can be recognized by their inclusion in analyses 

of both education and noneducation organizations (for example, Downs 1967; Hoxby et 

al., 2009; Wolf, 1993). Operational features are related to the logistics of how a school 

functions. Such features include the length of the school day and school year as well as 

whether daily transportation is provided to students. Instructional features are related 

to how a school implements its mission of educating students. Such features, specific to 

schools, include classroom size, student supports, and the qualifications of teachers. 

These features were selected because prior research found them to be associated with 

student achievement growth (see box 1) and because policymakers and practitioners have 

at least some ability to shape them at the school level. (Comparisons between the population of New Orleans public charter schools and the entire population of Louisiana public 

schools on these and other features can be found in appendix C.) 

New Orleans public charter schools tend to serve a distinct population of Louisiana students and typically do so in ways that differ from the ways in which traditional public 

schools do so. Charter school students in New Orleans are more likely than traditional 

public school students in Louisiana to be Black and to come from low-income households 

but are less likely to be Hispanic. On average, New Orleans charter schools have wider 

grade spans than traditional public schools in Louisiana and are likely to include both kindergarten and grade 6, and sometimes grade 12 as well, suggesting that the charter school 

model is to enroll students for an extended number of years before transitioning them to 

another school. New Orleans charter schools have fewer student supports, suggesting that 

they rely more on classroom teachers in addressing the needs of disadvantaged students 

than do traditional public schools in Louisiana. The proportion of teachers rated highly 

effective in New Orleans charter schools is lower than the statewide average. 

The analysis focused on the extent to which the selected features are correlated with the 

school-level test-score gains of elementary and middle school students in New Orleans 

charter schools.2 Key terms used in this report are described in box 2. A summary of data 

sources and analysis methods is in box 3. 

New Orleans 

public charter 

schools tend to 

serve a distinct 

population of 

Louisiana students 

and typically do 

so in ways that 

differ from the 

ways in which 

traditional public 

schools do so 

3 
Box 2. Key terms 

Charter school. A public school authorized to operate with some degree of independence from 

district or state public school regulations while being held accountable for student outcomes 

by an authorizing entity. This study counts each school campus of a charter school operator 

within the geographic boundary of New Orleans as an individual New Orleans public charter 

school, because the Louisiana Department of Education assigns each school campus a unique 

school identification number. 

Indicator. A feature of a school that is statistically associated with school-level outcomes, in 

this case student achievement growth in English language arts, math, or science (value-added 

measure scores). Features of schools that research has indicated might explain school effectiveness and that can be altered by policymakers or practitioners are potential indicators. This 

study examines potential indicators of effectiveness among New Orleans charter schools. 

Open enrollment. A type of public charter school that does not apply any restrictive admissions standards on students. Such schools admit all interested applicants to the grades that 

they serve and enroll students by lottery if they have more applicants than seats. Language 

immersion schools that required applicants to have some previous exposure to the language 

were classified as open enrollment because such a minimum requirement is not likely to be 

restrictive for the subpopulation of students applying to such schools. Schools that required 

applicants to score above a certain level on a general achievement test were classified as 

selective-admissions schools rather than open-enrollment schools because such a test is 

likely to be restrictive for the subpopulation of students applying. 

School authorizer. A legal entity with the responsibility to decide whether new charter schools 

should open, to monitor charter school operation and performance, and to decide whether 

existing charter schools should close. Charter school authorizers in Louisiana include public 

school boards, the Recovery School District, and the Louisiana Board of Elementary and Secondary Education. 

Student support staff index. Count of how many of the following student supports are offered 

at the school: speech therapy, occupational therapy, adapted physical education, mental 

health coordination, on-site counseling, school psychologist, mentoring, and peer supports. 

Value-added measure. Average learning gains that could reasonably be attributed to students’ education experiences in school. Value-added measures were calculated by the Louisiana Department of Education for English language arts, math, and science. This study uses 

school-level value-added measure scores provided by the Louisiana Department of Education. 

4 
Box 3. Data sources and research methods 

Data 

This study used school-level data provided by the Louisiana Department of Education and by 

the New Orleans Parents’ Guide, a nonprofit community organization that collects information 

from all public schools in the city and produces the annual New Orleans Parents’ Guide to Public 

Schools (New Orleans Parents’ Guide, 2013; see appendix B for more information). 

The Louisiana Department of Education data included: 

•	 School-level value-added measure scores in English language arts, math, and science that 

measure student achievement growth in scale score units while controlling for student 

background characteristics. 

•	 Measures of the following potential indicators: school age, size, and number and range of 

grades served; inclusion of prekindergarten, kindergarten, grade 6, and grade 12; school 

authorizer type; percentage of teachers certified; teacher experience; student–teacher 

ratio; and percentage of teachers rated highly effective in practice. 

•	 Measures of the characteristics of the student population served by New Orleans charter 

schools. These measures are listed in appendix B for descriptive purposes, but they were 

not used in the analysis. These descriptive variables included percentage of students from 

low-income households (identified by eligibility for the federal school lunch program), percentage of Black and Hispanic students, percentage of English learner students, percentage of female students, and percentage of students in special education. 

The New Orleans Parents’ Guide (2013) data covered the potential indicator variables of 

schools’ use of an extended school day or year, student supports offered, and the provision of 

transportation or tutoring. 

Sample 

This report presents analytic results for a sample of open-enrollment New Orleans public charter 

schools operating during the 2012/13 school year that enrolled a sufficient number of students 

in tested grades to produce at least one school-level student achievement growth score from 

the spring 2013 test administration (see descriptive information about the sample in table B2 

in appendix B). The analytic sample consisted of 49 or 50 of the 92 charter school campuses, 

depending on the subject area with student achievement growth scores (see table B2). Seven 

charter school campuses were excluded because they had exclusionary admissions requirements, 

and thirty-five were excluded because they lacked student achievement growth scores. Descriptive statistics are provided in appendix B for the entire population of 92 charter school campuses, 

but only the analytic sample of 49 or 50 campuses was used in the potential indicator analysis. 

Methodology 

The potential indicator analysis used ordinary least-squares regression to estimate the extent to 

which each potential indicator variable was systematically associated (covaried) with variation in 

school-level student achievement growth scores, while the association between all other potential 

indicators and school-level student achievement growth scores was controlled for. An indicator 

variable for whether the school offered prekindergarten was included as a control variable in the 

regressions so that the effect of kindergarten alone would be clear. Associations are judged to be 

statistically significant if there is a low likelihood that the association is due to random chance or 

statistical noise. The standard threshold for statistical significance in education evaluations of p < 

.05 was used in this analysis. Only school features whose association with a student achievement 

growth outcome was statistically significant were considered potential indicators of charter school 

effectiveness. See appendix B for additional information on study methods. 

5 
What the study found 

Regression analysis found that several New Orleans public charter school features were statistically significantly associated with school-level student achievement growth, especially 

in English language arts achievement (table 1). Those features are considered potential 

indicators of New Orleans charter school effectiveness. 

The organizational feature of kindergarten as an entry grade, a subindicator of grades 

served, was statistically significantly associated with higher student achievement growth in 

English language arts. The operational feature of an extended school year also was statistically significantly associated with higher student achievement growth in English language 

arts. The instructional features of a lower percentage of teachers with a graduate degree, 

more experienced teachers, and a lower student–teacher ratio were statistically significantly associated with higher student achievement growth in English language arts (see table 

D1 in appendix D for detailed statistical results). 

Two organizational school features had associations with English language arts student 

achievement growth that were statistically significant at the 90 percent confidence level 

Table 1. Association of potential indicators of effectiveness of New Orleans 

public charter schools with English language arts, math, and science value-added 

measure scores, 2012/13 (unstandardized regression coefficient estimates) 

Feature 

English 

language arts Math Science 

Organizational 

Authorized by Recovery School Districta 3.78 4.65 2.85 

School age –0.78 –0.78 0.13 

School size in number of students 1.48 2.08 0.08 

Number of grades served –1.83 –2.21 0.59 

Includes prekindergarten –4.51 –7.06 –2.12 

Includes kindergarten 15.30** 22.59* 0.31 

Operational 

Offers extended learning day –3.11 –0.95 –2.42 

Offers extended school year 5.22* 6.45 3.09 

Kindergarten as 

an entry grade, an 

extended school 

year, a lower 

percentage of 

teachers with a 

graduate degree, 

more experienced 

teachers, and a 

lower student– 

teacher ratio were 

all statistically 

significantly 

associated with 

higher student 

achievement 

growth in English 

language arts 

Instructional 

Percentage of teachers traditionally certified 12.20 11.15 –0.74 

Percentage of teachers with graduate degree –25.55* –33.83 –4.49 

Teacher experience in years 1.09* 0.98 0.08 

Percentage of teachers rated highly effective 2.87 1.15 –1.86 

Student–teacher ratio –0.92* –0.82 –1.24*

Student support staff indexb –0.15 –0.90 –2.00*

* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units. F-statistic for English language arts model = 3.10; 

probability > F = .00. F-statistic for math model = 1.09; probability > F = .41. F-statistic for science model = 1.27; 

probability > F = .28. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education 

and New Orleans Parents’ Guide (2013). 

6 
but fell short of statistical significance at the 95 percent standard used in this study: larger 

school size was associated with higher student achievement growth in English language 

arts, and more grades served was associated with lower student achievement growth in 

English language arts. These results should be interpreted with caution because they fail to 

meet the conventional standard for statistical significance. 

The analysis revealed fewer potential indicators of charter school effectiveness associated 

with student achievement growth in math and science than in English language arts (see 

tables D1–D3 in appendix D for detailed statistical results). Having kindergarten as an 

entry grade was the only school feature that had a statistically significant association with 

higher student achievement growth in math. Having a lower student–teacher ratio and 

fewer student supports offered were the only school features that had a statistically significant association with higher student achievement growth in science. 

The features of New Orleans charter schools in 2012/13 had no statistically significant 

associations with student achievement growth in 2013/14. This could be due to the original 

findings being false discoveries of associations by mere chance, misalignment of 2012/13 

measures of the potential indicator variables with the 2013/14 student achievement growth 

outcome data, or less precision in the 2013/14 value-added measures of student achievement growth. The analysis could not distinguish among these three possible explanations 

for the nonrobustness of the findings across outcome years. Specifically, the regression 

coefficients for the eight cases of associations that were statistically significant for the main 

analysis on the 2012/13 outcome data were all smaller when the analysis was repeated on 

the 2013/14 outcome data, consistent with both false discoveries and misalignment as possible explanations for the differences. The standard errors of the regression estimates from 

the repeat of the analysis on the 2013/14 outcome data were larger than those from the 

main analysis on the 2012/13 outcome data in every case except for one, consistent with 

the possible explanation that less precise student achievement growth measures in 2013/14 

explain the discrepancy. 

Having kindergarten as an entry grade and having a lower student–teacher ratio were the 

only potential indicators that had a statistically significant association with more than one 

student achievement growth outcome. Having kindergarten as an entry grade was associated with higher student achievement growth in English language arts and math. Having 

a lower student–teacher ratio was associated with higher student achievement growth in 

English language arts and science. 

Implications of the study findings 

This study revealed statistically significant associations between student achievement 

growth (especially in English language arts) and six New Orleans public charter school 

features in 2012/13 that may be used as potential indicators of charter school effectiveness 

in the future. Charter schools that had kindergarten as an entry grade, an extended school 

year, and more experienced teachers tended to have higher student achievement growth 

scores. Charter schools with a higher proportion of teachers with a graduate degree, higher 

student–teacher ratios, and more student supports offered tended to have lower student 

achievement growth scores. 

Having 

kindergarten as an 

entry grade was 

the only school 

feature that had 

a statistically 

significant 

association with 

higher student 

achievement 

growth in math, 

and having a lower 

student–teacher 

ratio and fewer 

student supports 

offered were 

the only school 

features that had 

a statistically 

significant 

association with 

higher student 

achievement 

growth in science 

7 
 

One exploratory finding from the current study was inconsistent with the prior literature. 

Having more student supports was associated with lower achievement, which contrasts 

with previous research findings that having more support staff was associated with higher 

student test scores (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 2000; Rockoff, 

2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). The exploratory results in the 

current study are correlational and may be due largely to variables that could not be 

included in the analyses rather than to causal connections. It also is possible that student 

outcomes are influencing potential indicators and not vice-versa. For example, persisting 

low student achievement may have led some schools to offer more supports to help struggling students. 

Other exploratory findings of the current study were consistent with prior research on 

school effectiveness, either in general or specifically for public charter schools. For example, 

previous studies have found that years of teacher experience are predictive of larger student 

test score gains (Staiger & Rockoff, 2010; Steele et al., 2011). Another study found that 

an extended school year was associated with greater charter school effectiveness in New 

York City (Hoxby et al., 2009), as was the case in the current study for charter schools in 

New Orleans. And other studies using more precise classroom-by-classroom counts of students per teacher than the overall student–teacher ratio in each school used in the current 

study have also found that a smaller class size is associated with higher test score gains 

(Mosteller, 1997). 

These exploratory results identify features that should be tested with more rigorous 

research methods before they are used in making decisions about the design and operation 

of public charter schools. 

More potential indicators of charter school effectiveness were identified in the analysis 

of associations with student achievement growth in English language arts (five potential 

indicators) than in analyses of associations with student achievement growth in math 

(one potential indicator) and science (two potential indicators). Four potential indicators 

have a statistically significant association with student achievement growth in English language arts but not student achievement growth in math, even though the math coefficient 

estimates are larger (for having an extended school year and the percentage of teachers 

with a graduate degree) or very similar in size (for teacher experience and student–teacher 

ratio), suggesting—but not proving—that the measures of student achievement growth 

may have been more precise for English language arts than for math. Since the reasons 

for the discrepancy are not certain, additional studies that explore associations between 

potential indicators of charter school effectiveness and student achievement growth in 

all three domains (English language arts, math, and science) would be helpful in determining whether this pattern is systematic or limited to the case examined here. Research 

on potential indicators that can replicate results across multiple years of student achievement growth also would be welcome, since this study’s findings were not consistent across 

outcome years. 

Research has identified multiple potential indicators of charter school effectiveness (see 

appendix A). Thus, it would be helpful if schools collected and reported a rich set of information about these potential indicators, such as the amount and difficulty of assigned 

homework and formative assessment practices, on a regular basis to facilitate a more comprehensive assessment. 

The finding that 

having more 

student supports 

was associated 

with lower 

achievement 

contrasts with 

previous research 

findings that 

having more 

support staff was 

associated with 

higher student 

test scores 

8 
Limitations of the study 

The analysis was limited by the potential indicator variables for which data were available. 

Some indicators, including transportation and tutoring, were almost universal features 

among New Orleans charter schools so that they could not be used to distinguish more 

effective schools from less effective schools. Data were not available on other measures 

of potential indicators, such as universally high expectations for students, the quality of 

principal leadership, and the amount of parent involvement in a child’s education. Analyzing these potential indicators would have required administering surveys to teachers and 

students or parents, original data collection activities that were beyond the scope of this 

study. 

Because the study relied on school-level data, the sample size for the regression analysis 

was small (49 or 50 observations). It is possible that some potential indicators of charter 

school effectiveness were not identified because of low study power. A direct analysis of 

student-level data would have permitted a more fine-grained and higher-powered exploration of the potential indicators of New Orleans charter school effectiveness. However, 

recently enacted student privacy laws in Louisiana prevented the Louisiana Department 

of Education from releasing student-level achievement data to the study team. Instead, 

the department used student-level achievement data to generate the school-level student 

achievement growth scores that were then shared with the study team. 

The data for this study were limited to the 2012/13 and 2013/14 school years, with the 

main analysis focused on the 2012/13 school year, when the potential indicator measures 

and student achievement outcome measures were aligned. It is possible that some of the 

findings were particular to conditions that students experienced that year and might not 

hold in other years. A longitudinal study with multiple years of data on student achievement growth outcomes and indicator data aligned with those outcomes would produce 

more reliable estimates of potential indicators of charter school effectiveness. Such a study 

was not possible in this case, so readers are cautioned not to draw firm conclusions from 

these findings. 

All the variables examined likely contained some degree of measurement error (Kmenta 

1986). In this study the data for the dependent variables were provided by the Louisiana 

Department of Education, which calculated the student achievement growth scores for 

individual schools based on aggregate student performance on the Louisiana Educational 

Assessment Program and Integrated Louisiana Assessment Program accountability tests. 

These are typical criterion-referenced tests designed by psychometrics consultants working 

with Louisiana educators and government officials (Louisiana Department of Education, 

2007). The procedure used to generate school-level student achievement growth scores was 

consistent with professional practice and drew on data for student prior-year achievement 

and background characteristics (see appendix B). Still, some degree of measurement error 

is likely since the dependent variables used in this study were themselves the product of 

statistical estimation. Staff of the Louisiana Department of Education suggested to the 

study team that they had more confidence in the reliability of the student achievement 

growth estimations for English language arts and math than in the reliability of the student 

achievement growth estimations for science. Student achievement growth estimations 

for English language arts and math count double in the Louisiana school accountability 

Because the study 

relied on schoollevel data, the 

sample size for the 

regression analysis 

was small (49 or 

50 observations). 

It is possible that 

some potential 

indicators of 

charter school 

effectiveness were 

not identified 

because of low 

study power 

9 
metric, so school personnel have an incentive to focus on producing learning gains in 

those areas. 

For the potential indicators used in the analysis, the magnitude of error is likely of an 

acceptable level given the involvement of the Louisiana Department of Education in data 

collection quality control for all the variables in the study, including those from the New 

Orleans Parents’ Guide to Public Schools (New Orleans Parents’ Guide, 2013). Potential 

indicators of charter school effectiveness that likely are not subject to precise measurement, such as disciplinary policy, were excluded from the study at the planning stage due 

to concerns about severe measurement error. 

The student achievement growth that informed the potential indicator analysis excluded 

certain important student subgroups. Students in grades K–3 and 9–12 in 2013, students 

who changed schools in 2013, and students with disabilities who took an alternative assessment in place of the standard state test were not represented in the baseline and outcome 

tests that generated the school-level student achievement growth calculations. The results 

of the potential indicator portion of this study apply directly only to the population of 

stable New Orleans charter school students in a regular education program in grades 4–8 

in 2012/13. 

The study is limited to the New Orleans open-enrollment charter schools with 2012/13 

student achievement growth scores generated from 2011/12 and 2012/13 accountability testing. Schools that did not operate in both 2011/12 and 2012/13, because they were 

new or because they closed in 2012/13, were excluded from the sample, as were schools 

that required an academic entrance exam and schools that enrolled too few students in 

tested grades to report school-level scores without violating privacy laws. These schools 

could not be included in the analytic sample because their selective nature would have 

made them too different from the other charter schools to include (in the case of schools 

with entrance exams) or because there were no data for them (in the case of schools that 

enrolled too few students to report scores). Thus, the results of this study do not necessary 

apply to these types of New Orleans charter schools. 

Many of the potential indicators analyzed are not fully independent of each other. For 

example, class size, school size, and school grade span all will be strongly correlated with 

each other in charter schools with just one classroom per grade. Such multicollinearity 

among explanatory variables in regression estimations does not generate bias but does 

increase the random error of the estimates (Kmenta, 1986). Since the analysis already is 

underpowered, inefficiency due to a high degree of multicollinearity among some of the 

potential indicators is an added limitation of the study. 

Given the limitations of this exploratory study, practitioners are cautioned to consider 

the findings as only a first step in identifying potential indicators of school effectiveness. 

The most important function of these findings is to signal to researchers which features 

of public charter schools should be evaluated more rigorously. Should follow-up studies 

confirm that certain school features are causes of public charter effectiveness rather than 

merely correlates, practitioners would then be well advised to adopt those organizational, 

operational, or instructional features in their schools. 

The results of 

the potential 

indicator portion 

of this study apply 

directly only to 

the population 

of stable New 

Orleans charter 

school students in 

a regular education 

program in grades 

4–8 in 2012/13 

10 
Appendix A. Literature review 

This appendix summarizes the research literature on potential indicators of charter school 

effectiveness and identifies which indicators could not be included in the research. It also 

discusses previous research on education in New Orleans and the role of public charter 

schools in the city. 

Potential indicators of student achievement 

A systematic literature review identified 23 potential indicators of charter school student 

achievement. Twelve of these indicators were included in the potential indicator analysis 

in this study, either through indicator or subindicator variables; eleven were not. 

The following indicators were included in the analysis: 

1.	 Authorizer type. 

2.	 School age. 

3.	 School size. 

4.	 Grades served, represented in the analysis by the subindicators of number of grades 

served and inclusion of kindergarten as an entry grade. 

5.	 Extended learning day. 

6.	 Extended school year. 

7.	 Percentage of teachers traditionally certified 

8.	 Percentage of teachers with a graduate degree. 

9.	 Teacher experience in years. 

10.	 Teacher quality, represented in the analysis by the percentage of teachers rated 

highly effective. 

11.	 Class size, represented in the analysis by student–teacher ratio. 

12.	 Availability of student support staff (such as counselors, speech therapists, and 

occupational therapists). 

The following indicators were not included in the analysis: 

1.	 Availability of tutoring. 

2.	 Knowledge is Power Program (KIPP) school model. 

3.	 For-profit or nonprofit status. 

4.	 Administrative structure of the charter management organization. 

5.	 Academic press. 

6.	 Extent of instructional innovation. 

7.	 Principal leadership. 

8.	 Parent involvement. 

9.	 Disciplinary policy. 

10.	 Amount and difficulty of homework. 

11.	 Use of formative assessment. 

Potential indicators of student achievement in charter schools that were included in the analysis 

1. Authorizer type. Two studies have found that charter schools authorized by government 

entities tend to produce higher achievement growth than do charter schools authorized by 

independent nonprofit organizations (Witte et al., 2007; Zimmer et al., 2014). New Orleans 

charter schools are all authorized by government agencies: the Recovery School District (a 

A-1 
special school district run by the Louisiana Department of Education that intervenes in 

the management of chronically low-performing schools), the Orleans Parish School Board, 

or the Louisiana Board of Education. 

2. School age. School age is the most commonly reported potential indicator of charter 

school effectiveness. Mature charter schools produce higher student achievement growth 

(Bifulco & Ladd, 2006; Buddin & Zimmer, 2005; Cremata et al., 2013; Hanushek et al., 

2007; Kelly & Loveless, 2012; Zimmer et al., 2008). As with the association between teacher 

experience and student test scores, researchers hypothesize that older charter schools have 

higher student achievement growth both because less effective charter schools close down 

and because young charter schools learn how to operate more effectively over time. 

3. School size. All else equal, smaller public charter schools tend to perform more effectively (Berends et al., 2010; Gleason et al., 2010). Smaller schools are most clearly predictive 

of higher student achievement growth in the high school grades, when schools tend to be 

largest (Egalite & Kisida, 2016). Research suggests that smaller (but not extremely small) 

education communities are especially helpful in boosting the achievement of students 

from low-income households and racial/ethnic minority students (Lee & Smith, 1997). 

4. Grades served. Three studies have found significantly higher student achievement 

growth in charter schools that serve a broader range of grades, especially if they include 

both junior high and high school grades, than in charter schools that serve a narrower 

range of grades (Abdulkadiroglu et al., 2009; Angrist et al., 2013; Zimmer et al., 2009). The 

researchers hypothesize that eliminating the transition from junior high to high school 

produces less social and education disruption and therefore enables greater continuity in 

learning. Having kindergarten as an entry grade also is hypothesized to improve student 

learning outcomes by enabling students to enter a school at kindergarten without the disruption of transitioning from a previous school. Only 4 of the 50 public charter schools in 

the analytic sample for this study included grade ranges that spanned both junior high and 

high school grades, so the study was unable to explore the specific claim that combining 

those grade levels is a potential indicator of New Orleans charter school effectiveness. 

The subindicators of the number of grades served and kindergarten as an entry grade were 

included in the potential indicator analysis. 

5. Extended learning day. Several studies have reported that charter schools with added time 

for formal academic instruction, beyond the standard seven-hour school day, have a stronger positive effect on student achievement (Angrist et al., 2011; Gleason et al., 2014; Macey 

et al., 2009; Musher et al., 2005). The extra time needs to be structured and to have academic 

content for extended school days to be positively associated with student achievement. 

6. Extended school year. Increasing learning time by lengthening the school year also 

has been identified as a potential indicator of charter school effectiveness (Hoxby et al., 

2009; Musher et al., 2005). Distinct from year-round schooling models, which involve the 

same number of instruction days as conventional school calendars, extended school years 

add instruction days either in the summer or on Saturdays. Charter schools in New York 

City provide as many as 220 instruction days, compared with the typical 180–185 days 

in traditional public schools, and their extended school years have been identified as the 

most likely reason for charter schools outperforming traditional public schools in that city 

(Hoxby et al., 2009). 

A-2 
 

7. Percentage of teachers traditionally certified. California schools that rely less on 

alternatively licensed teachers have been found to produce higher achievement growth 

(Zimmer & Buddin, 2007). The effect was consistent across the samples of public charter 

and traditional public schools. 

8. Percentage of teachers with a graduate degree. Teachers with a graduate degree have 

been found to produce higher student achievement growth (Betts et al., 2003; Dee, 2004). 

However, findings have not been consistent across studies with different methodologies or 

samples (Buddin & Zamarro, 2009; Harris & Sass, 2011). 

9. Teacher experience in years. Studies have concluded that teachers become more effective at increasing student achievement the longer they work as classroom teachers (Greenwald, Hedges & Lane, 1996; Staiger & Rockoff, 2010). Researchers suggest two reasons 

for the association between teacher experience and higher student achievement growth. 

First, administrators are able to observe young teachers and counsel out of teaching those 

who perform poorly (Staiger & Rockoff, 2010). Thus, teachers who continue past their formative years tend to be higher in the student performance distribution. Second, teachers 

learn by doing. Those who continue in the profession become more effective in boosting 

student performance, with a large share of the gains occurring in the first five years of their 

career (Staiger & Rockoff, 2010; Rivkin et al., 2005). Higher levels of teacher experience 

might be especially strongly associated with higher student performance growth in New 

Orleans public charter schools because charter school teachers in the city averaged 9.7 

years of experience compared with an average of 13 for traditional public school teachers 

in 2008/2009 (Steele et al., 2011). New Orleans charter schools with teachers that average 

more years of experience might be particularly effective, relative to peer charter schools 

with teachers that average less years of experience, because there are so few teachers with 

many years of experience in the charter school sector. 

10. Teacher quality. The quality of teachers has a strong association with student 

achievement growth (Rockoff, 2004; Rivkin et al., 2005). In this study, teacher quality was 

measured based on ratings of whether teachers were highly effective in the classroom by 

independent observers using a rubric developed by the Louisiana Department of Education. 

11. Class size. All else equal, smaller class size tends to be associated with higher student 

achievement growth, especially in early elementary grades (Mosteller, 1997). The effects of 

class size on student achievement growth appear to be highly context dependent, varying 

with the maximum class size established by policy, with implementation fidelity, and with 

scale (Bohrnstedt & Stecher, 2002). This study used teacher–student ratio as the measure 

of class size. 

12. Availability of student support staff (such as counselors, speech therapists, and 

occupational therapists). Schools with more specialized support staff tend to generate 

higher student achievement growth (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 

2000; Rockoff, 2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). Education specialists, school counselors, and nurses all have been linked to higher student test scores in 

various contexts. 

A-3 
Potential indicators of student achievement in charter schools that were not included in the analysis 

Eleven potential indicators of charter school effectiveness identified in the literature could 

not be included in this study for the reasons given below. 

1. Availability of tutoring. Charter schools that provide tutoring to struggling students 

have been shown to increase student achievement (Dobbie & Fryer, 2011). The current 

study could not examine tutoring as a potential indicator of charter school effectiveness 

because all but five New Orleans charter schools in the analytic sample (90 percent) provided in-school tutoring to their students. 

2. Knowledge is Power Program (KIPP) school model. KIPP schools are one of the 

best known national charter school networks. Researchers have found the KIPP education model to have a positive effect on student achievement (Macey et al., 2009; Angrist, 

Dynarski, Kane, Pathak, & Walters, 2010). The KIPP model could not be included in this 

analysis because too few New Orleans charter schools are KIPP schools. 

3. For-profit or nonprofit status. Research suggests that charter schools that are incorporated as nonprofit organizations are more effective at raising student achievement than are 

charter schools incorporated as for-profit companies (King, 2007). The for-profit or nonprofit status of schools could not be included in this analysis because too few New Orleans 

charter schools were for-profit. 

4. Administrative structure of the charter management organization. Charter management organizations (CMOs) are administrative units that provide back-office services and 

oversight to charter school franchises. The administrative structure of the CMO has been 

linked to higher student achievement in public charter schools (Furgeson et al., 2012). An 

indicator variable for administrative structure of the CMO could not be included in this 

analysis because information about that feature was not available from the databases used 

in the study. 

5. Academic press. Academic press is the extent to which school personnel emphasize academic achievement in the culture and daily operations of the school. Academic press has 

been linked to higher test score performance in public charter schools (Berends et al., 2010; 

Cheng, Hitt, & Kisida, 2015). Academic press could not be included in this analysis because 

information about that feature was not available from the databases used in the study. 

6. Extent of instructional innovation. Instructional innovation involves adopting a novel 

or unconventional approach to delivering educational content. One previous charter 

school effectiveness study that examined whether instructional innovation was associated 

with learning gains in public charter schools found innovation measures to be negatively 

associated with gains, suggesting that the disruption from such changes might not be worth 

the benefits of adopting a new instructional approach. (Berends et al., 2010). Instructional 

innovation could not be included in the study because information about that feature was 

not available from the databases used in the study. 

7. Principal leadership. Charter school principals are the organizational and instructional leaders of their schools. A few charter school studies have evaluated the quality of principal leadership, either through teacher evaluations of their principals or through principal 

A-4 
self-reports of engaging or not engaging in a defined set of leadership practices (Campbell & Gross, 2008; Macey et al., 2009; Witte et al., 2007). Charter schools with higher 

ratings for principal leadership were more likely to outperform their local traditional public 

schools in promoting student achievement (Macey et al., 2009; Witte et al., 2007). Principal leadership could not be included in this study because information about the feature 

was not available in the study databases. 

8. Parent involvement. Parents can be involved in their child’s education in a variety 

of ways, including volunteering at school and assisting with homework. Several charter 

school studies have examined the association between level of parent involvement and 

student achievement, finding that more involvement was associated with higher achievement (Downing, Spencer, & Cavallaro, 2004; Zimmer & Buddin, 2007). Parent involvement could not be included in this study because information about the feature was not 

available in the study databases. 

9. Disciplinary policy. Disciplinary policies that do not exclude students from their school 

environment are associated with higher student achievement growth (Arcia, 2006; Beck & 

Muschkin, 2012; Skiba & Rausch, 2004). School disciplinary policy could not be included 

in this study because the measure of this feature available in the study databases was based 

on the descriptions of principals, raising concerns about precision and reliability. 

10. Amount and difficulty of homework. Assigning a greater amount of, and more challenging, homework has been associated with higher student achievement growth (Wolf & 

Hoople, 2006). Amount and difficulty of homework could not be included in this study 

because information about the feature was not available in the study databases. 

11. Use of formative assessment. The frequent use of diagnostic tests to gauge student 

mastery of content, often called formative assessment, has been linked to higher student 

achievement growth in general (Wolf, 2007) and specifically for students in charter schools 

(Angrist et al., 2013; Chabrier, Cohodes, & Oreopoulos, 2016; Hoxby et al., 2009). The 

frequency of the use of formative assessment could not be included in this study because 

no information about this feature was available in the study databases. 

Charter school student achievement in New Orleans 

The post–Hurricane Katrina education landscape has resulted in an unprecedented proliferation of charter schools in New Orleans. Public charter schools constitute 88 percent of 

public schools in New Orleans and enroll 84 percent of the city’s public school students. 

In addition, 75 of the 104 public charter school campuses in Louisiana are in New Orleans 

(Louisiana Department of Education, 2013). 

Several studies have examined features of the New Orleans charter school landscape since 

Hurricane Katrina (for example, Kingsland & Donovan, 2013; Rasheed, 2006; Tillotson, 

2006). Harris (2015, p. 12), writing on behalf of an extensive Education Research Alliance 

team, reported that the replacement of the New Orleans traditional public school system 

with an overwhelmingly charter school alternative produced “an upward change in the 

trajectory of student test scores in New Orleans after the reforms of about 0.2 to 0.4 standard deviations, enough to improve the typical student’s performance by 8 to 15 percentile 

A-5 
points.” The study was not able to identify potential indicators of student achievement 

growth in New Orleans charter schools. 

Two Center for Research on Education Outcomes studies compared student achievement 

in New Orleans charter schools with the achievement of matched students in traditional 

public schools (Center for Research on Education Outcomes, 2009; Cremata et al., 2013). 

These studies reported that the typical charter school student in Louisiana outperformed 

his or her matched comparison peer in both English language arts and math, with effect 

sizes in the 2013 study of 0.07 standard deviation in English language arts (equivalent to 50 

days of learning) and 0.09 standard deviation in math (equivalent to 65 days of learning) 

and effect sizes in the 2009 study of 0.06 standard deviation in both English language arts 

and math. Because 72 percent of Louisiana charter schools in the 2013 study (Cremata 

et al., 2013) were in New Orleans, this statewide Louisiana charter school achievement 

effect is likely to reflect results specific to New Orleans as well. 

In partnership with the Cowen Institute at Tulane University, RAND conducted an 

exploratory study of the features of New Orleans public charter schools that distinguish 

them from traditional public schools in the city (Steele et al., 2011). New Orleans charter 

schools had principals who reported higher decisionmaking autonomy and more contracting-out for services, teachers who reported greater ease in maintaining discipline, school 

personnel who were less likely to say that the school faced serious challenges in promoting 

student achievement (that is, they evidenced a “no excuses” attitude to boosting student 

learning), a smaller percentage of teachers who were alternatively licensed, a larger percentage of less-experienced teachers, less professional development for teachers, a higher 

percentage of parents who listed academic reasons for choosing the school as opposed to 

ease of transportation, and a larger percentage of satisfied parents. Not all of these indicators were included in the list at the beginning of this appendix because the Steele et al. 

study was limited by its nonexperimental research methodology and low survey response 

rates (52 percent for teachers, 36 percent for parents, and 32 percent for principals). 

Conclusion 

The literature examining student achievement in charter schools continues to grow. 

Several of the reviewed studies compared student achievement or achievement growth in 

charter schools and traditional public schools. Research that identifies causal relationships 

between indicators (or moderators) of charter school effectiveness and student outcomes is 

more limited. However, preliminary results from this literature review revealed 18 potential 

indicators that have been associated (or hypothesized to be associated) with differences 

in student outcomes in charter schools. Data were available to include 12 of the potential 

indicators in this exploratory study. 

A-6 
 

Appendix B. Literature review process, data, and methodology 

The first step in this exploration of the potential indicators of public charter school effectiveness in New Orleans was to review the research literature systematically to determine 

what is known about the school features that are associated with charter school effectiveness (see appendix A). Those previous findings then informed the study team’s data 

requests and analytic models. This appendix describes the data and methodology employed 

to conduct the literature review and the exploratory analysis that followed. 

Literature review 

The study team followed a structured protocol for searching academic databases for peerreviewed journal articles or reports published between 2000 and 2014 on charter school 

performance and potential indicators of effectiveness. Initial searches returned more than 

1,000 potentially relevant results. After reviewing abstracts for each article, the study team 

identified 32 articles as relevant for a more detailed review. 

In addition to academic journal articles, the review included research reports from nonpartisan university-affiliated and nonprofit research organizations, such as RAND, Mathematica Policy Research, the Center for Research on Education Outcomes, and the Center 

on Reinventing Public Education. This search uncovered 26 reports for review. 

Across all sources the study team identified 58 articles and reports for final review. The 

systematic review of each article or report focused on identifying primary research questions, methodology employed, and main findings—including factors influencing student 

achievement, if applicable. In addition, the review included three articles and reports that 

did not focus explicitly on influences on student achievement but that the study team 

deemed relevant because of a focus on charter schools in New Orleans. 

Overview of study characteristics. The methodologies applied in the 58 studies reviewed 

are summarized in figure B1. A large majority of the studies (49 of 58) used some kind 

of quantitative method to explore relationships between charter school variables (such as 

extended instruction time and size of school) and student achievement. Potential indicators were not randomly assigned to treatment or control groups, even in the studies identified as experimental in this review, as the condition of random assignment was to a charter 

school or not. Thus, the findings from the literature on the potential indicators of charter 

school effectiveness should be viewed as correlational and not necessarily causal. 

Twenty-six studies employed nonexperimental quantitative methods. Twenty-three 

studies employed either quasi-experimental methods (15) or experimental methods (8). 

For the experimental studies, lottery systems to control entry into oversubscribed charter 

schools provided researchers with natural randomized treatment and control groups. A 

common quasi-experimental method involved matching charter school students (the 

treatment group) with otherwise similar traditional public school students (the control 

group). Researchers compared students across the two groups who are similar in age, 

socioeconomic status, race/ethnicity, gender, and other background characteristics. An 

additional 9 studies used other methods (qualitative, meta-analysis, literature review, and 

nonempirical). 

B-1 
Figure B1. The most common methodology applied in the 58 studies reviewed was 

nonexperimental quantitative analysis 

 

 

 

 

 

 

 

 

 

Source: Authors’ compilation from literature review. 

A majority of the 58 studies examined only the effect of attending charter schools on 

student achievement growth, in some cases also exploring student characteristics that 

might moderate that effect. Only 23 of the reviewed studies identified potential indicators 

of charter school effectiveness. It was from those 23 studies that the potential indicators 

for this study were drawn. Because only 5 of the 23 studies identifying potential indicators 

were experimental, factors reported in any of the studies regardless of their methodological 

rigor were included in the potential indicators list. 

Potential indicators from the studies. The literature review identified 23 potential indicators of charter school student achievement. Three potential indicators were excluded 

from the analyses in this study because either too few or too many charter schools had 

that feature: availability of tutoring, Knowledge is Power Program (KIPP) school model, 

and for-profit or nonprofit status. Eight potential indicators identified by prior research 

were excluded from this study because reliable measures were unavailable: administrative 

structure of the charter management organization, academic press (universally high expectations for student achievement), extent of instructional innovation, principal leadership, 

parent involvement, disciplinary policy, amount and difficulty of homework, and use of 

formative assessment. The 12 remaining potential indicators included in one or more of 

the analyses in this study were: 

•	 Authorizer type. 

•	 School age. 

•	 School size. 

•	 Grades served, represented in the analysis by the subindicators of number of grades 

served and inclusion of kindergarten as an entry grade. 

•	 Extended learning day. 

•	 Extended school year. 

•	 Percentage of teachers traditionally certified. 

B-2 
•	 Percentage of teachers with a graduate degree. 

•	 Teacher experience in years. 

•	 Teacher quality, represented in the analysis by the percentage of teachers rated 

highly effective. 

•	 Class size, represented in the analysis by student–teacher ratio. 

•	 Availability of student support staff (such as counselors, speech therapists, and 

occupational therapists). 

Data 

The analysis of the indicators of charter school effectiveness drew on the following data 

sources (as indicated in table B1): 

•	 Data available on the Louisiana Department of Education (LDE) website Louisiana Believes (https://www.louisianabelieves.com). 

•	 The LDE’s student accountability testing database. 

•	 The LDE’s Student Information System database. 

•	 The LDE’s Profile of Educational Professionals database. 

•	 The LDE’s School Report Cards. 

•	 National Center for Education Statistics Common Core of Data (U.S. Department of Education, 2013). 

•	 New Orleans Parents’ Guide to Public Schools (New Orleans Parents’ Guide, 2013).3 

All data for the main analysis were for the 2012/13 school year and were measured at or 

aggregated to the school level. Data on student achievement growth also were obtained for 

the 2013/14 school year to test the robustness of the findings of the main analysis. 

The LDE calculated school-level value-added measures (VAM) using a student-level model 

and then aggregated the results by individual school campus (there were 75 chartered 

schools in New Orleans in 2012/13, supporting 92 different campuses). Student performance on the 2013 outcome test in a given domain was estimated as a function of student 

performance on the prior-year test in that same domain as well as student background 

characteristics such as race/ethnicity, gender, low-income status (identified by eligibility 

for the federal school lunch program), English learner status, and special education status. 

The student-level residuals from the LDE regression were then converted to z-scores by 

student grade level and grade-promotion history to adjust for higher learning trajectories 

for younger students and students who were never retained in grade.4 Finally, the z-scores 

for student-level residuals were aggregated to the school level. Achievement growth estimated in this way isolates variance in student learning that might reasonably be attributed 

to the school the student currently attends and to its various features (Chetty, Friedman, 

& Rockoff, 2012; Kane, McCaffrey, Miller, & Staiger, 2013). 

Achievement in Louisiana was assessed using two state-administered standardized tests, 

the Integrated Louisiana Educational Assessment Program (iLEAP) and the Louisiana 

Educational Assessment Program (LEAP). The iLEAP, which is administered in grades 3, 

5, 6, and 7, combines a criterion-based component and the normative Iowa Test of Basic 

Skills. The LEAP test is a high-stakes, criterion-based assessment administered in grades 4 

and 8. The criterion-based elements of both tests are based on Louisiana state standards. 

According to the LDE testing officers, the LEAP and iLEAP are similar enough in content 

and scoring to generate valid school-level VAM scores even if most individual student 

B-3 
Table B1. Variables included in the current study of potential indicators of the effectiveness of New 

Orleans charter schools, 2012/13 

Variable Description Source 

Organizational 

Authorized by Recovery School Indicator for whether the Recovery School Districta was the official Louisiana Believes 

District authorizer of the charter school website (https://www.

louisianabelieves.com) 

School age Total years from the year of the school’s founding, in any sector, un

2013 

til U.S. Department of 

Education (2013) 

School size in number of Student enrollment on the fall 2012 enrollment count day U.S. Department of 

students Education (2013) 

Grades served 

Number of grades Calculated from the reported grade range of the school U.S. Department of 

Education (2013) 

Includes prekindergarten Indicator variable calculated from the reported grade range of the school U.S. Department of 

used exclusively as a control variable in the analysis Education (2013) 

Includes kindergarten Indicator variable calculated from the reported grade range of the school U.S. Department of 

Offers extended learning day Indicator of whether the school had a formal school day longer than the New Orleans Parents’ 

conventional seven hours Guide (2013) 

Offers extended school year Indicator of whether the school operated for more days than the state 

required 

New Orleans Parents’ 

Guide (2013) 

Education (2013) 

Operational 

Instructional 

Percentage of teachers Proportion of the teaching staff with formal certification Louisiana Department 

traditionally certified of Education 

Percentage of teachers with a Proportion of the teaching staff with a master’s or doctorate in any field Louisiana Department 

graduate degree of Education 

Teacher experience in years Average years of experience for the teaching staff Louisiana Department 

of Education 

Percentage of teachers rated 

highly effective 

Proportion of teachers rated highly effective at educational practice based 

on state’s observational rating system 

Louisiana Department 

of Education 

Student–teacher ratio Total student enrollment divided by total number of teachers, both 

instructional and noninstructional 

Louisiana Department 

of Education 

Student support staff index Count of each of the following supports offered at the school: speech 

therapy, occupational therapy, adapted physical education, mental health 

coordinator, on-site counseling, school psychologist, mentoring, and peer 

Louisiana Department 

of Education 

English language arts VAM Average student value-added after prior-year achievement and student 

background characteristics expected to affect learning are controlled for 

Louisiana Department 

of Education 

Math VAM Average student value-added after prior-year achievement and student 

background characteristics expected to affect learning are controlled for 

Louisiana Department 

of Education 

supports 

Outcome 

Science VAM Average student value-added after prior-year achievement and student Louisiana Department 

background characteristics expected to affect learning are controlled for of Education 

VAM is value-added measure of school-level student achievement growth. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management of chronically lowperforming schools. 

Source: Authors’ compilation. 

B-4 
VAM estimates use iLEAP as a prior-year control and LEAP as the outcome or vice versa. 

This study excluded social studies as an outcome measure because the research literature 

supporting the identification of potential indicators was based on studies that used English 

language arts, math, or science as student outcomes. 

Before the study analysis, the study team determined that potential indicators would be 

excluded from the ordinary least squares (OLS) estimations if 90 percent or more of the 

observations had the same value for that variable. Any variable that varied so little across 

cases would be highly unlikely to explain variation in school-level VAM scores, which 

varied greatly and continuously across observations. Such a variable likely would have 

unacceptably high multicollinearity with other explanatory variables in the regression 

model. Two potential indicators were excluded from the analysis on this basis: providing 

free transportation to students (92  percent of the observations contained the common 

value of 1) and providing tutoring (90 percent of the observations contained the common 

value of 1). 

Sampling 

This study included two samples. Sample 1 was designed to capture the profile of all the 

public charter schools operating in New Orleans in 2012/13. All 92 charter school campuses with a unique LDE site code and data indicating that they operated during the 2012/13 

school year were included. The fact that the exploratory analysis of potential indicators 

was limited to charter schools in New Orleans holds constant all city-specific factors that 

otherwise might confound the relationships in a potential indicator study with a larger 

geographic scope. 

Sample 2 was designed to inform the research question on the potential indicators of 

charter school effectiveness. Because school-level VAM scores were the measure of charter 

school effectiveness, 36 charter campuses that lacked VAM scores for 2013 for all three of 

the domains of English language arts, math, and science were excluded from this analytic 

sample. Six 2012/13 New Orleans charter schools with VAM scores were open only to 

students who passed an admissions test. Since the ability of such schools to control the 

quality of the students they admit could confound any relationship between other school 

characteristics and school effectiveness, those schools were excluded from the analytic 

sample. The 50 school campuses that remained in sample 2 represented the complete set 

of open-enrollment charter schools in New Orleans in 2012/13 with VAM outcome data 

for at least one of the three subjects. VAM scores were available in all three subjects for 47 

public charter schools in the sample. Those schools contributed to the exploratory results 

for English language arts, math, and science. VAM scores were available in English language arts and math but not science for one school, for math and science but not English 

language arts for another school, and for English language arts and science but not math 

for a third school. Thus, three schools informed the exploration of potential indicators for 

two of the subjects but not the third one. 

Language immersion charter schools were included in sample 2. Those schools require 

some familiarity with a foreign language as a minimal condition of admission. This language requirement differs in two ways from the admission-test requirements of the six 

charter schools excluded from sample 2. First, basic language familiarity is a less rigorous screen on student ability than the requirement to achieve a high overall score on a 

B-5 
general academic ability test. Second, test-admission charter schools are selecting students 

on initial achievement in the areas that generate the VAM scores that are the dependent 

variables in the analysis. That makes them fundamentally different from language immersion schools, conceptually and in terms of the exploratory statistical analysis. For example, 

the VAM scores have less variance in the test-admission charter schools than in the other 

charter schools, including the language immersion schools. Including both groups in the 

same statistical regression analysis would produce heteroskedastic error terms, thus violating a necessary condition of regression that “the variance of the disturbance is constant for all observations” (Kmenta, 1986, p. 269). Heteroskedasticity reduces the efficiency 

of regression estimations, which is a serious concern for studies with small numbers of 

observations such as this one. For these reasons, language immersion charter schools were 

included in the sample while general test-admission charter schools were excluded. 

Descriptive statistics for sample 2 are shown in table B2. Student counts and composition 

for each analytic sample are in table B3. 

Table B2. Descriptive statistics for analytic sample of open-enrollment charter 

schools in New Orleans with at least some value-added measure data in 2012/13 

Statistic Mean 

Standard 

deviation Minimum Maximum 

Number 

of school 

campuses 

Organizational 

Authorized by Recovery School Districta (percent) 84.0 na na na 50 

School age (years) 7.3 2.6 3.0 14.0 50 

School size (number of students) 498.6 167.4 130.0 938.0 49 

Grades served 

Number of gradesb 8.5 2.0 4.0 14.0 50 

Includes prekindergarten (percent) 52.0 na na na 50 

Includes kindergarten (percent) 88.0 na na na 50 

Includes grade 6 (percent) 84.0 na na na 50 

Includes grade 12 (percent) 6.0 na na na 50 

Female students (percent) 47.7 8.2 0.0 58.8 46 

Students from low-income households (percent) 91.4 10.7 51.0 98.0 49 

Black students (percent) 93.7 12.6 47.7 100.0 46 

Hispanic students (percent) 2.9 8.1 0.0 51.4 46 

English learner students (percent) 1.5 5.3 0.0 35.6 46 

Students in special education (percent) 10.6 3.9 2.5 21.0 49 

Operational 

Schools providing transportation (percent) 92.0 na na na 50 

Schools offering extended learning day (percent) 74.0 na na na 50 

Schools offering extended school year (percent) 56.0 na na na 50 

Instructional 

Teachers traditionally certified (percent) 77.4 15.3 31.0 98.0 46 

Teachers with a graduate degree (percent) 24.5 13.7 0.0 67.0 46 

Teacher experience (years) 7.4 4.3 1.2 19.3 50 

Teachers rated highly effective (percent) 14.3 22.5 0.0 83.0 40 

Student–teacher ratio 15.0 2.6 7.0 21.0 48 

Student support staff indexc 1.8 1.3 0.0 8.0 50 

Schools providing tutoring (percent) 90.0 na na na 50 

(continued) 

B-6 
1 

2 

Table B2. Descriptive statistics for analytic sample of open-enrollment charter 

schools in New Orleans with at least some value-added measure data in 2012/13 

(continued) 

Statistic Mean 

Standard 

deviation Minimum Maximum 

Number 

of school 

campuses 

Outcome 

English language arts VAMd 2.8 8.1 –14.3 29.8 49 

Math VAMd 7.5 13.7 –29.2 44.2 49 

Science VAMd 1.9 7.3 –18.0 17.0 49 

na means that the descriptive statistic was not meaningful for the variable because it takes only a value of 

1 or 0; VAM is value-added measure of student achievement growth. 

Note: The highlighted variables were included in the statistical models for the potential indicator analysis. Of 

the 50 school campuses that had value-added measure scores in one or more subject area, 47 had them for 

all three domains and 3 had them for two domains. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Grade configurations varied, but every school included at least one of the indicator grades 4–8 and most 

included two of them (for example, K and 6, 6 and 12). 

c. Count of each of the student supports offered at a school. 

d. Expressed as scaled score units, which account for different rates of learning in various grades. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education 

and New Orleans Parents’ Guide (2013). 

Table B3. Size and composition of study samples of New Orleans charter schools in 

2012/13 

Sample Focus 

Possible 

grades 

Number of 

schools 

Number of 

students 

Descriptive statistics (see appendix C) K–12 92 46,064 

Potential indicators of effectiveness 4–8 50 12,560 

Note: The number of students informing the sample used to generate the value-added measure scores was 

estimated by multiplying the average number of students per grade at a school by the number of grades 

served in the grade 4–8 range. The number of students informing the calculation of other school-level variables used in the study varied and was higher to the extent that the school included grades outside of the 

testing window of 4–8. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education. 

Methodology 

The research question for this study was addressed using OLS statistical regression analysis 

on school-level data. The purpose of OLS is to identify the pattern of association between 

a set of explanatory variables and a single dependent variable that minimizes the sum of 

the squared errors from the calculation. 

The following equation describes the OLS regression approach used here: 

VAMsk = B0 + B1

Xjk + ek 

where VAMsk is the 2012/13 VAM score in subject s for school k was estimated as a function of a constant B0

 and various B1s that measure the association between j indicator 

B-7 
 

 

 

variables measured at school-level k and VAMsk. The expression ek is the residual error 

after factoring in the extent to which the constant and the indicator variables explain the 

levels of VAMsk. The B1

s indicate the direction of the relationship—whether the specific 

potential indicator variable is associated with higher or lower levels of VAM—as well as 

the size of the association. 

It was not necessary to control for the effects of student demographic characteristics in 

the statistical models. Student demographic characteristics, such as race/ethnicity, English 

learner status, and special education status, were all factored into the LDE’s calculation 

of school-level VAM. Student achievement in the previous year, 2011/12, also was a key 

element of the VAM calculations and therefore did not need to be controlled for in the 

regression models. 

An average of 3.2 percent of the data for the 12 potential indicator variables in the statistical model was missing. Missing baseline data were multiple-imputed to preserve the 

size and representativeness of the achievement samples (King, Honaker, Joseph, & Scheve, 

2001; Puma, Olsen, Bell, & Price, 2009). None of the school-level VAM dependent variables were imputed. 

A supplemental analysis using descriptive statistics was also performed comparing the 

features of the New Orleans charter school population with all Louisiana public schools 

(see appendix C). School-level mean values on VAM scores and potential indicators of 

charter school effectiveness were calculated for all charter school campuses in operation during 2012/13. Those New Orleans charter population means were compared with 

the school-level averages across all of Louisiana. Differences were calculated as the New 

Orleans charter average minus the statewide school average, and those differences were 

divided by the statewide school average to generate a percentage. 

Comparable statewide data were obtained for only half of the 26 school features for which 

descriptive information was available in sample 1 of all New Orleans public charter school 

campuses. State data privacy laws prevent the LDE from publishing the school-level state 

averages for school-level VAM scores for English language arts, math, or science as well 

as the school-level state averages for teacher years of experience, teacher certification 

percentage, and teacher graduate degree percentage. The LDE was able to provide these 

school-level data for this study with the restriction that the report would not present any 

aggregate state-level calculation of school-level VAM or those teacher characteristics. The 

measures of whether a New Orleans charter school offered tutoring, free transportation, 

extended learning day, and extended school year were taken from the New Orleans Parents’ 

Guide to Public Schools (New Orleans Parents’ Guide, 2013). Comparable information was 

not available statewide. The school-level state average for the percentage of English learner 

students was published as “less than 5 percent,” a description too vague to be used in the 

comparison with New Orleans charter schools. Finally, the indicator variable for a school 

being authorized by the Recovery School District distinguishes public charter schools 

within New Orleans but not public schools outside the city. 

This study is observational and exploratory, as it lacks the design elements required to 

identify causal relationships. It is possible that the associations identified in the analysis 

between potential indicators and student achievement growth are spurious, that is, are 

B-8 
caused by factors outside of the analysis, or endogenous, with greater effectiveness causing 

the school to adopt the indicator feature. 

Sensitivity tests 

Two sensitivity tests were run on the results from the potential indicator statistical analysis. First, the analysis was rerun on the data using listwise deletion instead of multiple 

imputation to deal with observations with missing data. The results did not change substantively. Second, the analysis was rerun on 2013/14 school-level VAM data provided by 

the LDE. None of the potential indicators of charter school effectiveness as measured in 

2012/13 were associated with variation in 2013/14 English language arts, math, or science 

VAM. 

B-9 
Appendix C. Characteristics of New Orleans 

charter schools and Louisiana public schools 

This appendix describes the features of the New Orleans charter school population and 

then compares them with all Louisiana public schools. School-level mean values on value-added measures (VAM) and potential indicators of charter school effectiveness were 

calculated for all 92 New Orleans charter school campuses in operation during 2012/13. 

The averages for the New Orleans charter population were compared with Louisiana 

school-level averages. Differences were calculated as the New Orleans charter school 

average minus the statewide school average divided by the statewide school average to 

generate a percentage difference between New Orleans charter schools and all Louisiana 

public schools relative to the average value for all schools. Significance tests could not be 

conducted on the population differences because data on the population variance for the 

census of Louisiana public schools were not available. Any apparent differences between 

the populations of New Orleans public charter schools and all Louisiana public schools 

could be due to the New Orleans schools being urban, the charter schools being public 

schools, some combination of the two, or other unknown factors. 

Although data challenges limited the number of school features the study could examine 

(see appendix A), many salient features were included (table C1). The average age of New 

Orleans charter schools was 7 years, with the youngest having operated for 3 years and 

the oldest for 14 years. Enrollment averaged 501 students and ranged from 119 to 1,697 

students. The average number of grades served was 8, with 29  percent including grade 

12. The Recovery School District authorized 78 percent of New Orleans charter schools. 

The proportion of students from low-income households (as indicated by eligibility for the 

federal school lunch program) averaged 91 percent, the proportion of Black students averaged 91 percent, and the proportion of students in special education averaged 11 percent. 

Table C1. Descriptive statistics for the census of 2012/13 New Orleans public 

charter school campuses, 2012/13 

Standard 

Number 

of school 

Statistic Mean deviation Minimum Maximum campuses 

Organizational 

Authorized by Recovery School Districta (percent) 77.8 na na na 90 

School age (years) 7.4 2.3 3.0 14.0 86 

School size (number of students) 500.7 240.6 119.0 1,697.0 81 

Grades served 

Number of gradesb 7.5 2.8 1.0 14.0 89 

Includes prekindergarten (percent) 41.6 na na na 89 

Includes kindergarten (percent) 65.2 na na na 89 

Includes grade 6 (percent) 65.2 na na na 89 

Includes grade 12 (percent) 29.2 na na na 89 

Female students (percent) 47.6 8.3 0.0 67.3 61 

Students from low-income households (percent) 90.6 10.3 51.0 99.7 78 

Black students (percent) 90.9 17.7 18.3 100.0 61 

Hispanic students (percent) 2.9 7.2 0.0 51.4 61 

English learner students (percent) 1.7 4.9 0.0 35.6 61 

(continued) 

C-1 
Table C1. Descriptive statistics for the census of 2012/13 New Orleans public 

charter school campuses, 2012/13 (continued) 

Statistic Mean 

Standard 

deviation Minimum Maximum 

Number 

of school 

campuses 

Students in special education (percent) 10.8 4.2 2.5 22.0 78 

Operational 

Schools providing transportation (percent) 95.5 na na na 89 

Schools offering extended learning day (percent) 71.9 na na na 89 

Teachers traditionally certified (percent) 79.6 14.8 31.0 98.0 58 

Teachers with a graduate degree (percent) 24.9 14.8 0.0 67.0 58 

Schools offering extended school year (percent) 52.8 na na na 89 

Instructional 

Teacher experience (years) 7.7 4.3 1.2 19.3 70 

Teachers rated highly effective (percent) 12.7 20.1 0.0 83.0 56 

Student–teacher ratio 14.8 2.6 7.0 21.0 78 

Student support staff indexc 1.8 1.1 0.0 8.0 89 

Schools providing tutoring (percent) 93.3 na na na 89 

Outcome 

English language arts VAMd 2.9 7.9 –14.3 29.8 55 

Math VAMd 7.3 13.3 –29.2 44.2 55 

Science VAMd 1.7 7.2 –18.0 17.0 55 

na means that the descriptive statistic was not meaningful for the variable because it only takes the value 

1 or 0; VAM is value-added measure of student achievement growth. 

Note: A total of 92 public charter school campuses operated in New Orleans in 2012/13. The number of 

school campuses with data for each characteristic varied substantially because many campuses were not 

required to report some information to the state, they were required but neglected to do so, or they reported 

the information but it was lost or corrupted in the process of data aggregation. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Grade configurations varied, but every school included at least one of the indicator grades 4–8 and most 

included two of them (for example, K and 6, 6 and 12). 

c. Count of each of the student supports offered at a school. 

d. Expressed as scaled score units, which account for different rates of learning in various grades. 

Source: Authors’ compilation from school-level data provided by the Louisiana Department of Education and 

New Orleans Parents’ Guide (2013). 

A total of 72 percent of New Orleans charter schools provided an extended learning day, and 

53 percent provided an extended school year. The proportion of teachers with traditional 

certification averaged 80 percent, with an average of 25 percent having a graduate degree. 

Teachers averaged 7.7 years of teaching experience, and 13 percent of teachers were rated 

highly effective in practice. The average student–teacher ratio was 15, ranging from 7 to 21. 

The average New Orleans charter school offered more grades than the average Louisiana school 

In 2012/13 New Orleans charter schools were similar to all public schools in the state on some 

key characteristics and distinctive on others (table C2). The two populations of schools were 

considered approximately similar on a characteristic if the difference was less than 5 percent 

and descriptively distinctive if the difference was 5 percent or more.5

 The average for New 

Orleans charter schools was similar to the statewide average for school size, percentage of 

female students, percentage of students in special education, and student–teacher ratio. 

C-2 
Table C2. Comparison of New Orleans public charter schools and state public 

schools on selected characteristics, 2012/13 

New Orleans Louisiana 

Characteristic charter mean mean Difference 

Percent 

difference 

Organizational 

School size (number of students) 500.7 493.3 7.4 1.5 

Number of gradesa 7.5 5.9 1.6 27.1 

Includes kindergarten (percent) 65.2 49.6 15.6 31.5 

Includes grade 6 (percent) 65.2 36.4 28.8 79.1 

Includes grade 12 (percent) 29.2 23.0 6.2 27.0 

Female students (percent) 47.6 48.7 –1.1 –2.3 

Low-income students (percent) 

Black students (percent) 

90.6 

90.9 

66.3 

45.0 

24.3 

45.9 

36.7 

102.0 

Hispanic students (percent) 2.9 4.3 –1.5 –32.6 

Teachers rated highly effective (percent) 12.7 18.3 –5.6 –30.6 

Student–teacher ratio 14.8 15.2 –0.4 –2.6 

Students in special education (percent) 10.8 11.3 –0.5 –4.4 

Instructional 

Student support staff indexb 1.8 3.2 –1.4 –43.8 

Note: The two populations of schools were considered approximately similar on a characteristic if the difference was less than 5 percent and descriptively distinctive if the difference was 5 percent or more. Descriptively distinctive characteristics are shown in bold. 

a. Grade configurations varied, but every school included at least one of the indicator grades 4–8 and most 

included two of them (for example, K and 6, 6 and 12). 

b. Count of each of the student supports offered at a school. 

Source: Authors’ compilation from school-level data provided by the Louisiana Department of Education and 

New Orleans Parents’ Guide (2013). 

New Orleans charter schools were descriptively distinctive in that they offered 27 percent 

more grades, on average, than the statewide norm (7.5 compared with 5.9). The average 

New Orleans charter school was much more likely to include kindergarten (32  percent 

more likely), grade 6 (79 percent more likely), and grade 12 (27 percent more likely) than 

the average Louisiana public school. Charter schools more often included broad grade 

ranges such as K–6, K–8, or even K–12 that cut across typical school-building transition 

grades. 

The average New Orleans charter school served more Black students and more students from lowincome households than the average Louisiana school did 

The proportion of Black students in the average New Orleans charter school (91 percent) 

was more than twice the statewide average (45 percent), and the proportion of students 

from low-income households was more than a third higher (91 percent) than the statewide 

average (66 percent). The average proportion of Hispanic students in New Orleans charter 

schools (3 percent) was lower than the statewide average (4 percent). The average score on 

the student support staff index (count of student supports offered at a particular school) for 

New Orleans charter schools (1.8) was 1.4 points lower than the statewide average (3.2). 

On average, 12.7 percent of teachers in New Orleans charter schools were rated highly 

effective, 5.6 percentage points less than the average of 18.3 percent for all public schools 

in Louisiana. 

C-3 
-

Appendix D. Detailed results 

This appendix reports the results of the regression analysis of potential indicators of New 

Orleans charter school effectiveness for English language arts value-added measures (table 

D1), math VAM (table D2), and science VAM (table D3). 

Table D1. Association of potential indicators of effectiveness in New Orleans 

public charter schools for English language arts value-added measure, 2012/13 

(unstandardized regression coefficient estimates) 

Characteristic Beta Standard error p value 

Organizational 

Authorized by Recovery School Districta 3.78 3.50 .29 

School age –0.78 0.49 .12 

School size in number of students 1.48 0.78 .07 

Number of grades served –1.83 0.95 .06 

Includes prekindergarten –4.51 2.95 .14 

Offers extended learning day –3.11 2.95 .30 

Offers extended school year 5.22 2.33* .03 

Includes kindergarten 15.30 4.21** .00 

Operational 

Instructional 

Percentage of teachers traditionally certified 12.20 9.33 .21 

Percentage of teachers with a graduate degree –25.55 10.16* .02 

Teacher experience in years 1.09 0.46* .02 

Percentage of teachers rated highly effective 2.87 4.14 .59 

Student–teacher ratio –0.92 0.39* .03 

Student support staff indexb –0.15 0.83 .86 

* Significant at p < .05; ** significant at p < .01. 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 

various grades. F-statistic = 3.10; probability > F = .00. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education 

and New Orleans Parents’ Guide (2013). 

D-1 
-

Table D2. Association of potential indicators of effectiveness in New Orleans 

public charter schools for math value-added measure, 2012/13 (unstandardized 

regression coefficient estimates) 

Characteristic Beta Standard error p value 

Organizational 

Authorized by Recovery School Districta 4.65 7.11 .52 

School age –0.78 1.06 .47 

School size in number of students 2.08 1.57 .20 

Number of grades served –2.21 2.03 .29 

Includes prekindergarten –7.06 6.03 .25 

Offers extended learning day –0.95 6.25 .88 

Offers extended school year 6.45 4.94 .20 

Includes kindergarten 22.59 9.04* .02 

Operational 

Instructional 

Percentage of teachers traditionally certified 11.15 20.65 .60 

Percentage of teachers with a graduate degree –33.83 22.57 .15 

Teacher experience in years 0.98 1.00 .34 

Percentage of teachers rated highly effective 1.15 9.31 .90 

Student–teacher ratio –0.82 0.82 .33 

Student support staff indexb –0.90 1.76 .61 

* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 

various grades. F-statistic = 1.09; probability > F = .41. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education 

and New Orleans Parents’ Guide (2013). 

D-2 
-

Table D3. Association of potential indicators of effectiveness in New Orleans 

public charter schools for science value-added measure, 2012/13 (unstandardized 

regression coefficient estimates) 

Characteristic Beta Standard error p value 

Organizational 

Authorized by Recovery School Districta 2.85 4.25 .51 

School age 0.13 0.59 .83 

School size in number of students 0.08 0.86 .92 

Number of grades served 0.59 1.12 .60 

Includes prekindergarten –2.12 3.37 .53 

Offers extended learning day –2.42 3.43 .49 

Offers extended school year 3.09 2.75 .27 

Includes kindergarten 0.31 4.97 .95 

Operational 

Instructional 

Percentage of teachers traditionally certified –0.74 10.86 .95 

Percentage of teachers with a graduate degree –4.49 12.05 .71 

Teacher experience in years 0.08 0.56 .88 

Percentage of teachers rated highly effective –1.86 4.57 .69 

Student–teacher ratio –1.24 0.45* .01 

Student support staff indexb –2.00 0.97* .05 

* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 

various grades. F-statistic = 1.27; probability > F = .28. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 

of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors’ calculations based on school-level data provided by the Louisiana Department of Education 

and New Orleans Parents’ Guide (2013). 

D-3 
Notes 

1.	 Core alliance members represent the following institutions: Andrew H. Wilson 

Charter School, Audubon Charter School, Benjamin Franklin High School, Cypress 

Academy, Einstein Elementary Charter School, Greater New Orleans Collaborative of 

Charter Schools, Hynes Charter School, International School of Louisiana, Jefferson 

Parish School District, Lake Forest Elementary Charter School, Louisiana Department of Education, Lusher Charter School, Magnolia School of Excellence, Morris 

Jeff Community School, New Beginnings Schools Foundation, New Orleans Charter 

Science and Mathematics High School, School Leadership Center of Greater New 

Orleans, Tulane University, Warren Easton Charter High School, West Feliciana 

Parish Schools, and Xavier University of Louisiana. 

2.	 High school students in grades 9–12 are assessed by end-of-course exams that are 

administered by subject and are not comparable from year to year. As such, high 

schools that do not include grade 7 or 8 are not included in this analysis. 

3.	 New Orleans Parents’ Guide to Public Schools is produced annually by the New Orleans 

Parents’ Guide, a nonprofit community organization, and includes an associated 

smartphone app. Staff and volunteers of the New Orleans Parents’ Guide collect information directly from all the public schools in the city and verify those data with LDE 

prior to publishing the guide. 

4.	 Grade promotion history was accounted for in the conversion to a z-score of the dependent variable and not controlled for in the regression estimations themselves because 

the analysis already suffered from having a small number of degrees of freedom. 

5.	 Five percent was selected as the benchmark for differences being distinctive because a 

difference of that magnitude can be noticeable to the observer, whereas a difference of 

less than 5 percent tends to be imperceptible. Information about the variance around 

the mean for the statewide averages was not available, preventing the use of formal 

significance tests in judging the reliability of the differences. 

Notes-1 
References 

Abdulkadiroglu, A., Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Fullerton, J., Kane, 

T. J., et al. (2009). Informing the debate: Comparing Boston’s charter, pilot and traditional 

schools. Boston, MA: Boston Foundation. 

Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. R. (2013). 

Charter schools and the road to college readiness: The effects on college preparation, attendance and choice. Boston, MA: The Boston Foundation and NewSchools Venture Fund. 

Angrist, J. D., Dynarski, S. M., Kane, T. J., Pathak, P. A., & Walters, C. R. (2010). Inputs 

and impacts in charter schools: KIPP Lynn. The American Economic Review, 100(2), 

239–243. 

Angrist, J. D., & Lavy, V. (1999). Using Maimonides’ Rule to estimate the effect of class 

size on scholastic achievement. Quarterly Journal of Economics, 114(2), 533–575. 

Angrist, J. D., Pathak, P. A., & Walters, C. R. (2011). Explaining charter school effectiveness. 

(NBER No. 17332). Cambridge, MA: National Bureau of Economic Research. http://

eric.ed.gov/?id=ED559928 

Arcia, E. (2006). Achievement and enrollment status of suspended students: Outcomes in 

a large, multicultural school district. Education and Urban Society, 38(3), 359–369. 

Beck, A. N., & Muschkin, C. G. (2012). The enduring impact of race: Understanding the 

disparities in student disciplinary infractions and achievement. Sociological Perspectives, 55(4), 637–662. 

Berends, M., Goldring, E., Stein, M., & Cravens, X. (2010). Instructional conditions in 

charter schools and students’ mathematics achievement gains. American Journal of 

Education, 116(3), 303–336. http://eric.ed.gov/?id=EJ890455 

Betts, J. R., Zau, A. C., & Rice, L. A. (2003). Determinants of student achievement: New evidence from San Diego. Working Paper. San Diego: Public Policy Institute of California. 

Bifulco, R., & Ladd, H. F. (2006). The impacts of charter schools on student achievement: 

Evidence from North Carolina. Education, Finance, and Policy, 1(1), 50–90. http://eric.

ed.gov/?id=EJ902821 

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. 

Bohrnstedt, G. W., & Stecher, B. M. (Eds.). (2002). What we have learned about class size 

reduction in California. Sacramento, CA: California Department of Education. 

Buddin, R., & Zamarro, G. (2009). Teacher qualifications and student achievement in 

urban elementary schools. Journal of Urban Economics, 66(2), 103–115. 

Ref-1 
Buddin, R., & Zimmer, R. (2005). Student achievement in charter schools: A complex 

picture. Journal of Policy Analysis and Management, 24(2), 351–371. 

Campbell, C., & Gross, B. (2008). Working without a safety net: How charter school leaders 

can best survive on the high wire. Seattle, WA: Center on Reinventing Public Education. 

Center for Research on Education Outcomes. (2009). Charter school performance in 16 

states. Stanford, CA: Author. 

Chabrier, J., Cohodes, S., & Oreopoulos, P. (2016). What can we learn from charter school 

lotteries? Journal of Economic Perspectives, 30(3), 57–84. 

Cheng, A., Hitt, C., & Kisida, B. (2015). “No excuses” charter schools: A meta-analysis of the 

experimental evidence on student achievement (Working Paper No. 226). New York, NY: 

National Center for the Study of Privatization in Education. Retrieved September 8, 

2016, from http://ncspe.tc.columbia.edu/working-papers/OP226.pdf. 

Chetty, R., Friedman, J. N., & Rockoff, E. J. (2012). The long-term impacts of teachers: 

Teacher value-added and student outcomes in adulthood (NBER No. 17699). Cambridge, 

MA: National Bureau of Economic Research. Retrieved July 6, 2016, from http://www.

nber.org/papers/w17699. 

Cremata, E., Davis, D., Dickey, K., Lawyer, K., Negassi, Y., Raymond, M. E., et al. (2013). 

National charter school study 2013. Center for Research on Education Outcomes. Stanford, CA: Stanford University. 

Darling-Hammond, L., Holtzman, D. J., Gatlin, S. J., & Heilig, J. V. (2005). Does teacher 

preparation matter? Evidence about teacher certification, Teach for America, and teacher 

effectiveness. Education Policy Analysis Archives, 13(42). http://eric.ed.gov/?id=EJ846746 

Dee, T. S. (2004). Teachers, race and student achievement in a randomized experiment. 

Review of Economics and Statistics, 86(1), 195210. 

Dobbie, W., & Fryer, R. G. (2011). Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem Children’s Zone. American Economic Journal: Applied Economics, 3(3), 158–187. http://eric.ed.gov/?id=EJ846746 

Downing, J. E., Spencer, S., & Cavallaro, C. (2004). The development of an inclusive 

charter elementary school: Lessons learned. Research & Practice for Persons with Severe 

Disabilities, 29(1), 11–24. 

Downs, A. D. (1967). Inside bureaucracy. Boston, MA: Little, Brown & Company. 

Egalite, A. J., & Kisida, B. (2016). School size and student achievement: A longitudinal 

analysis. School Effectiveness and School Improvement, 27(3), 406–417. 

Furgeson, J., Gill, B., Haimson, J., Killewald, A., McCullough, M., Nichols-Barrer, I., et al. 

(2012). Charter-school management organizations: Diverse strategies and diverse student 

impacts. Princeton, NJ: Mathematica Policy Research. http://eric.ed.gov/?id=ED528536 

Ref-2 
 

 

Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school 

impacts: Final report (NCEE 2010–4029). Washington, DC: U.S. Department of Education, Institute of Education Sciences. 

Gleason, P. M., Tuttle, C. C., Gill, B., Nichols-Barrer, I., & Teh, B. (2014). Do KIPP schools 

boost student achievement? Education Finance and Policy, 9(1), 36–58. http://eric.

ed.gov/?id=EJ1016285 

Greenwald, R., Hedges, L., & Laine, R. (1996). The effect of school resources on student 

achievement. Review of Educational Research, 66(3), 361–396. 

Hamilton, L. S., & Guin, K. (2005). Understanding how families choose. In J. R. Betts & 

T. Loveless (Eds.), Getting choice right: Ensuring equity and efficiency in education policy. 

Washington, DC: Brookings Institution Press. 

Hanushek, E. A., Kain, J. F., Rivkin, S. G., & Branch, G. F. (2007). Charter school quality 

and parental decision making with school choice. Journal of Public Economics, 91(5–6), 

823–848. 

Harris, D. N. (2015). Good news for New Orleans: Early evidence shows reforms lifting 

student achievement. Education Next, 15(4), 8–15. http://eric.ed.gov/?id=EJ1073904 

Harris, D. N., & Larsen, M. (2016). The effects of the New Orleans post-Katrina school 

reforms on student academic outcomes. New Orleans, LA: Education Research Alliance for New Orleans, Tulane University. Retrieved August 21, 2016, from http://

educationresearchalliancenola.org/files/publications/The-Effects-of-the-New-Orleans

-Post-Katrina-School-Reforms-on-Student-Academic-Outcomes.pdf. 

Harris, D. N., & Sass, T. R. (2011). Teacher training, teacher quality and student achievement. Journal of Public Economics, 95(7–8), 798–812. 

Hoxby, C. M., Murarka, S., & Kang, J. (2009). How New York City’s charter schools affect 

student achievement: August 2009 report. Cambridge, MA: New York City Charter 

Schools Evaluation Project. 

Jacob, B. A., & Rockoff, J. E. (2011). Organizing schools to improve student achievement: Start 

times, grade configurations, and teacher assignments. Washington, DC: The Hamilton 

Project at the Brookings Institution. 

Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have we identified effective 

teachers? Validating measures of effective teaching using random assignment. Seattle, WA: 

Bill & Melinda Gates Foundation. http://eric.ed.gov/?id=ED540959 

Kelly, A. P., & Loveless, T. (2012). Comparing new school effects in charter and traditional public schools. American Journal of Education, 118(4), 427–453. http://eric.ed.gov/

?id=EJ986706 

Ref-3 
 

 

King, G., Honaker, J., Joseph, A., & Scheve, K. (2001). Analyzing incomplete political 

science data: An alternative algorithm for multiple imputation. American Political 

Science Review, 95(1), 49–69. 

King, K. (2007). Charter schools in Arizona: Does being a for-profit institution make a 

difference? Journal of Economic Issues, 41(3), 729–746. 

Kingsland, N., & Donovan, W. (2013). Charter schools in New Orleans: Lessons for Massachusetts. Boston, MA: Pioneer Institute. Retrieved July 24, 2014, from http://pioneerinstitute

.org/download/charter-schools-in-new-orleans-lessons-for-massachusetts. 

Kmenta, J. (1986). Elements of econometrics (2nd edition). New York, NY: Macmillan. 

Lee, V. E., & Smith, J. B. (1997). High school size: Which works best and for whom? Educational Evaluation and Policy Analysis, 19(3), 205–227. 

Louisiana Department of Education. (2007). LEAP/GEE 2006–2007 annual report section 

1: The Louisiana educational assessment program. Baton Rouge, LA: Author. Retrieved 

September 12, 2016, from http://www.doe.state.la.us/lde/uploads/1703.pdf. 

Louisiana Department of Education. (2013). Louisiana charter schools annual report, 2013. 

Baton Rouge, LA: Office of School Choice. 

Macey, E., Decker, J., & Eckes, S. (2009). The Knowledge Is Power Program (KIPP): An 

analysis of one model’s efforts to promote achievement in underserved communities. 

Journal of School Choice, 3(3), 212–241. 

Maranto, R., Milliman, S., Hess, F., & Gresham, A. (2001). School choice in the real world: 

Lessons from Arizona charter schools. Boulder, CO: Westview Press. 

Mosteller, F. (1997). The Tennessee study of class size in the early school grades. Bulletin of 

the American Academy of Arts and Sciences, 50(7), 14–25. 

Musher, K. K., Musher, D. M., Graviss, E. A., & Strudler, R. M. (2005). Can an intense 

educational experience improve performance on objective tests? Results from one 

charter school. The Educational Forum, 69(4), 352–366. http://eric.ed.gov/?id=EJ724885 

New Orleans Parents’ Guide. (2013). New Orleans parents’ guide to public schools. New 

Orleans, LA: Author. Retrieved September 20, 2013, from http://www.neworleans

parentsguide.org/downloads.php. 

Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to do when data are missing in 

group randomized controlled trials (NCEE No. 2009–0049). Washington, DC: Government Printing Office. Retrieved July 6, 2016, from http://ies.ed.gov/ncee/pdf/20090049.

pdf. 

Purkey, S. C., & Smith, M. S. (1983). Effective schools: A review. Elementary School Journal, 

83(4), 427–452. 

Ref-4 
Rasheed, A. (2006). Education in New Orleans: Some background. The High School 

Journal, 90(2), 4–7. 

Ritter, G. W. (2000). The academic impact of volunteer tutoring in urban public elementary 

schools: Results of an experimental design evaluation. Ann Arbor, MI: Bell & Howell 

Information and Learning Company. 

Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools and academic 

achievement. Econometrica, 73(2), 417–458. 

Rockoff, J. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247–252. http://eric.ed.gov/

?id=ED475274 

Rumberger, R. W., & Palardy, G. J. (2005). Does segregation still matter? The impact of 

student composition on academic achievement in high school. Teachers College Record, 

107(9), 1999–2045. 

Skiba, R. J., & Rausch, M. K. (2004). The relationship between achievement, discipline, and 

race: An analysis of factors predicting ISTEP scores. Bloomington, IN: Center for Evaluation and Education Policy, Indiana University. 

Staiger, D. O., & Rockoff, J. E. (2010). Searching for effective teachers with imperfect 

information. Journal of Economic Perspectives, 24(3), 97–118. 

Steele, J. L., Vernez, G., Gottfried, M. A., & Schwam-Baird, M. (2011). The transformation 

of a school system: Principal, teacher, and parent perceptions of charter and traditional 

schools in post-Katrina New Orleans. RAND technical report prepared for the Scott S. 

Cowen Institute for Public Education Initiatives. New Orleans, LA: Tulane University. 

http://eric.ed.gov/?id=ED525090 

Taylor, E. S., & Tyler, J. H. (2012). The effect of evaluation on teacher performance. American Economic Review, 102(7), 3628–3651. 

Tillotson, D. (2006). What’s next for New Orleans? The High School Journal, 90(2), 69–74. 

U.S. Department of Education, National Center for Education Statistics. (2013). Common 

Core of Data (CCD). State Nonfiscal Public Elementary/Secondary Education Survey 

Data. Retrieved October 30, 2013, from https://nces.ed.gov/ccd/stnfis.asp. 

Witte, J., Weimer, D., Shober, A., & Schlomer, P. (2007). The performance of charter 

schools in Wisconsin. Journal of Policy Analysis and Management, 26(3), 557–573. 

Wolf, P. J. (1993). A case survey of bureaucratic effectiveness in U.S. cabinet agencies: 

Preliminary results. Journal of Public Administration Research and Theory, 3(2), 161–181. 

Wolf, P. J. (2007). Academic improvement through regular assessment. Peabody Journal of 

Education, 82(4), 690–702. 

Ref-5 
Wolf, P. J., & Hoople, D. S. (2006). Looking inside the black box: What schooling factors 

explain voucher gains in Washington, DC. Peabody Journal of Education 81(1), 7–26. 

Zimmer, R., Blanc, S., Gill, B., & Christman, J. (2008). Evaluating the performance of Philadelphia’s charter schools. Santa Monica, CA: RAND Corporation. 

Zimmer, R., & Buddin, R. (2007). Getting inside the black box: Examining how the operation of charter schools affects performance. Peabody Journal of Education, 82(2–3), 

231–273. 

Zimmer, R., Gill, B., Attridge, J., & Obenauf, K. (2014). Charter school authorizers and 

student achievement. Education Finance and Policy, 9(1), 59–85. http://eric.ed.gov/?id=

EJ1016278 

Zimmer, R., Gill, B., Booker, K., Lavertu, S., Sass, T. R., & Witte, J. (2009). Charter schools 

in eight states: Effects on achievement, attainment, integration, and competition. Santa 

Monica, CA: RAND Corporation. http://eric.ed.gov/?id=ED504625 

Ref-6 
The Regional Educational Laboratory Program produces 7 types of reports
 

Making Connections 

Studies of correlational relationships 

Making an Impact 

Studies of cause and effect 

What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 

What’s Known 

Summaries of previous research 

Stated Briefly 

Summaries of research findings for specific audiences 

Applied Research Methods 

Research methods for educational settings 

Tools 

Help for planning, gathering, analyzing, or reporting data or research