NBER WORKING PAPER SERIES MEASURING THE IMPACTS OF TEACHERS I: EVALUATING BIAS IN TEACHER VALUE-ADDED ESTIMATES

Are teachers impacts on students test scores (“value-added”) a good measure of their quality? One reason this question has sparked debate is disagreement about whether value-added (VA) measures provide unbiased estimates of teachers causal impacts on student achievement. We test for bias in VA using previously unobserved parent characteristics and a quasi-experimental design based on changes in teaching staff. Using school district and tax records for more than one million children, we find that VA models which control for a student s prior test scores exhibit little bias in forecasting teachers impacts on student achievement. Raj Chetty Department of Economics Harvard University 1805 Cambridge St. Cambridge, MA 02138 and NBER chetty@fas.harvard.edu John N. Friedman Harvard Kennedy School Taubman 356 79 JFK St. Cambridge, MA 02138 john_friedman@harvard.edu Jonah E. Rockoff Columbia University Graduate School of Business 3022 Broadway #603 New York, NY 10027-6903 and NBER jonah.rockoff@columbia.edu How can we measure and improve the quality of teaching in primary schools? One prominent but controversial method is to evaluate teachers based on their impacts on students’ test scores, commonly termed the “value-added” (VA) approach. School districts from Washington D.C. to Los Angeles have begun to calculate VA measures and use them to evaluate teachers. Advocates argue that selecting teachers on the basis of their VA can generate substantial gains in achievement (e.g., Gordon, Kane, and Staiger 2006, Hanushek 2009), while critics contend that VA measures are poor proxies for teacher quality (e.g., Baker et al. 2010, Corcoran 2010). The debate about teacher VA stems primarily from two questions. First, do the differences in test-score gains across teachers measured by VA capture causal impacts of teachers or are they biased by student sorting? Second, do teachers who raise test scores improve their students’ outcomes in adulthood or are they simply better at teaching to the test? This paper addresses the first of these two questions. Prior work has reached conflicting conclusions about the degree of bias in VA estimates (Kane and Staiger 2008, Rothstein 2010, Kane et al. 2013). Resolving this debate is critical for policy because biased VA measures will systematically reward or penalize teachers based on the mix of students in their classrooms. We develop new methods to estimate the degree of bias in VA estimates and implement them using information from two administrative databases. The first is a dataset on test scores and teacher assignments in grades 3-8 from a large urban school district in the U.S. These data cover more than 2.5 million students and include over 18 million tests of math and English achievement spanning 1989-2009. We match 90% of the observations in the school district data to selected data from United States tax records spanning 1996-2011. These data contain information on parent characteristics such as household income, retirement savings, and mother’s age at child’s birth. We begin our analysis by constructing VA estimates for the teachers in our data. We predict each teacher’s VA in a given school year based on the mean test scores of students she taught in other years. We control for student characteristics such as prior test scores and demographic variables when constructing this prediction to separate the teacher’s impact from (observable) student selection. Our approach to estimating VA closely parallels that currently used by school districts, except in one respect. Existing value-added models typically assume that each teacher’s quality is fixed over time and thus place equal weight on test scores in all classes taught by the teacher when forecasting teacher quality. In practice, test scores from more recent classes are better predictors of current teacher quality, indicating that teacher quality fluctuates over time. We account for such “drift” in teacher quality by estimating the autocovariance of scores across classrooms taught by a 1Value-added models of teacher quality were pioneered by Hanushek (1971) and Murnane (1975). More recent examples include Rockoff (2004), Rivkin, Hanushek, and Kain (2005), Aaronson, Barrow, and Sander (2007), and Kane and Staiger (2008). 2We address the second question in our companion paper (Chetty, Friedman, and Rockoff 2014). There are also other important concerns about VA. Most importantly, as with other measures of labor productivity, the signal in value-added measures may be degraded by behavioral responses if high-stakes incentives are put in place (Barlevy and Neal 2012).