How Methodology Decisions Affect the Variability of Schools Identified as Beating the Odds. REL 2015-071.rev
Author: Abe, Yasuyo
Date: 2015-07-00
Type: Reports - Research
Subjects: School Effectiveness,Statistical Analysis,Identification,Academic Achievement,Prediction,Comparative Analysis,Institutional Characteristics,Public Schools,Elementary Secondary Education

A number of states and school districts have identified schools that perform better than expected, given the populations they serve, in order to recognize school performance or to learn from local school practices and policies. These schools have been labeled &quot;beating the odds,&quot; &quot;high-performing/high-poverty,&quot; &quot;high-flying,&quot; and other terms that reflect their demonstration of higher academic achievement than schools with similar student demographic characteristics. If administrators are to learn from these schools, it is important to correctly identify the schools that perform above expectations. However, there is no one right approach to identifying these schools. Typical identification approaches often consider many factors, including policy priorities, available data, resources and capacity (including technical analysis), and stakeholders' preferences. These choices can affect which schools are identified and labeled as exceeding performance expectations. This report considers the Michigan Department of Education's approach to identifying beating-the-odds schools by using two statistical methods. The first method, the prediction method, identifies a school as beating the odds if it outperforms its predicted level of performance given school demographics by comparing the predicted performance of each school to its actual performance. The second method, the comparison method, identifies a school as beating the odds if it outperforms other demographically similar schools by comparing the performance of each school to the performance levels of the 29 demographically most similar schools in the state. This report uses Michigan's approach as an example to demonstrate how the choice of statistical methods and technical specifications can change which schools are identified as beating the odds. Michigan's two statistical models produced different results: the comparison method identified fewer than half as many as the prediction method (28 versus 75), with a 39 percent agreement rate. When a change was made to the school performance measure, school characteristic indicator, or school sample configuration, the schools identified as beating the odds changed by varying degrees, with changes in school performance measures causing the biggest difference. Identification results also varied across time. For year-to-year variation from school year 2007/08 through 2010/11, the agreement rate between one year and the next was, on average, less than 50 percent. The findings confirm the importance of carefully considering the conceptual criteria and technical specifications and measures to be used in identifying schools exceeding performance expectations. Different policy and technical choices may lead to wide variations in resulting lists of schools labeled as beating the odds. The following are appended: (1) Literature review; and (2) Technical details on methods and additional results. [This version of this report replaces the version released in February 2015. Revisions are indicated by a dagger (†). For the original version, see ED554442.]