Using Simulation to Understand Consistency in Treatment Effects: An Application to School Choice
Author: Maroulis, Spiro
Date: 2012-00-00
Type: Reports - Research
Subjects: Public Schools,School Choice,Effect Size,Enrollment,Open Enrollment,Simulation,Experiments,Academic Achievement,Outcomes of Education,Models

The authors examine the sensitivity of the school choice treatment effects--as defined as the difference between participants and non-participants in open enrollment programs--to differences in i) the underlying student/household preferences of a school districts, and ii) the program participation rates of the district. Data detailed and broad enough to directly estimate these relationships across many districts do not exist. Instead, the authors use student- and school-level data from Chicago Public Schools to initialize an agent-based, computational model of the transition to public school choice; and then conduct computational experiments with hypothetical districts that would be otherwise difficult or impossible to execute in the field. To be clear, their intent is not to perform a secondary analysis of the Chicago open enrollment program, but instead to gain a better understanding of the connection between the contextual features in which school choice programs are implemented, and the outcome measures used in social experiments that take place in those contexts. To initialize the model, the authors used student-level data made available by Chicago Public Schools (CPS). This included achievement, school enrollment, and demographic information on all students enrolled in a CPS school between 2001 and 2003 (See Appendix B.3). Analysis of the model finds that treatment effects calculated by comparing choosers to non-choosers are highly dependent on both the household participation rates in the program and the distribution of available capacity across schools. In particular, as participation rates rise, the magnitude of the treatment effect falls, because capacity constraints increasingly limit the amount of choosers who are able to attend the highest value-added schools. From a policy perspective, this finding highlights the importance of connecting an understanding of the mechanisms in each context that give rise to aggregate school choice outcomes. From a research perspective, the most direct implication of this finding is that one must account for the amount of &quot;better&quot; capacity available to choosers when using treatment effects estimated from existing programs to either a) project the impact of a larger scale program, or b) synthesize effect sizes estimated across programs. The measure the authors developed to characterize the better available capacity of the district in the model, &quot;bac&quot;, could be applied to program data to aid with both these purposes. Future work should include a more refined understanding of student preferences is required. More specifically, the current model only partially addresses heterogeneity in the decision-making rules of households. Additional heterogeneity could come in the form of categories of agents weighing elements of the existing preference function differently, or in the form of additional and varied criteria on which to judge schools that go beyond mean achievement and geographic proximity. Appended are: (1) Estimating Achievement Growth By School; (2) Results from Model; and (3) Additional Model Information. (Contains 1 table and 5 figures.)