Two Perspectives on the Generalizability of Lessons from Scaling Up SimCalc
Author: Roschelle, Jeremy
Date: 2010-00-00
Type: Reports - Research
Subjects: African Americans,Middle Schools,Educational Research,Scaling,Academic Achievement,Statistical Analysis,Geographic Regions,Sampling,White Students,African American Students,Whites,Hispanic American Students,Hispanic Americans,Disproportionate Representation,Middle School Students,Student Characteristics,Institutional Characteristics,Intervention,Mathematics Education,Computer Software,Mathematics Curriculum,Faculty Development,Research Design,Research Methodology

One purpose of educational research is to provide information about the likely impact of   interventions or treatments on policy-relevant populations of students. Randomized experiments are useful for estimating the causal effects of interventions on the students in schools that participate in the experiments. Unfortunately, the samples of schools and students participating in experiments are typically not probability (random) samples. Thus, even well-conducted experiments may not yield results that generalize to populations of interest. In the Scaling Up SimCalc experiments, one concern about the sample is that teachers were volunteers and potentially not representative of a broader teaching population. Although the volunteer teachers were randomly assigned to condition (reducing the chance that results were due to selection bias), the properties of the volunteer pool as a whole might limit generalizability to broader or differently-selected populations. A second concern is that, because pragmatic issues unrelated to sampling led to recruitment in regions with high proportions of Hispanic and Caucasian students and teachers, other groups of interest, such as African-American students and teachers, were underrepresented in the studies. In light of these and other concerns, this paper examines generalizability from two   complementary perspectives. First, the authors have conducted detailed analyses of the characteristics of teachers and schools participating in the sample in comparison to others in the state in which the experiments took place. Second, they present findings from a novel statistical method developed to permit principled generalization from research samples to well-defined populations. The studies took place during the 2005-06 and 2006-07 school years in 115 middle schools throughout several geographic regions across the state of Texas. This research has led the authors to propose the use of complementary approaches for examining generalizability. A foundational approach is to start out with the best sampling procedure possible, striving to achieve a broad and representative sample. The authors provided their recruiters with randomized lists of schools, but found that the actual schools they contacted reflected a tension between random selection and convenience. They complemented this procedure with two additional analyses. The first found some ways in which their samples do not reflect the full diversity of Texas; in particular they did not have large urban districts and did not have many African American participants. However, with regard to many other characteristics, their sample is not systematically different from the full population in the state of Texas. Their hypothesis is that the second analysis will predict positive effects for all populations of interest, but with wider confidence intervals for populations that were undersampled. Thus, they should have good confidence in how their results generalize to Hispanic schools but less confidence as to how the generalize to African American schools. Overall, this affects how they share the results of their research with the practitioner community. (Contains 4 tables and 3 figures.)