Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions That Need to Be Powered to Detect Small Impacts
Author: Deke, John
Date: 2018-00-00
Type: Reports - Research
Subjects: Intervention,Program Evaluation,Sample Size,Randomized Controlled Trials,Research Design,Attrition (Research Studies),Educational Research

Evaluators of education interventions increasingly need to design studies to detect impacts much smaller than the 0.20 standard deviations that Cohen (1988) characterized as &quot;small.&quot; For example, an evaluation of Response to Intervention from the Institute of Education Sciences (IES) detected impacts ranging from 0.13 to 0.17 standard deviations (Balu et al. 2015), and IES' evaluation of the Teacher Incentive Fund detected impacts of just 0.03 standard deviations (Chiang et al. 2015). The drive to detect smaller impacts is in response to strong arguments that in many contexts, impacts once deemed &quot;small&quot; can still be meaningful (Kane 2015). Though based on a compelling rationale, the drive to detect smaller impacts may create a new challenge for researchers: the need to guard against relatively smaller biases. When studies were designed to detect impacts of 0.20 standard deviations or larger, it may have been reasonable for researchers to regard small biases as ignorable. For example, a bias of 0.03 standard deviations might have been ignorable in a study that could only detect an impact of 0.20 standard deviations. But in a study designed to detect much smaller impacts, such as Chiang et al. (2015) in which the impact estimate was 0.03 standard deviations, a bias of 0.03 standard deviations is no longer small--it is enormous. The authors define asymdystopia as a context in which a larger sample size is not necessarily better and could even be worse from the perspective of controlling the Type 1 error rate. The authors examine the potential for asymdystopia as studies are powered to detect smaller impacts, where even small biases may lead to false inferences about the existence or magnitude of an impact. The authors focus on the potential for bias from attrition in the case of randomized controlled trials (RCTs) and bias from regression misspecification in the case of regression discontinuity designs (RDDs). While the methodological details are distinct, in both cases they are unpacking a source of bias that may become increasingly problematic when studies are designed to detect smaller impacts. The two main research questions are: (1) How problematic is attrition bias in RCTs as studies are powered to detect smaller impacts? and (2) How problematic is functional form misspecification bias in RDDs as studies are powered to detect smaller impacts? Overall, the findings suggest that biases that might have once been reasonably ignorable can pose a real threat in evaluations that are powered to detect small impacts. This paper identifies and quantifies some of these biases, and shows that they are important to consider when designing evaluations and when analyzing and interpreting evaluation findings. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]