Evidence-Based Education: Postcards From the Edge

This article reports on the research design and implications stemming from a statewide evaluation developed during a Department of Education Title IID (EETT) grant. This article introduces critical issues to consider while developing research and evaluation methods for technology integration projects. The North Carolina IMPACT project, an evaluation of 11 technology-infused schools in low income districts, serves as a case illustrating the complexity of research under No Child Left Behind legislation. In this article, the authors explore the practical challenges involved in research that complies with new federal standards. This work is supported in part by a grant from the Elementary and Secondary Education Act, Title II, D. ********** Under the 2001 No Child Left Behind Act (NCLB), districts receiving federal funding are required to adopt "scientifically proven" programs, and conduct "scientifically-based research" on educational innovations (U.S. Department of Education, 2001). One of the six goals of the Department of Education's Strategic Plan (2002) to carry out the accountability focus of NCLB involves transforming education "into an evidence-based field." Drawing upon medical models of research, the U.S. Department of Education (2002) has provided administrators and researchers with a rank-ordered set of methods for conducting such research. Within this rank-ordered list, qualitative approaches such as the case study are given the lowest priority, while experimental methods are touted as the most desirable. Based on the argument that experimental approaches allow researchers to make causal inferences (Frankel & Wallen, 2003), the models favored by the Department of Education (DE) have a strong health sciences flavor, emphasizing the clinical trial and defining approaches using random assignment as the "gold standard" (National Council of Teachers of Mathematics [NCTM], 2003; Oakley, 2002). A recent position statement by the DE (2003) underscored the preferential status given to experimental and quasi-experimental designs: Evaluation methods using an experimental design are best for determining project effectiveness ... If random assignment is not feasible, the project may use a quasi-experimental design with carefully matched comparison conditions. This alternative design attempts to approximate a randomly assigned control group by matching participants--e.g., students, teachers, classrooms, or schools--with non-participants having similar pre-program characteristics ... In assessing the potential costs and benefits--both quantitative and qualitative--of this notice of proposed priority, we have determined that the benefits of the proposed priority justify the costs. (pp.62446-32447) Many researchers have applauded the shift toward experimental and quasi-experimental design in educational research, arguing that educators and policy-makers need more reliable evidence upon which to base their decisions and make causal inferences about treatment effects (Lewis, 2003; National Research Council [NRC], 2002; Slavin, 2002). Indeed, there are relatively few large-scale educational studies relying on experimental models, and the paucity of this type of research makes it difficult for educators and policy-makers to apply the kinds of inferential assumptions commonly found in the "hard" sciences. However, a growing number of voices have called attention to the problematic nature of using experimental research models to evaluate educational interventions (Berliner, 2002; Gardner, 2002; Kilpatrick, 2001; Morrison, 2001), citing ethical and inferential considerations. In light of the current legislation's strong emphasis on applying scientific research models to educational contexts, it is important to examine the issues that may arise as researchers attempt to comply with these standards. Many scholars on both sides of the issue have explored the theoretical and conceptual territory surrounding the research designs favored by the current administration, and this type of discussion is both fruitful and necessary, given the importance of addressing the epistemological issues involved in the kinds of questions that educational researchers want to ask, and the degree to which their methods address those questions. …